RuntimeConfig
Defined in: packages/ai.matey.types/src/model-runner.ts:151
Runtime configuration for model execution.
Properties
Section titled “Properties”batchSize?
Section titled “batchSize?”
optionalbatchSize:number
Defined in: packages/ai.matey.types/src/model-runner.ts:175
Batch size for prompt processing.
Default
Section titled “Default”512contextSize?
Section titled “contextSize?”
optionalcontextSize:number
Defined in: packages/ai.matey.types/src/model-runner.ts:156
Context size (max tokens in context window).
Default
Section titled “Default”2048gpuLayers?
Section titled “gpuLayers?”
optionalgpuLayers:number
Defined in: packages/ai.matey.types/src/model-runner.ts:163
Number of GPU layers to offload. -1 = all layers, 0 = CPU only
Default
Section titled “Default”0keepAlive?
Section titled “keepAlive?”
optionalkeepAlive:boolean
Defined in: packages/ai.matey.types/src/model-runner.ts:181
Keep model loaded in memory.
Default
Section titled “Default”truemlock?
Section titled “mlock?”
optionalmlock:boolean
Defined in: packages/ai.matey.types/src/model-runner.ts:193
Lock model in memory (prevent swapping).
Default
Section titled “Default”false
optionalmmap:boolean
Defined in: packages/ai.matey.types/src/model-runner.ts:187
Memory map the model file.
Default
Section titled “Default”truethreads?
Section titled “threads?”
optionalthreads:number
Defined in: packages/ai.matey.types/src/model-runner.ts:169
Number of threads to use.
Default
Section titled “Default”(CPU cores)