Hardware & context

Local AI quality depends on what fits in your GPU memory. Olace measures your hardware and makes the sizing decisions for you, so models load reliably instead of crashing into memory limits.

What Olace detects

The daemon reads your GPU vendor, model, and memory, and on Apple Silicon the unified memory available to the GPU. Check what it sees:

olace status

The Model hub uses the same information: models that need more GPU memory than you have are labeled before you download, and the setup flow only recommends models that fit.

Context windows

Context is sized automatically per model and VRAM available on your machine. Two models on the same GPU get different context windows, and the same model gets a different window on a different runtime. Olace works this out for you:

Your hardware sets the budget. GPU memory, and whether auto-optimize (Flash Attention plus KV cache quantization) is on, decide how much there is to spend.
The model sets the cost. Its size on disk, its architecture, and its own maximum supported context all change how far that budget stretches. A compact 3B model and a dense 9B model on the same card land in very different places.
Olace learns the real number. The first estimate comes from the model's memory profile. After that, each load is measured on your own machine and the window is adjusted, growing when there is room and pulling back if a load fails. Models you use often converge on a number tuned to your machine instead of a generic guess. These measurements are stored locally and never leave your computer.
Loaded context only grows during a conversation. Olace never shrinks a loaded model's context mid-session, so long conversations do not silently lose memory. Starting a fresh chat may reload smaller if that lets the model fit entirely in GPU memory, which is faster.
Out of memory is recoverable. If a load hits a memory limit, Olace retries at a smaller size and grows again when conditions allow. One bad moment, such as a game or browser holding GPU memory, does not permanently lower your ceiling.

This is why the context shown for a model can change over the first few conversations, and why it differs between two computers with the same GPU.

Want a floor? Ask for one:

olace settings context auto     # default: size automatically
olace settings context 32768    # request at least 32k tokens

Values between 2048 and 131072 are accepted. This is a request, not a guarantee: the model's own maximum, your GPU memory, and out-of-memory recovery still win.

Keep-alive

Models stay in GPU memory between messages so follow-ups are instant. Control residency:

olace settings keep-alive 30      # minutes: 5, 10, 15, 30, 60, 120
olace settings keep-alive always  # stay loaded until unloaded
olace unload                      # free GPU memory now

Apple Silicon

On Apple Silicon, GPU memory is a share of unified memory. Olace uses a conservative share of your RAM for model sizing (larger machines get a larger share), and respects a custom GPU memory limit if you have set one on the system.

When a chat exceeds the context

If a conversation with attachments outgrows the model's context, Olace trims the least essential context first and tells you when something had to be dropped, rather than failing the turn.