MLXDocs
Local AI · mlx-lm

Models

Markdown

GET /v1/models lists the models mlx-lm's server has locally. Which ones fit your Mac, and how to save memory on long chats.

List models

GEThttp://127.0.0.1:8080/v1/models
Request
curl localhost:8080/v1/models -H "Content-Type: application/json"

The answer is a list of the models available locally (SERVER.md, mlx-lm 0.32.2). Each has:

FieldNotes
idThe Hugging Face repo id. Send it as model to use that model.
createdWhen the model was created.

Choosing a model

The server takes a local path or a Hugging Face repo (--model, or model in a request). Builds made for MLX, often 4-bit, are published on Hugging Face; the guide's example is mlx-community/Mistral-7B-Instruct-v0.3-4bit.

Will it fit?

A model needs about its parameter count times the bits stored per weight, plus room for the context, and macOS lets the GPU use about three quarters of unified memory by default. mlx.app/compare does that arithmetic for every Apple chip and popular open model, and mlx.app/run has a page per model with the smallest Mac that holds it. Both are estimates from published specifications.

Long chats: a quantized KV cache

On long contexts the KV cache can take a lot of memory. --kv-bits stores it quantized:

bash
mlx_lm.server --model <path_to_model_or_hf_repo> --kv-bits 4 --prefill-step-size 512
  • Attention on a quantized cache is not fused, so it keeps a score matrix of prefill_step_size × context_length. Lower --prefill-step-size if that matrix would cost more than the cache saves.
  • --kv-group-size sets the group size and --quantized-kv-start the token position where quantization starts (default 5000).
  • A quantized cache doesn't batch: with --kv-bits set, the server handles one request at a time.