MLXDocs
Local AI · mlx-lm

Local quickstart

Markdown

Install mlx-lm, start its server with a model, and call http://127.0.0.1:8080/v1/chat/completions from curl, Python or JavaScript.

Get a model running

  1. Install mlx-lm

    On a Mac with Apple silicon, install the package from PyPI:

    bash
    pip install mlx-lm

  2. Start the server

    Give it a local path or a Hugging Face repo. This is mlx-lm's own example:

    bash
    mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit

    The model is downloaded the first time, then the server listens on port 8080 of localhost. To check before downloading that a model fits your Mac's memory, see Models.

  3. See every option

    bash
    mlx_lm.server --help

Call it

The request is OpenAI-style chat JSON. model is optional: without it the server uses the model it was started with.

POSThttp://127.0.0.1:8080/v1/chat/completions
curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

max_tokens defaults to 512, so ask for more when you want a long answer. Every field is on Chat completions.

Stream the reply

Set "stream": true to get the answer as it is generated; each piece is a chat.completion.chunk.

curl -N localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "stream": true,
     "messages": [{"role": "user", "content": "Write a haiku about a Mac mini."}]
   }'