Local AI · mlx-lm
Chat completions
POST /v1/chat/completions on mlx-lm's server takes a chat and returns the model's answer. Every field mlx-lm documents, with its default.
POST
http://127.0.0.1:8080/v1/chat/completionsThe fields below are the ones listed in mlx-lm's server guide (SERVER.md, mlx-lm 0.32.2). The API is meant to be similar to OpenAI's chat API; a field that isn't listed here isn't documented by mlx-lm.
Request
| Parameter | Type | Notes |
|---|---|---|
messages | array | The conversation so far. Each message has a role (for example user, assistant) and content (the text). |
model | string | Optional. A local path (relative to the folder the server was started in) or a Hugging Face repo id. |
role_mapping | object | Optional. Custom role prefixes for the prompt; the defaults are used without it. |
stop | string or array | Optional. Sequences on which generation stops. |
max_tokens | integer | Optional. Most tokens to generate. Default 512. |
stream | boolean | Optional. Send the answer as it is generated. Default false. |
temperature | number | Optional. Sampling temperature. Default 0.0. |
top_p | number | Optional. Nucleus sampling. Default 1.0. |
top_k | integer | Optional. Top-k sampling. Default 0 (off). |
min_p | number | Optional. Min-p sampling. Default 0.0 (off). |
repetition_penalty | number | Optional. Multiplicative penalty on repeated tokens. Default 0.0 (off). |
repetition_context_size | integer | Optional. Window for the repetition penalty. Default 20. |
presence_penalty | number | Optional. Additive penalty on tokens that appeared before. Default 0.0 (off). |
presence_context_size | integer | Optional. Window for the presence penalty. Default 20. |
frequency_penalty | number | Optional. Additive penalty in proportion to how often a token appeared. Default 0.0 (off). |
frequency_context_size | integer | Optional. Window for the frequency penalty. Default 20. |
logit_bias | object | Optional. Token id → bias. |
logprobs | integer | Optional. Return this many top tokens with their log probabilities for each output token, 1 to 10. |
adapters | string | Optional. Path to low-rank adapters, relative to the folder the server was started in. |
draft_model | string | Optional. A smaller model for speculative decoding; null unloads it. |
num_draft_tokens | integer | Optional. Tokens the draft model predicts at once. Default 3. |
Response
| Field | Notes |
|---|---|
id | A unique id for the chat. |
system_fingerprint | A unique id for the system. |
object | chat.completion, or chat.completion.chunk when streaming. |
model | The model repo or path. |
created | When the request was processed. |
choices | A list of outputs, each with index, logprobs (token_logprobs, tokens, top_logprobs), finish_reason (stop or length) and message. |
usage | prompt_tokens, completion_tokens and total_tokens. |
Example
curl localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'A finish_reason of length means the answer hit max_tokens: raise it and ask again.