Streaming

Set stream: true and the gateway relays server-sent events as the model generates. This page covers the parts SDKs do not handle for you: the final usage chunk, reasoning deltas, and errors that happen after HTTP 200 is already committed.

Enabling streaming

curl -N https://api.clfaigateway.dev/v1/chat/completions \
  -H "Authorization: Bearer sk-gw-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2.6",
    "messages": [{"role": "user", "content": "Explain SSE in one sentence."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Chunks arrive as data: events and the stream ends with data: [DONE]. The final chunk always carries the authoritative token counts in usage — it is sent on every stream whether or not you ask for it. stream_options.include_usage only changes the shape of the chunks in between: with it, they carry "usage": null (exact OpenAI schema); without it, they omit the field.

JSON mode cannot stream: response_format of a JSON type together with stream: true returns 400 json_mode_no_stream.

Reasoning models

Reasoning models (all nine open models) stream their thinking as delta.reasoning_content before the answer starts in delta.content. The gateway forwards it verbatim so you can render it live. Each reasoning delta increments reasoning_tokens, reported inside completion_tokens_details of the final usage chunk.

Thinking is spent from max_tokens — budget for it

The whole completion budget covers reasoning plus the answer. If the budget runs out mid-thought you get finish_reason "length" with little or no content, and the tokens are still billed — the model really did consume them. The appetite differs by model: we measured kimi-k2.6 spending 1,500-3,000 tokens thinking on analytical prompts and up to 8,000 on a logic puzzle, at every reasoning_effort except none — so give it max_tokens of 8,000 or more for that kind of work, or send none when you do not need the thinking. GLM 5.3 is hungrier still on huge contexts — against a ~240K-token prompt we watched it spend 7,000+ tokens thinking before any visible text, so agentic work at that scale wants 8,000+ of headroom; several agent tools default lower and starve the answer. If you see an empty answer, check finish_reason before blaming the model: "length" means the budget died, not the model.

Nothing on screen yet? Prefill and thinking come first

On very large prompts the first token is late by design: prefill runs at roughly 72-77 ms per 1,000 input tokens on the 1M-window models, so a 250K-token prompt spends ~20 seconds before anything streams — and a reasoning model may then think for minutes before delta.content starts. Render delta.reasoning_content as it arrives so users can see progress, and keep client read timeouts at 60 s or more for large prompts. In our logs the most common "hang" is actually a client giving up mid-think: it lands in your usage log as a success flagged client-disconnected, billed only for the tokens delivered.

reasoning_tokens is informational

Reasoning tokens are a subset of completion_tokens and are never billed as a separate dimension — the count exists so you can see why an answer was longer than its visible text. On non-streaming requests the split cannot be observed, so reasoning_tokens is 0 there.

reasoning_effort by model

Each model accepts its own set of reasoning_effort levels, and the same name does not mean the same depth everywhere. We ran every accepted level on 25 Sep 2026 — a short logic puzzle, 3-8 runs per level — and report the median completion tokens (thinking plus answer). Thinking length swings a lot from run to run, so read these as tendencies, not guarantees.

Model	Accepted levels	What the level actually changes
`deepseek-v4-flash`	low, medium, high, xhigh, max	Barely steers it — low even thought longer (~2,600 tokens) than high (~1,100). Thinking cannot be switched off.
`deepseek-v4-pro`	low, medium, high, xhigh, max	Real but uneven: low ~880 → high ~1,300 → xhigh ~4,100 tokens; max landed in between (~1,600). Thinking cannot be switched off.
`glm-4.7-flash`	low, medium, high	No consistent difference — it thinks long (4,000-7,500 tokens) at every level.
`glm-5.2`	low, medium, high, xhigh, max	Two tiers: low, medium and high run alike (~1,800-2,900 tokens); xhigh and max think longer (~3,100-4,500). Thinking cannot be switched off.
`glm-5.3`	low, medium, high, max	Two tiers: low and high think briefly (~460 tokens); medium and max think fully (~1,300). Thinking cannot be switched off.
`glm-5.3-flash`	low, medium, high, xhigh, max	Three tiers: low barely thinks (~95 tokens — fastest, but it missed our puzzle 2 times out of 3); high thinks briefly (~415); medium, xhigh and max think fully (~630-1,200).
`kimi-k2.6`	none, low, medium, high	`none` switches thinking off (~880 tokens, less accurate on multi-step problems); low, medium and high all think at full depth (~6,600-7,100).
`kimi-k2.7-code`	low, medium, high	Always thinks at full depth — the level makes no measurable difference.
`qwen3.8-27b`	low, medium, xhigh	A real, gentle ramp: low ~650 → medium ~740 → xhigh ~1,000 tokens.

When you do not send reasoning_effort, the gateway sends low — on GLM 5.3 Flash that means almost no thinking, so send high or max for multi-step work there. A level the model does not accept returns 400 reasoning_effort_not_supported listing the valid ones. That includes none on the models that cannot stop thinking: upstream would quietly run it as a full-depth level and bill the thinking, so the gateway refuses it instead.

Errors after the stream has started

Once the first byte is sent, the HTTP status is already 200 and cannot change. If the upstream fails mid-generation, the gateway emits one final event with finish_reason: "error" and an error object, then [DONE]:

the last event before [DONE]

data: {"id":"req_01j9zxg0aabbccddeeff00112233","object":"chat.completion.chunk","created":1774694600,"model":"kimi-k2.6","choices":[{"index":0,"delta":{},"finish_reason":"error"}],"error":{"message":"Upstream provider error while streaming. You were charged only for tokens already delivered.","type":"api_error","code":"upstream_error","param":null}}

data: [DONE]

SDKs will not raise this for you

An SDK that only checks the HTTP status sees a successful stream that simply ended. Check finish_reason on every chunk — the Python and JavaScript samples above do — and treat "error" as a failed request. You are charged only for the tokens delivered before the failure.

If you disconnect first

Disconnecting does not cancel what was already generated: you are charged for the tokens delivered up to the disconnect, counted exactly from per-chunk usage — never estimated. Cap your worst case with max_tokens. The full money-side policy, including zero-completion insurance, is on Billing.

How a stream settles: the status field

Every request appears in GET /v1/generation?id={x-request-id} after it settles, with a status:

status	Meaning	What you pay
`success`	The stream finished — including the case where you disconnected early.	All delivered tokens.
`partial`	The upstream failed mid-stream; you received `finish_reason: "error"`.	Only the tokens delivered before the failure.
`error`	The request failed before any output.	$0 when zero-completion insurance applies — see Billing.

Stream costs are read after the fact

Streams do not carry the x-gw-cost-nano response header — headers are sent before the cost is known. Read the settled cost from GET /v1/generation using the x-request-id header (also the id field of every chunk).

One more stream-specific rule: Idempotency-Key is not supported on streaming requests — Idempotency explains why.

If your stream arrives all at once

The gateway flushes every chunk immediately and never compresses text/event-stream. If chunks still arrive in one burst at the end, something between us and your code is buffering — almost always a corporate proxy, or your own reverse proxy (nginx buffers responses by default: set proxy_buffering off; for SSE routes, or honor the X-Accel-Buffering: no convention). Serverless platforms that buffer function responses have the same effect.

A quick differential: run the same request with curl -N from the affected network. If curl streams smoothly, the buffering lives in your stack, not on the wire.

Behind a corporate proxy, configure the SDK explicitly (httpx ≥ 0.28 renamed the option to singular proxy): Python OpenAI(http_client=httpx.Client(proxy="http://proxy:8080")), Node via fetchOptions. If you get APIConnectionError, the request never reached us — check proxy/DNS/TLS on your side; APIStatusError means it did.

Client timeouts and reasoning models

The SDK defaults (10-minute overall timeout, Python applies it per-read while streaming) are safe for reasoning models. The common mistake is lowering it globally — timeout=30 — because reasoning models can think for 30–70 s before the first visible token on long prompts (we stream reasoning_content early precisely so the connection is never silent for long). A too-low timeout raises APITimeoutError mid-generation, and the SDK then retries the whole request — on a non-streaming call without an Idempotency-Key, that can bill twice.

granular timeout

# Python — tight connect, generous read (instead of one small global timeout)
import httpx
client = OpenAI(
    base_url="https://api.clfaigateway.dev/v1",
    api_key=os.environ["CLF_API_KEY"],
    timeout=httpx.Timeout(600.0, connect=5.0),
)

PreviousCoding agents NextBilling