API Reference

Streaming

Set one field and the answer arrives as server-sent events, so the interface can update while the model is still writing.

Streaming is switched on by one field in the request body: "stream": true. The reply then comes back as text/event-stream — a sequence of events, each carrying its payload on a data: line — instead of one JSON document at the end. Nothing else about the request changes: same route, same header, same body fields.

What that buys you depends on the model, not on the route, and the difference is not cosmetic — a text model writes in fragments as it goes, on either of the two routes that serve it; a frame is finished before its first byte leaves, so an image model streams nothing sooner.

Where streaming works

Streaming support per route and kind of model
Route	stream: true	What arrives
POST /api/v1/messages	Incremental	Native events as the model writes them; the stream ends with message_stop.
POST /api/v1/chat/completions text model	Incremental	chat.completion.chunk as the model writes; the last chunk carries usage, then data: [DONE].
POST /api/v1/chat/completions image model	One terminal chunk	A single chat.completion.chunk with the finished frames, then data: [DONE].
POST /api/v1/images	One event per frame	An image_generation.completed per frame, then data: [DONE].

Scroll the table horizontally for the last column.

Text is incremental in both shapes. For image models the flag changes the envelope, not the timing: frames are never partial, so the first byte leaves when the picture is already drawn. The rest of this page is about text; the last section covers frames.

A minimal example

Open the stream, read it line by line, keep the text deltas. On the Messages route:

PythonTypeScript (fetch)Shell

main.py

import json

import requests

with requests.post(
    "https://api.gtokens.io/api/v1/messages",
    headers={"Authorization": "Bearer sk-sl-your-key"},
    json={
        "model": "anthropic/claude-sonnet-5",
        "max_tokens": 1024,
        "messages": [{"role": "user", "content": "Describe a lighthouse at dawn"}],
        "stream": True,
    },
    stream=True,          # do not buffer the body
    timeout=600,
) as r:
    r.raise_for_status()  # nothing has been streamed yet, so this is a real status
    text = ""
    for line in r.iter_lines(decode_unicode=True):
        # Everything that is not a "data:" line is framing: the "event:" name
        # repeats the type that is already inside the payload, and blank lines
        # separate events.
        if not line.startswith("data:"):
            continue
        event = json.loads(line[5:].strip())
        if event["type"] == "content_block_delta":
            text += event["delta"]["text"]
            print(event["delta"]["text"], end="", flush=True)

print()
print(len(text), "characters")

main.ts

const r = await fetch('https://api.gtokens.io/api/v1/messages', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer sk-sl-your-key',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'anthropic/claude-sonnet-5',
    max_tokens: 1024,
    messages: [{ role: 'user', content: 'Describe a lighthouse at dawn' }],
    stream: true,
  }),
})
// Nothing has been streamed yet, so a failure is still a real status.
if (!r.ok) throw new Error((await r.json()).error.message)

const reader = r.body!.getReader()
const decoder = new TextDecoder()
let buffer = ''
let text = ''

for (;;) {
  const { done, value } = await reader.read()
  if (done) break
  buffer += decoder.decode(value, { stream: true })

  for (let nl = buffer.indexOf('\n'); nl !== -1; nl = buffer.indexOf('\n')) {
    const line = buffer.slice(0, nl).trim()
    buffer = buffer.slice(nl + 1)
    // Skip framing: blank separators and "event:" lines.
    if (!line.startsWith('data:')) continue

    const event = JSON.parse(line.slice(5).trim())
    if (event.type === 'content_block_delta') {
      text += event.delta.text
      process.stdout.write(event.delta.text)
    }
  }
}

console.log('\n', text.length, 'characters')

shell

# -N turns off curl's own buffering, so the events print as they arrive.
curl -N https://api.gtokens.io/api/v1/messages \
  -H "Authorization: Bearer sk-sl-your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Describe a lighthouse at dawn"}],
    "stream": true
  }'

Two habits make hand-written parsers survive: skip every line that does not start with data:, and switch on the type field inside the payload rather than on the event: line — the two carry the same name, and the payload is the one you have already parsed. A spec-compliant SSE parser does the same framing work for you and is worth using for anything long-lived.

The same request in the OpenAI shape

On chat/completions every event is a chat.completion.chunk, the text is in choices[0].delta.content, and the stream is closed by the line data: [DONE] — which is not JSON, so a parser has to check for it before decoding. The OpenAI client library does all of that itself:

PythonTypeScript (openai)

main.py

import json

import requests

with requests.post(
    "https://api.gtokens.io/api/v1/chat/completions",
    headers={"Authorization": "Bearer sk-sl-your-key"},
    json={
        "model": "anthropic/claude-sonnet-5",
        "messages": [{"role": "user", "content": "Describe a lighthouse at dawn"}],
        "stream": True,
    },
    stream=True,
    timeout=600,
) as r:
    r.raise_for_status()  # a real status: nothing has been streamed yet
    text, cost = "", None
    for line in r.iter_lines(decode_unicode=True):
        if not line.startswith("data:"):
            continue
        payload = line[5:].strip()
        if payload == "[DONE]":
            break                 # the terminator; nothing follows it
        chunk = json.loads(payload)
        if "error" in chunk:      # a failure after the first byte, see below
            raise RuntimeError(chunk["error"]["message"])
        delta = chunk["choices"][0]["delta"]
        if delta.get("content"):
            text += delta["content"]
            print(delta["content"], end="", flush=True)
        if "usage" in chunk:      # only the last chunk carries it
            cost = chunk["usage"]["cost"]

print()
print(len(text), "characters, charged", cost)

main.ts

import OpenAI from 'openai'

const client = new OpenAI({
  baseURL: 'https://api.gtokens.io/api/v1',
  apiKey: 'sk-sl-your-key',
})

// The library reads the chunks and stops at [DONE] by itself.
const stream = await client.chat.completions.create({
  model: 'anthropic/claude-sonnet-5',
  messages: [{ role: 'user', content: 'Describe a lighthouse at dawn' }],
  stream: true,
})

let text = ''
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta.content ?? ''
  text += delta
  process.stdout.write(delta)
  if (chunk.usage) console.log('\ncharged', chunk.usage.cost)
}

Stream events

On the Messages route one answer is one sequence of events in a fixed order. Content blocks repeat: a long answer opens, fills and closes several of them before the message ends.

Server-sent events on the Messages route
Event	When	What it carries
message_start	First	The message id, the model, and usage.input_tokens.
content_block_start	Opens a block	The block index and its type.
content_block_delta	Repeatedly	The next fragment in delta.text — this is what you append.
content_block_stop	Closes a block	The block index, nothing else.
message_delta	Second to last	delta.stop_reason and usage.output_tokens.
message_stop	Last	Nothing. Its arrival is the end of the stream.
ping	Between any two	Nothing to act on: it keeps the connection alive.
error	Instead of the rest	A failure after the headers left. See below.

Scroll the table horizontally for the last column.

The whole stream, as it arrives

text/event-stream

event: message_start
data: {"type":"message_start","message":{"id":"msg_014htqPCz8ZfKq2wR9nJ1bYs","type":"message","role":"assistant","model":"claude-sonnet-5","content":[],"stop_reason":null,"usage":{"input_tokens":26,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: ping
data: {"type":"ping"}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"A lighthouse"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" at dawn."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":9}}

event: message_stop
data: {"type":"message_stop"}

Where the counters are

On the Messages route the token counters ride inside the events, not in a separate accounting frame at the end: input in message_start, output in message_delta — the event just before message_stop. Read them there, or the request looks free. That stream has no terminator line: it ends with message_stop and the connection closing, so a client waiting for a sentinel waits until its own timeout. The amount charged is not in the stream at all; read it from the credits route or the account area.

On chat/completions there is an accounting frame: the last chunk before data: [DONE] carries usage with all the counters and cost — the amount charged for that request, the same figure the account log shows. It is sent whether or not stream_options.include_usage asked for it. The images route does the same on its last event.

Chunks on chat/completions

The same answer as above, in the OpenAI shape. Every chunk repeats the id, the model and a one-element choices; what changes is delta:

text/event-stream

data: {"id":"gen-3b7e9c1d5a2f4086","object":"chat.completion.chunk","created":1772668800,
"model":"anthropic/claude-sonnet-5","choices":[{"index":0,"delta":{"role":"assistant","content":""},
"finish_reason":null}]}

data: {"id":"gen-3b7e9c1d5a2f4086","object":"chat.completion.chunk","created":1772668800,
"model":"anthropic/claude-sonnet-5","choices":[{"index":0,"delta":{"content":"A lighthouse at dawn."},
"finish_reason":null}]}

data: {"id":"gen-3b7e9c1d5a2f4086","object":"chat.completion.chunk","created":1772668800,
"model":"anthropic/claude-sonnet-5","choices":[{"index":0,"delta":{},"finish_reason":"stop",
"native_finish_reason":"end_turn"}],"usage":{"prompt_tokens":26,"completion_tokens":9,
"total_tokens":35,"cost":0.000071}}

data: [DONE]

delta.role: Named once, in the first chunk, with an empty content.
delta.content: The next fragment of text — this is what you append.
delta.tool_calls[]: Opens with a chunk carrying index, id and function.name with empty arguments; the chunks after it carry only function.arguments in pieces under the same index. Joined in order, the pieces are the JSON string of the non-streaming reply.
finish_reason: null on every chunk but the last, which carries stop, length, tool_calls or content_filter and native_finish_reason, the executor's own word, next to an empty delta.
usage: On the last chunk only: the token counters and cost.
data: [DONE]: The terminator. Nothing follows it.

Cancelling a stream

There is no cancel call: you close the connection. Stop reading and close the response, or abort the request through a signal — the gateway sees the client leave and ends the request instead of finishing it into nowhere. The same holds in both shapes.

PythonTypeScript (fetch)

main.py

import json

import requests

r = requests.post(
    "https://api.gtokens.io/api/v1/messages",
    headers={"Authorization": "Bearer sk-sl-your-key"},
    json={
        "model": "anthropic/claude-sonnet-5",
        "max_tokens": 4096,
        "messages": [{"role": "user", "content": "Write a long story"}],
        "stream": True,
    },
    stream=True,
    timeout=600,
)
r.raise_for_status()

try:
    words = 0
    for line in r.iter_lines(decode_unicode=True):
        if not line.startswith("data:"):
            continue
        event = json.loads(line[5:].strip())
        if event["type"] != "content_block_delta":
            continue
        words += len(event["delta"]["text"].split())
        if words > 200:
            break                # leaving the loop is the cancellation
finally:
    r.close()                    # closing the connection ends the request

main.ts

const controller = new AbortController()

// Cancel from anywhere: the request and the reader both stop.
setTimeout(() => controller.abort(), 5000)

try {
  const r = await fetch('https://api.gtokens.io/api/v1/messages', {
    method: 'POST',
    headers: {
      Authorization: 'Bearer sk-sl-your-key',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'anthropic/claude-sonnet-5',
      max_tokens: 4096,
      messages: [{ role: 'user', content: 'Write a long story' }],
      stream: true,
    }),
    signal: controller.signal,
  })

  const reader = r.body!.getReader()
  for (;;) {
    const { done } = await reader.read()
    if (done) break
  }
} catch (e) {
  if ((e as Error).name !== 'AbortError') throw e
  console.log('cancelled; whatever already arrived is billed')
}

Cancelling does not cancel the charge

Whatever the stream reported before you left is billed: the request is written to your log with the counters that arrived, and the part of the reserve it did not need goes back to your balance. Cancelling saves the remainder of the answer, not the part already generated.

Leaving before the first event is the one case that costs nothing: nothing had arrived to count, so the whole reserve is released and no request is recorded at all. For image models there is no such window — the frame is finished before the first byte, so an aborted stream is a paid frame you did not collect.

Errors before and after the first byte

Until the stream has started, nothing has been committed: the upstream connection is opened and its status checked before the response headers leave. A failure at that point is an ordinary HTTP status with the ordinary error body for that route, which means the same code that handles a non-streaming request handles it — the request simply never became a stream.

400 invalid_request: The body or a field is not usable, or the upstream refused the request itself.
401 authentication: No key, or a key that is not ours, or a revoked one.
402 payment_required: The balance cannot cover the request, or an account ceiling is reached.
403 permission_denied: The account is suspended.
404 not_found: No such model identifier in the catalogue.
429 rate_limit_exceeded: The provider refused without naming when to come back, or asked for extra usage. Retry-After is set, but a repeat on a schedule will not change the answer.
502 provider_unavailable: The executor answered with something unusable, or turned the request away before it reached the model. No Retry-After: a repeat will not help.
503 provider_overloaded: Every account for the model is busy or paused after a limit, or the provider refused. Retry-After names when the nearest one comes back; repeat the same request after it.
504 timeout: The upstream did not answer in time. A repeat of the same heavy request will not go faster.

The body shape depends on the route, not on streaming. Status codes and error bodies

After the first byte the status is already spent

The headers leave together with the first event, and a 200 already sent cannot be taken back. So anything that fails after that — the connection dropping mid-answer, the provider overloading while the model was writing, a fault on our side — arrives inside the stream, and the stream ends immediately after it. This is the one thing worth writing a branch for: without it a client cannot tell a refusal from a network hiccup, and the two are handled differently. The shape of that failure follows the route:

Messages routechat/completions

text/event-stream

event: error
data: {"type":"error","error":{"type":"overloaded_error","message":"provider is temporarily unavailable, retry later","error_type":"provider_overloaded"},"request_id":"gen-6f2c1b90a4d7e315","metadata":{"reason":"no_provider_available"}}

text/event-stream

data: {"id":"gen-6f2c1b90a4d7e315","object":"chat.completion.chunk","created":1772668800,"model":"anthropic/claude-sonnet-5","choices":[{"index":0,"delta":{},"finish_reason":"error"}],"error":{"code":503,"message":"provider is temporarily unavailable, retry later","metadata":{"error_type":"provider_overloaded","reason":"no_provider_available"}}}

data: [DONE]

The HTTP status stays 200, and so does the content type. Only the payload says something went wrong.
On the Messages route the event is named error on its event: line and carries "type": "error" in the payload, so a parser that reads only data: lines still sees it. Nothing follows it, and no terminator is sent.
On chat/completions it is a chunk with finish_reason: "error", an empty delta, and the usual error body of the OpenAI-shaped routes beside choices. data: [DONE] still follows it, and nothing follows that.
error_type is the same typed code you would have got as a status, so one table of reactions serves both cases. There is no Retry-After here: headers are long gone, so for a retryable code back off on your own schedule. What the model produced before the failure is billed, in both shapes.
A stream can also end without a single event — the upstream answered and said nothing. A 200 that carried no content is a failure, not an empty answer; it is charged nothing, and it is worth retrying.

Handling both

One function, three checks: the status before the stream, the failure inside it, and an empty result at the end. On the Messages route the failure is the error event; on chat/completions it is a chunk with an error key, as the example above reads it.

PythonTypeScript (fetch)

main.py

import json

import requests


def stream_message(prompt: str) -> str:
    with requests.post(
        "https://api.gtokens.io/api/v1/messages",
        headers={"Authorization": "Bearer sk-sl-your-key"},
        json={
            "model": "anthropic/claude-sonnet-5",
            "max_tokens": 1024,
            "messages": [{"role": "user", "content": prompt}],
            "stream": True,
        },
        stream=True,
        timeout=600,
    ) as r:
        # 1. Before the first byte: a real status and the usual error body.
        if r.status_code != 200:
            raise RuntimeError(f"{r.status_code}: {r.json()['error']['message']}")

        # 2. After the first byte the status is already 200, so a failure
        #    arrives as an event and the stream ends right after it.
        text = ""
        for line in r.iter_lines(decode_unicode=True):
            if not line.startswith("data:"):
                continue
            event = json.loads(line[5:].strip())
            if event["type"] == "error":
                raise RuntimeError(event["error"]["message"])
            if event["type"] == "content_block_delta":
                text += event["delta"]["text"]

        # 3. A 200 that carried nothing is a failure too, not an empty answer.
        if not text:
            raise RuntimeError("the stream ended without any content")
        return text

main.ts

async function streamMessage(prompt: string): Promise<string> {
  const r = await fetch('https://api.gtokens.io/api/v1/messages', {
    method: 'POST',
    headers: {
      Authorization: 'Bearer sk-sl-your-key',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'anthropic/claude-sonnet-5',
      max_tokens: 1024,
      messages: [{ role: 'user', content: prompt }],
      stream: true,
    }),
  })

  // 1. Before the first byte: a real status and the usual error body.
  if (!r.ok) throw new Error(`${r.status}: ${(await r.json()).error.message}`)

  const reader = r.body!.getReader()
  const decoder = new TextDecoder()
  let buffer = ''
  let text = ''

  for (;;) {
    const { done, value } = await reader.read()
    if (done) break
    buffer += decoder.decode(value, { stream: true })

    for (let nl = buffer.indexOf('\n'); nl !== -1; nl = buffer.indexOf('\n')) {
      const line = buffer.slice(0, nl).trim()
      buffer = buffer.slice(nl + 1)
      if (!line.startsWith('data:')) continue

      const event = JSON.parse(line.slice(5).trim())
      // 2. The status is already 200, so a failure comes as an event.
      if (event.type === 'error') throw new Error(event.error.message)
      if (event.type === 'content_block_delta') text += event.delta.text
    }
  }

  // 3. A 200 that carried nothing is a failure too, not an empty answer.
  if (!text) throw new Error('the stream ended without any content')
  return text
}

What does not stream

Frames are never partial. For an image model on chat/completions and on images the flag is honoured — the reply comes back as text/event-stream and ends with data: [DONE], carrying one terminal chunk with every frame in it on the first route and one completed event per frame on the second — but nothing arrives sooner than it would have without the flag. There is no progress to report, so we do not invent events to report it. On those two routes a stream value that is not a boolean is refused with 400 before anything is generated or charged, because guessing between a stream and a document breaks the client either way.

The read-only routes — the catalogue, credits and key — always answer with one JSON document; they have nothing to stream.

Status codes and error bodies Point an agent runtime at the gateway