Customer API docs

Connection details

Current support matrix

• OpenAI-compatible chat completions: /v1/chat/completions.
• Anthropic-compatible Messages: /v1/messages with x-api-key or bearer auth.
• Streaming: supported on both routes with "stream": true.
• Prompt caching: enabled for Claude requests when cacheable stable context is present and the upstream model reports cache usage.
• Usage visibility: portal usage rows include request count, normal input tokens, output tokens, cache write tokens, cache read tokens, latency, time to first token, and output tokens/sec when reported.
• Gateway location: the current primary gateway is hosted in Helsinki, Finland. Additional regional gateways may be tested separately before cutover.

List available models

curl https://api.canaryllm.com/v1/models \
  -H "Authorization: Bearer CUSTOMER_KEY"

The response includes the models allowed for the key and customer-visible pricing fields where configured.

Switch models

Each request chooses its model with the model field. To switch models, send another request with a different allowed model ID.

{
  "model": "claude-opus",
  "messages": [
    {"role": "user", "content": "Use this model for this request."}
  ],
  "max_tokens": 200
}

The customer portal also has a model dropdown for test requests. If a key is restricted, unavailable models are rejected with a model-permission error.

Create a chat completion

curl https://api.canaryllm.com/v1/chat/completions \
  -H "Authorization: Bearer CUSTOMER_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {"role": "user", "content": "Reply with OK only."}
    ],
    "max_tokens": 8,
    "metadata": {
      "session_id": "sess_123",
      "user_id": "customer_user_123",
      "turn_index": 1
    }
  }'

Metadata fields are optional, but using stable session and user IDs makes usage and traces easier to review in the customer dashboard.

Anthropic-compatible Messages

curl https://api.canaryllm.com/v1/messages \
  -H "x-api-key: CUSTOMER_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {"role": "user", "content": "Reply with OK only."}
    ],
    "max_tokens": 8,
    "stream": true
  }'

Use this route for Anthropic SDKs or clients that expect /v1/messages and x-api-key.

Claude Code CLI

For Anthropic-compatible CLI clients, use the CanaryLLM gateway as the base URL and your customer API key as the auth token.

export ANTHROPIC_BASE_URL="https://api.canaryllm.com"
export ANTHROPIC_AUTH_TOKEN="CUSTOMER_KEY"

claude

Credits and limits

• Requests require an active API key and enough remaining prepaid credits.
• Each key can have its own monthly credit cap.
• If credit is exhausted, the API returns a payment or credit-limit error instead of calling the upstream model.
• Streaming is supported with "stream": true on OpenAI-compatible chat completions and Anthropic-compatible messages.
• Default gateway guardrails are 180 requests/minute and 500,000 tokens/minute per key, plus 1,000 requests/minute and 2,000,000 tokens/minute per organization, unless a customer-specific limit is configured.
• The gateway accepts large max_tokens values up to its configured ceiling, but the selected model and upstream route may enforce lower model-specific limits.
• Committed TPM, RPM, TPD, and SLA numbers must be confirmed for the specific account, model, and route before production commitments.

Prompt caching

Claude prompt caching is enabled automatically for repeated stable context. To improve cache hit rate, keep large reusable instructions, tools, examples, and reference text at the beginning of the conversation, then place changing user input near the end.

• Cache hits reduce cached input-token cost and often reduce latency.
• Changing timestamps, random IDs, reordered tools, or edited context inside the reusable prefix can break cache hits.
• Usage responses include cache_creation_input_tokens and cache_read_input_tokens when the upstream model reports them.
• Advanced clients may send "cache_control": {"type": "ephemeral"}; otherwise automatic caching is applied for Claude requests.
• For very large single text blocks, the gateway may split a stable prefix into a cached text block and leave the changing suffix uncached. This happens within the same model request; requests are not decomposed into multiple model calls.
• For best results, still send stable context as separate system/message blocks when possible.

Performance guidance

• Use "stream": true for interactive chat so users see the first tokens as soon as the model starts generating.
• Keep stable system prompts, tool definitions, examples, and reference material at the start of the request to improve cache reuse.
• Avoid changing timestamps, random IDs, request IDs, or ordering inside the stable prefix; small prefix changes can reset cache hits.
• Set max_tokens close to the expected answer size. Very high values can increase reservation and tail latency.
• Usage rows expose latency, time to first token, output tokens/sec, cache write tokens, and cache read tokens when reported.

What you need from us

• Your customer API key.
• Your starting credit allocation and any monthly key cap.
• The list of models enabled for your key.
• The customer portal link for usage, billing, and child key management.

Support facts

• Every API response includes an internal request ID header: X-Provider-Request-ID.
• For support, include request IDs, timestamps, model IDs, route style, and the visible key prefix. Do not send full API keys.
• Prompt and response traces are retained under the configured platform retention policy. Current raw trace retention is 120 days.
• API key secrets are shown once when issued; the platform stores hashes, not recoverable full key secrets.