LLM Providers

Configure LLM providers for Runra Runtime — OpenAI, Anthropic, Gemini, OpenRouter, local models (Ollama, vLLM), and custom provider configuration

Runra Runtime supports multiple LLM providers. Configure them once and any agent adapter can use any provider. This decouples your agent choice from your model choice.

Built-in Providers

Provider	Key	Models
OpenAI	`openai`	gpt-4o, gpt-4o-mini, gpt-5, o1, o3, o4-mini
Anthropic	`anthropic`	claude-sonnet-4-20250514, claude-opus-4-20250514, claude-haiku-3-5
Gemini	`gemini`	gemini-2.5-pro, gemini-2.5-flash
OpenRouter	`openrouter`	Any OpenRouter model
Ollama	`ollama`	Any local Ollama model
vLLM	`vllm`	Any vLLM-served model
Custom	`custom`	OpenAI-compatible endpoint

Configuration

OpenAI

import { Runra } from "@runra/runtime";

const runra = new Runra({
  llm: {
    provider: "openai",
    config: {
      apiKey: process.env.OPENAI_API_KEY,
      model: "gpt-5",
      baseUrl: "https://api.openai.com/v1",     // optional
      organization: "org-abc123",                // optional
      temperature: 0.7,                          // optional (default: 0.3)
      maxTokens: 4096,                           // optional
      requestTimeoutMs: 30000,                   // optional (default: 30000)
    },
  },
  agent: {
    provider: "claude-code",
    config: {
      model: "gpt-5",  // Override: agent uses this model via the OpenAI provider
    },
  },
  // ... sandbox and observability config
});

Anthropic

const runra = new Runra({
  llm: {
    provider: "anthropic",
    config: {
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: "claude-sonnet-4-20250514",
      maxTokens: 8192,
      thinking: {
        type: "enabled",
        budgetTokens: 4000,
      },
    },
  },
  agent: {
    provider: "claude-code",
    config: {
      model: "claude-sonnet-4-20250514",
      permissionMode: "auto-approve",
    },
  },
});

Gemini

const runra = new Runra({
  llm: {
    provider: "gemini",
    config: {
      apiKey: process.env.GEMINI_API_KEY,
      model: "gemini-2.5-pro",
      temperature: 0.3,
      maxOutputTokens: 8192,
      safetySettings: [
        { category: "HARM_CATEGORY_DANGEROUS_CONTENT", threshold: "BLOCK_ONLY_HIGH" },
      ],
    },
  },
});

OpenRouter

Use OpenRouter to access hundreds of models through a single API key:

const runra = new Runra({
  llm: {
    provider: "openrouter",
    config: {
      apiKey: process.env.OPENROUTER_API_KEY,
      model: "anthropic/claude-sonnet-4-20250514",
      baseUrl: "https://openrouter.ai/api/v1",
      appName: "my-runra-agent",                  // Required by OpenRouter
      headers: {                                   // Optional extra headers
        "HTTP-Referer": "https://myapp.com",
      },
    },
  },
});

You can switch models without changing providers:

// Swap models by changing one value
const llmConfig = {
  provider: "openrouter",
  config: {
    apiKey: process.env.OPENROUTER_API_KEY,
    // model: "anthropic/claude-sonnet-4-20250514",
    // model: "openai/gpt-5",
    // model: "google/gemini-2.5-pro",
    model: "meta-llama/llama-4-maverick",
  },
};

Local Models

Ollama

Run models locally with Ollama:

const runra = new Runra({
  llm: {
    provider: "ollama",
    config: {
      model: "llama3.2:70b",
      baseUrl: "http://localhost:11434",      // Ollama default
      temperature: 0.1,
      numCtx: 32768,                           // Context window size
    },
  },
});

Make sure Ollama is running and the model is pulled:

ollama pull llama3.2:70b
ollama serve

vLLM

Use a vLLM server for high-throughput local inference:

const runra = new Runra({
  llm: {
    provider: "vllm",
    config: {
      model: "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
      baseUrl: "http://localhost:8000/v1",
      apiKey: "not-needed",                    // vLLM typically doesn't require auth
      maxTokens: 4096,
    },
  },
});

Provider Configuration Reference

interface LLMConfig {
  /** Provider key */
  provider: "openai" | "anthropic" | "gemini" | "openrouter" | "ollama" | "vllm" | "custom";

  /** Provider-specific configuration */
  config: {
    /** API key for the provider */
    apiKey?: string;

    /** Override base URL (for proxies, local models, custom endpoints) */
    baseUrl?: string;

    /** Model name / ID */
    model: string;

    /** Temperature (0.0 - 2.0). Lower = more deterministic */
    temperature?: number;

    /** Maximum tokens per response */
    maxTokens?: number;

    /** Request timeout in milliseconds */
    requestTimeoutMs?: number;

    /** Maximum retries on transient errors */
    maxRetries?: number;

    /** Additional HTTP headers */
    headers?: Record<string, string>;
  };
}

Custom OpenAI-Compatible Provider

Any service with an OpenAI-compatible API can be used as a provider:

const runra = new Runra({
  llm: {
    provider: "custom",
    config: {
      baseUrl: "https://my-proxy.internal/v1",
      apiKey: process.env.MY_PROXY_KEY,
      model: "custom-fine-tuned-model",
      temperature: 0.3,
      headers: {
        "X-Custom-Header": "value",
      },
    },
  },
});

Writing a Full Custom Provider

Implement the LLMProvider interface for complete control:

import type { LLMProvider } from "@runra/runtime";

interface LLMRequest {
  messages: Array<{ role: string; content: string | Array<unknown> }>;
  tools?: Array<{ name: string; description: string; input_schema: unknown }>;
  systemPrompt?: string;
  temperature?: number;
  maxTokens?: number;
}

interface LLMResponse {
  content: string;
  toolCalls?: Array<{ id: string; name: string; input: unknown }>;
  stopReason: "end_turn" | "tool_use" | "max_tokens" | "stop";
  usage: {
    inputTokens: number;
    outputTokens: number;
  };
}

class MyCustomLLMProvider implements LLMProvider {
  readonly id = "my-custom-provider";

  private apiKey!: string;
  private baseUrl!: string;

  async initialize(config: Record<string, unknown>): Promise<void> {
    this.apiKey = config.apiKey as string;
    this.baseUrl = (config.baseUrl as string) || "https://my-api.example.com";
  }

  async chat(request: LLMRequest): Promise<LLMResponse> {
    const response = await fetch(`${this.baseUrl}/chat`, {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${this.apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "my-model",
        messages: this.buildMessages(request),
        tools: request.tools,
        temperature: request.temperature ?? 0.3,
        max_tokens: request.maxTokens ?? 4096,
      }),
    });

    const data = await response.json();
    return this.parseResponse(data);
  }

  private buildMessages(request: LLMRequest) {
    const messages = [];
    if (request.systemPrompt) {
      messages.push({ role: "system", content: request.systemPrompt });
    }
    messages.push(...request.messages);
    return messages;
  }

  private parseResponse(data: any): LLMResponse {
    const choice = data.choices[0];
    const message = choice.message;

    return {
      content: message.content || "",
      toolCalls: message.tool_calls?.map((tc: any) => ({
        id: tc.id,
        name: tc.function.name,
        input: JSON.parse(tc.function.arguments),
      })),
      stopReason: choice.finish_reason,
      usage: {
        inputTokens: data.usage?.prompt_tokens || 0,
        outputTokens: data.usage?.completion_tokens || 0,
      },
    };
  }

  async dispose(): Promise<void> {}
}

// Register and use
Runra.registerLLMProvider("my-custom-provider", () => new MyCustomLLMProvider());

Fallback Providers

Configure fallbacks when your primary provider is unavailable:

const runra = new Runra({
  llm: {
    provider: "openai",
    config: {
      apiKey: process.env.OPENAI_API_KEY,
      model: "gpt-5",
    },
    fallbacks: [
      {
        provider: "anthropic",
        config: {
          apiKey: process.env.ANTHROPIC_API_KEY,
          model: "claude-sonnet-4-20250514",
        },
      },
      {
        provider: "openrouter",
        config: {
          apiKey: process.env.OPENROUTER_API_KEY,
          model: "meta-llama/llama-4-maverick",
        },
      },
    ],
  },
});

Fallbacks trigger when the primary provider returns 5xx errors or times out. They're tried in order.

Environment Variables

All providers can read credentials from environment variables:

Provider	Environment Variable
OpenAI	`OPENAI_API_KEY`
Anthropic	`ANTHROPIC_API_KEY`
Gemini	`GEMINI_API_KEY`
OpenRouter	`OPENROUTER_API_KEY`

If apiKey is not set in config, the runtime reads it from the environment automatically.

Next Steps

Agent Adapters — connect AI agents
Runtime Architecture — full stack overview
Configuration Reference — all env vars