Provider Routing

Automatic model routing across 100+ providers for reliability and performance

Never worry about provider outages again. The AI Gateway automatically routes your requests to the best available provider, with instant failover when things go wrong.

Supports BYOK (Bring Your Own Keys), passthrough routing, and pass-through billing (PTB). To enable PTB with Helicone’s API keys, request access in Settings → Credits.

The Problem

Using LLMs in production means dealing with:

• Provider outages that break your app
• Rate limits that block your users
• Regional restrictions that limit availability
• Vendor lock-in that prevents optimization

The Solution

Provider routing gives you access to the same model across multiple providers. When OpenAI goes down, your app automatically switches to Azure or AWS Bedrock. When you hit rate limits, traffic flows to another provider. All without changing your code.

How It Works

1. You request a model

   Your app asks for gpt-4o-mini just like normal

2. Gateway finds providers

   Consults the Model Registry to find all providers offering this model

3. Smart routing

   Applies sorting algorithm (cheapest first) then attempts providers

4. Automatic failover

   If a provider fails, instantly tries the next one

The result? Your request succeeds even when providers fail.

You request a model

The simplest approach lets the gateway handle everything:

// Just specify the model - gateway handles the rest
const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Hello!" }]
});

// Behind the scenes, the gateway tries:
// OpenAI → Azure OpenAI → AWS Bedrock → Others
// Until one succeeds

The gateway only tries providers where you’ve configured API keys. See Provider Setup to add your keys.

Routing Options

Automatic

Format: model: "gpt-4o-mini"

Best for:

• Maximum uptime in production
• Automatic cost optimization
• Zero-config reliability

How it works: The gateway tries ALL providers offering this model (sorted by cheapest first). Requests almost never fail because if one provider is down, another takes over instantly.

Example scenario: Your production chat app needs to stay online no matter what. You don’t care which provider serves the request as long as it works.

How the Gateway Finds Models

The Model Registry is our source of truth for which providers support which models. This powers intelligent routing.

Two Ways to Access Models

Option 1: Passthrough Billing (PTB)

Use Helicone’s API keys in supported regions. Zero configuration required - just request access in Settings → Credits.

model: "gpt-4o-mini"  // Will automatically use Helicone's keys

Available in: Major US/EU regions for popular providers

Option 2: Your Own Keys (BYOK)

Add your provider keys in Provider Settings. The gateway uses YOUR keys for all requests.

When you add a provider deployment, ALL models and regions that provider supports become available through your deployment.

Example:

You add an Azure deployment in Brazil.

When you request any Azure-supported model:

model: "gpt-4o-mini"  // Uses your Brazil deployment

The gateway uses your configured deployment for all requests.

Passthrough Routing (Unknown Models)

The gateway forwards ANY model/provider combination, even if not in our registry:

// Brand new model
model: "o3-preview/openai"

// Custom fine-tuned model
model: "ft:gpt-3.5-turbo:my-org::abc123/openai"

Important: Unknown models ONLY route through YOUR deployments (BYOK). The provider’s API determines if the model is valid.

Cost tracking is best-effort for unknown models.

Smart Routing Algorithm

When multiple deployments are available, the gateway intelligently selects which to use:

Routing Priority

1. Your deployments (BYOK) - Always tried first
2. PTB endpoints - Automatic fallback for reliability

Selection Logic

Within each priority level, we:

1. Sort by cost - Cheapest deployments first
2. Load balance - If costs are equal or unknown, randomly distribute requests

Example with multiple Azure deployments:

You have Azure Brazil + Azure US deployments.

Routing order:

1. Your cheapest deployment (e.g., Brazil if cheaper)
2. Your other deployments (e.g., US)
3. Helicone PTB endpoints

This ensures optimal cost while maintaining reliability.

Failover Triggers

The gateway automatically tries the next provider when encountering these errors:

Error	Description
429	Rate limit errors
401	Authentication errors
400	Context length errors
408	Timeout errors
500+	Server errors

The gateway attempts providers where you have configured API keys (BYOK) first, then falls back to Helicone’s API keys via Pass-through Billing (PTB) if enabled.

Real World Examples

Scenario: OpenAI Outage

Your production app uses GPT-4. OpenAI goes down at 3am.

// Your code doesn't change
const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Process this customer request" }]
});

What happens: Gateway automatically fails over to Azure OpenAI, then AWS Bedrock if needed. Your app stays online, customers never notice.

Scenario: Using Azure Credits

Your company has $100k in Azure credits to burn before year-end.

// Prioritize Azure but keep fallback for reliability
const response = await client.chat.completions.create({
  model: "gpt-4o-mini/azure,gpt-4o-mini",  
  messages: messages
});

What happens: Tries your Azure deployment first (using credits), but falls back to other providers if Azure fails. Balances credit usage with reliability.

Scenario: EU Compliance Requirements

GDPR requires EU customer data to stay in EU regions.

// Use your custom EU deployment
await client.chat.completions.create({
  model: "gpt-4o/azure/eu-frankfurt-deployment",  // Your CUID
  messages: messages
});

What happens: Requests ONLY go through your Frankfurt deployment. No data leaves the EU.

Next Steps

Browse Models

Explore all available models and providers

Add Provider Keys

Connect your provider accounts

Prompt Management

Combine routing with managed prompts