Custom LLM Rate Limits

Set custom rate limits for model provider API calls. Control usage by request count, cost, or custom properties to manage expenses and prevent unintended overuse.

Who can use this feature: Anyone on any plan.

Introduction

Rate limits are an important feature that allows you to control the number of requests made with your API key within a specific time window. For example, you can limit users to 1000 requests per day or 60 requests per minute. By implementing rate limits, you can prevent abuse while protecting your resources from being overwhelmed by excessive traffic.

Why Rate Limit

• Prevent abuse of the API: Limit the total requests a user can make in a given period to control cost.
• Protect resources from excessive traffic: Maintain availability for all users.
• Control operational cost: Limit the total number of requests sent and total cost.
• Comply with third-party API usage policies: Each model provider has their own rate limit for your key. Helicone’s rate limit is bounded by your provider’s policy.

How Rate Limits Works

To set up rate limiting, simply add the Helicone-RateLimit-Policy header in your request. This will rate limit all requests made with the specified API key.

The header follows this format:

"Helicone-RateLimit-Policy": "[quota];w=[time_window];u=[unit];s=[segment]"

Rate Limit Parameters

Parameter	Description
quota (required)	The maximum number of requests allowed within the specified time window.
time_window (required)	The unit is seconds. For example, you would use w=86400 (60 _ 60 _ 24 = 86400) to set the time window for a single day. The minimum is 60 seconds.
unit (optional)	Must be request or cents. If left blank, unit is set to request by default.
segment (optional)	Must be user or a custom property. If left blank, segment is set to global by default. We’ll explain the difference in the Filtering By Segments section.

Example: A policy that allows 10 cents of requests per 1000 seconds per user.

Fun fact: this policy format is an IETF standard for specifying rate limits! Except for the segment field, that’s a Helicone special twist 🍬

Filtering By Segments

The s=[segment] parameter is used to specify the scope in which you want to apply rate limits to all requests made with an API key. You can apply rate limits globally, by users, or by a custom property.

• Global rate limiting: leave the segment field empty (i.e. 1000;w=60).
• Rate limiting by user: set the segment field to “user” (i.e. 1000;w=60;s=user).
  • In addition, the user ID must be included as a parameter in the request or in the helicone-user-id header (i.e. "Helicone-User-Id": "username").
  • See User Metrics for more details.
• Rate limiting by a custom property: set the segment field to the desired property name (i.e. 1000;w=60;s=[property_name]).
  • In addition, include a corresponding header in the request (i.e. "Helicone-Property-{property_name}": "some label").
  • See Custom Properties for more details.

Examples: Here is a list of example policies to use for the header Helicone-RateLimit-Policy:

Rate Limiting Globally

• Quota: 10k requests
• Time window: 1 hour (3600 seconds)
• Segment: global (default)

Header policy value: 10000;w=3600

Notice the s=[segment] parameter is ignored since the default is global.

Rate Limiting By User

• Quota: 500k requests
• Time window: 1 day (86400 seconds)
• Segment: user

Header policy value: 500000;w=86400;s=user

Don’t forget to add User Metrics.

Rate Limiting By Custom Property

• Quota: 300 requests
• Time window: 30 minutes (1800 seconds)
• Segment: organization (custom property)

Header policy value: 300;w=1800;s=organization

Don’t forget to set the custom property for organization in the request, see Custom Properties.

Extracting Rate Limit Response Headers

Extracting the headers allows you to test your rate limit policy in a local environment before deploying to production.

If your rate limit policy is active, the following headers will be returned:

Helicone-RateLimit-Limit: "number" // the request/cost quota allowed in the time window.
Helicone-RateLimit-Policy: "[quota];w=[time_window];u=[unit];s=[segment]" // the active rate limit policy.
Helicone-RateLimit-Remaining: "number" // the remaining quota in the time window.

• Helicone-RateLimit-Limit: The quota for the number of requests allowed in the time window.
• Helicone-RateLimit-Policy: The active rate limit policy.
• Helicone-RateLimit-Remaining: The remaining quota in the current window.

If a request is rate-limited, a 429 rate limit error will be returned.

Example: Extracting headers from python with OpenAI

client = OpenAI(
    api_key="<OPENAI_API_KEY>",
    base_url="https://oai.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": f"Bearer <HELICONE_API_KEY>",
    }
)

# 1. add `.with_raw_response` here
chat_completion_raw = client.chat.completions.with_raw_response.create(
    model="gpt-4-vision-preview",
    messages=[
        {"role": "user", "content": "Hello world!"}
    ],
    extra_headers={
        "Helicone-RateLimit-Policy": "10000;w=3600" # add rate limit policy here
    },
)

# This is the original parsed response as expected...
chat_completion = chat_completion_raw.parse()

# 2. get header response
rate_limit = chat_completion_raw.http_response.headers.get(
    'Helicone-RateLimit')

print(rate_limit) # will print the Rate Limit header responses

Latency Considerations

Using rate limits adds a small amount of latency to your requests. This feature is deployed with Cloudflare’s key-value data store, which is a low-latency service that stores data in a small number of centralized data centers and caches that data in Cloudflare’s data centers after access. The latency add-on is minimal compared to multi-second OpenAI requests.

Coming Soon

Soon, we will support rate limiting by tokens and cost. Additionally, you will be able to see how close your requests, users, and properties are to hitting their rate limits in the web UI.