Serverless Rate Limits

When using Serverless, you may experience 429 Too Many Requests or 503 Service Overloaded. To avoid 429s, you need to stay below our adaptive rate limits. To reduce the likelihood of 503s, you can upgrade to Priority tier.

​

What are your rate limits?

There are three metrics we use to rate limit accounts:

• Total Prompt TPM — input tokens per minute (cached + uncached).
• Uncached Prompt TPM — uncached input tokens per minute.
• Generated TPM — output tokens per minute.

All limits are measured and enforced in tokens per minute (TPM). Adaptive rate limit ceilings depend on the model’s total parameter count. Smaller models get higher ceilings: Fast, Priority, and US-only variants of a model share the same tier and ceilings as the base model. Models without a known parameter count use Large ceilings.

​

Models by tier

Models not listed here are tiered by total parameter count using the thresholds above. Based on your usage, your adaptive limits will grow and shrink within these ceilings. If your traffic ramps up too quickly, you will get 429s. Your current limits for each model are in the Serverless dashboard. See Checking your current limit. Adaptive rate limits have an upper and lower bound. A higher account Spending Tier correlates with higher upper bound rate limits; enterprise accounts get higher upper bounds automatically.

​

How do adaptive limits change?

Your limit for each model starts at a default value and grows in steps as you use it. If your recent one-minute usage is above 50% of your current limit, the limit increases by 25%, and no more than once every 15 minutes, until it reaches your ceiling. If you stop using it, the limit shrinks slowly and eventually resets. Each of the three metrics (Total Prompt TPM, Uncached Prompt TPM, and Generated TPM) has its own limit and changes on its own. Heavy usage on one metric does not change the other two.

​

Your ceiling

The ceiling is the highest your limit can grow. Two things set it:

• Model size picks the row in the table above.
• Your spending tier sets how much of that row you can reach. The table shows the maximum ceilings, available at Tier 3 and above. Tiers below Tier 3 have lower ceilings.

​

When limits go up, down, or reset

A few details:

• A decrease is skipped if your recent usage would be at least 50% of the lower limit. This keeps the limit from dropping and then immediately climbing back up.
• With Reserved Throughput, these thresholds apply to burst usage above your reservation and to the adaptive portion of your limit. Your reservation is added to that adaptive limit, and usage within the reservation does not count toward the thresholds.
• Resets do not apply to accounts with a custom limit or a reservation for that model.
• If you have reserved throughput, your limit never drops below your reservation.

​

How slowly limits shrink

Say your Total Prompt TPM limit is 10,000,000 and your traffic drops to a steady 1,000,000 tokens per minute: If you stop sending traffic completely, you don’t go through these steps. After more than 72 hours with no traffic, the limit resets to the starting default.

​

Example: ramping up

Say your current Total Prompt TPM limit is 1,000,000.

1. Send more than 500,000 but less than 1,000,000 prompt tokens per minute. This keeps you above half your limit without hitting 429s.
2. More than 15 minutes later, your limit rises to 1,250,000.
3. Raise your traffic to match, staying between 625,000 and 1,250,000.
4. Repeat until your limit stops rising.

At this pace, you reach your ceiling in a few hours. If you are using more than half your limit and it has stopped rising after an increase interval, you are at the ceiling for your spending tier. Moving to a higher spending tier, up to Tier 3, raises the ceiling, and the limit keeps growing in the same 25% steps. A short burst does not lock in a higher limit. If you go 72 hours without traffic, the limit resets to the default.

​

Checking your current limit

Check your current limit in the Serverless dashboard. The same limits are also returned in the response headers on each request. Header values are tokens per minute, so you can compare them directly with the TPM ceilings above.

​

FAQ