southseact-3d
|
ab7799ca58
|
feat: Add rate limiting and automatic provider switching
- Add rate limit detection and parsing from HTTP 429 responses
- Implement automatic retry with exponential backoff for short-term rate limits
- Implement automatic provider switching for long-term rate limits
- Add circuit breaker pattern for failing providers
- Integrate with existing admin panel rate limit configuration
- Add allProviders parameter to LLM.stream calls to enable provider fallback
Rate limit behavior:
- Short-term (< 5 min): Retry with configured backoff strategy
- Long-term (≥ 5 min): Switch to next available provider
- Max retries: 3 (configurable via admin panel)
- Max wait time: 5 minutes (configurable via admin panel)
- Provider switching: Enabled by default (configurable via admin panel)
Provider priority:
1. Anthropic
2. OpenAI
3. Google
4. OpenRouter
5. Groq
6. xAI
7. Together AI
8. Perplexity
9. DeepInfra
10. Cerebras
11. Mistral
12. Cohere
13. Amazon Bedrock
14. Azure
15. GitHub Copilot
16. GitHub Copilot Enterprise
17. OpenCode
18. ZenMux
19. Google Vertex
20. GitLab
|
2026-02-08 14:37:02 +00:00 |
|