Skip to content

Feature: Add tier-based provider prioritization #349

@etwk

Description

@etwk

Add support for configuring providers in priority tiers, where lower-tier (higher priority) providers are used first, and higher-tier providers serve as fallbacks only when all lower-tier providers are exhausted.

Use Case

Many users have multiple API access methods with different cost structures:

  • Tier 1: Subscription-based accounts (e.g., Claude Max) - cheaper/included usage
  • Tier 2: Usage-based API keys (e.g., pay-per-token) - more expensive

Currently, CLIProxyAPI uses round-robin across all providers equally, which means expensive usage-based credentials get consumed even when cheaper subscription-based options are available.

Proposed Configuration

claude-api-key:
  - api-key: "AIzaSy...subscription"
    tier: 1  # Use first (subscription-based, cheaper)
  - api-key: "AIzaSy...payasyougo"
    tier: 2  # Fallback only (usage-based, expensive)

openai-compatibility:
  - name: "openrouter"
    tier: 2  # Fallback provider
    # ...
  - name: "local-llm"
    tier: 1  # Preferred provider
    # ...

Expected behavior:

  1. Always prefer tier 1 providers (round-robin within tier 1)
  2. Only use tier 2 when ALL tier 1 providers are:
    • Quota exceeded (in cooldown)
    • Temporarily unavailable (auth errors, rate limits)
  3. Automatically return to tier 1 when any tier 1 provider recovers
  4. Default tier: 1 if not specified (backward compatible)
  5. Able to add more tiers (tier 3, tier 4, etc.) following the same pattern

Alternatives considered:

  • Manually disabling expensive credentials and re-enabling when needed (inconvenient)
  • Running separate proxy instances per tier (complex, doesn't solve automatic fallback)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions