Skip to content

Circuit breaker

The circuit breaker is a protection mechanism that stops routing traffic to a provider that is consistently failing. When a provider returns repeated errors, the gateway skips that provider for a cooldown period instead of letting every request fail against it.

Function

Without a circuit breaker, a provider outage slows every request: each request retries against the failing provider and only then falls back. The circuit breaker counts provider failures, opens after the failure threshold is reached, and routes traffic to the remaining healthy targets immediately. This keeps latency stable during a provider incident.

Rules and facts

  • The breaker is enabled per gateway and is inactive until an administrator turns it on.
  • The breaker tracks state per provider on each gateway. One failing provider does not affect the others. The one exception is the built-in Myra model fleet: because every gateway shares that same infrastructure, its breaker is global — a fleet outage sheds it once for everyone rather than per gateway.
  • The breaker moves through three states: Closed (healthy, requests route normally), Open (requests skip the provider), and Half-open (after the cooldown, probe requests test recovery). A probe is judged on what it actually delivered: the first attempt that returns a real answer closes the breaker, the first failure re-opens it, and an attempt that returns nothing — an empty answer, a broken stream, a cancelled turn — counts as neither and leaves the breaker under probation.
  • Server errors count as failures (the exact status-code set is configurable); connection errors and timeouts always count; a self-hosted model that ends a turn cleanly with no content at all counts once the automatic retry has also failed (a turn cut off by the caller's max_tokens does not count); client errors (4xx) count only if you list them.
  • The breaker works together with fallback and retry: fallback handles a single failed request, the breaker handles a failing provider.
  • The status API lists the providers whose breaker is currently open or half-open; a provider absent from the response is healthy.

See also

  • Circuit breaker — thresholds, cooldown, failure codes, and the status API.
  • Gateways — the gateway detail page where the breaker is enabled.