Intelligence (LLM Runtime)
The intelligence layer is Athena’s LLM dispatch system. It owns provider routing, circuit breaking, fallback, the tool loop, and budget tracking. All AI node executions pass through it.
Provider Matrix
Athena supports 15 LLM / media providers:
| Provider | Modalities |
|---|---|
| OpenAI | chat, vision, tools, streaming |
| Anthropic | chat, vision, tools, streaming, prompt caching |
| Google (Gemini) | chat, vision, tools, streaming |
| Groq | chat, tools, streaming |
| Cohere | chat, tools, streaming |
| Perplexity | chat, streaming |
| xAI (Grok) | chat, tools, streaming |
| OpenRouter | chat, tools, streaming (model-dependent) |
| DeepSeek | chat, reasoning, tools, streaming |
| Mistral | chat, tools, streaming |
| MiniMax | chat, tools, streaming |
| Moonshot (Kimi) | chat, tools, streaming |
| ZAi (GLM) | chat, tools, streaming |
| XiaomiMimo | chat, tools, streaming |
| Fal.ai | media generation only (image/video) |
Provider API keys are stored as workspace secrets and resolved at execution time.
Key lookup order: group config → workspace secret → 401 Unauthorized.
Circuit Breaker
Each provider has an independent circuit breaker:
| Parameter | Value |
|---|---|
| Failure threshold to open | 5 failures in 60 s |
| Open duration before probe | 30 s |
| Probe successes to close | 2 |
States: Closed (normal) → Open (failures exceeded) → HalfOpen (probe)
→ Closed (2 successes) or back to Open (probe fails).
When a circuit is open, callers receive an ExternalService error immediately
(no LLM call made). This error is in the fallback-trigger set, so node-level
fallback still fires if configured.
Fallback Chain
Each AI node can optionally specify a (fallback_provider, fallback_model). If
the primary call fails with a retryable error, the engine retries automatically
with the fallback.
Errors that trigger fallback:
| Error type | Fallback triggered? |
|---|---|
RateLimit (429) | Yes |
ExternalService (500 / 502 / 503 / 504, timeout, connection error) | Yes |
| Circuit breaker open | Yes |
Validation (400, prompt too long) | No |
Unauthorized / Forbidden / NotFound | No |
If the fallback also fails, the primary error is returned (fallback errors do not mask the root cause).
Tool Loop Limits
| Limit | Default | Env override |
|---|---|---|
| Rounds (LLM call ↔ tool result cycles) | 8 | ATHENA_TOOL_MAX_ROUNDS |
| Tool calls per round | 8 | ATHENA_TOOL_MAX_CALLS_PER_ROUND |
| Total tool calls across all rounds | 20 | ATHENA_TOOL_MAX_TOTAL_CALLS |
Exceeding any limit terminates the loop with the last assistant message as
output. The execution is marked Completed, not Failed.
Budget Tracking
Every execution tracks token and cost accumulation in execution variables:
| Variable | Type | Updated |
|---|---|---|
__accumulated_tokens | u64 | After every LLM round |
__accumulated_cost_microdollars | u64 | After every LLM round |
Cost is calculated in microdollars from the model_pricing table (admin-managed,
5-minute in-process cache).
Budget enforcement happens before every AI node:
| Action | Effect |
|---|---|
End | Execution fails with TokenLimitExceeded or CostLimitExceeded |
Warn | Logged once via a latch variable; execution continues |
Handoff | Execution fails with TerminationReason::Handoff |
Pre-call token estimation uses a 1.2x safety buffer:
estimated = (prompt_chars / 4 × 1.2) + 4096 output tokens.
Anthropic Prompt Caching
Athena uses a direct Anthropic HTTP client (bypassing the SDK) to inject
cache_control breakpoints into prompts. This is enabled by default and can
be disabled via ATHENA_ANTHROPIC_PROMPT_CACHING=off.
Cache breakpoints are placed at:
- System prompt — if 4096+ characters
- Last tool definition — if more than one tool is defined
- Last assistant message — if 4096+ characters
Maximum 4 breakpoints per request. Cache hit rates are tracked in cost accounting
(cache_read_input_tokens vs cache_creation_input_tokens).
Streaming vs Sync
Both execution paths share the same intelligence layer. The streaming path emits token chunks via SSE as the LLM generates them. The sync path waits for the complete response.
SSE token stream event types:
| Event | Meaning |
|---|---|
text | LLM text chunk |
reasoning | Thinking / reasoning chunk (DeepSeek, Claude extended thinking) |
tool_call | LLM invoked a server-side tool |
tool_result | Server-side tool returned a result |
llm_final | LLM call complete with token usage |
Related Docs
- Architecture — how the execution pipeline works
- Workflows — BFS loop that calls the intelligence layer
- Credits — cost accounting and credit pre-deduction
- Error Codes —
external_service_error,rate_limitcodes