Workflow Engine

Every agent execution is driven by the workflow engine: a BFS state-graph executor that runs a queue of nodes, checkpointing after each one, and dispatching to one of 11 node type handlers.

---

Execution Lifecycle

Create execution record  →  Validate workflow graph
  ↓
BFS loop
  ├── pop node from queue
  ├── enforce timeout / visit limits / budget
  ├── dispatch to node handler
  ├── checkpoint to Redis (hot)
  └── queue activated downstream nodes
  ↓
Terminate (Completed / Failed / paused state)

Termination States

State	Trigger
Completed	BFS queue fully drained
Failed	Timeout, visit limit, budget cap, or unrecoverable error
AwaitingApproval	HumanGate node paused — waiting for human decision
WaitingForExternalEvent	MediaGeneration node submitted — waiting for Fal.ai webhook
AwaitingClientTool	LLM emitted a client-tool call — waiting for your app to fulfil
Cancelled	Cancelled via API

---

Node Types

#	Node	Purpose
1	Trigger	Entry point — emits initial parameters and input
2	Ai (LLM)	LLM call via intelligence layer — sync or streaming, with output-schema validation retry
3	Router	Branch selection — rules engine, round-robin, random, or LLM-based
4	Parallel	Fan-out — run sub-graphs concurrently under a configurable semaphore
5	Join	Merge parallel results — concat, select_best, aggregate, first, last, llm_judge, or custom template
6	Memory	Recall (hybrid retrieval) or Learn (async episode queue)
7	HumanGate	Human-in-the-Loop pause — execution resumes on approved/rejected/skipped decision
8	Action	HTTP / webhook dispatch with SSRF protection and configurable timeout
9	AgentInvocation	Nested agent execution as a child — max depth 3
10	Logic	Conditionals, loops, variable mutation, schema validation, pass-through
11	MediaGeneration	Fal.ai image/video submission — the only node that pauses for an external event

---

Execution Limits

Limit	Default	Termination
Wall-clock timeout	300 s	Failed — Timeout
Total node visits	25	Failed — RecursionLimitExceeded
Per-node visits	10	Failed — NodeVisitLimitExceeded
Aggregate output bytes	50 MB	Failed — BadRequest
AgentInvocation depth	3	Validation error
Action node timeout	30 s default, 600 s max	—
LLM stream buffer per node	256 KB	Truncated, not failed

Note

Cycles in the workflow graph are intentionally permitted (orchestrator-loop pattern: Router → LLM → Router). The per-node visit limit (10) is the ceiling on any individual node. The total visit limit (25) is the ceiling across the whole execution.

---

Checkpointing

After every node execution, the engine writes a checkpoint so paused and crashed executions can resume:

Tier	Store	TTL	Compression
Hot	Cache layer	24 h	zstd-3
Cold	Primary datastore	30 d	—

Write triggers (in addition to every-node saves):

Trigger	When
save_at_human_gate	HumanGate pause — includes gate info and timeout
save_at_external_event	MediaGen pause
save_at_client_tool_pause	LLM yields to client-tool fulfilment

On resume, the hot checkpoint is tried first; cold is the fallback.

---

HumanGate (HITL)

When execution reaches a HumanGate node:

1. Status → AwaitingApproval
2. human_gate_info saved to execution record with timeout_at
3. Your app shows an approval UI

To resume, POST to /executions/{id}/resume:

{
  "decision": "approved",
  "comment": "Looks good — proceed",
  "variables": {}
}

Decision	Effect
approved	Execution continues downstream
rejected	Continues via fallback_branch edge if configured
skipped	Continues via fallback_branch edge if configured

If timeout_at has passed when you call /resume, the server applies the node’s configured timeout_action (Approve / Reject / Skip / Route) automatically.

Caution

Rejected and Skipped decisions do not set the execution to Failed. The engine continues downstream so a fallback branch can fire. Only configure a fallback edge if your workflow needs to handle these cases.

---

MediaGeneration Node

MediaGeneration is the only async-pausing node:

1. Path A (first visit): Submits job to Fal.ai with a webhook URL. Returns immediately with status WaitingForExternalEvent.
2. Fal.ai delivers result via webhook — Athena injects ai_result:{node_id} into execution variables and resumes from checkpoint.
3. Path B (resume): Node detects ai_result:{node_id} in variables and processes the result, continuing the workflow.

---

Client-Tool Calls

If an LLM node emits a tool call for a client-registered tool, execution suspends:

1. Status → AwaitingClientTool
2. SSE emits tool_call_awaiting_client event
3. Your app runs the tool locally and POSTs result to POST /executions/{id}/tool-result

{
  "tool_call_id": "tc_abc123",
  "outcome": "success",
  "result": { "balance": 42.50 }
}

The last fulfilled tool call resumes execution automatically.

---

Retry Policy

Individual nodes can be configured with retry and timeout settings:

Retryable errors	Non-retryable
ExternalService, RateLimit, ServiceUnavailable	Validation, NotFound, Unauthorized, Forbidden, PaymentRequired, Conflict

Retry delay is linear by default; configurable to exponential backoff (base_delay × 2^attempt, capped at 5 minutes).

---

Related Docs

• Architecture — 3-layer execution pipeline
• Intelligence — LLM dispatch invoked by Ai/LLM node
• Credits — credit pre-deduct/reconcile wired into every AI node
• Execute — client-tool calls, HITL resume, SSE events
• Error Codes — workflow error codes