Agent Orchestration & State Management

Orchestration is the system that runs the agent loop:

Core responsibilities:

Prompt management:

• Constructing prompts with context
• Including tool definitions
• Managing conversation history
• Injecting system instructions

LLM interaction:

• Calling the model
• Parsing responses
• Handling function calls
• Managing retries

Tool execution:

• Validating tool calls
• Executing functions
• Formatting results
• Error handling

State management:

• Tracking current state
• Persisting progress
• Managing memory

Control flow:

• Loop continuation logic
• Termination conditions
• Timeout handling

Agents need different types of state:

Conversation state:

• Messages exchanged with user
• Agent's responses and reasoning
• Typically in-memory during session

Workflow state:

• Current step in multi-step process
• Intermediate results
• Decisions made
• Needs persistence for long-running tasks

Working memory:

• Scratchpad for current task
• Accumulated information
• Temporary calculations

Long-term memory:

• User preferences
• Historical interactions
• Learned patterns
• Stored in database/vector store

State persistence options:

• In-memory (simple, lost on restart)
• Database (durable, queryable)
• Redis (fast, good for sessions)
• File system (simple, for development)

Model agent workflows as explicit state machines:

Benefits:

• Clear understanding of possible states
• Defined transitions prevent undefined behavior
• Easy to visualize and debug
• Natural checkpointing

State machine components:

• States: Defined workflow positions
• Transitions: Rules for moving between states
• Guards: Conditions that must be true for transition
• Actions: Work done during transitions

Example workflow states:

• INIT → RESEARCHING → DRAFTING → REVIEWING → COMPLETE
• Each state has defined entry/exit actions
• Transitions happen on specific events

LangGraph approach: LangGraph makes state machines explicit:

• Define nodes (processing steps)
• Define edges (transitions)
• State passed between nodes
• Conditional edges for branching
• Built-in persistence and replay

Some agent tasks take minutes, hours, or days:

Challenges:

• Can't keep connection open
• Need to survive restarts
• Users need status updates
• Must handle timeouts

Patterns:

Async execution:

• Start workflow, return job ID immediately
• Poll or webhook for completion
• Store all state durably

Checkpointing:

• Save state after each significant step
• Can resume from last checkpoint
• Handles crashes and deployments

Time-based triggers:

• Workflow waits for external event
• Timer triggers continuation
• Scheduled follow-ups

Implementation:

• Durable execution frameworks (Temporal, Inngest)
• Database-backed state machines
• Message queues for async steps
• Scheduled jobs for time-based logic

Managing what goes into the LLM context:

The problem:

• Context windows are limited (128K tokens, etc.)
• Agent history grows with each step
• Tools return variable amounts of data
• Long contexts increase cost and latency

Strategies:

Summarization:

• Compress old conversation history
• Summarize tool results to key points
• Keep recent details, compress older

Relevance filtering:

• Only include relevant history
• Use embeddings to find related past context
• Drop clearly irrelevant information

Structured state:

• Keep state in structured format outside context
• Only load what's needed for current step
• Agent explicitly asks for specific context

Tiered memory:

• Recent: Full detail in context
• Medium: Summarized in context
• Old: In vector store, retrieved as needed

Monitoring: Track token usage per step. Alert when approaching limits.