Agent Orchestration & State Management

Orchestration is the system that runs the agent loop:

Core responsibilities:

Prompt management: - Constructing prompts with context - Including tool definitions - Managing conversation history - Injecting system instructions

LLM interaction: - Calling the model - Parsing responses - Handling function calls - Managing retries

Tool execution: - Validating tool calls - Executing functions - Formatting results - Error handling

State management: - Tracking current state - Persisting progress - Managing memory

Control flow: - Loop continuation logic - Termination conditions - Timeout handling

Agents need different types of state:

Conversation state: - Messages exchanged with user - Agent's responses and reasoning - Typically in-memory during session

Workflow state: - Current step in multi-step process - Intermediate results - Decisions made - Needs persistence for long-running tasks

Working memory: - Scratchpad for current task - Accumulated information - Temporary calculations

Long-term memory: - User preferences - Historical interactions - Learned patterns - Stored in database/vector store

State persistence options: - In-memory (simple, lost on restart) - Database (durable, queryable) - Redis (fast, good for sessions) - File system (simple, for development)

Model agent workflows as explicit state machines:

Benefits: - Clear understanding of possible states - Defined transitions prevent undefined behavior - Easy to visualize and debug - Natural checkpointing

State machine components: - States: Defined workflow positions - Transitions: Rules for moving between states - Guards: Conditions that must be true for transition - Actions: Work done during transitions

Example workflow states: - INIT → RESEARCHING → DRAFTING → REVIEWING → COMPLETE - Each state has defined entry/exit actions - Transitions happen on specific events

LangGraph approach: LangGraph makes state machines explicit: - Define nodes (processing steps) - Define edges (transitions) - State passed between nodes - Conditional edges for branching - Built-in persistence and replay

Some agent tasks take minutes, hours, or days:

Challenges: - Can't keep connection open - Need to survive restarts - Users need status updates - Must handle timeouts

Patterns:

Async execution: - Start workflow, return job ID immediately - Poll or webhook for completion - Store all state durably

Checkpointing: - Save state after each significant step - Can resume from last checkpoint - Handles crashes and deployments

Time-based triggers: - Workflow waits for external event - Timer triggers continuation - Scheduled follow-ups

Implementation: - Durable execution frameworks (Temporal, Inngest) - Database-backed state machines - Message queues for async steps - Scheduled jobs for time-based logic

Managing what goes into the LLM context:

The problem: - Context windows are limited (128K tokens, etc.) - Agent history grows with each step - Tools return variable amounts of data - Long contexts increase cost and latency

Strategies:

Summarization: - Compress old conversation history - Summarize tool results to key points - Keep recent details, compress older

Relevance filtering: - Only include relevant history - Use embeddings to find related past context - Drop clearly irrelevant information

Structured state: - Keep state in structured format outside context - Only load what's needed for current step - Agent explicitly asks for specific context

Tiered memory: - Recent: Full detail in context - Medium: Summarized in context - Old: In vector store, retrieved as needed

Monitoring: Track token usage per step. Alert when approaching limits.