The Runtime Gap: Why LLM Orchestrators Are Not Enough

LangGraph gives you a powerful way to define agent logic as a graph of nodes and edges. But defining the graph is only half the problem. Running it reliably in production -- with real users, real failures, and real latency requirements -- exposes a gap between what the orchestrator provides and what a production system needs.

This post examines what is missing between LangGraph's orchestration layer and a production-ready agent runtime.

What LangGraph Gives You

LangGraph handles the core orchestration problem well. You define nodes (functions that transform state), edges (transitions between nodes), and conditional branching. The framework manages execution order, passes state between nodes, and supports checkpointing for long-running workflows.

from langgraph.graph import StateGraph, END

graph = StateGraph(AgentState)
graph.add_node("search", search_web)
graph.add_node("analyze", analyze_results)
graph.add_conditional_edges(
    "analyze",
    should_continue,
    {"continue": "search", "done": END}
)
app = graph.compile()

This is a clean abstraction for defining what your agent does. The question is how it runs.

The Deployment Gap

LangGraph compiles to a Python object. Deploying that object as a service requires solving problems that the framework does not address:

• HTTP serving: You need a web server (FastAPI, Flask) to expose the graph as an API endpoint
• Authentication: Who is allowed to invoke this agent and how do you verify their identity
• Concurrency: How many agent sessions can run in parallel without exhausting memory or API rate limits
• Timeouts: What happens when a node takes 5 minutes because an upstream API is slow
• Scaling: How do you run multiple instances behind a load balancer when state is in memory

LangGraph Platform (formerly LangGraph Cloud) addresses some of these, but it is a managed service with its own constraints. Self-hosting means solving these problems yourself.

The Observability Gap

Production agents need observability beyond print statements. You need to know which node is executing, how long each tool call takes, what the LLM decided at each branch point, and why a particular run failed.

# What you need in production
{
    "run_id": "abc-123",
    "node": "search",
    "tool": "search_google",
    "args": {"query": "best laptops 2026"},
    "latency_ms": 1200,
    "status": "success",
    "result_count": 10,
    "tokens_used": 450,
    "timestamp": "2026-04-19T10:30:00Z"
}

LangSmith provides tracing for LangGraph, but it is a separate product with its own pricing. If you want to send traces to your existing observability stack (Datadog, Grafana, OpenTelemetry), you need to instrument the graph yourself.

The Error Recovery Gap

LangGraph supports checkpointing, which means you can resume a graph from a saved state after a failure. But checkpointing alone does not solve error recovery. You also need:

• Dead letter queues for runs that fail after all retries are exhausted
• Partial result handling -- if 3 of 4 tool calls succeeded, do you return partial results or fail the entire run
• Graceful degradation -- if the search API is down, can the agent still answer from its training data
• User notification -- how do you tell the user their agent is stuck and needs intervention

These are application-level concerns that no orchestration framework can fully solve. They require design decisions specific to your use case.

Bridging the Gap

The practical approach is to treat LangGraph as the orchestration layer inside a larger application architecture:

• Wrap the graph in a FastAPI service with proper auth, rate limiting, and request validation
• Use a task queue (Celery, Temporal) for long-running agent sessions that might outlive a single HTTP request
• Instrument every node with structured logging that feeds into your existing observability platform
• Use reliable data sources -- structured APIs like Scavio return consistent JSON responses, reducing the error surface area in your tool nodes
• Define an SLA for each node and add circuit breakers that trip when latency exceeds the SLA

The Bigger Picture

The runtime gap is not a LangGraph-specific problem. Every LLM orchestrator -- LangChain, CrewAI, AutoGen -- has the same gap between defining agent logic and running it in production. The orchestrator gives you a programming model. The runtime is your responsibility. Accepting this early saves you from the surprise of discovering it in production.