Single AI agents are impressive. Multi-agent systems that work together? That's where real operational leverage lives. The challenge isn't building individual agents—it's orchestrating them. How do you coordinate five, ten, or twenty specialized agents without creating a tangled mess of dependencies, race conditions, and communication failures? This isn't theoretical. We've deployed multi-agent systems handling everything from content pipelines to DevOps workflows to customer success operations. What follows are the battle-tested patterns that survived production. Why Single Agents Hit a Ceiling Before diving into orchestration, let's understand why multi-agent architectures exist in the first place. Single agents face fundamental constraints: Context window limits. Even with 200K token windows, complex operations requiring domain expertise across multiple areas exhaust context fast. An agent trying to handle research, writing, editing, SEO optimization, and publishing burns through tokens retrieving and maintaining state across all these domains. Specialization tradeoffs. An agent optimized for code generation has different prompt engineering, tool access, and behavioral patterns than one optimized for customer communication. Trying to do everything creates a jack-of-all-trades that excels at nothing. Latency multiplication. Sequential operations in a single agent create compounding delays. A task requiring research, analysis, drafting, and review takes four times as long when one agent handles everything serially versus four agents working their phases in parallel where possible. Failure isolation. When a monolithic agent fails, everything fails. When a specialized agent in an orchestrated system fails, you can retry that specific operation, substitute another agent, or degrade gracefully. Multi-agent systems solve these problems—but only if you orchestrate them correctly. Pattern 1: Hub-and-Spoke (Coordinator Model) The most common starting pattern. One central coordinator agent receives tasks, delegates to specialized worker agents, and synthesizes results. Architecture ┌─────────────┐ │ Coordinator │ │ (Hub) │ └──────┬──────┘ ┌───────────────┼───────────────┐ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐ │ Worker │ │ Worker │ │ Worker │ │ Agent A │ │ Agent B │ │ Agent C │ └───────────┘ └───────────┘ └───────────┘ How It Works The coordinator receives a task like "research competitor pricing and create a comparison document." It decomposes this into subtasks: Dispatch to Research Agent: "Find pricing information for competitors X, Y, Z" Wait for research results Dispatch to Analysis Agent: "Compare pricing structures, identify positioning opportunities" Wait for analysis Dispatch to Content Agent: "Create comparison document from analysis" Receive final output, perform any synthesis needed Implementation Details Task decomposition logic sits in the coordinator. This is the hardest part to get right. Too granular, and you're micromanaging with excessive overhead. Too coarse, and you lose the benefits of specialization. We use a task complexity scoring system: function shouldDecompose(task) { const domains = identifyDomains(task); // ['research', 'analysis', 'writing'] const estimatedTokens = estimateTokenUsage(task); const parallelizationPotential = assessParallelism(task); return domains.length > 1 || estimatedTokens > SINGLE_AGENT_THRESHOLD || parallelizationPotential > 0.5; } Communication protocol needs structure. We use a standard message format: { "task_id": "uuid", "parent_task_id": "uuid | null", "agent_target": "research-agent", "priority": "normal | high | critical", "payload": { "objective": "string", "context": "string", "constraints": ["string"], "output_format": "string" }, "deadline": "ISO timestamp", "retry_policy": { "max_attempts": 3, "backoff_ms": 1000 } } State management is critical. The coordinator maintains: Active task registry (what's currently dispatched) Completion status per subtask Aggregated results waiting for synthesis Failure/retry state When to Use Hub-and-Spoke Teams of 3-7 specialized agents Clear hierarchy with one decision-maker Tasks that decompose cleanly into independent subtasks When you need centralized logging and observability Failure Modes to Watch Coordinator becomes bottleneck. All communication routes through one agent. If it's slow or overwhelmed, the entire system stalls. Solution: implement async dispatch and don't wait for coordinator acknowledgment on fire-and-forget tasks. Over-coordination. Coordinators that try to micromanage every step waste tokens and time. Trust your specialists. Dispatch objectives, not instructions. Single point of failure. If the coordinator dies, everything stops. Implement coordinator health checks and failover to a backup coordinator, or use persistent task queues that survive coordinator restarts. Pattern 2: Pipeline (Assembly Line) When work flows in one direction through discrete stages, pipelines beat hub-and-spoke for simplicity and throughput. Architecture ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Stage 1 │───▶│ Stage 2 │───▶│ Stage 3 │───▶│ Stage 4 │ │ Intake │ │ Process │ │ Enrich │ │ Output │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ How It Works Each agent owns one transformation. Work enters the pipeline, flows through stages, and exits as finished output. No coordinator needed—each stage knows what comes before and after. A content pipeline example: Research Agent: Takes topic, outputs raw research with sources Outline Agent: Takes research, outputs structured outline Draft Agent: Takes outline + research, outputs draft content Edit Agent: Takes draft, outputs polished final content Implementation Details Inter-stage contracts are essential. Each stage must produce output that the next stage can consume. Define schemas: interface ResearchOutput { topic: string; sources: Source[]; key_findings: string[]; raw_data: Record<string, unknown>; confidence_score: number; } interface OutlineInput extends ResearchOutput {} interface OutlineOutput { topic: string; sections: Section[]; word_count_target: number; research_ref: ResearchOutput; } Queue-based handoffs decouple stages. Instead of direct agent-to-agent calls, each stage writes to an output queue that the next stage reads from: Research Agent → [Research Queue] → Outline Agent → [Outline Queue] → ... This provides: Natural buffering under load Easy stage-by-stage scaling (run 3 outline agents if that's the bottleneck) Clean failure isolation (dead letter queue for failed items) Backpressure handling prevents cascade failures. If Stage 3 is slow, Stage 2's output queue grows. Implement: Queue depth monitoring Automatic throttling of upstream stages Alerts when queues exceed thresholds When to Use Pipelines Work naturally flows through sequential transformations Each stage is independently valuable (can save/resume mid-pipeline) High throughput requirements (easy to parallelize stages) Simple operational model (each agent has one job) Pipeline Optimizations Parallel execution within stages. If you have 10 articles to research, spin up 10 Research Agent instances. The pipeline architecture makes this trivial—just scale the workers reading from each queue. Speculative execution. Start Stage 2 before Stage 1 fully completes if you can predict the output shape. The Edit Agent might begin setting up style checks while the Draft Agent is still writing. Circuit breakers. If a stage fails repeatedly, stop sending it work. Better to accumulate a queue than to keep hammering a broken service. Pattern 3: Swarm (Collaborative Consensus) When there's no clear sequence and multiple perspectives improve output quality, swarm patterns excel. Architecture ┌───────────────────────────────────┐ │ Shared Context │ │ (Blackboard/State) │ └───────────────────────────────────┘ ▲ ▲ ▲ ▲ │ │ │ │ ┌─────┴─┐ ┌───┴───┐ ┌─┴─────┐ ┌┴──────┐ │Agent 1│ │Agent 2│ │Agent 3│ │Agent 4│ └───────┘ └───────┘ └───────┘ └───────┘ How It Works All agents have access to a shared context (sometimes called a "blackboard"). They read current state, contribute their expertise, and write updates. No single agent controls the flow—emergence from collective contribution produces the output. Example: Code review swarm Security Agent scans for vulnerabilities Performance Agent identifies optimization opportunities Style Agent checks conventions Logic Agent verifies correctness Each agent reads the code and existing reviews, then adds their findings. The final review is the aggregate of all perspectives. Implementation Details Blackboard structure needs careful design: { "artifact_id": "uuid", "artifact_type": "code_review", "artifact_content": "...", "contributions": [ { "agent_id": "security-agent", "timestamp": "ISO", "findings": [...], "confidence": 0.92 }, { "agent_id": "performance-agent", "timestamp": "ISO", "findings": [...], "confidence": 0.87 } ], "consensus_state": "gathering | synthesizing | complete", "synthesis": null } Contribution ordering matters. Options: Round-robin: Each agent gets a turn in sequence Parallel with merge: All agents work simultaneously, conflicts resolved at synthesis Iterative refinement: Multiple rounds where agents react to each other's contributions Consensus mechanisms determine when the swarm is "done": Time-boxed: Stop after N minutes regardless Contribution-based: Stop when no agent has new input Quality threshold: Stop when confidence score exceeds target Vote-based: Stop when majority of agents agree on output When to Use Swarms Problems benefiting from multiple perspectives No clear sequential dependency between contributions Quality matters more than speed Creative or analytical tasks (not mechanical transformations) Swarm Pitfalls Infinite loops. Agent A's contribution triggers Agent B, which triggers Agent A again. Implement contribution deduplication and iteration limits. Groupthink. If agents can see each other's contributions, they may converge prematurely. Consider blind contribution phases before synthesis. Coordination overhead. Shared state requires synchronization. At scale, the blackboard becomes a bottleneck. Consider sharding by artifact or using CRDTs for conflict-free updates. Pattern 4: Hierarchical (Nested Coordination) For large agent ecosystems, flat structures collapse. Hierarchical patterns introduce management layers. Architecture ┌──────────────┐ │ Executive │ │ (Level 0) │ └───────┬──────┘ ┌───────────────┼───────────────┐ │ │ │ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ │ Manager A │ │ Manager B │ │ Manager C │ │ (Level 1) │ │ (Level 1) │ │ (Level 1) │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ ┌───┴───┐ ┌───┴───┐ ┌───┴───┐ │ │ │ │ │ │ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ ┌──▼──┐ │ W1 │ │ W2 │ │ W3 │ │ W4 │ │ W5 │ │ W6 │ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ How It Works Executive-level agents handle strategic decisions and cross-domain coordination. Manager-level agents coordinate teams of workers in their domain. Workers execute specific tasks. This mirrors organizational structures because it solves the same problem: span of control. One coordinator can effectively manage 5-7 direct reports. Beyond that, you need hierarchy. Implementation Details Clear authority boundaries prevent conflicts: executive: authority: - cross_domain_prioritization - resource_allocation - escalation_handling delegates_to: [manager_content, manager_engineering, manager_ops] manager_content: authority: - content_task_assignment - quality_decisions - scheduling_within_domain delegates_to: [research_agent, writing_agent, edit_agent] escalates_to: executive Escalation protocols handle cross-boundary issues: async function handleTask(task) { if (isWithinAuthority(task)) { return await executeOrDelegate(task); } if (requiresCrossDomainCoordination(task)) { return await escalate(task, this.manager); } if (exceedsCapacity(task)) { return await requestResources(task, this.manager); } } Information flow typically moves: Commands: Down (executive → managers → workers) Status: Up (workers → managers → executive) Coordination: Lateral at same level (manager ↔ manager) When to Use Hierarchies More than 10 agents in the system Multiple distinct domains requiring coordination Need for strategic oversight and resource allocation Complex escalation paths and exception handling Hierarchy Anti-Patterns Too many levels. Every level adds latency and potential miscommunication. Most systems work with 2-3 levels maximum. Rigid boundaries. Sometimes workers need to collaborate directly across domains. Build in peer-to-peer channels for efficiency. Bottleneck managers. If every decision flows through managers, they become the constraint. Push authority down; managers should handle exceptions, not routine operations. Pattern 5: Event-Driven (Reactive Choreography) Instead of explicit coordination, agents react to events. No orchestrator tells them what to do—they subscribe to relevant events and act autonomously. Architecture ┌────────────────────────────────────────────────────┐ │ Event Bus │ └─────┬─────────┬──────────┬──────────┬─────────────┘ │ │ │ │ ┌──▼──┐ ┌──▼──┐ ┌───▼──┐ ┌───▼──┐ │ A1 │ │ A2 │ │ A3 │ │ A4 │ │sub: │ │sub: │ │ sub: │ │ sub: │ │ X,Y │ │ Y,Z │ │ X │ │ W,Z │ └─────┘ └─────┘ └──────┘ └──────┘ How It Works When something happens (new lead arrives, deployment completes, error detected), an event fires. Agents subscribed to that event type react: Event: new_lead_captured → Lead Scoring Agent: Calculate score → CRM Agent: Create contact record → Notification Agent: Alert sales team → Research Agent: Background check on company No coordinator specified these actions. Each agent knows its triggers and responsibilities. Implementation Details Event schema standardization is critical: interface SystemEvent { event_id: string; event_type: string; timestamp: string; source_agent: string; payload: unknown; correlation_id: string; // Links related events causation_id: string; // The event that caused this one } Subscription management: // Agent declares its subscriptions at startup const subscriptions = [ { event_type: 'content.draft.completed', handler: handleDraftCompleted, filter: (e) => e.payload.priority === 'high' }, { event_type: 'content.*.failed', // Wildcard subscription handler: handleContentFailure } ]; Event sourcing for state reconstruction. Instead of storing current state, store the event stream. Any agent can rebuild state by replaying events. This provides: Complete audit trail Easy debugging (replay events to reproduce issues) Temporal queries (what was the state at time T?) When to Use Event-Driven Highly decoupled agents that shouldn't know about each other Many-to-many reaction patterns (one event triggers multiple agents) Audit and compliance requirements Systems that evolve frequently (adding agents doesn't require coordinator changes) Event-Driven Challenges Event storms. Agent A fires event, Agent B reacts and fires event, Agent A reacts... Implement circuit breakers and event rate limiting. Debugging complexity. Without a coordinator, tracing why something happened requires following event chains. Invest in correlation IDs and distributed tracing. Eventual consistency. Agents react asynchronously. At any moment, different agents may have different views of system state. Design for this reality. Hybrid Patterns: Mixing and Matching Real systems rarely use one pure pattern. They compose: Hub-and-spoke with pipeline workers: Coordinator dispatches to specialized pipelines rather than individual agents. Hierarchical with event-driven leaf nodes: Managers use explicit coordination, but workers react to events within their domain. Swarm synthesis with pipeline production: Multiple agents collaborate on planning/design, then hand off to a pipeline for execution. The key is matching pattern to problem shape: Clear sequence? Pipeline. Need oversight? Hub-and-spoke or hierarchy. Multiple perspectives? Swarm. Loose coupling? Event-driven. Practical Implementation Checklist Before deploying any multi-agent system: Communication Defined message/event schemas Serialization format chosen (JSON, protobuf, etc.) Transport mechanism selected (queues, pub/sub, direct HTTP) Timeout and retry policies configured State Management State storage selected (Redis, database, file system) Consistency model understood (strong, eventual) State recovery procedures documented Conflict resolution strategy defined Observability Centralized logging configured Correlation IDs implemented Metrics exposed (task counts, latencies, error rates) Alerting thresholds set Failure Handling Dead letter queues for failed tasks Circuit breakers for degraded services Fallback behaviors defined Graceful degradation tested Operations Agent health checks implemented Deployment procedure documented Scaling strategy defined Runbooks for common issues Conclusion Orchestration patterns aren't academic exercises. They're the difference between a multi-agent system that scales to production and one that collapses under real load. Start simple. Hub-and-spoke handles most cases with 3-7 agents. As complexity grows, evolve to hierarchies or event-driven architectures. Use pipelines when work flows naturally through stages. Add swarms when quality requires multiple perspectives. The pattern matters less than the principles: clear contracts between agents, explicit state management, robust failure handling, and comprehensive observability. Build the simplest orchestration that solves your problem. Then iterate as you learn what actually breaks in production. Your agents are only as good as their coordination. Get orchestration right, and you unlock operational leverage that single agents can never achieve.