How enterprise AI architecture is actually being laid out this year , the layers underneath it, the frameworks teams are shipping agents with, and the platforms those agents run on in production.
From pipelines to systems
Through 2023‚ 2024, most enterprise generative-AI work took the shape of a pipeline: a prompt went in, passed through a retrieval step against some indexed documents, hit a foundation model once, and a response came out. That pattern still exists and still ships value, but it’s no longer the architecture teams reach for by default. The default in 2026 is agentic , systems where a model plans across several steps, decides which tools to call and in what order, holds state across a session, and in some designs hands work off to other agents entirely, all inside guardrails an enterprise has to define, log, and be able to audit.
That shift changes what “architecture” means for this stack. It’s no longer just model selection and a vector index. A useful way to see the current landscape is as five concerns layered on top of each other, four of them stacked and one , governance , running the full height of the stack rather than sitting at any one level of it.
Two design principles recur across nearly every enterprise write-up of this stack, and they’re worth stating plainly because they drive most of the concrete decisions in the rest of this document: bounded autonomy , agents operate inside explicitly defined decision boundaries and escalate anything above their authority to a human , and contextual grounding , an agent’s reasoning is only as trustworthy as the enterprise data it’s actually connected to, which is why the tools/integration layer gets as much architectural attention as the model layer itself.
The other pattern worth naming here is the AI gateway. Rather than each application wiring itself directly to a model provider, most enterprise builds now route every model and tool call through a central gateway that enforces guardrails at the point of invocation, handles multi-provider routing and cost accounting, and logs every interaction for audit. It’s the piece that makes the governance frame in the diagram above actually enforceable rather than aspirational.
Architecture patterns underneath the layer
Two families of pattern do most of the work inside the stack above: how an agent gets grounded in enterprise knowledge, and how multiple agents divide a piece of work between them.
Retrieval, from naive to agentic
Retrieval-augmented generation hasn’t gone away , it’s specialized. Enterprise builds now pick from a spectrum rather than treating “RAG” as one technique:
- Naive RAG , top-k semantic search over embedded chunks, then generate. Cheap, fast to stand up, fine for FAQ-shaped questions, and still where most proofs of concept start.
- Hybrid RAG , vector search combined with keyword/lexical search. This is the production baseline for most enterprise search and copilot use cases in 2026, because pure-vector retrieval misses exact terminology that regulated and technical domains depend on.
- Graph RAG , retrieval over a structured knowledge graph rather than flat chunks, enabling multi-hop reasoning across entity relationships. Justifies its indexing cost in relationship-heavy domains: legal research, supply chain, org knowledge.
- Agentic RAG , retrieval stops being a single passive lookup. The agent classifies a query’s complexity, routes it to a faster or deeper retrieval path, and can iteratively re-query across CRM, databases and APIs until it has enough to answer.
- Self-RAG , the model evaluates its own retrieval quality and answer confidence before responding, and re-retrieves when that confidence is low. Used where hallucination risk is the binding constraint.
Dividing work across agents
When a single agent with a tool loop isn’t enough, three coordination shapes cover most production systems: a supervisor/worker hierarchy, where one orchestrating agent decomposes a task and dispatches to specialist sub-agents; a role-based crew, where agents are defined by function (researcher, analyst, writer) and coordinate as peers; and a handoff/routing model, where agents pass a conversation to one another as the task’s needs change, closer to a call-center transfer than a management hierarchy. Which shape fits maps fairly directly onto which framework a team reaches for , that’s Part 03.
Contextual grounding and bounded autonomy aren’t separate initiatives from the retrieval and coordination choices above , they’re what those choices are graded against. A hybrid-RAG pipeline with no escalation path is still ungoverned; a supervisor/worker hierarchy with no audit log is still unaccountable.
Building agents: the framework layer
Frameworks occupy the orchestration band of the stack , they implement the plan/act/observe loop, manage memory, and (for the multi-agent ones) implement one of the coordination shapes above. Six are seeing real production use this year, each built around a different execution model.
| Framework | Execution model | Strength | Best fit | 2026 status |
|---|---|---|---|---|
| LangGraph | Graph , explicit nodes & transitions | Durable execution with checkpointing; strong audit trail | Regulated workflows needing human-approval steps | Enterprise default where auditability matters |
| CrewAI | Crew , roles, framework coordinates | Fastest idea-to-prototype path; low boilerplate | Role-based multi-agent teams, rapid prototyping | Established, Python-only |
| OpenAI Agents SDK | Handoff , agents route to each other | Minimal ceremony; built-in tracing | GPT-centric apps, support-style routing | Production-ready successor to Swarm |
| Google ADK | Code-first, multimodal | Native Vertex AI deploy; strong A2A interop | GCP-native and multimodal agents | GA; Python, Java & more |
| Microsoft Agent Framework | Unified graph/handoff, enterprise state mgmt | Official AutoGen + Semantic Kernel successor | Azure-native enterprises, .NET teams | Reached GA (1.0) in April 2026 |
| Claude Agent SDK | Brain/Hands/Session loop | Tight dev iteration loop; full local visibility | Prototyping, then promote to Managed Agents | Paired with Anthropic’s hosted Managed Agents |
Worth knowing about even if they don’t make the primary six: LlamaIndex Workflows (event-driven, retrieval-first), Pydantic AI (type-safe, minimal-overhead Python), Smolagents (Hugging Face’s lightweight, code-writing agents), and Dify / n8n as low-code builders for teams without dedicated engineering capacity.
Before adopting any of them, it’s worth checking a project actually needs one , direct API calls plus a simple loop still cover a lot of ground, and a framework earns its place only once the plumbing it removes outweighs the abstraction it adds. When picking one, the questions that actually discriminate are: code-first or low-code, tolerance for provider lock-in, whether the workflow needs durable checkpointing for crash recovery, single- versus multi-agent complexity, the language ecosystem the team already lives in, where it ultimately needs to deploy, and how mature its observability story is. Part 05 covers that last point.
Building agents with the Claude Agent SDK
Anthropic’s own path is worth calling out because it names its three components explicitly, and that decomposition generalizes well beyond Anthropic’s own stack: a Brain (the model plus the decision loop that picks the next action and processes tool results), Hands (the sandboxes and tools that actually execute , file operations, code execution, external API calls), and a Session (a durable, append-only event log of everything that happened). Separating those three means any one of them can fail or be swapped without losing the other two , the SDK runs the loop in your own process for fast local iteration, and the same architecture is what Managed Agents (Part 05) runs as a hosted service once that loop needs to survive process restarts and scale to many concurrent sessions.
The protocol layer: MCP and A2A
The reason framework choice stopped being a bet-the-architecture decision is that two protocols have standardized the two edges every agent system needs: reaching a tool, and reaching another agent. MCP (Model Context Protocol) gives an agent a standard way to call tools and read data sources. A2A (Agent-to-Agent Protocol) gives agents built on different frameworks , even by different vendors , a standard way to discover each other’s capabilities and delegate work.
Both protocols moved from “promising standard” to genuinely load-bearing infrastructure this year. MCP is the more mature of the two: it now sees on the order of tens of millions of monthly SDK downloads, more than 5,800 public MCP servers exist, and it ships built into Claude Desktop, VS Code and most major IDEs , every major model provider supports it. A2A, which Google originated, has grown faster than expected in enterprise pilots since its introduction; governance of both protocols now sits with the Linux Foundation’s Agentic AI Foundation (launched December 2025), with OpenAI, Anthropic, Google, Microsoft, AWS and Block all backing it , which is a meaningful signal, since none of those companies typically co-govern infrastructure they don’t expect to depend on.
The practical consequence for an architecture decision: a framework is now a component choice rather than a foundational bet, as long as the tools and agents underneath it speak MCP and A2A. That’s what makes the framework comparison in Part 03 a lower-stakes decision in 2026 than it would have been two years earlier.
Getting agents into production
Once an agent works in a notebook or a local SDK session, it needs somewhere durable to run. Two postures cover almost every production deployment this year: self-managed, where the agent runs in infrastructure the team operates , containers, Kubernetes, serverless functions, behind the gateway from Part 01 , and hosted/managed, where a cloud provider runs the execution runtime, memory, and credential handling on your behalf.
| Platform | Architecture | Model flexibility | Best fit |
|---|---|---|---|
| AWS Bedrock AgentCore | Modular services , Runtime, Memory, Gateway, Browser, Code Interpreter, Identity | Model-agnostic: Claude, Nova, Gemini, Llama, Mistral, GPT | Multi-model routing, GovCloud compliance, dynamic tool discovery |
| Azure AI Foundry Agent Service | Unified layer on the Responses API | Tight to Azure OpenAI, limited third-party catalog | Existing Azure OpenAI estates; Microsoft 365 integration |
| Google Vertex AI Agent Builder + Agent Engine | Design layer (Builder) + managed runtime (Engine) | Gemini-native, tightly integrated with Vertex AI Search | Document-heavy RAG, internal knowledge bases |
| Claude Agent SDK + Managed Agents | Brain / Hands / Session, self-hosted or Anthropic-hosted | Claude models; MCP servers for custom tools | Long-running async work, multi-tenant apps needing durable recovery |
The recommended path across nearly all of these vendors is the same shape: prototype with an SDK or open framework locally, where iteration is fast and debugging is direct; containerize once the logic is stable; put it behind the gateway so guardrails and logging apply uniformly; and move it to a managed runtime once it needs things that are expensive to build yourself , automatic recovery from failure, concurrent session scaling, and credential isolation (secrets live in a vault the agent never sees directly, rather than in its own process). Most production teams end up running both postures at once: an SDK for development, a managed service for the workloads that actually need to stay up.
Operational concerns that don’t show up in a framework comparison
- Non-human identity. Agents increasingly get their own scoped credentials and permission boundaries rather than inheriting a developer’s or a service account’s full access , this is where the governance frame from Part 01 becomes an actual access-control system rather than a policy document.
- Cost and latency budgets per run. A multi-step agentic call can fan out into dozens of model and tool invocations; teams that skip per-run budgets discover this in a bill, not a design review.
- Human-in-the-loop checkpoints. Bounded autonomy needs an actual escalation path , a defined point where a high-stakes action pauses for a person, not just a policy stating that it should.
- Eval gates in CI/CD. Prompt and agent changes go through the same regression discipline as code changes before promotion, using the same tools covered next.
Watching production agents: observability & evaluation
This tooling category matured fastest of anything in this document , by mid-2026, the large majority of organizations running agents in production had some form of agent observability in place, even though rigorous evaluation practice still lags behind basic tracing.
| Tool | License | What it does | Best for |
|---|---|---|---|
| Langfuse | Open-source | Nested tracing over OpenTelemetry, LLM-as-judge evals, CI regression tests | Self-hosting, data residency |
| LangSmith | Commercial | Deep tracing, AI-assisted debugging, online evals, unified cost tracking | LangChain/LangGraph shops |
| Braintrust | Commercial | Eval-first: versioned datasets, automated scoring, prompt-improvement loop | Evals as the workflow center |
| Arize (AX + Phoenix) | Source-available | ML-observability heritage, drift detection, RAG-quality plots | Regulated workloads, classic ML + LLM |
| MLflow | Open-source | Native agent tracing, OTel GenAI export, prompt optimization | Teams standardized on MLflow already |
| W&B Weave | Commercial | Multi-agent tracing linked to experiment history | ML research teams on Weights & Biases |
| Datadog LLM Obs. | APM extension | Correlates AI traces with infra, APM and security | Enterprises already standardized on Datadog |
Where this is actually landing
Two tensions are doing most of the work in real architecture decisions right now, and neither resolves cleanly.
Build versus buy is the obvious one: hyperscaler-native platforms (Bedrock AgentCore, Azure AI Foundry, Vertex AI Agent Builder) cut integration effort dramatically if an organization already lives in that cloud, at the cost of tighter coupling to that vendor’s model catalog and pricing. Open frameworks plus the MCP/A2A protocol layer reduce that lock-in, but shift real engineering and operational burden back onto the team , someone still has to run the gateway, the observability stack, and the identity system that the hyperscaler platforms bundle in. Neither answer is wrong; they’re different trades, and the right one depends more on how much platform team an organization already has than on which framework benchmarks best.
Governance maturity is running ahead of evaluation maturity. Most enterprises running agents in production now have tracing and logging in place , that part of the governance frame is real, not aspirational, for the majority of shops. Rigorous evaluation , the discipline of scoring agent output against a dataset before and after every change, and gating deploys on it , is adopted by meaningfully fewer of them. That gap is worth planning for explicitly rather than assuming observability tooling alone covers it; a trace tells you what an agent did, not whether it did the right thing.
The protocol layer is what actually de-risked this whole space for a CTO. Two years ago, choosing an agent framework meant betting your integration surface on one vendor’s abstractions. With MCP and A2A now under neutral, multi-vendor governance, that bet has shrunk to a component decision , which is the reason this document can cover six frameworks and four deployment platforms without any one of them being the “wrong” choice.
Vector databases: consolidating, not disappearing
The infrastructure & data layer from Part 01 deserves one more pass, because the vector-database market underneath agentic RAG went through its own version of the framework story this year. The 2023‚Äì2025 assumption was that a dedicated vector database was a required piece of any serious RAG stack. That assumption softened in 2026: Postgres, MongoDB, Oracle and every major cloud data warehouse now embed vector search directly, to the point where the more accurate description is that a vector became a data type, not a database category. Hyperscalers backed that shift with money, not just features , the wave of acquisitions of Postgres-focused vector companies by Databricks, Snowflake and AWS over the past year was a bet that most enterprise vector workloads belong next to the relational data they’re already grounding agents in, not in a separate system.
That doesn’t retire purpose-built vector databases , it narrows what they’re for. The decision now turns on workload shape rather than category loyalty:
| Option | Where it runs | Sweet spot | Watch out for |
|---|---|---|---|
| pgvector (Postgres) | Inside your existing relational database | Under ~2M vectors; vectors and transactional data in one query | p95 latency climbs sharply past a few million vectors |
| Pinecone | Fully managed, cloud-only | Sub-20ms latency at 5M+ vectors with no tuning | 3–8x the cost of a Postgres instance at equivalent scale |
| Qdrant | Self-hosted (Docker/Kubernetes) or managed | High throughput with selective metadata filtering, on a budget | Needs real operational familiarity with containers |
| Weaviate | Self-hosted or managed | Multimodal retrieval , text and image in one index | Schema-first design and SDK breaking changes add friction for text-only RAG |
The other force pushing on this layer is agentic usage itself: an agent loop that re-queries, re-ranks and self-corrects (the agentic and self-RAG patterns from Part 02) generates on the order of ten times the query volume of a human-driven search box, which shifts the engineering problem from raw latency toward sustained throughput and concurrency , connection pooling and caching strategies tuned for a person clicking search don’t hold up under a fleet of agents doing it continuously. Practically, that argues for starting with whatever store already sits under an organization’s transactional data (pgvector, or a warehouse’s native vector type) and graduating to a purpose-built database only once actual scale or latency numbers demand it , the same “don’t adopt the specialized tool until the plumbing it removes outweighs what it costs” logic from the framework discussion in Part 03 applies here too.
If there’s a single practical takeaway for standing up or re-architecting an agent program this year, it’s to sequence the decisions in roughly this order: settle the governance frame and the gateway pattern first, since retrofitting audit and identity onto a live system is far more expensive than building it in; pick a framework based on the coordination shape the actual workflow needs rather than which one is trending; build against MCP and A2A from day one so the framework and deployment-platform choices stay reversible; and treat the observability/eval stack as production infrastructure, not an add-on, from the first agent that touches a real customer or real data.
Don't Be So Serious 
Leave a comment