Inside an AI Agent's Architecture: LLM, Memory, Tools and the Planning Loop (AI Agents for Enterprise, Part 2)

This is Part 2 of our series, "AI Agents for Enterprise." In Part 1 we drew the line between an AI agent and a traditional chatbot or RPA bot: instead of executing a fixed script, an agent receives a goal and figures out the steps itself. Now we open the hood and look at the four building blocks that make an agent run.

By 2026 the industry has settled on a single formula: Agent = LLM + Memory + Planning + Tool use. The piece that ties all four together at runtime—the agent loop—is the real heart of the system. Let's take them in order.

1. The LLM core: the agent's "brain"

At the center sits a large language model. The crucial difference from ordinary chat is that here the LLM acts as a controller that decides what to do at every step. It doesn't just generate a reply; it reasons about which tool to call and why, reads the result, and decides the next move.

For enterprises, model selection drives both quality and cost. "Model routing"—using a powerful model for complex reasoning and a lightweight one for routine work—is becoming standard practice in 2026. Remember: the intelligence of the core sets the ceiling for the whole agent.

2. Memory: a two-layer architecture

An LLM only has its context window as working memory, and it forgets everything once a session ends. That's why we bolt on a memory layer. In 2026 memory is no longer an afterthought—it is a core component that is benchmarked and evaluated independently.

• Short-term (working) memory: holds the context of the current task or conversation.
• Long-term memory: persists across sessions, and splits further into episodic memory (past events and dialogues) and semantic memory (facts such as user preferences and business knowledge).

The common implementation stores embeddings in a vector database (Qdrant, Pinecone) and retrieves semantically related memories. But a real limitation has surfaced: vector search alone lacks governance, versioning and workflow state. Dedicated memory products—Mem0, Letta, Zep—have matured into standalone tools. RAG isn't disappearing, but long-context ("context architecture") memory is overtaking it for agentic use cases.

3. Tools: the "hands and feet" to the outside world

Without tools, an agent can only produce text. With them it can query databases, call APIs, search the web, execute code and pull real-time data. Technically, function calling is the foundation.

The biggest shift in 2026 is the rise of the Model Context Protocol (MCP). This open standard, released by Anthropic in November 2024, unifies how agents connect to tools and data. It isn't a rival to function calling—it layers standardization on top of it. OpenAI adopted MCP in 2025, and it is now the de-facto standard for connecting agents to tools. The enterprise upside is clear: expose an internal system once as an MCP server, and many agents can reuse it the same way.

4. Planning and the loop: the runtime that binds it all

The loop is what binds the four components together on every step. An agent runs a cycle: receive a goal, break it down, call tools, store results in memory, and repeat until it has enough to answer. Three planning patterns dominate in 2026.

• ReAct (Reasoning + Acting): alternates thought, action and observation. The most widely used pattern, prized because its decision process is inspectable and easy to debug. Start here for anything interactive.
• Plan-and-Execute: builds the full plan first, then executes each step. Useful when a ReAct agent keeps re-deriving the same plan for every request.
• Reflexion: the agent critiques its own output and retries on failure. Wrap it as an outer loop when final quality matters more than wall-clock time.

In 2026 production systems these patterns are composed, not used alone. A typical coding assistant runs a Plan-and-Execute outer loop, where each executor step is a ReAct agent with its own tools, and the whole run is wrapped in a Reflexion pass that re-runs the failing tests.

Quick reference

What enterprises should take away

A key lesson: the most autonomous system is not the most reliable one. What earns trust in 2026 production is a design that places autonomy exactly where it creates value and constrains it everywhere else. Test memory independently, standardize tools through MCP, and keep the loop observable and debuggable—these three principles are the foundation for moving past a PoC and into real production.

In Part 3 we'll take this same architecture and dig into the question of self-hosting versus cloud, comparing the two deployment models across three axes: security, cost and data sovereignty. Stay tuned.