Build a Documentation RAG Agent with Search Tools

We build a RAG system over real documentation and evolve it into an agentic search flow. For the knowledge base we use the Evidently AI documentation, a real set of Markdown files that keeps changing as the library evolves. This is where RAG helps most, since LLMs can't keep up with library docs on their own.

What we cover:

• Build a classic RAG pipeline: index documents with minsearch and retrieve highlighted snippets
• Implement search (returns snippets) and get_file (returns the full document)
• Let the LLM decide which tool to call
• Add the tools to an agent using toyaikit and PydanticAI
• The agent mirrors how humans read docs: search, scan snippets, open the promising one

We originally delivered this workshop at DataMakersFest 2026 in Porto, Portugal. There's no recording available.

The system you will build

The architecture has one LLM agent with two tools and two data sources:

The agent has two tools. search returns short highlighted snippets so the agent can decide which document to open, and get_file returns the full document so it can read that one end-to-end. This mirrors how humans read documentation: search, scan snippets, open the promising one.

Links

Resources not included in the workshop materials list:

• The Agentic Loop from Scratch - the same loop built by hand, with no framework.
• toyaikit - teaching framework for agents
• PydanticAI - production agent framework