Course External course 167 enrolled

AI Engineering Buildcamp: From RAG to Agents

Alexey Grigorev

AI/ML engineer, founder of AI Shipping Labs and DataTalks.Club

ai-agents rag llm testing monitoring evaluation

A hands-on course on building production AI applications with large language models: retrieval-augmented generation (RAG), structured output, AI agents, testing, monitoring, and evaluation. Learners build real projects -- a documentation assistant, a YouTube summarizer, a books RAG system, and their own capstone -- across seven core modules, from foundational RAG through agentic flows to production-grade testing, monitoring, and evaluation.

Past cohort

Cohort 1

Started October 6, 2025

Past cohort

Cohort 2

Started January 26, 2026

Past cohort

Cohort 3

Started April 13, 2026

Current cohort

Cohort 4

Started September 21, 2026

Syllabus

Pre-work and logistics

Course Logistics 6 lessons

Course Overview Communication Homework and Project Submissions Learning in Public Office Hours Optional Workshops Optional

Main content

Week 1 · Foundations: LLMs, RAG, and Structured Output 4 topics · 27 lessons

Session 1 Week 1 Overview

Environment Preparation 6 lessons

GitHub Codespaces Windows Linux MacOS AI-Assisted Development Libraries Overview

Foundation: LLMs, RAG and Search 8 lessons

Section Overview Documentation Agent: Running Example for the Course OpenAI API RAG (Retrieval-Augmented Generation) Getting Documentation Data from GitHub Indexing Data for Search with minsearch Chunking Documents for Better Retrieval End-to-End Document Assistant

Structured Output 3 lessons

Structured Output Structured Output for Documentation RAG Streaming Structured Output

Project Work 5 lessons

Project Work From Idea to Submission Project Ideas for Your AI Capstone Cohort 2 Project Ideas Other Project Ideas

Homework: Document Processing with AI Capstone: Your AI Project Optional Content Optional

Week 2 · RAG in Practice: Use Cases and Technologies 1 topic · 4 lessons

Session 2

RAG Use-Cases Overview 2 lessons

Week overview Reusable RAG Class

Optional Content Optional

Week 3 · Agentic Flows 2 topics · 16 lessons

Session 3

Agentic RAG 6 lessons

Agent Basics Overview Agentic RAG with Tool Calling Agentic Loop Q&A Loop and an Agent Library Visual Summary Structured Output for Agents (Optional) Optional

Agent Frameworks 6 lessons

Agent Frameworks Overview ToyAIKit for Agent Implementation Organizing Tools in a Class Agentic Search: Going Beyond RAG Pydantic AI: Production-Ready Agents Event Stream Handlers

Homework: Building a Wikipedia Agent Capstone: Adding Agents to Your Project Optional Content Optional

Week 4 · Testing AI Applications 3 topics · 23 lessons

Session 4 Testing and Evaluation Overview

Project Structure 7 lessons

Preparing the Project Converting a Jupyter Notebook into a Python Project Streaming with PydanticAI Optional Streaming Methods in PydanticAI Optional Structured Streaming with PydanticAI Optional Streamlit Frontend for Documentation Agent Optional How the Project and Tests Fit Together Optional

Testing for Agents 6 lessons

Testing for Agents First Agent Test Testing Tool Calls Fixtures and Refactoring More Tests: Code Blocks and Adding References Tracking Costs Optional

Test Judges 5 lessons

LLM Judges Creating an LLM Judge From Usage to Test Cases LLM Judge Tests Improving Judge Tests

Homework: Testing a DuckDB SQL Agent Capstone: Testing Your Agent Optional Content Optional

Week 5 · Monitoring 1 topic · 9 lessons

Session 5

Monitoring with Pydantic Logfire 5 lessons

Logfire Overview Logfire Integration with PydanticAI Joining Multiple Runs Under One Span Tree Tracking User Feedback Downloading Traces Data from Logfire

Homework: Monitoring a Trivia Agent with Logfire Capstone: Monitoring Your Agent Optional Content Optional

Week 6 · Evaluation 2 topics · 16 lessons

Session 6 Overview

Measure and Align: Offline Evaluation for Agents 6 lessons

Offline Evaluation for Agents Collecting Evaluation Data Running the Agent on Scenarios Evaluating AI Responses Judge and Alignment Measuring Agent Quality with the Judge

Synthetic Data Generation for Evaluations 4 lessons

Synthetic Evaluation Data Generating Synthetic User Questions Running the Agent on Synthetic Data Running the Judge on Synthetic Data

Homework: Evaluating a Recipe Assistant Capstone: Evaluating Your Agent Optional Content Optional Evaluate Retrieval Before Generation Optional

Project · weeks 7–9

Capstone Project 1 topic · 10 lessons

Session 7

Capstone Project 6 lessons

Project Work Project Guidelines Evaluation Criteria Documentation Tips Submission and Peer Review Hackathon

Session 8 Capstone Presentations Optional Content Optional

Optional

Optional 27 topics · 133 lessons

Workshop Projects 1 lesson

More Workshop Projects

Week 1

OpenAI Alternatives 9 lessons

Alternative Providers Overview OpenAI Chat Completions API Groq Anthropic Amazon Bedrock Google Gemini Z.ai Grok Ollama

Search Materials 3 lessons

Advanced Search Overview Vector Search Advanced Chunking

Week 2

FAQ Assistant 5 lessons

FAQ Assistant Overview Dataset Q&A RAG Use Case: DataTalks.Club FAQ Chatbot Use Case: RAG for Content Management

YouTube Transcripts 5 lessons

YouTube Transcripts Getting Transcripts from YouTube Videos Prompt Engineering for Video Summaries Structured Output for Video Summaries Search and RAG with Transcripts

PDF Data Extraction (Math Book) 9 lessons

Extracting Information from PDF Books Extracting Text from PDFs Structured Output Cleaning and Refactoring Processing All Book Pages Parallel Processing for Book Pages RAG on a Math Book External Persistent Knowledge Bases Persistent RAG with SQLiteSearch

Text Search with Elasticsearch 4 lessons

Elasticsearch Overview Docker Setup for Elasticsearch Ingesting Documents into Elasticsearch Querying Elasticsearch

Vector Search with Qdrant 4 lessons

Qdrant Overview Docker Setup for Qdrant Ingesting Documents into Qdrant Querying Qdrant with Vector Search

Week 3

Web and YouTube Researcher 6 lessons

Web Research Overview Web Search with OpenAI Built-in Tool Web Search Agent with PydanticAI Web Search with Brave API Web Page Content Extraction YouTube Search and Web Research Agent

Tool Calling with Alternative Providers 6 lessons

Tool Calling with Alternative Providers Preparing the Agents Library Tool Calling with OpenAI Responses API Tool Calling with Chat Completions API Tool Calling with Anthropic Tool Calling with Gemini

Agent Frameworks - Alternatives 4 lessons

Agent Framework Alternatives OpenAI Agents SDK LangChain Google ADK

Model Context Protocol (MCP) 5 lessons

MCP Host, Client, and Server Model Context Protocol Overview Stdio Transport Sse Transport Cursor Integration

Week 4

Libraries for Agent Testing 2 lessons

LangWatch Scenario for Agent Testing Evaluation with Evidently

Multi-Agent Systems 18 lessons

Multi-Agent Patterns Workflows and Orchestrated Patterns Group 1: Routing Patterns Simple Routing Agents as Tools Handoffs Group 2: Pipeline Patterns Prompt Chaining Static Plan-and-Execute Dynamic Plan-and-Execute Group 3: Feedback Patterns Human-in-the-Loop Fixed Evaluator-Optimizer Dynamic Evaluator-Optimizer Group 4: Coordination Patterns Parallelization Orchestrator-Workers Multi-Agent Collaboration

Additional Frameworks and Graphs 8 lessons

CrewAI Multi-Agent LangChain Multi-Agent Google ADK Multi-Agent Agentic Graphs Pydantic Graph: Retry Loop Pydantic Graph: Routing Pydantic Graph: Human in the Loop LangGraph

Week 5

Pydantic Logfire and Jaeger 1 lesson

Using Jaeger as OTel Backend

DIY Logging and Monitoring 8 lessons

DIY Monitoring Defining the Log Records Integrating the Monitoring Service into the Application Saving Logs and Events to SQLite Creating a Streamlit Dashboard Fake Logs Generator Online Judge Building a Dashboard with Grafana and Postgres

Other Observability Platforms 2 lessons

Langwatch Evidently (previous lesson)

Week 6

Code Improvements 2 lessons

Refactoring the Evaluation Code Using AI Assistant to Improve the Judge

Week 7

My Daily Tasks: End-to-End Project Example 7 lessons

Environment Preparation Implementing Tools from OpenAPI Specs Implementing the Agent with Pydantic AI Organizing the Code into a Project and Creating Streamlit App Testing the Agent Test Judge Monitoring with Pydantic Logfire

Guardrails 1 lesson

Guardrails workshop

Deployment 4 lessons

Deploying Your Agent Deploying to Streamlit Community Cloud Deploying to Render Deploying to AWS

Week 8

Build a Coding Agent from Scratch 6 lessons

Intro Django Template Agent Tools Coding Agent Zai Multi Agent

Code Analysis Agent 6 lessons

Introduction Project Scorer Tools Project Scorer Agent File Tools for Code Analysis The Code Analysis Agent Streamlit Demo

Deep Research Agent 3 lessons

Deep Research Multi Agent Research Orchestration

Student Project Examples 1 lesson

Cohort 1

Week 8 · Deployment 3 lessons

Choose Your Deployment Prepare Your Project for Hosting Launch and Verify Your Demo

External course

Enroll via Maven to access this course

This is an external course, run on another platform — it isn't included in any AI Shipping Labs membership plan.

Enroll now