Back to Courses
Course External course

AI Engineering Buildcamp: From RAG to Agents

Alexey Grigorev

AI/ML engineer, founder of AI Shipping Labs and DataTalks.Club

ai-agents rag llm testing monitoring evaluation

A hands-on course on building production AI applications with large language models: retrieval-augmented generation (RAG), structured output, AI agents, testing, monitoring, and evaluation. Learners build real projects -- a documentation assistant, a YouTube summarizer, a books RAG system, and their own capstone -- across nine modules, from foundational RAG through agentic flows to production-grade testing, monitoring, and evaluation.

Next cohort

Cohort 4

Starts September 21, 2026

Syllabus

1 Foundations: LLMs, RAG, and Structured Output
Course Overview
Communication
Course Management Platform
Learning in Public
Office Hours
Week 1 Overview
Environment Preparation (GitHub Codespaces)
Environment Preparation (Windows)
Environment Preparation (Linux)
Environment Preparation (MacOS)
AI-Assisted Development
Libraries Overview
Section Overview
Documentation Agent: Running Example for the Course
OpenAI API
RAG (Retrieval-Augmented Generation)
Getting Documentation Data from GitHub
Indexing Data for Search with minsearch
Chunking Documents for Better Retrieval
End-to-End Document Assistant
Structured Output
Structured Output for Documentation RAG
Streaming Structured Output
Advanced Search Overview
Vector Search
Advanced Chunking
Alternative Providers Overview
OpenAI Chat Completions API
Groq
Anthropic
Amazon Bedrock
Google Gemini
Z.ai
Grok
Ollama
Project Work
From Idea to Submission
Project Ideas for Your AI Capstone
Cohort 2 Project Ideas
Other Project Ideas
Module 1 Homework: Document Processing with AI
Module 1 Capstone: Your AI Project
2 RAG in Practice: Use Cases and Technologies
Reusable RAG Class
FAQ Assistant Overview
Dataset
Q&A RAG
Use Case: DataTalks.Club FAQ Chatbot
Use Case: RAG for Content Management
YouTube Transcripts
Getting Transcripts from YouTube Videos
Prompt Engineering for Video Summaries
Structured Output for Video Summaries
Search and RAG with Transcripts
Extracting Information from PDF Books
Extracting Text from PDFs
Structured Output
Cleaning and Refactoring
Processing All Book Pages
Parallel Processing for Book Pages
RAG on a Math Book
External Persistent Knowledge Bases
Persistent RAG with SQLiteSearch
Elasticsearch Overview
Docker Setup for Elasticsearch
Ingesting Documents into Elasticsearch
Querying Elasticsearch
Qdrant Overview
Docker Setup for Qdrant
Ingesting Documents into Qdrant
Querying Qdrant with Vector Search
3 Agentic Flows
Overview
Agent Basics Overview
Agentic RAG with Tool Calling
Agentic Loop
Q&A Loop and an Agent Library
Structured Output for Agents (Optional)
Visual Summary
Agent Frameworks Overview
ToyAIKit for Agent Implementation
Organizing Tools in a Class
Agentic Search: Going Beyond RAG
Pydantic AI: Production-Ready Agents
Event Stream Handlers
Web Research Overview
Web Search with OpenAI Built-in Tool
Web Search Agent with PydanticAI
Web Search with Brave API
Web Page Content Extraction
YouTube Search and Web Research Agent
Tool Calling with Alternative Providers
Preparing the Agents Library
Tool Calling with OpenAI Responses API
Tool Calling with Chat Completions API
Tool Calling with Anthropic
Tool Calling with Gemini
Agent Framework Alternatives
OpenAI Agents SDK
LangChain
Google ADK
Model Context Protocol Overview
Stdio Transport
Sse Transport
Cursor Integration
Week 3 Homework: Building a Wikipedia Agent
Week 3 Capstone: Adding Agents to Your Project
4 Testing AI Applications
Testing and Evaluation Overview
Preparing the Project
Converting a Jupyter Notebook into a Python Project
Streaming with PydanticAI
Streaming Methods in PydanticAI
Structured Streaming with PydanticAI
Streamlit Frontend for Documentation Agent
Testing for Agents
First Agent Test
Testing Tool Calls
Fixtures and Refactoring
More Tests: Code Blocks and Adding References
Tracking Costs
LLM Judges
Creating an LLM Judge
From Usage to Test Cases
LLM Judge Tests
Improving Judge Tests
Evaluation with Evidently
Multi-Agent Patterns
Workflows and Orchestrated Patterns
Group 1: Routing Patterns
Simple Routing
Agents as Tools
Handoffs
Group 2: Pipeline Patterns
Prompt Chaining
Static Plan-and-Execute
Dynamic Plan-and-Execute
Group 3: Feedback Patterns
Human-in-the-Loop
Fixed Evaluator-Optimizer
Dynamic Evaluator-Optimizer
Group 4: Coordination Patterns
Parallelization
Orchestrator-Workers
Multi-Agent Collaboration
CrewAI Multi-Agent
LangChain Multi-Agent
Google ADK Multi-Agent
Agentic Graphs
Pydantic Graph: Retry Loop
Pydantic Graph: Routing
Pydantic Graph: Human in the Loop
LangGraph
Week 4 Homework: Testing a DuckDB SQL Agent
Week 4 Capstone: Testing Your Agent
5 Monitoring
Overview
Logfire Overview
Logfire Integration with PydanticAI
Joining Multiple Runs Under One Span Tree
Tracking User Feedback
Downloading Traces Data from Logfire
Using Jaeger as OTel Backend
DIY Monitoring
Defining the Log Records
Integrating the Monitoring Service into the Application
Saving Logs and Events to SQLite
Creating a Streamlit Dashboard
Fake Logs Generator
Online Judge
Building a Dashboard with Grafana and Postgres
Evidently
Langwatch
Week 5 Homework: Monitoring a Trivia Agent with Logfire
Week 5 Capstone: Monitoring Your Agent
6 Evaluation
Overview
Offline Evaluation for Agents
Collecting Evaluation Data
Running the Agent on Scenarios
Evaluating AI Responses
Judge and Alignment
Measuring Agent Quality with the Judge
Synthetic Evaluation Data
Generating Synthetic User Questions
Running the Agent on Synthetic Data
Running the Judge on Synthetic Data
Refactoring the Evaluation Code
Using AI Assistant to Improve the Judge
Week 6 Homework: Evaluating a Recipe Assistant
Week 6 Capstone: Evaluating Your Agent
7 Project Work and Guardrails (Bonus)
Overview
Project Work
Project Guidelines
Evaluation Criteria
Documentation Tips
Submission and Peer Review
Deploying Your Agent
Deploying to Streamlit Community Cloud
Deploying to Render
Deploying to AWS
8 Use Cases
Overview
Intro
Django Template
Agent Tools
Coding Agent
Zai
Multi Agent
File Tools for Code Analysis
The Code Analysis Agent
Streamlit Demo
Deep Research
Multi Agent Research
Orchestration
9 Capstone Demo Day
Overview
External course

Enroll via Maven to access this course

This is an external course, run on another platform — it isn't included in any AI Shipping Labs membership plan.

Enroll now