Evals in CI
Event times are shown in Europe/Berlin. Sign in to use your timezone
This event is for Main members
Membership is required before you can register for this event.
You change a prompt, the tests pass, and the agent quietly gets worse. Most teams run evals by hand, if at all, so regressions slip into main.
In this workshop we take an eval set and run it in CI, so a pull request that makes the agent worse doesn't get merged.
What we'll cover: - Start from an existing eval set for an agent (we built one in the eval framework workshop) - Make the eval run reproducible and fast enough for CI - Run the eval suite with pytest in GitHub Actions on every pull request - Define thresholds and fail the build when quality drops - Report the results on the pull request so you can see what changed - Keep the cost of running evals on every PR under control
You'll leave with a CI setup that gates merges on agent quality, which you can copy into your own project.
Who it's for: people who already have an agent or RAG app with some tests or evals and want to stop regressions.
Prerequisites: Python, Git and GitHub (we use GitHub Actions), an LLM API key. Watching the eval framework workshop first helps. The code and recording are shared with members after the session.
Hosted by
Alexey Grigorev
Chief Agent Officer at AI Shipping Labs
Software engineer and machine learning practitioner with 15+ years of experience building production ML systems. I focus on practical, production-grade ML and AI systems, from early prototypes to reliable systems in production.
I'm the founder of DataTalks.Club, a free community that connects tens of thousands of practitioners worldwide, and the creator of the Zoomcamp series, free, code-first programs that have reached 100,000+ learners globally.
At AI Shipping Labs, I'm building the kind of environment that would have accelerated my own career growth. After years of teaching at scale, I wanted something more focused: a space for action-oriented builders who want to turn AI ideas into real projects. The community gives members the structure, accountability, and peer support to ship practical AI products consistently, even alongside their main jobs.
Related content
Durability
Agents run for minutes or hours, call slow tools and sometimes wait for a human. If the process dies halfway, the run is gone and you start over, sometimes repeating actions that …
January 12, 2027
MCP
MCP lets you write a tool once and use it from many clients: your own agent, Claude Code, Cursor, VS Code. Our previous MCP workshop is more than a year old, and a lot has changed…
January 5, 2027
Freestyle Workshop
A freestyle workshop: you come with your own topic, problem or question, and we work on it live. We try to turn something useful into working code during the session. Previous fre…
January 4, 2027