Tag: evaluation | AI Shipping Labs

Jev for LLM Evaluations

In the batch and flex workshop we built an eval for the FAQ agent and scored it with an LLM judge. Here we swap that judge for Jev, a model that returns typed decisions instead of generating text. We keep the same eval, ten agent answers with their tool calls and three good/bad checks each. One Jev request scores all three checks for one answer. On our ten-entry run it cost $0.000429 while the OpenAI judge cost $0.02439, about 57 times cheaper. Jev is TypeSafe AI's decision model. It reads a piece of application state and returns a class per question. It generates no output tokens, so there's no output-token cost and no wait for streaming text. We ask a judge only to say good or bad, and Jev is built for exactly that. We keep the OpenAI judge as the baseline and add a Jev judge through OpenRouter's decisions API. The same checks also run through Laya, an open-weights decision model that runs locally on a CPU. All three judges score the same run, so we can compare verdicts and costs directly. On our sample they agree almost everywhere. All three mark the same answer incorrect, and Jev and Laya return identical scores on every check. We dictated most of the code as prompts to an AI assistant. We include the prompts, so you can reproduce the workflow with any agent you like. We tried one approach that didn't work: batching ten records into one request by making state an array. It looks natural, but the model returns one mixed answer. The final code sends one request per entry and runs them concurrently. Links Resources used here: Saving Money with Batch and Flex - the eval framework we port from. End-to-End Agent Deployment - the FAQ agent we evaluate. Jev on OpenRouter - the model page, with pricing and the decisions API. Laya - the open-weights decision model we run locally.

Sep 22, 2026

evaluation llm-as-a-judge jev openrouter coding-agents

Course

AI Hero: 7-Day AI Agents Crash-Course

Build a complete AI agent from data ingestion to deployment in 7 focused days. You will use a real GitHub repository as the source material, then build an assistant that can read the project, search through its docs and code, answer questions with citations, and ship as a small web app. This is not a toy chatbot course. By the end, you will have a portfolio-ready AI project with code, evaluation, a demo, and a clear README. What your agent will be able to do Your agent will understand a real codebase, not a tiny example file. You can use your own repository, or choose an open-source project if you do not have one ready. During the course, you will build an assistant that can: Read documentation, code, issues, and comments from a GitHub repository Split large files into useful chunks for retrieval Search with lexical search, semantic search, and hybrid search Answer project-specific questions using RAG Use tools and function calling for more useful responses Keep answers grounded with citations and logs Evaluate the quality of search and agent responses Run behind a simple Streamlit interface Be shared as a portfolio project Example question: How does authentication work in this project? By the end of the course, your agent should be able to answer questions like this by looking at the project files it indexed and pointing you to the relevant parts of the codebase. What you need Plan for 1-2 hours per day. The lessons are practical and build on each other, so it is better to do a little every day than to read everything at once. You will need: Basic Python skills: functions, classes, virtual environments, and pip Comfort with the command line A GitHub repository to analyze, or an open-source repository you want to study An OpenAI API key for the hands-on exercises A laptop where you can run Python locally The OpenAI API cost for the course is usually small. For most projects, expect a few euros or dollars, depending on the size of the repository and how much you experiment. What you will build in 7 days Each day adds one piece to the same project. Day 1: ingest and index your data from GitHub Day 2: chunk and prepare the data for search Day 3: add lexical, semantic, and hybrid search Day 4: turn the search system into an agent with tools Day 5: add logging, citations, and evaluations Day 6: publish the agent with a Streamlit UI Day 7: write the README, record a demo, and share the result Certificate: submit the finished project and get the completion certificate The syllabus below contains the detailed lessons and homework for each day. What you will have at the end When you finish, you will have a working AI assistant that understands a GitHub project and can answer questions about it. You will also have the supporting pieces that make it useful outside a notebook: Source code you can keep extending A documented ingestion and search pipeline An agent with tools and prompts Evaluation data and checks A deployed web interface A README with screenshots or a demo video A certificate of completion after project submission The goal is to leave with something you can show: in a portfolio, in a job conversation, or as a base for your next AI engineering project. Frequently asked questions What makes this different from other AI tutorials? You build with a real GitHub project, not a generic demo. The course connects the full path from data ingestion to search, agent behavior, evaluation, deployment, and portfolio packaging. What if I do not have a GitHub project? Use an open-source project that interests you. The important part is learning with real code and real documentation. Do I need an OpenAI API key? Yes. The hands-on exercises use the OpenAI API. The course shows how to keep usage small while still building the full workflow. Can I do it on a phone or tablet? You can read the lessons anywhere, but you need a computer for the coding exercises. Why is it free? This course gives you a practical foundation and shows the teaching style we use at AI Shipping Labs. If you want more structure, feedback, and accountability after that, join the community and continue with sprints, workshops, and project reviews.

Sep 21, 2026

ai-agents rag python pydantic-ai evaluation

Course

AI Engineering Buildcamp: From RAG to Agents

A hands-on course on building production AI applications with large language models: retrieval-augmented generation (RAG), structured output, AI agents, testing, monitoring, and evaluation. Learners build real projects -- a documentation assistant, a YouTube summarizer, a books RAG system, and their own capstone -- across seven core modules, from foundational RAG through agentic flows to production-grade testing, monitoring, and evaluation.

Sep 21, 2026

ai-agents rag llm testing monitoring evaluation