Build an Agent Evaluation Framework with OpenAI Batch and Flex

We take the FAQ agent from the first deployment workshop and build an evaluation framework around it. We leave the agent unchanged, so all the new work goes into the surrounding pipeline. That pipeline generates synthetic student questions, runs the agent over them, then has an LLM judge score the answers. The pipeline produces a repeatable eval set we can rerun after every change to the agent.

Two OpenAI pricing features make the eval cheaper to run. The Batch API processes a whole file of requests at about half the standard price, with results returned within 24 hours, usually a few minutes. Flex processing charges the same batch rate but returns results in real time, with a longer and less predictable wait. Both can combine with prompt caching, which applies automatically when requests share a prompt prefix.

In this session we explore rather than ship a polished build. We port a working synthetic-data generator from the AI Engineering Buildcamp to this project. Then we try Batch and Flex against the same task to see what each gives us.

Generation and judging are batches of independent one-shot LLM calls, so both fit the Batch API. Running the agent needs live back-and-forth tool calls, so we keep it on the standard endpoint.

For most of the coding we used Claude Code with Opus. Most prompts are dictated, and the agent both implemented the scripts and tested them. The code is exploration quality. The speaker calls out places to refactor for a cleaner version.

Links

Resources used here:

• End-to-End Agent Deployment - the FAQ agent we evaluate.
• AI Engineering Buildcamp synthetic eval code - the working version we port from.
• OpenAI Batch API guide - the docs for the batch endpoint.
• OpenAI Flex processing guide - the docs for the flex tier.
• OpenAI prompt caching - how prompt caching applies across Batch and Flex.