Saving Money on API Calls with Batch and Flex Processing
API costs get expensive fast when you run many model calls. In this workshop we'll look at where you can deliberately trade latency for lower cost, especially for evaluations where results usually do not need to be immediate. We'll focus on: - When evaluations can run asynchronously instead of inline - Using OpenAI Batch API for eval runs and other delayed jobs - Using Flex processing for lower-priority Responses / Chat Completions calls - Prompt caching, model choice, and request shaping as cost levers - How to structure an eval pipeline so slower, cheaper calls are acceptable - Failure modes: timeouts, retries, expired batches, and when not to delay Bring an existing eval idea or a workflow with repeated API calls if you want to reason through it live. References: - https://developers.openai.com/api/docs/guides/batch - https://developers.openai.com/api/docs/guides/flex-processing