Back to Events
Completed

Build a Production-Ready YouTube AI Agent with Temporal

December 16, 2025, 01:00 Europe/Berlin
ai-agents data-engineering temporal elasticsearch production-systems

Full workshop writeup

Pages, code repo, and split-level access for Main and above.

View workshop writeup

We build a deep research agent over the DataTalks.Club podcast archive. The workshop starts by downloading and indexing YouTube transcripts, then turns that ingestion code into a durable Temporal workflow. After the data is searchable, we build a Pydantic AI research agent, add a summarization sub-agent for long transcripts, and wrap the agent run in Temporal too.

Links

The main resources for this workshop:

The system you will build

The final system looks like this:

flowchart LR YT["YouTube transcripts"] CACHE["Cached transcript files<br/>GitHub fallback"] FLOW["flow/<br/>Temporal ingestion"] ES["Elasticsearch<br/>podcasts index"] AGENT["agent/<br/>Pydantic AI research agent"] SUM["Summarization sub-agent"] TW["Temporal workflow<br/>agent run"] OPENAI["OpenAI"] YT -->|fetch_subtitles| FLOW CACHE -->|fetch_transcript_cached| FLOW FLOW -->|index_video| ES AGENT -->|search_videos| ES AGENT -->|summarize| SUM AGENT -->|model call| OPENAI SUM -->|model call| OPENAI TW -->|durable execution| AGENT

The ingestion side has the parts that usually fail in production: network calls, YouTube blocking cloud IPs, proxies, Elasticsearch writes, and long loops over many videos. Temporal gives that side retries, observability, and durable execution. The agent side uses the indexed data to answer questions from the podcast archive and then uses Temporal again so long agent runs can survive failures.

Walkthrough

Follow the numbered files in order. Each file is one self-contained step.

  1. Overview and setup - what you will build, prerequisites, local setup, secrets, and the two-project folder layout.
  2. Part 1: Fetch one transcript - fetch a YouTube transcript, convert it to subtitle text, and use cached transcript files when direct YouTube access is blocked.
  3. Part 2: Run Elasticsearch - run Elasticsearch in Docker, create the podcasts index, index one transcript, and search with snippets.
  4. Part 3: Discover podcast videos - discover podcast videos from the DataTalks.Club YAML file, skip problematic videos, process all episodes, and add proxy support after YouTube blocks the IP.
  5. Part 4: Temporal motivation - install Temporal, convert the notebook to Python, identify activities, and create activities.py.
  6. Part 5: Define the workflow - define the Temporal workflow, run it through a worker, handle sync activities, and pass imports through the Temporal sandbox.
  7. Part 6: Create the agent project - create the Pydantic AI research agent with Elasticsearch-backed search tools and tool-call logging.
  8. Part 7: Summarize long transcripts - add a summarization sub-agent so the research agent can work with full transcripts without exhausting context.
  9. Part 8: Wrap the agent in Temporal - wrap the Pydantic AI agent in a Temporal workflow and serialize the run context used by the summarization tool.
  10. Q&A: questions from the workshop - useful questions about Temporal, proxies, secrets, and orchestration choices.
  11. What to improve next - items intentionally left for a follow-up: parallel batching, structured references, stronger agent output, tests, and production hardening.
  12. Appendix: file inventory - file inventory, resource links, and learning-in-public notes.