Tracing a FAQ agent with Langfuse

We add Langfuse tracing to a FAQ agent, then turn its traces into reviewed evaluation examples. We start with the existing FAQ application, add the Cloud integration and repeat the setup with a local Langfuse instance.

Follow the code changes to capture model calls and searches, then look at their traces before choosing which answers belong in an evaluation dataset.

Links

Useful resources for this workshop:

• Workshop recording and chapter markers
• End-to-End Agent Deployment - the FAQ application we add tracing to.
• Data Engineering Zoomcamp FAQ - the questions and answers our agent searches.
• Langfuse getting started - Cloud projects, credentials and regional endpoints.
• Langfuse SDK overview - instrumentation and OpenTelemetry.
• Self-host Langfuse with Docker Compose - the local setup.
• Langfuse annotation queues - human review of captured traces.
• Langfuse datasets - reviewed examples for evaluations.

The app you will build

We capture the FAQ agent's calls automatically and review its answers before creating reference answers:

We use the FastAPI application from End-to-End Agent Deployment. It searches the DataTalks.Club FAQ and streams answers to a browser. We add the Langfuse client wrapper and observation decorators to that application. Then we create the scripts for moving reviewed traces into an evaluation dataset.