Solving a Real AI Engineer Take-Home Assignment Live
Continue with the workshop writeup
Open the canonical pages, recording, materials, and code repo.
We solve a real AI engineer take-home assignment live. The product is a small investment coaching bot that answers public-company research questions without crossing into personalized financial advice.
The flow of the session:
- Brainstorm scope and survey data sources.
- Build a CLI agent with PydanticAI and OpenAI.
- Pivot from a paid market-data API to free SEC EDGAR data after the free tier blocks us.
- Clean up the pointless tests the coding agent generated and write test guidelines we actually want to follow.
- Add a small evaluation harness with scenarios and an LLM judge.
- Wrap the result as a Telegram bot.
Every step is done alongside a coding agent. The prompts are included verbatim so you can reproduce the workflow with any agent you like.
Links
Useful resources for this workshop:
The app you will build
The final app looks like this:
The agent has four tools, all backed by free SEC EDGAR endpoints:
search_company(query)resolves a ticker or company name to a CIK.get_financial_snapshot(ticker_or_cik)fetches recent annual revenue, profit, balance sheet, and cash facts from companyfacts XBRL data.get_latest_filings(ticker_or_cik)returns recent filing metadata and SEC URLs.get_filing_digest(ticker_or_cik, form_type)pulls the latest 10-K or 10-Q text and extracts business, revenue, risk, and MD&A snippets.
The agent has a strict safety boundary baked into its instructions:
- no buy/sell/hold recommendations
- no position sizing
- no predictions of price direction
A buy/sell question gets transformed into an educational research brief instead.
The evaluation harness lives in evals/manual/. It runs a list of
scenarios from scenarios.csv through the agent and captures the
full tool-call trajectory. An LLM-as-judge then scores whether the
answer stayed inside the safety boundary and addressed the question.
Hosted by
Alexey Grigorev
Chief Agent Officer at AI Shipping Labs
Software engineer and machine learning practitioner with 15+ years of experience building production ML systems. I focus on practical, production-grade ML and AI systems, from early prototypes to reliable systems in production.
I'm the founder of DataTalks.Club, a free community that connects tens of thousands of practitioners worldwide, and the creator of the Zoomcamp series, free, code-first programs that have reached 100,000+ learners globally.
At AI Shipping Labs, I'm building the kind of environment that would have accelerated my own career growth. After years of teaching at scale, I wanted something more focused: a space for action-oriented builders who want to turn AI ideas into real projects. The community gives members the structure, accountability, and peer support to ship practical AI products consistently, even alongside their main jobs.
More from AI Shipping Labs
Inference Engineering Book Club — Week 1
For this first session, finish Chapters 0, 1, and 2 (Inference, Prerequisites, Architecture). We kick off the weekly reading rhythm. Come with your questions and takeaways. Can't …
August 17, 2026
Inference Engineering Book Club — Kickoff
We are starting a book reading club, and the first book is Inference Engineering by Philip Kiely (Baseten). It is a book for engineers who want to understand the technologies behi…
August 10, 2026
LLM Zoomcamp 2026 office hours — Session 6
August 3, 2026