Human checkpoints and model tiering for agent teams

We met on 15 September 2026 for the second office hours of the AI dev tools course. One member joined the Q&A and live demo. The host shared his screen and walked through his agent pipeline, from a raw idea to a deployed and checked change. You can follow this workflow without watching the recording.

Add a human checkpoint where assumptions are made

The most valuable human intervention comes during grooming, before agents spend tokens building the wrong feature.

1. When the PM needs to make an assumption during grooming, it reports the assumption to the orchestrator before an engineer starts work.
2. Tag the issue needs human input so the orchestrator skips it until you respond.
3. Approve, deny, or correct the assumption. Remove the tag to return the issue to the pipeline.
4. Write rules for when the PM should ask you and when it should continue. Otherwise, it may ask at every step or let the team build the wrong thing.
5. Check the tagged issues yourself. GitHub issues give you no dedicated approval queue, which was the limitation discussed in the call.

Changing direction during implementation only helps if every working agent receives the correction immediately. Today's tools don't reliably do that, so place the checkpoint before implementation.

Keep the process in an editable markdown file

Write the pipeline in a markdown file so you can change it for each project. The host drew this flow during the call:

• A human reports a bug or idea → the orchestrator (your main session) creates a GitHub issue.
• The PM grooms the issue into acceptance criteria → a software engineer implements → QA accepts or rejects (a rejection loops back to the engineer) → the PM signs off that the acceptance criteria are actually covered by the work and the tests.
• Keep the role order fixed within one issue (PM → software engineer → QA → PM). Have the orchestrator enforce it from the process file. Earlier models sometimes skipped roles. Current models usually follow the written order.
• Insert a human gate where an early decision can prevent wasted work. For example, build a frontend mockup with a mocked backend after grooming, then ask for a decision before implementing the backend. The course's module-2 full-stack app followed that frontend-first sequence.

Give every issue its own git worktree

Give each issue a git worktree on a local branch. Agents can then work on different issues in the same repository without editing each other's files. The orchestrator creates each worktree, tracks the issue's stage, and removes the worktree when the issue is done.

Tier your models by decision weight

Use stronger models for decisions and smaller ones for well-specified execution:

1. Keep the orchestrator on a frontier-class model. It tracks parallel worktrees and issue stages, handles interruptions, and needs a large context window. You can give it a new problem during other work. It files an issue and resumes.
2. Ask a strong model to analyze a complicated task and write a detailed plan. Then give the specified steps to a small model. Smaller models have higher quotas, which makes long execution runs practical.
3. Small models forget things. The PM and QA checks around them are what compensate for that.
4. Use a small local model for execution only after a stronger model has broken the backlog into atomic tasks. The call mentioned an open model around 20B parameters with reasonable throughput. Keep the PM and orchestrator on stronger models.

The screenshot shows the orchestrator's per-project sessions on the left. On the right, an agent reports QA results, migration and type checks, and the remaining review steps before merge.

The on-call engineer checks the deployment

The on-call engineer runs a script that watches CI/CD and deployment. If both pass, it exits. If either fails, it can fix the problem, ask the orchestrator to return it to an engineer, or roll back a bad deployment. The agent starts the script and handles its result. Keep this final check because QA can miss failures that CI catches. The screenshot shows a pull request that completed this pipeline:

Verdict on the teamwork tool from the webinar

The webinar tool puts agents in rooms with a lead, team leads, and workers. The host had tried it after its team contacted him about a month earlier. He found it excessive for solo work because he doesn't need Slack to talk to his own agents. It can help a real team or a large task with a clear goal that can be divided among many agents. The example discussed was proving a hard mathematical theorem with a hundred agents.

The same principle applies to a "head of department" orchestrator across projects. A single project doesn't need one. It becomes useful when several projects share functionality and you want one place to submit requests such as "start planning this new course".

Next actions

Try one or two of these changes in your own agent pipeline this week:

• [ ] Write your escalation rules: list the situations where your PM role must stop and ask you, and the ones where it must decide on its own.
• [ ] Add a needs human input tag gate. The orchestrator skips tagged issues until you approve, deny, or correct them and remove the tag.
• [ ] Pick one recurring step in your pipeline and convert it into an intermediate human gate (a mockup, a plan review) before full implementation.
• [ ] Split one task you would normally give a strong model into plan (strong model) + execute (small model), with a QA check in between, and compare the cost and quality.
• [ ] If you're taking the course, finish homework 2. It was reported working with ZCode. Start scoping your capstone. One plan discussed refactoring an older arcade game with an agent team.