Chapter

Evaluation & calibration

Rubric design, LLM-as-judge grading, gaming resistance, and real-time signal extraction.

Live work-skills calibration with merit-based XP

Calibration system that maintains a live read of how each user actually performs on five work-skills and turns it into XP that ladders up toward real job-readiness.

Major contributor

Behavioral state detection that drives manager mode

Real-time classifier that reads confusion, stress, disengagement, and flow from activity signals and switches the manager NPC between supportive and Socratic mode accordingly.

Tech lead

Manager-learner relationship calibration

Per-learner running read of the working relationship with each manager NPC, so the manager treats the tenth conversation differently from the first.

Sole author

Stage submission grading with anti-gaming

Submission system that extracts the user's artefact, grades it against the stage rubric, blocks resubmit-gaming, and ships back a verdict with a celebratory tagline and an XP award when the user passes.

Sole author

End-of-scenario performance review

Review system that steps back at the end of a whole scenario and names the pattern across every stage, conversation, and submission the user moved through.

Sole author

Pre-stage understanding gates

Two quick understanding checks that catch a misread of the assignment before it turns into days of wrong work.

Tech lead

Per-learner journey narrative

Running per-learner story that compounds across every touchpoint so the rest of the platform knows how the user is actually moving through their focus, not just where they passed or failed.

Major contributor

Eval observability: run log and cohort analytics

Two observability tools that watch every evaluation the platform runs: a per-call run log and a cohort analytics view that reads the log to surface where the curriculum is breaking down.

Major contributor

NPC walkthrough of the stage submission verdict

Realtime feedback flow where the user's manager NPC walks them through the stage-submission verdict with audio, evidence highlights on the original file, and a lazy positives sequence.