How AI Actually Writes Trivia Questions (and How We Stop It Lying)

"Where do the questions come from?" is the single most common email we get. The short answer is: a large language model writes them, our system validates them, and a human review pass catches the rest. The long answer — what actually happens between you tapping Start Quiz and seeing a question — is more interesting than that summary makes it sound.

Step 1: Topic decomposition

When you choose "NBA → Medium," we do not ask the model for "five NBA questions." That produces lazy, recyclable questions every time. Instead we decompose the topic into a stratified set of sub-themes — Finals history, statistical leaders, draft trivia, rule changes, franchise relocations, coaching tenures, jersey numbers — and ask for one question per sub-theme. The result is a balanced quiz instead of five questions about LeBron.

Step 2: Constrained generation

The model gets a structured prompt that demands: a question, four answer options, the index of the correct one, a one-sentence factual explanation, and a difficulty self-rating. We constrain the output as JSON so we can parse and validate it programmatically. Free-form text is where models drift; tight schemas keep them honest.

Step 3: Automated fact-checking

Every generated question is run through a verification step before it is ever shown to a player. We re-ask the model the question without showing it the answer, ask it to justify its answer with a sourceable claim, and compare both answers. If they disagree, the question is discarded. We also reject questions whose explanation contains hedging language ("believed to be," "approximately," "around") for factual categories like sports records and geography, where the answer should be exact.

Step 4: Difficulty calibration

The model is a bad judge of its own difficulty. So we calibrate empirically. Every question's true difficulty is learned from real play data — how often players get it right at each declared level, how long they take, and how often they swap answers before submitting. Questions that 80% of players get right on Hard mode get demoted; questions that 30% of players get right on Easy mode get retired entirely. The system is constantly retuning itself.

Step 5: Diversity and anti-repetition

If the model has seen the same prompt yesterday, it will happily produce the same question today. We hash every question we have ever generated and refuse duplicates at the database level. We also track which specific facts have been used recently per category, so you should not see "the 1972 Dolphins were the only undefeated NFL team" twice in a week.

What the model is genuinely good at

• Phrasing. Models are excellent at writing clear, unambiguous questions with parallel answer options that are not too easy to spot by elimination.
• Distractor generation. Plausible-but-wrong answer choices are surprisingly hard to write. LLMs are great at "give me three teammates of this player in 2008."
• Stable knowledge. Facts that have been true for 20+ years (Olympic medal counts, classic film cast lists, capital cities) come out essentially perfect.

What we still do not trust the model with

• Anything time-sensitive. Current league standings, who won the most recent championship, this season's stats. Those questions are pulled from authoritative APIs and templated in, not generated.
• Counts and exact numbers. "How many career goals does X have" is a trap. We avoid generating those and prefer ranges or all-time-record phrasings where the answer is stable.
• Living-people biography details. Birthdays, marriages, exact net-worth claims. Too risky, low payoff.

The human in the loop

A subset of every day's freshly generated questions is reviewed by a person before going live in high-traffic categories. Most of what we catch is not factual error — it is phrasing that is technically correct but unfair ("Which of these did NOT happen…" with a triple negative, that kind of thing). We rewrite, we ship.

Why this matters

Anyone can wire a chatbot up to a quiz UI. Doing it in a way that you can actually trust — where the answer is correct, the difficulty matches what is promised, and you are not seeing the same question every other day — is a real engineering problem. We are obsessive about it because the entire product collapses the moment you stop trusting the questions.

If you ever spot a wrong answer, please use the "report this question" link. Every report goes into the calibration system and helps us retire bad questions faster.