Machine Learning Engineer interview questionsModelling review — 40,000 labels and a foundation model round

Purpose

Sets the round as a decision under constraints rather than a survey of techniques, so a candidate cannot spend the hour describing the options.

Purpose

The load-bearing question. The strongest separator is not which option is chosen — two of the three are defensible on this pack — but whether the candidate reads the per-class table and the agreement figure before choosing, and whether the reversal condition is specific enough to be tested.

Signals to score

• Reads the per-class table before naming an approach, and says that 80% overall is 93% on three queues and 35–50% on four, which are different problems
• Treats the 0.78 kappa as a ceiling: the labels disagree with each other often enough that any single accuracy figure above the low 90s should be doubted
• Notices that the 500-ticket zero-shot sample contains roughly a dozen tickets per rare queue, so the 35–50% figures have intervals of twenty points or more
• Brings the data-residency line into the decision unprompted, and says what it rules out or what it costs to satisfy
• Converts the cost sheet into cost per correctly routed ticket rather than cost per thousand tokens
• Checks the latency budget against the frontier option and says whether 300 ms at p95 is achievable with a retrieval step in front of it
• Says which option the 40,000 labels actually favour and why eighteen months of labels may not describe this month's tickets
• States a reversal condition that names a metric, a threshold and a sample, not a feeling
• Proposes a hybrid or a staged answer only if they can say what the seam costs, and does not use it to avoid the decision
• Asks what a misroute costs on each queue before treating all fourteen as equal

Follow-up questions

• The four small queues are the ones where a misroute is a legal escalation. Does that change the call?
• The kappa was 0.78. What is the highest accuracy you would believe from any model on these labels?
• The 500-ticket sample had eleven tickets in the smallest queue. What is 40% accuracy on eleven tickets worth?
• Tickets cannot leave the region. Which of your three options survives that as written?
• You have picked one. What would you have to see in two weeks to pick a different one?

Purpose

Moves the frame from the model to the training data, where most of the real risk in this project lives and where candidates who have only run tutorials have nothing to say.

Purpose

Tests whether the candidate audits labels as a matter of course. Practitioners have a routine here — sample, re-label, measure drift, look at the confusions — and can predict what it turns up. People who have not done it propose to train and see.

Signals to score

• Proposes re-labelling a fresh stratified sample now, and comparing against both the old labels and the old agreement figure, before trusting either
• Predicts that the queue definitions changed over eighteen months and proposes to check by measuring agreement on old tickets against the current guidelines
• Asks who labelled — support agents routing live, or annotators after the fact — and says how the answer changes what the labels mean
• Says that the 0.78 kappa is an average and asks for it per queue, expecting the rare queues to be where the disagreement lives
• Proposes to look at the confusion structure between the three big queues, since that is where most labelled tickets and most disagreements are
• Plans a clean, held-out, recently labelled evaluation set with enough rare-queue tickets to bound their accuracy, and says how many
• Treats multiply-labelled tickets as a resource for measuring the ceiling, not as a nuisance to deduplicate
• Names the decision the audit feeds — whether the rare queues get more labels, a merged queue, or a human in the loop — rather than auditing for its own sake
• Says what they would do with tickets the labellers cannot agree on, and does not simply drop them

Follow-up questions

• The labels were applied by the agents who handled the tickets, at the time. What does that do to them?
• Kappa was 0.78 overall. Sketch what you expect it to be on the smallest queue and say why.
• Two labellers disagree on a ticket. Is that a bad label, or information?
• How many recently labelled tickets do you want in your evaluation set, per queue, and where does that number come from?
• The queue guidelines were rewritten nine months ago. What does that do to the first nine months of labels?

Purpose

Tests whether the candidate can write a decision rule before seeing the results, choose a metric the business would recognise, and make the comparison fair. This is where "we should A/B test it" gets separated from knowing how.

Signals to score

• Evaluates both options on the same fresh, stratified, held-out set rather than on each option's own convenient sample
• Reports per-queue accuracy and a cost-weighted misroute figure, not a single overall accuracy
• Fixes the decision rule before the run: which metric, what margin, on how many tickets, and who signs it off
• Uses a paired comparison on the same tickets and counts where the two options disagree, rather than comparing two aggregate percentages
• Includes cost per correctly routed ticket and p95 latency as reported outcomes with thresholds, not as afterthoughts
• Proposes a shadow-mode week on live tickets after the offline result, with the label being where the human actually routed
• Says what the frontier model is allowed to see during the experiment given the residency constraint — redacted text, in-region, or nothing
• Names what would make the result inconclusive and what they would do then, instead of assuming a winner
• Keeps the rare queues from being decided by a handful of tickets — either enough labels or an explicit statement that they are undecided

Follow-up questions

• Both options score within a point of each other overall. What breaks the tie?
• The frontier model wins the offline comparison and loses in shadow mode. What happened?
• Who decides, and what did they agree to before the run started?
• What sample size makes a two-point difference on the big queues believable?
• Two weeks is not enough. What do you cut, and what do you refuse to cut?

Purpose

A candidate who has made this call before asks about label ownership and past reversals; one who has not asks about the model zoo. Naming the topics makes the signal about the choice.

Purpose

Ends on a specific, true, unflattering fact. It is the strongest pitch to the candidate this round exists to hire and a deterrent to one who wanted decisions already made. A generic version reads as a script; check yours is still true before you say it.