Machine Learning Engineer interview questionsSystem design — the retrain that has to be reversible round

Purpose

Sets ownership as the frame and removes blame as a topic, so the candidate can spend the time on action rather than on defending a job they did not actually run.

Purpose

Tests ownership against an incident the candidate's own system caused, and tests whether they stop the bleed before they hunt the cause. Almost everyone goes to root cause first; the strong candidate goes to the discount pipeline first.

Signals to score

• Stops the discounts before investigating the model — pauses the automation or raises the threshold — and names who has to agree to that within the hour
• Tells the retention team and their finance partner what is known, what is not, and when the next update is, before knowing the cause
• Tries to recover the previous model and discovers, or already knows from the sketch, that it was overwritten — and says what they do instead
• Reconstructs the previous behaviour from something durable: the training run's logged artifacts, a retrain on last month's snapshot, or last week's stored scores
• Forms cause hypotheses in a testable order: a source table changed shape, the training window swallowed a promotion period, the label definition moved, a feature's fill rate collapsed
• Checks whether the model is wrong or the threshold is, since a better-calibrated model with the same threshold would also triple the volume
• Quantifies the exposure — how many customers, how much money, whether it is still running — before the first update goes out
• Writes down the timeline as it happens rather than reconstructing it afterwards
• By Friday has a guard in place that would have stopped Tuesday's model from being promoted, and can name it
• Owns the outcome in the first person without either apologising for the design or blaming the team that set the threshold

Follow-up questions

• Hour one. What is the very first thing you do, and who has to say yes?
• The previous model was overwritten. How do you get last week's behaviour back?
• The retention team says the threshold was never yours to touch. Are they right?
• It turns out the new model is better calibrated and the old one was under-scoring. Was this an incident?
• What is different about the system on Friday, not about the write-up?

Purpose

Redirects from tooling to controls, where the design decisions actually live and where a candidate who has only used a managed pipeline has less to say.

Purpose

Tests production ML systems design at the level of controls: what is checked, where, against what, and who is paged. The separator is whether the score-distribution check is a gate or a graph.

Signals to score

• Versions every model artifact immutably and makes the scorer read a pointer, so rollback is repointing and takes minutes
• Puts a score-distribution comparison against the incumbent in the promotion path as a hard gate with a stated threshold, not a dashboard panel
• Validates the training data before training — schema, fill rates, row counts, date range — and fails the run rather than training on a surprise
• Runs the candidate model in shadow against live traffic for a period before it takes over, and says what is compared
• Checks calibration against a holdout with a known base rate, because a threshold downstream depends on it
• Gives the downstream consumer a contract — a score distribution they can rely on, or a percentile-based trigger instead of a raw threshold — and names who owns it
• Canaries the new model on a cohort before all customers, and says what is watched and for how long
• Separates the training schedule from the promotion decision, so a retrain can complete without becoming the production model
• Names what pages a human, what stops automatically, and what merely logs — and puts the $180k case in the second category
• Keeps the design proportionate: says what they would not build for a model of this size and blast radius

Follow-up questions

• Where exactly is the score-distribution check, and what happens when it fails at 3am?
• How long does it take to serve last week's model, and who can do it?
• The retention team still thresholds at 0.7. How do you stop the next calibration change from doing this again?
• What do you check before training, and what do you refuse to train on?
• You have described a lot of machinery. Which piece would you cut for a model that touches ten customers a day?

Purpose

Tests whether the candidate understands the difference between drift in inputs, drift in the relationship, and delayed labels — and whether they know that retraining on a schedule can chase noise and retraining on drift can chase itself.

Signals to score

• Separates input drift, a change in the relationship between inputs and outcome, and label delay, and says which of the three can be measured this week
• Proposes monitoring the input distributions and the score distribution weekly, with a stated comparison method, as the proxy while labels are delayed
• Says that a change in inputs is not by itself a reason to retrain, and gives a case where retraining on it would make things worse
• Uses the delayed labels when they arrive — backtests the model that was live thirty days ago against the churn that actually happened — and treats that as the real accuracy signal on a lag
• Names the feedback loop: the discounts this model triggers change who churns, so the labels it will later train on are partly its own doing
• Proposes a retrain trigger that is either a measured accuracy decay on the lagged backtest or a data-validation failure, rather than a calendar
• Keeps a fixed reference model or holdout cohort so that drift can be measured against something that does not itself move
• Says what a real change in the customer base looks like in the data versus what a broken upstream looks like, and how they would tell

Follow-up questions

• The input distribution shifted this week. Do you retrain?
• Labels from thirty days ago just arrived. What do you do with them?
• The customers who got a discount mostly did not churn. What has that done to your training data?
• Weekly retraining has been running for a year. Has it ever helped, and how would you know?
• A feature's mean moved by 40% overnight. Drift, or a broken pipeline?

Purpose

A candidate who has owned a production model asks about rollback and ownership of the seam; one who has not asks about the orchestrator. Naming the topics makes the signal about the choice.

Purpose

Ends on a specific, true, unflattering fact. It is the strongest pitch available to the candidate this round exists to hire and a deterrent to one who wanted the platform already built. Check it is still true before you say it.