Machine Learning Engineer interview questionsCoding — the training-serving skew round

Purpose

Re-aims a candidate who prepared for a data-structures round, so the first minutes are spent on the repository rather than on recalling the interview they expected.

Purpose

Gives permission to read rather than perform, so silence is not penalised and the first thing said is a finding rather than a reflex.

Purpose

Separates candidates who read the serving path against the training path line by line from candidates who read the training path and reason about what serving probably does. The whole discriminator is whether they compare or infer.

Signals to score

• Opens both files side by side and reads the serving path against the training path rather than reading one and inferring the other
• Names the scaler fitted before the split, and says which number it inflates — the offline AUC, not the serving score
• Names the wall-clock recency feature, and says its effect grows with the age of the training snapshot
• Names the two null-fill strategies and says what a customer with no basket history looks like to each path
• Names the unseen-category mapping and says which category it silently becomes
• Flags `refund_flag` as arriving after the label window and treats it as a different class of problem from the other four
• Orders the list by damage rather than by line number, and can say why
• Asks which of the five would show up as the 71% agreement figure and which would not, since a pure offline leak leaves shadow agreement intact
• Says out loud what is correct in the pack, not only what is wrong

Follow-up questions

• The scaler is fitted on the whole frame. Which reported number does that move — the offline AUC or the serving score — and does it explain the 71%?
• Of the five, which would you have caught with a unit test, and which needs data to see?
• Pick the one that costs the business the most on a normal Tuesday. Why that one?
• Is the refund flag the same kind of bug as the other four?
• If the shadow-agreement figure were 99% instead of 71%, which of these would you still expect to be in the code?

Purpose

Sets the deliverable as a failing test rather than a fix, so the candidate has to reproduce before repairing and cannot hand back a patch whose effect nobody measured.

Purpose

The load-bearing exercise of the round. A parity test is the one instrument that catches this class before shadow serving does, and most engineers who have shipped a model have written one badly — with a float equality, a live clock, or a sample that never includes a null.

Signals to score

• Replays real training rows rather than synthetic ones, and says why the null and the unseen category have to be in the sample
• Freezes the clock in the test, or names that the test is flaky by construction until it does
• Compares floats with a tolerance and can say what tolerance and where it came from
• Asserts feature by feature and names the failing feature in the assertion message, rather than asserting on the final score
• Makes the test fail first, and reads the failure back before touching either path
• Fixes the disagreement in one place both paths share, or says explicitly why the two paths cannot share code here and what guards the seam instead
• Says how many rows the test should replay and how they should be chosen, with a reason
• Names where the test runs — on every change to either file, and again before a model is promoted — and what happens when it fails there
• Distinguishes a parity test from a skew monitor in production, and says which of the five plants only the monitor would catch

Follow-up questions

• You compared the two feature vectors with `==`. Will that pass tomorrow?
• The test replays a hundred rows and passes. None of them had a missing basket. Is the bug fixed?
• Where does this test run, and what does a failure block?
• You have made both paths call one function. What is the argument against that?
• The recency feature is right in the test and wrong in production a week later. What did the test not do?

Purpose

Tests whether the candidate reasons about when data becomes available rather than about what a column is called. This is the most expensive class of ML bug and the one least visible in code review.

Signals to score

• States that in training the flag is populated for orders that later refunded, and at serving time it is almost always false because the refund has not happened yet
• Names this as the feature encoding the label, or a near relative of it, rather than as a generic leak
• Predicts the direction of the damage: offline metrics too good, production scores systematically shifted for the segment the flag marks
• Reaches for a point-in-time join — reconstruct what each row's features were as of its prediction time — as the fix, and can say what it needs to exist
• Says the honest fix makes the offline number go down and that the lower number is the true one
• Asks how the label itself is defined and when it becomes known, since the same problem may be in the target
• Generalises: asks which other features in the frame come from tables that are written after the event
• Names a check that would catch the class — feature availability logged at serving time and compared with training

Follow-up questions

• Fix it properly. What happens to the offline AUC, and what do you tell the person who put 0.81 in a deck?
• Which other columns in this frame would you now go and check, and how?
• Is the label defined the same way in training and in the product?
• You cannot rebuild the history of that table. What do you do instead?
• Could a parity test have caught this?

Purpose

Attaches sequencing and ownership to the findings. A candidate who has done this work has an order and a reason; one who has read about it has a list.

Signals to score

• Puts the production-facing skews first and the offline-only inflation second, and says why in terms of who is affected
• Names the refund flag as the one whose fix needs a conversation before code, because it changes a number someone has already reported
• Proposes the parity test as the first thing shipped, before any fix, so each fix is measured
• Distinguishes what is fixed in this repository from what is fixed in the platform — a shared feature definition, a point-in-time store, an `as_of` convention
• Says what goes in the post-mortem and who reads it, and does not call it a post-mortem for a bug that had no incident
• Names the person who owns the model's number and tells them before the number moves
• Sets a check that runs on the next model, not only on this one

Follow-up questions

• Someone wants the recency fix shipped today without the test. Do you?
• Which of the five is a platform problem rather than a model problem?
• Who do you tell before the offline AUC changes, and what do you say?
• What would you want the next person who trains a model here to be unable to get wrong?

Purpose

A candidate who has shipped a model asks about the seam between training and serving; one who has not asks about the model zoo. Naming the topics makes the signal about what they choose.

Purpose

Ends on a real, specific fact about the team's own pipeline. It is the strongest pitch available to the candidate this round exists to hire, and a deterrent to one who wanted the problem already solved. It only works if it is true — check before you say it.