AI Engineer interview questionsSystem design — caching and reuse round

Purpose

Names the subject narrowly so a candidate who prepared a RAG-architecture answer redirects inside thirty seconds, and so retrieval quality does not become a place to hide.

Purpose

Establishes two motivations — money and latency — without ranking them. Which one the candidate optimises for, and whether they ask, is an early read.

Purpose

The whole signal is what happens before the first box. A candidate who starts at embeddings has skipped the only step that determines the design; a candidate who asks what a wrong cached answer costs has already done the hard part.

Signals to score

• Asks what a wrong cached answer costs before proposing any mechanism
• Asks whether answers are specific to the asking account, and gives an example of one that would be
• Asks how stale an answer may be, and ties it to something that actually changes — pricing, a release, a policy
• Asks whether the prompt carries anything that is not the question: account state, retrieved documents, tool results, the time, the conversation so far
• Separates the question from the rendered prompt when talking about what would be keyed on
• Asks whether this is a first turn or a message inside a conversation
• Names latency and cost as separate goals and asks which one is being bought
• Proposes a first cut that is narrower than the whole feature — one topic, one surface, read-only questions
• Gives a rough order of magnitude for the saving unprompted, or asks for the numbers needed to compute one

Follow-up questions

• If I told you a wrong answer here misquotes our refund policy, what changes about your design?
• What in that prompt is not the user's question, and which parts of it move between two users asking the same thing?
• Suppose the answer is right but nine days old. Is that a cache problem, and whose problem is it?
• Give me a number for what this saves, and tell me what you had to assume to produce it.
• Where would you turn it on first, and what would make you turn it off?

Purpose

Converts a talkable design question into one with answers, so the section can be scored rather than impressioned.

Purpose

Removes puzzle-hunting and explicitly licenses revision, which is what the last two pairs are meant to provoke.

Purpose

The two identical-string pairs are the discriminator. Almost every candidate defends exact match as the free, safe layer in the section above; these two require them to revise that in the room, and how they revise is worth more than whether they got the pairs right.

Signals to score

• Works pair by pair with a stated reason rather than sorting them by feel
• Catches the entity swap and says why an embedding does not separate the two plan names
• Catches that trial and subscription are different flows with different refunds, not two words for one thing
• Catches the temporal scope change and notes it is one word and one that no lexical distance would weight
• Notices that two questions can be about one topic and still want differently shaped answers — a number against a procedure
• Refuses the identical string asked at two times of day, and locates the reason in the answer being a function of something the question does not contain
• Treats the identical string from two accounts as a different class of problem from the others, not as a harder version of the same one
• Serves the low-overlap paraphrase, or says plainly that their design would miss it and that this is a cost
• Generalises before the end into a rule about which questions are cacheable at all, rather than leaving eight verdicts
• Revises an earlier answer out loud after a later pair contradicts it

Follow-up questions

• Which of these does raising or lowering a similarity threshold fix?
• Two of these are the identical string. Is exact match still your safe layer?
• The quota pair — is that a wrong answer, or is it something else?
• You have refused seven of eight. What is the hit rate of the thing you have just designed?
• Give me the rule. Before you match anything, how do you decide whether this question is a candidate at all?

Purpose

Grants the candidate the gate they just built, so the section is about the instrument and not about re-litigating the previous one.

Purpose

Asked flatly, as if the number were an implementation detail. The strong response treats it as the output of a measurement rather than as an input to the design.

Signals to score

• Declines to name a number without data, or names one and immediately says how they would find out whether it is right
• Proposes labelling real pairs from real traffic rather than reasoning about the score's distribution
• Runs the cache in shadow first — logging what would have hit and serving nothing — before any traffic is served from it
• Derives the cutoff from the false-positive cost established in the first section rather than from a target hit rate
• Holds precision and recall separately and knows which one the cost asymmetry is buying
• Proposes different cutoffs for different topics, with a reason for the split
• Prices the verification step itself, and notices a verifier can consume the saving it protects
• Asks who labels the pairs, and would not hand it to someone without the domain knowledge to judge the trial-against-subscription case
• Puts an interval on a rate estimated from a sample, or asks how many labels they get
• Says what would make them move the cutoff again after launch

Follow-up questions

• Where does that number come from? Talk me through how you would find out whether it is right.
• You are serving 12,000 hits a day at a 2% false-positive rate. What does that look like from the support queue?
• How many pairs do you need to label before you would trust the number, and who labels them?
• You verify every candidate with a second model call. What has that done to the saving?
• Same cutoff for billing questions and for how-do-I questions?

Purpose

States the property plainly so the question that follows is about a category and not about a tuning parameter.

Purpose

Separates candidates who have reused answers from candidates who have reused computation. The three mechanisms that qualify are all the second kind, and a candidate who names one has almost certainly run an LLM feature in production.

Signals to score

• Distinguishes reusing the answer from reusing the work that produced it, in those terms or their own
• Names prefix or key-value caching and knows it is a different mechanism from the one discussed all hour
• Proposes feeding the nearest neighbour to the model as a draft it must still agree with, rather than as the response
• Or proposes serving the cached answer as retrieved context the model writes from, and says what that changes about the risk
• Explains why each named mechanism cannot produce a false positive, rather than asserting that it cannot
• Prices what each one actually saves, and does not claim the same saving as answer substitution
• Says which of the three caches a "hit rate" number would be about, once three exist
• Notes that a small model trained on past answers is a cascade and not a cache, if they raise it
• Ends with a recommendation rather than a menu, and says what it costs relative to the risky version

Follow-up questions

• What fraction of the prompt is identical across two users asking different questions, and what is that worth?
• If the model still generates every token, where did the saving come from?
• What is the risk profile of putting the old answer in the context window instead of in the response?
• Three caches now. When someone reports a hit rate, which one are they talking about?
• Given all of that, what do you actually ship first?

Purpose

Establishes that the project succeeded, so the incident is not a referendum on the idea and the candidate has something to protect rather than something to abandon.

Purpose

The first hour reads reversibility; the permanent change reads whether they learned the rule from the pair pack. A candidate who cannot say how they would find out which entry served it has built something nobody can operate.

Signals to score

• Turns it off before diagnosing, and can say whether that is a flag or a deploy
• Knows whether the switch is global or can be scoped to a topic or a namespace
• Treats it as a data incident rather than a quality bug, and says who else needs telling
• Names what a hit log would have to contain to attribute it: the matched entry, the score, the layer, both questions, and which account the entry was generated for
• Computes or asks for the blast radius — how many responses that entry served since it was written
• Distinguishes "one bad entry" from "the classifier is passing a whole class of questions" and says how the log tells them apart
• Fixes it permanently at the classification gate, not by raising the threshold
• States the rule that account-scoped answers never enter a shared namespace, and ideally that they are generated without another account's context in scope
• Proposes what goes on the dashboard so this is visible before a customer reports it
• Says that hit rate on its own is a vanity metric, or otherwise notices that a cache which serves everything scores 100%

Follow-up questions

• Is that switch a deploy or a flag? How long from decision to off?
• Which entry served it, and how do you find out?
• How many other people got that answer?
• Is raising the threshold the fix here?
• Six weeks in, what is on the dashboard, and what would have caught this in week one?