P4 · 8 questions

Evaluation, Testing & Optimization — Attempt 3

CCAR-P P4 practice. Some items are “select all that apply” — they tell you how many to pick and are graded all-or-nothing. Commit before revealing, then read every explanation.

0 of 8 answered

P4 · Q1golden set design
What best describes the purpose of a 'golden set' in LLM evaluation?
P4 · Q2canary vs A/B rollout
A team wants to release a new prompt version to a small slice of live traffic first, mainly to catch severe failures before a wider rollout, without necessarily comparing two variants statistically. Which approach fits best?
P4 · Q3schema-valid ≠ correct
A model returns structured JSON that passes schema validation on every field. What can you correctly conclude about the output's quality?
P4 · Q4continuous eval / monitoring loop
A production assistant was strong at launch but has quietly drifted as user questions evolved. What is the most appropriate practice to keep quality from silently degrading over time?
P4 · Q5overfitting to the eval set
Over months, a team repeatedly tweaks prompts until scores on their fixed eval set reach 98%, yet real users report no improvement lately. Each explanation is plausible, but what is the most likely root cause?
P4 · Q6model right-sizing / routing
An analytics team runs one workload that mixes trivial lookups with a minority of genuinely hard reasoning queries, all currently sent to the largest model. They must cut cost without dropping accuracy on the hard queries. All options reduce cost, but which is the least risky?
P4 · Q7RAG retrieval metricsSelect 2
You want to evaluate only the retrieval stage of a RAG pipeline, independent of the generator. Which two metrics are appropriate?
P4 · Q8cost-optimization leversSelect 2
A batch pipeline summarizes millions of documents nightly and is over budget, with a generous overnight completion window. Which two levers most appropriately cut cost given this workload?

More P4 attempts