P4 · 8 questions

Evaluation, Testing & Optimization — Attempt 2

CCAR-P P4 practice. Some items are “select all that apply” — they tell you how many to pick and are graded all-or-nothing. Commit before revealing, then read every explanation.

0 of 8 answered

P4 · Q1metric selection
A model extracts a single ISO date (YYYY-MM-DD) from invoices, with exactly one correct answer per document. Which evaluation metric is most appropriate?
P4 · Q2faithfulness / groundedness
In a document-QA system, 'faithfulness' (or groundedness) most precisely measures which of the following?
P4 · Q3LLM-as-judge — calibration
Before trusting an LLM judge's scores across thousands of eval items, what is the most appropriate calibration step?
P4 · Q4latency optimization
A chat assistant produces long answers and users complain it feels slow to respond, though total generation time is acceptable. Which optimization most directly improves the perceived latency with the least behavioral risk?
P4 · Q5statistical rigor
Version B beats version A by 1.5 percentage points of accuracy on a 40-item eval set, and the team wants to ship B as clearly better. All of the following sound reasonable, but which is the most appropriate conclusion?
P4 · Q6agent / tool-trajectory evaluationSelect 2
You are evaluating a multi-step tool-using agent, not just its final answer. Which two signals are most important to capture in a trajectory evaluation?
P4 · Q7offline vs online evaluationSelect 2
A team wants both fast pre-release feedback and confidence that changes help real users. Which two statements correctly characterize offline versus online evaluation?
P4 · Q8diagnosing RAG eval failuresSort each into a bucket
Sort each observed RAG failure into the evaluation dimension it belongs to.
The correct source passage was never returned in the top-k results
Many irrelevant passages were returned, diluting the useful ones
The answer asserted a fact that appears in none of the retrieved passages
The LLM scorer rated a fabricated answer as 'fully grounded' where humans rated it wrong
The judge consistently gives higher scores to the longer of two equally correct answers

More P4 attempts