P4 warm-up · 8 questions

Warm up: Evaluation, Testing & Optimization

A quick 8-question drill on CCAR-P P4, easy to hard. Rate your confidence as you go — a 5-minute win to find where you stand before a full practice set.

0 of 8 answered

P4 · Q1eval-driven development
A team is about to start building a customer-support summarization feature. Before writing any prompts, what is the most appropriate first step under an eval-driven development approach?
How confident are you?
P4 · Q2LLM-as-judge — position bias
An LLM judge compares two candidate answers (A vs B) and picks the better one. In an audit, the judge favors whichever answer is presented first about 65% of the time, regardless of content. Which mitigation most directly addresses this?
How confident are you?
P4 · Q3eval dataset designSelect 2
You are designing a golden eval set for a contract-clause classifier. Which two practices most improve the set's ability to catch real-world failures?
How confident are you?
P4 · Q4faithfulness / groundedness
In a document-QA system, 'faithfulness' (or groundedness) most precisely measures which of the following?
How confident are you?
P4 · Q5statistical rigor
Version B beats version A by 1.5 percentage points of accuracy on a 40-item eval set, and the team wants to ship B as clearly better. All of the following sound reasonable, but which is the most appropriate conclusion?
How confident are you?
P4 · Q6diagnosing RAG eval failuresSort each into a bucket
Sort each observed RAG failure into the evaluation dimension it belongs to.
The correct source passage was never returned in the top-k results
Many irrelevant passages were returned, diluting the useful ones
The answer asserted a fact that appears in none of the retrieved passages
The LLM scorer rated a fabricated answer as 'fully grounded' where humans rated it wrong
The judge consistently gives higher scores to the longer of two equally correct answers
How confident are you?
P4 · Q7schema-valid ≠ correct
A model returns structured JSON that passes schema validation on every field. What can you correctly conclude about the output's quality?
How confident are you?
P4 · Q8model right-sizing / routing
An analytics team runs one workload that mixes trivial lookups with a minority of genuinely hard reasoning queries, all currently sent to the largest model. They must cut cost without dropping accuracy on the hard queries. All options reduce cost, but which is the least risky?
How confident are you?

Full P4 practice (3 attempts)

Back to readiness check