Claude Certified Engineers logoCCE

CCAO-F · Study Guide

← Domain 2: Output Evaluation and Validation

2.2 · Lesson 2 of 6

Hallucinations, Inconsistencies, and Bias

What you need to know

Claude can produce text that sounds right but is not grounded in your source or in fact. Lesson 2.2 trains you to spot red flags in claude.ai outputs: invented numbers and citations, detail beyond the source, internal contradictions, uniform confidence, and one-sided framing.

Hallucination red flags

  • Precise statistics, quotes, case names, URLs, or citations you cannot trace
  • Detail that goes beyond what you provided or attached
  • Higher risk on obscure facts, recent events, and exact figures without a source

Each red flag is a prompt to check, not a reason to trust. Treat “not found” as a finding.

Inconsistencies

Watch for numbers that change between sections, a name spelled two ways, or conclusions that contradict earlier statements. Asking the same question in a fresh chat with different wording can surface unstable figures — disagreement shows where the risk is, not which run is automatically correct.

Bias and framing

Bias appears as one-sided framing, stereotyped assumptions, or missing perspectives. Ask whose view is absent and whether the framing would change with different inputs. A tidy brief that only argues one side still fails evaluation.

Worked example

A research brief invents “Section 12(b)” of a regulation and states an exact market share with no source. Both look authoritative. The correct move is to open the official text and a primary market source: if neither supports the claim, remove or flag it before anyone acts.

Exam traps

  • Ask Claude whether it is sure about a citation or figure

    Self-reported confidence is not validation. Locate the claim in a source or treat it as unverified.

  • Uniform confident tone means every claim is safe

    Real knowledge has uneven certainty. Equal confidence across claims is a signal to dig harder.

  • Accept beyond-source detail as helpful enrichment

    If you asked for a summary of a document, facts the document never states are suspect until verified.

  • Ignore one-sided framing if the numbers look tidy

    Bias includes missing perspectives and unbalanced framing. Ask who is left out.

Practice scenario

A customer contract is attached. Claude’s summary mentions a 30-day termination clause that is not in the document. What is this, and what should happen next?

Choose one answer

Build exercise

Hunt for Hallucinations and Bias in a Research Brief

2.2 · Intermediate · ~40 min · claude.ai

What you will practise

  • Red flags: invented statistics, citations that cannot be found, detail beyond the source, and uniform confidence.
  • Why confident wording is not a sign of accuracy.
  • How to test for inconsistency by asking the same thing two ways.
  • How to notice one-sided framing.

Use the Domain 2 exercise to log each number and citation as confirmed, not found, or contradicted.

Open Domain 2 exercises

Path: /learn/claude-associate/exercises/2-output-evaluation-validation

Sources

View progress