Skip to content
Skip to content

Practical LLM operations guide · By Yes AI

Check whether AI source citations support the answer

A reference beside an answer should help the reader verify it. Your review process needs to check whether the source actually supports the claim, not merely whether a link appears. Use this guide to define citation checks for an assistant answering from approved business documents.

Define which claims need evidence

Decide which parts of an answer should be traceable to approved material. Instructions, eligibility rules, dates and stated conditions usually deserve particular attention in a business workflow. Distinguish a quoted rule from a suggestion or inference. A fictional staff assistant explaining equipment requests should show the source for the approval steps, while labelling any proposed next action that is not written in the procedure. Avoid presenting a mixed paragraph as entirely sourced when only its first sentence is supported.

Open the referenced passage

Ask the reviewer to read the relevant source section and compare its meaning with the answer. Look for omitted conditions, changed exceptions and statements stronger than the original. A document mentioning a topic does not necessarily support every claim about it. Check the full rule around the cited sentence rather than an isolated fragment. If the answer combines several sources, verify which one supports each part. Record unsupported additions explicitly so the team can distinguish a retrieval problem from an answer that went beyond the retrieved material.

Verify the applicable version

Check whether the document is current for the question and whether it applies to the relevant team, product or time period. A correctly quoted retired instruction can still give the wrong operational answer. Include version and ownership information in the review where available. If current sources conflict, route that conflict to the document owner rather than allowing the assistant to invent precedence. Historical questions may need older material, but the answer should make the period clear instead of mixing past and present instructions.

Make references usable to the reader

Use labels that identify the document and section rather than a generic link saying source. Confirm that the intended reader can open the reference through their normal authorised access. Do not broaden access merely to make a citation work. Where a reference cannot be shared, use the approved response or review route. Test links from an ordinary user account, not only the administrator view. A citation that points to an inaccessible workspace or the wrong document leaves the person unable to check an otherwise plausible answer.

Test misleading but plausible answers

Prepare a review set with a correct citation, a relevant but insufficient source, an outdated version and an answer that omits an important condition. Include a multi-part question where only one part is supported. Use fictional or approved internal examples and keep expected judgements separate from the answer under review. Record both missing support and unnecessary rejection of a valid answer. The evaluation should assess whether the evidence establishes the claim, not reward an answer simply for containing many references.

Turn findings into specific corrections

If the wrong document was selected, investigate the source organisation and retrieval setup. If the right passage was available but the answer added a rule, address the answer behaviour and retest. If the source itself is unclear, assign it to its owner. Keep these causes separate so the team does not rewrite good documents to compensate for an unrelated failure. Report the number and type of claims reviewed, with any access limits. A clean sample supports only that review scope, not a guarantee that every future answer will be grounded.

Before you approve the workflow

  • Material claims map to supporting passages.
  • Conditions and exceptions retain their original meaning.
  • The applicable version is identified.
  • References work for the intended authorised reader.

Continue planning

Plan the checks before your next LLM release

Describe the task, source documents and people who will review the answers. We can discuss evaluation, access boundaries and a staged implementation. Keep confidential documents and access credentials out of this enquiry.