Get Your Documents Ready for Retrieval
Retrieval-augmented generation answers from whatever you index. Index duplicates and the model contradicts itself. Index a stale policy and it confidently quotes the wrong one. Index a spreadsheet of customer details and you have a privacy problem with a search box on it. This checklist is the preparation work that stops each of those.
Work through it in order. The first two sections decide what goes in and who may see it, and everything after depends on them. Ticks are saved in your browser so you can return to it over several sessions.
RAG Data Preparation Checklist
Inventory and permissions first. Everything else depends on them.
Your ticks are saved in this browser, so you can work through the list over several sessions.
01Source inventory and ownership
0/5Know what goes in, which copy is authoritative and who answers for it.
02Deduplication and cleaning
0/5Remove what would make the assistant contradict itself.
03PII and secrets scan
0/5Find personal information and credentials before they become retrievable.
04Chunking, metadata and access mapping
0/5Make each piece retrievable, filterable and correctly restricted.
05Evaluation, freshness and rollback
0/4Prove it works, keep it current, and be able to undo a bad load.
A preparation checklist, not a legal or security assessment. Involve whoever owns privacy and security in the decisions it records. Ticks are saved in your browser only.
Why Preparation Decides the Result
Retrieval quality is mostly a property of the corpus, not the model. Most of the answer problems that show up in a pilot trace back to something on this list being skipped.
Duplicates cause contradictions
Three versions of the same procedure in three folders means the retriever may return the 2019 version for one question and the current one for the next. Deduplication and a single authoritative copy remove an entire class of wrong answers before any tuning.
Permissions must be mapped before indexing
Once a document is embedded it is retrievable by anyone the index serves. If access control is bolted on afterwards, restricted content has already leaked into a shared index. Map who may see what first, then index.
The evaluation set is the acceptance test
Fifty real questions with expert-written correct answers, collected before the build, is how you tell whether ingestion worked. Without it, quality is an opinion and every change to chunking or retrieval is a guess.
How to Use the Checklist
Five sections in the order the work is done. Each item has a one-line note on what done looks like.
Inventory and ownership
List every source, decide which copy is authoritative and name an owner for each. Anything without an owner is excluded until it has one.
Clean and scan
Deduplicate, remove superseded versions, and scan for personal information and secrets. Decide what is redacted, excluded or allowed with restricted access.
Chunk, tag and map access
Split documents into retrievable pieces that keep their meaning, attach metadata the retriever can filter on, and carry document permissions through to every chunk.
Evaluate, refresh and be able to roll back
Run the evaluation set, agree how updates reach the index and how fast, and make sure a bad ingestion can be reversed without downtime.
The Items Teams Most Often Skip
The list is long because ingestion has many small decisions. These four are the ones that come back to bite when they are skipped.
Deciding what is authoritative
The inventory usually finds the same content in a document management system, a shared drive, an intranet page and several email attachments. Someone has to decide which one wins, and that decision is a business call, not a technical one. Make it explicit and record it.
- One authoritative location per document type
- Superseded copies excluded, not merely down-ranked
- Owner named for every included source
- Exclusion list kept alongside the inclusion list
Secrets in documents
Passwords in runbooks, API keys in onboarding guides, bank details in finance procedures. All of it is retrievable once indexed, and a model will happily read it out. A scan for credential patterns before embedding is quick and catches most of it.
- Scan for credential and key patterns, not only personal information
- Rotate any secret found, do not only redact it
- Treat runbooks and IT documentation as high risk
- Re-scan on every refresh, not only at first load
Chunking that keeps meaning
A chunk that starts mid-clause or separates a table from its heading retrieves badly and reads worse. Chunk on document structure where it exists, keep headings with their content, and store enough metadata that the retriever can filter by document type, date and audience.
- Split on headings and sections before falling back to fixed sizes
- Keep tables and their captions together
- Attach title, date, owner, audience and version to every chunk
- Test chunk boundaries against the evaluation set, not by eye
Freshness and rollback
Documents change. If the index does not follow, the assistant drifts out of date quietly. Agree how changes are detected, how quickly they must reach the index, and how to restore the previous index if a refresh goes wrong.
- Define the maximum acceptable lag between a document change and the index
- Automate detection from the source system where possible
- Keep the previous index version until the new one passes evaluation
- Log every ingestion run with what changed and who approved it
Next Steps
RAG Architecture in Australia
How the retrieval pipeline this checklist feeds is designed and hosted.
Read the architecture guide →LLM Security Review Checklist
The pre-deployment security review that follows once the corpus is prepared.
Run the security review →AI Data Sovereignty Checklist
Where the prepared documents and their embeddings are allowed to live.
Check data sovereignty →Frequently Asked Questions
It depends on how many sources there are and how much of the content has a clear owner. A single well-managed document library can be inventoried and cleaned in days. Content spread across drives, wikis and email with unclear ownership takes weeks, most of it spent on decisions rather than tooling. Start the inventory early because it is on the critical path.
Not always. Some use cases legitimately need it, for example an assistant that helps staff find a client file. What you need is a decision for each source: exclude it, redact the personal information, or include it with access restricted to the people who may already see it. The Australian Privacy Principles set the obligations for handling and securing personal information, so involve whoever owns privacy in that decision.
Around fifty questions that real users actually ask, each with a correct answer written by someone who knows the domain, and a note of which document the answer comes from. Include hard cases: questions the documents do not answer, questions with a recently changed answer, and questions that touch restricted content. Run it after every change to ingestion or retrieval.
They need text extraction first, and extraction quality varies with scan quality. Treat them as a separate stream: extract, check a sample by eye, and exclude anything where extraction is unreliable rather than indexing garbage. Where the same content exists as a text document, prefer that copy.
Every chunk inherits the access control of its source document as metadata, and retrieval filters on that metadata using the identity of the person asking, before anything reaches the model. The mapping from source system permissions to index metadata is written down and tested with a low-privilege account. If the source system cannot express permissions at document level, that is an inventory finding to resolve before indexing.
Only in your own browser, so the ticks are there when you come back. Nothing is transmitted to us. Use print to keep a copy or reset to clear it.
Sources and further reading
- Australian Privacy Principles (Office of the Australian Information Commissioner)
- The Privacy Act 1988 (Office of the Australian Information Commissioner)
Stuck on Inventory, Permissions or Evaluation?
Those three sections are where most preparation stalls. Send us the sources you have listed and where they live, and we will scope the preparation and the retrieval build as one piece of work, priced within our published range.