Custom LLM operation and review
Make your LLM service easier to check and maintain
Use these guides to decide how answers are reviewed, documents stay current and changes reach production. Each covers a practical operating decision, including what to record, who should act and how to check the result.
Practical LLM operations guides
Each guide includes a decision to make, specific operating checks and related reading for the next part of your process.
Decide when your business AI should decline to answer
Define answer boundaries, useful abstention wording and owned human-review tasks for a business knowledge assistant.
Read the guideCheck whether AI source citations support the answer
Review business AI answers for source support, accessible references, version accuracy and unsupported additions.
Read the guideControl document versions in a business AI knowledge base
Define current sources, retired documents, ownership and update checks so a knowledge assistant uses the intended business instructions.
Read the guideTest who can access information through your LLM
Plan permission tests for a business LLM using fictional records, separate user roles and evidence of both allowed and denied access.
Read the guideCheck an LLM release against the work it already handles
Build a practical regression review for changes to an LLM application, with task examples, acceptance criteria and a controlled rollback decision.
Read the guideTurn employee LLM feedback into reviewed improvements
Set up an owned feedback queue for incorrect or unhelpful LLM answers, with useful evidence, source review and verified closure.
Read the guideRespond to an unexpected rise in business AI costs
Define a cost-spike response that checks actual usage, contains affected work and preserves evidence before changing a business AI service.
Read the guidePrepare an exit plan for a business AI provider
Plan what must transfer, what needs rebuilding and how to test continuity before replacing a business AI provider.
Read the guideRemove retired documents from an AI knowledge service
Plan how retired documents stop appearing in business AI answers, with source checks, retrieval tests and clearly stated retention limits.
Read the guideWhich LLMs can run in an Australian cloud region?
How to check whether a model on Azure, AWS, Google Cloud or the OpenAI API stores and processes prompts in Australia, and which settings route them offshore.
Read the guideDo AI providers train on your business data?
What the main AI platforms say about training on business prompts, how long they keep them for abuse monitoring, and the terms to check before you sign.
Read the guideWhat support does a private LLM need after launch?
The ongoing work a private LLM needs after go-live: model retirements, document refresh, access changes, monitoring and incidents, and who should own each task.
Read the guideCustomer chatbot or staff assistant: which first?
Compare a customer-facing chatbot with an internal staff assistant on risk, data, oversight and effort, and decide which to build first from your documents.
Read the guideCan a private LLM read spreadsheets and databases?
Why document search handles prose well but struggles with tables, and how queries, code execution and row-level security let an LLM answer from structured data.
Read the guideShould your AI assistant act in other systems?
How to decide which actions an LLM assistant may take in your systems, where human approval belongs, and the permission and logging controls each action needs.
Read the guideLLMs and personal information: Privacy Act duties
Which Australian Privacy Principles apply when an LLM handles personal information, what the OAIC expects, and the automated decision rule due 10 December 2026.
Read the guideCan a private LLM work in languages besides English?
How to check a model's supported languages, why search across languages needs a multilingual embedding model, and when an interpreter is still needed.
Read the guideWho does what in a private LLM implementation?
The roles a private LLM project needs inside your organisation and from a provider, the decisions each role owns, and the staff time to plan for at each stage.
Read the guideWhere does a RAG system keep copies of your data?
Map every copy a RAG assistant makes of your documents, from extracted text and embeddings to prompt logs and backups, and decide how to secure and delete each.
Read the guideEarlier reading
Data sovereignty: earlier reading
Background discussion of data location and control. Verify current provider terms and applicable requirements for your proposed setup.
RAG and fine-tuning: planning the approach
Earlier reading on two ways to adapt an AI service. Test options against your own task and source material before choosing an implementation.
Scope your next LLM improvement
Describe the task, source systems and current limitations. We can discuss evaluation criteria, access requirements and the scope of a controlled pilot. Please leave confidential documents and credentials out of this enquiry.