Skip to content
Skip to content

Custom LLM operation and review

Make your LLM service easier to check and maintain

Use these guides to decide how answers are reviewed, documents stay current and changes reach production. Each covers a practical operating decision, including what to record, who should act and how to check the result.

Practical LLM operations guides

Each guide includes a decision to make, specific operating checks and related reading for the next part of your process.

Decide when your business AI should decline to answer

Define answer boundaries, useful abstention wording and owned human-review tasks for a business knowledge assistant.

Read the guide

Check whether AI source citations support the answer

Review business AI answers for source support, accessible references, version accuracy and unsupported additions.

Read the guide

Control document versions in a business AI knowledge base

Define current sources, retired documents, ownership and update checks so a knowledge assistant uses the intended business instructions.

Read the guide

Test who can access information through your LLM

Plan permission tests for a business LLM using fictional records, separate user roles and evidence of both allowed and denied access.

Read the guide

Check an LLM release against the work it already handles

Build a practical regression review for changes to an LLM application, with task examples, acceptance criteria and a controlled rollback decision.

Read the guide

Turn employee LLM feedback into reviewed improvements

Set up an owned feedback queue for incorrect or unhelpful LLM answers, with useful evidence, source review and verified closure.

Read the guide

Respond to an unexpected rise in business AI costs

Define a cost-spike response that checks actual usage, contains affected work and preserves evidence before changing a business AI service.

Read the guide

Prepare an exit plan for a business AI provider

Plan what must transfer, what needs rebuilding and how to test continuity before replacing a business AI provider.

Read the guide

Remove retired documents from an AI knowledge service

Plan how retired documents stop appearing in business AI answers, with source checks, retrieval tests and clearly stated retention limits.

Read the guide

Which LLMs can run in an Australian cloud region?

How to check whether a model on Azure, AWS, Google Cloud or the OpenAI API stores and processes prompts in Australia, and which settings route them offshore.

Read the guide

Do AI providers train on your business data?

What the main AI platforms say about training on business prompts, how long they keep them for abuse monitoring, and the terms to check before you sign.

Read the guide

What support does a private LLM need after launch?

The ongoing work a private LLM needs after go-live: model retirements, document refresh, access changes, monitoring and incidents, and who should own each task.

Read the guide

Customer chatbot or staff assistant: which first?

Compare a customer-facing chatbot with an internal staff assistant on risk, data, oversight and effort, and decide which to build first from your documents.

Read the guide

Can a private LLM read spreadsheets and databases?

Why document search handles prose well but struggles with tables, and how queries, code execution and row-level security let an LLM answer from structured data.

Read the guide

Should your AI assistant act in other systems?

How to decide which actions an LLM assistant may take in your systems, where human approval belongs, and the permission and logging controls each action needs.

Read the guide

LLMs and personal information: Privacy Act duties

Which Australian Privacy Principles apply when an LLM handles personal information, what the OAIC expects, and the automated decision rule due 10 December 2026.

Read the guide

Can a private LLM work in languages besides English?

How to check a model's supported languages, why search across languages needs a multilingual embedding model, and when an interpreter is still needed.

Read the guide

Who does what in a private LLM implementation?

The roles a private LLM project needs inside your organisation and from a provider, the decisions each role owns, and the staff time to plan for at each stage.

Read the guide

Where does a RAG system keep copies of your data?

Map every copy a RAG assistant makes of your documents, from extracted text and embeddings to prompt logs and backups, and decide how to secure and delete each.

Read the guide

Earlier reading

Data sovereignty: earlier reading

Background discussion of data location and control. Verify current provider terms and applicable requirements for your proposed setup.

RAG and fine-tuning: planning the approach

Earlier reading on two ways to adapt an AI service. Test options against your own task and source material before choosing an implementation.

Scope your next LLM improvement

Describe the task, source systems and current limitations. We can discuss evaluation criteria, access requirements and the scope of a controlled pilot. Please leave confidential documents and credentials out of this enquiry.