What Is a Private LLM? A Plain-English Guide
A private LLM is a large language model that runs on infrastructure your organisation controls, answers questions using your own documents and data, and never sends that data to a public AI provider. It is the same underlying technology as ChatGPT, deployed differently: on your servers or in an Australian cloud region, behind your access controls, with your audit logs, and with your information staying inside your legal jurisdiction.
How a Private LLM Differs From ChatGPT and Other Public AI Tools
Almost every question about private LLMs is really a question about ownership. The model architecture behind a private deployment and the model architecture behind a public chatbot are close cousins. What changes is where the model runs, who can see what you type into it, and what it knows about your organisation before you ask.
Same Technology, Different Ownership
A public AI tool is a service you rent. You send your text to a provider's servers, the provider runs the model, and the answer comes back. A private LLM is an asset you operate. The model weights sit on hardware you or your Australian hosting partner control, the inference happens inside that boundary, and you decide who can query it, what it can see, and how long conversations are retained. Nothing about the answer quality requires the public arrangement — that is a commercial and architectural choice, not a technical necessity.
Your Data Never Leaves Your Jurisdiction
When staff paste a client file, a contract, or a patient record into a public chatbot, that content crosses a border and lands with an overseas processor. Under Australian Privacy Principle 8, disclosing personal information to an overseas recipient makes your organisation accountable for what that recipient does with it. A private LLM removes the disclosure entirely: the prompt, the retrieved documents, and the generated answer all stay on infrastructure inside Australia, which makes the privacy position dramatically simpler to explain to a board or a regulator.
It Knows Your Business, Not Just the Internet
A public model knows what was on the open web when it was trained. It has never read your pricing schedule, your standard operating procedures, your Xero chart of accounts, or the seven years of matter files in your document management system. A private LLM is connected to those sources deliberately, so a question like "what did we agree on maintenance response times for this client?" gets answered from your actual contract rather than from a plausible-sounding guess about how such contracts usually read.
How a Private LLM Is Deployed and How It Learns Your Business
Two decisions define any private LLM project: where the model runs, and how it gets access to your knowledge. Everything else is detail built on top of those two answers.
Deployment Model 1: On-Premises
The model runs on GPU hardware in your own server room or your own rack in a colocation facility. There is no outbound network dependency at inference time — the system can run with the internet cable unplugged.
- Highest sovereignty: data never touches a third-party network
- Suits defence, government, and organisations with air-gapped requirements
- Capital purchase of GPU hardware rather than a monthly compute bill
- You own the operational burden: power, cooling, patching, hardware refresh
Deployment Model 2: Sovereign Australian Cloud
The model runs on GPU instances in an Australian cloud region — Sydney or Melbourne — inside your own tenancy. You get data residency and jurisdictional control without buying hardware.
- Data residency in an Australian region under Australian law
- No capital outlay; compute scales with actual query volume
- Faster to stand up than procuring and racking your own GPUs
- The most common starting point for a first private LLM deployment
Deployment Model 3: Hybrid
Sensitive workloads run on private infrastructure while lower-risk, general-purpose tasks are routed to a commercial model. A routing layer decides which requests are allowed to leave the boundary.
- Classify data first, then route by classification, not by convenience
- Keeps costs down on low-sensitivity work like generic drafting
- Requires disciplined routing rules and logging to stay defensible
- Only as private as its weakest routing rule — design it carefully
How It Learns, Part 1: RAG (Retrieval)
Retrieval Augmented Generation is the plain answer to "how does it know our stuff?". Your documents are indexed. When someone asks a question, the system finds the relevant passages first, then asks the model to answer using only those passages.
- Works from day one — no model retraining required
- New or updated documents are reflected as soon as they are indexed
- Answers can cite the source document, so staff can verify them
- Handles knowledge that changes weekly: policies, prices, procedures
How It Learns, Part 2: Fine-Tuning (Behaviour)
Fine-tuning adjusts the model itself so it reasons and writes the way your domain requires. It teaches style, vocabulary, and format — not facts. Most organisations do not need it initially.
- Teaches house style, tone, and specialist vocabulary
- Useful for dense professional language: legal, clinical, engineering
- Requires curated training examples, not a folder dump
- Best added after RAG, once you have measured the accuracy ceiling
What a Private LLM Cannot Do
Being honest about the limits is the fastest way to a project that succeeds. A private LLM is a very capable language system, not an oracle and not a replacement for your systems of record.
- It cannot answer from documents you never gave it access to
- It cannot fix bad source data — wrong policy in, wrong policy out
- It is not a compliance certificate; controls and audits still apply
- It should not make final decisions on legal, clinical, or credit matters
Where to Start With a Private LLM
The organisations that get value from private AI almost never start by buying hardware. They start by picking one question their staff ask constantly and answering it well.
Pick One Painful, Repetitive Question
Choose a single use case where staff repeatedly hunt through documents for answers — contract terms, policy interpretation, product specifications, or historical decisions. A narrow first use case is far more likely to prove value than an "AI for everything" mandate.
Classify the Data It Would Need to Read
Work out what the model must access and how sensitive it is. Personal information, health records, client-confidential material, and commercially sensitive IP are what push a deployment towards private infrastructure and away from a public subscription.
Choose the Deployment Model That Fits
Match the architecture to the data classification and your query volume. Most first deployments land on sovereign Australian cloud; on-premises is chosen where the sovereignty requirement or the volume economics justify the capital cost.
Build, Measure, Then Expand
Deploy the first use case, measure answer accuracy against a real test set before staff rely on it, then extend to the next use case on the same foundation. A typical first single-use-case implementation sits in the $25,000 to $60,000 range.
Who Needs a Private LLM, and the Myths That Confuse the Decision
A private LLM is the right answer for some organisations and an expensive answer for others. The honest test is whether your data sensitivity, query volume, or accuracy requirements genuinely exceed what an off-the-shelf subscription can deliver.
Who It Suits — and Who Is Better Off With a Subscription
The deciding factor is rarely headcount. It is whether the work involves information you cannot lawfully or commercially put into a public tool, and whether enough people ask enough questions to justify the build.
- Strong fit: firms handling client-confidential files — legal, accounting, insurance
- Strong fit: APRA-regulated entities and agencies with data residency obligations
- Strong fit: any organisation with a large, valuable, badly searchable document library
- Weaker fit: small teams using AI occasionally for generic drafting and summaries
- Weaker fit: organisations with no document corpus worth grounding a model in
Common Myths About Private LLMs
Most of the resistance to private AI comes from four misconceptions, and each one is straightforwardly wrong once you look at how modern deployments are actually built.
- Myth: you must train a model from scratch — you start from an open-weight base model
- Myth: it must live in your own server room — sovereign Australian cloud is far more common
- Myth: private models are noticeably worse — open-weight models are now competitive for most business tasks
- Myth: it is automatically compliant — sovereignty helps, but controls and audit still do the work
- Myth: it needs millions of documents — a few thousand well-chosen ones usually beats a bulk dump
Where to Go Next
Private AI vs ChatGPT
The direct comparison: capability, security, cost, and control across private deployment and the public platforms your staff already use.
Compare private and public AI →Private LLM Cost Australia
The honest numbers — implementation, infrastructure, and ongoing operation, plus where the break-even against API spend actually falls.
See the cost breakdown →RAG Architecture Australia
A deeper look at retrieval — chunking, vector databases, hybrid search, and the evaluation that proves the answers are accurate.
Understand how retrieval works →On-Premises LLM Deployment
What running the model in your own rack actually involves: GPU selection, sizing, air-gapped operation, and the operational burden.
Explore on-premises deployment →Open-Source LLM for Business
Where open-weight models stand against commercial ones, and how licensing works when you run a model inside your own boundary.
Review open-source options →Microsoft Copilot Alternative
If your organisation is weighing Copilot against a private deployment, this compares the two on data handling, grounding, and control.
Compare against Copilot →Frequently Asked Questions
It can absolutely run in the cloud, and for most Australian organisations that is the sensible starting point. "Private" refers to control and isolation, not to a physical location. A model running on GPU instances in an AWS Sydney, Azure Australia East, or Google Cloud australia-southeast region — inside your own tenancy, with your own keys and network controls — is a private LLM. Your data stays in Australia, under Australian law, and is never used to train a shared model. True on-premises deployment, where the hardware sits in your own rack, is reserved for organisations with air-gapped requirements or classified material: typically defence, national security, and some government workloads. It delivers the highest sovereignty but you take on the capital cost and the operational burden of running GPU infrastructure yourself.
No, and you almost certainly should not. Training a large language model from scratch costs tens of millions of dollars and requires a data centre of GPUs — it is something a handful of global labs do. Every practical private LLM deployment starts from an existing open-weight base model such as Llama, Mistral, or Qwen, which already understands English, reasoning, and general knowledge. Your project adds the part that matters: connecting that model to your documents through retrieval so it can answer from your actual information, and optionally fine-tuning it so it writes in your house style and understands your specialist vocabulary. Think of the base model as a capable graduate who already speaks the language — your deployment is the induction, not the schooling.
ChatGPT Enterprise and Teams are meaningful improvements over the consumer product: your prompts are contractually excluded from training, and there are admin controls and retention settings. But the architecture is unchanged — your text still travels to OpenAI's infrastructure for processing, which under Australian Privacy Principle 8 is a cross-border disclosure you remain accountable for. You are relying on a contractual promise rather than a technical boundary. A private LLM removes the disclosure entirely: the data never leaves your environment, so there is nothing to promise about. The second difference is grounding. Enterprise subscriptions know only what staff paste in or connect through limited integrations, whereas a private deployment is wired into your document management system, your practice software, and your records deliberately and permanently.
Less than most people expect, but it needs to be the right data. A useful first deployment typically needs a few thousand documents that are current, authoritative, and genuinely used — policies, contracts, procedures, product specifications, or historical advice. Quality beats volume decisively: a curated set of the documents your staff actually rely on will outperform a bulk dump of a twenty-year shared drive containing eleven conflicting versions of the same policy. Practically, the preparation work is deciding which repository is the source of truth, retiring superseded versions, confirming who is permitted to see what so those permissions carry through to the AI, and getting access to the systems where the documents live — commonly SharePoint, a document management system such as LEAP for law firms, or line-of-business software like Xero, MYOB, or JobAdder.
No, and any vendor who says otherwise is selling you a problem. Private deployment removes one significant risk — cross-border disclosure of personal information to an offshore processor — which makes your Privacy Act 1988 position substantially easier. But compliance is made of controls, not of geography. You still need role-based access so the model cannot surface a document to someone who could not open it directly, audit logging of prompts and retrievals, retention rules that match your existing obligations, and human review where decisions carry legal or clinical weight. For APRA-regulated entities, CPS 234 information security requirements apply to an AI system exactly as they do to any other material information asset. Sovereignty is the foundation the controls are built on, not a substitute for them.
Headcount is a poor predictor. A twenty-person law firm handling confidential client matters has a far stronger case than a two-hundred-person business whose AI use is drafting marketing copy. The three signals that actually matter are: sensitivity, where your data cannot lawfully or commercially go into a public tool; volume, where enough staff ask enough questions that per-query API costs compound; and depth, where a generic model simply cannot answer without your specific documents in front of it. If none of the three apply, a ChatGPT Teams or Microsoft 365 Copilot subscription is a more honest recommendation and we will tell you so. If one or more apply strongly, a first private deployment in the $25,000 to $60,000 implementation range is usually the right size of first step.
Still Not Sure Whether a Private LLM Is Right for Your Organisation?
Book a scoping conversation with our architects and we will map your data sensitivity, likely query volume, and realistic use cases against the three deployment models — and tell you honestly if an off-the-shelf subscription would serve you better. Call +61 3 9999 7398 or send us the details through the contact form.