LLM Security and Data Privacy for Australian Enterprise

Prompt injection is an attack where an adversary inserts instructions into data the LLM processes in order to override the model's intended behaviour. In an enterprise context, the most serious variants involve an attacker injecting instructions into documents that end up in the model's context, or into user inputs, that cause the model to perform actions it should not or reveal information it should not. The severity depends on what capabilities the model has: a model that can only answer questions has limited blast radius from prompt injection; a model that can take actions like sending emails or querying databases has much higher exposure. Architectural controls, specifically limiting the model's capabilities and implementing human review for consequential actions, are the most effective mitigations.

Training data extraction is a real but probabilistic risk. Research has shown that LLMs can reproduce verbatim training content through carefully crafted prompts, but typically only content that appeared many times in training. For fine-tuned models trained on your proprietary documents, the risk depends on: how many times each document appeared in training, how much the fine-tuning process has memorised vs generalised, and what access controls limit who can interact with the model. For most enterprise fine-tuning use cases involving moderate document volumes, verbatim extraction risk is low. However, the higher-risk approach is RAG: if your knowledge base is poorly access-controlled, an attacker can retrieve specific documents directly. Access controls on the RAG index are more important than fine-tuning memorisation risk.

The security profiles are different. A RAG system's primary risk is retrieval manipulation: an attacker who can influence retrieval to return documents they should not access, or who can inject content into the knowledge base to influence future retrievals, can exfiltrate data or manipulate model behaviour. Access controls on both the document index and the retrieval pipeline are essential. A fine-tuned model's primary risk is training data extraction and the fact that model behaviour can be influenced by training data characteristics. Both systems share the prompt injection risk. In practice, most enterprise deployments combine both, so both attack surfaces must be addressed.

Pre-deployment security testing for enterprise LLMs should cover at minimum: prompt injection testing using the OWASP LLM Top 10 as a framework, jailbreak testing to verify system prompt hardening, access control testing confirming document access restrictions are enforced, RAG manipulation testing including indirect injection via document content, output monitoring testing confirming sensitive data detection works as intended, and authentication and session management testing. For higher-security deployments involving sensitive data categories, red team testing by a specialist AI security team is worthwhile. We include red team testing as a standard part of our enterprise deployment process.

An LLM security incident response plan differs from a standard application incident response plan in that the primary evidence source is the audit log of prompts and responses. If an incident is suspected, the first step is to preserve and review audit logs to understand what queries were made, what documents were retrieved, and what outputs were generated. The second step is to determine whether the incident was an internal misuse, an external attack, or an accidental data leakage. Depending on the nature and data involved, you may have notification obligations under the Notifiable Data Breaches scheme. We include incident response procedures and audit log forensics guidance as part of our managed service.

On-premises deployment provides a fundamentally different security posture for the most sensitive data: the attack surface accessible from the internet is reduced to whatever management interface exists, model weights and training data cannot be accessed without physical access, and the entire inference pipeline is within your physical security perimeter. For government, defence, and highly regulated deployments, this is the appropriate architecture. Cloud-hosted sovereign deployment on Australian infrastructure provides strong data sovereignty and eliminates offshore access risk, but retains the network attack surface inherent in any cloud-connected system. For most commercial deployments, cloud-hosted with strong network controls is adequate; for the highest-sensitivity applications, on-premises is the appropriate choice.