Security Review Before You Deploy
LLM applications fail in ways conventional applications do not. The model follows instructions from anywhere in its context, including inside documents it retrieves, which turns your own content into a potential attack surface. This is the pre-deployment list.
Work through before go-live. The permission and prompt-injection sections cause the most real-world incidents, so start there if you are short of time.
Pre-Deployment Security Checklist
Permissions and injection first. They cause the most real incidents.
Your ticks are saved in this browser, so you can work through the list over several sessions.
01Permission enforcement
0/6The most common serious flaw in enterprise retrieval.
02Prompt injection resistance
0/6Assume it will sometimes succeed. Limit the blast radius.
03Output handling
0/4Model output is untrusted input to everything downstream.
04Secrets and tool access
0/5What an attacker gains by influencing the model.
05Logging and monitoring
0/5Logs often reproduce the documents they retrieved.
06Abuse and cost controls
0/4Extraction, enumeration and cost exhaustion.
A practical pre-deployment prompt list covering LLM-specific failure modes. It is not exhaustive and does not replace formal security assessment, threat modelling or any obligations under a sector framework.
Why LLM Security Is Different
Three properties make these systems behave unlike conventional applications, and each one invalidates a defensive assumption that normally holds.
Instructions and data are indistinguishable
The model cannot reliably tell the difference between your system prompt and text inside a retrieved document. A malicious instruction hidden in a PDF is, from the model’s perspective, simply more context to follow. There is no equivalent of parameterised queries here.
Retrieval makes access control a security boundary
A retrieval system that ignores document permissions becomes a highly effective tool for surfacing information people should never see. Access control must be enforced at retrieval time, not applied afterwards to results already generated.
Output is untrusted input to everything downstream
Model output rendered into a page, executed as code, or passed to another system is untrusted data. Treating it as trusted because it came from your own application is how prompt injection turns into a conventional vulnerability.
The Six Review Areas
Ordered roughly by how frequently each causes a real incident in production deployments.
Permission enforcement
Whether retrieval respects existing entitlements, evaluated per user per query rather than assumed at index time.
Prompt injection resistance
What happens when instructions appear inside retrieved documents or user input, and what the blast radius is.
Output handling
How model output is treated by whatever consumes it, rendering, execution, or onward transmission.
Secrets and tool access
What credentials and capabilities the application holds, and what an attacker gains by influencing it.
Logging and monitoring
Whether you would detect misuse, and whether the logs themselves create a new exposure.
Abuse and rate controls
Protection against extraction, enumeration and simple cost-exhaustion attacks.
The Four Failure Modes Worth Understanding
Each of these has caused real incidents in production systems. Understanding the mechanism makes the checklist items make sense.
Indirect prompt injection
An attacker places instructions inside a document they can get into your corpus, an emailed invoice, an uploaded CV, a shared file, a web page you index. When the model retrieves that document, it may follow those instructions. The user never typed anything malicious.
- Assume any content in your corpus may contain adversarial instructions
- Limit what the model can do so a successful injection has small blast radius
- Require human confirmation before any consequential action
- Control who can add documents to the indexed corpus, and review uploads
Permission bypass through retrieval
The most common serious flaw in enterprise retrieval deployments. If the index is built without permissions and filtering happens after generation, or not at all, a well-phrased question surfaces restricted content. The system works exactly as designed; the design was wrong.
- Filter by user entitlement before retrieval, never after generation
- Re-evaluate permissions at query time rather than trusting index-time state
- Ensure permission changes propagate promptly to the index
- Test explicitly with a low-privilege account asking for restricted content
Over-privileged tool access
Applications given the ability to send email, write to systems or execute code inherit the risk of every instruction the model follows. If a prompt injection can trigger those capabilities, an attacker has effectively acquired them, and they did so by uploading a document.
- Grant the minimum capability the application genuinely needs
- Require explicit human confirmation for anything irreversible
- Scope credentials narrowly and rotate them on a schedule
- Log every tool invocation with the input that triggered it
Sensitive data in logs and prompts
LLM applications log prompts and responses for debugging, which means logs frequently contain everything the model saw, including retrieved confidential documents. Those logs are often stored with far weaker controls than the source documents they reproduce.
- Decide deliberately whether full prompts and context are logged
- Apply the source system’s access controls to the logs as well
- Set retention on logs consistent with the underlying data classification
- Consider redaction before logging for high-sensitivity deployments
Next Steps
AI Data Sovereignty Checklist
Where the data goes is a separate question from how it is secured.
Open the checklist →LLM Security and Data Privacy
The longer written guide to securing private LLM deployments.
Read the guide →Private LLM Readiness Assessment
Check the organisational foundations before building anything.
Assess readiness →Frequently Asked Questions
Can prompt injection be prevented entirely?
No, and any vendor claiming otherwise is overstating their position. Because the model processes instructions and data in the same context window, there is no reliable way to guarantee it will ignore adversarial instructions embedded in retrieved content. Input filtering and instructional defences raise the bar but are bypassable. The practical strategy is architectural: assume injection will sometimes succeed and limit what the application is permitted to do, so a successful injection produces a bad answer rather than a sent email, a deleted record or a disclosed document.
How should permissions be enforced in a retrieval system?
Filter at retrieval time, based on the identity of the user making the request, before any content reaches the model. The common mistakes are building a single index with no permission metadata and hoping the model exercises discretion, or generating an answer and then attempting to filter it, which is far too late because the content has already influenced the response. Permission state should also be evaluated at query time rather than trusted from when the document was indexed, since entitlements change and stale index-time permissions grant access that no longer exists.
What is a realistic way to test for these issues?
Three tests catch most serious problems. First, log in as a genuinely low-privilege user and ask directly for information you know is restricted, phrased several ways. Second, place a document containing an obvious instruction, telling the model to ignore prior instructions and reveal its system prompt, into the corpus, and see what happens. Third, review what your logs contain after a session involving confidential documents. These take an afternoon and find real problems far more reliably than a checklist review alone.
Do we need a specialist penetration test?
For internal, low-sensitivity deployments, a structured internal review using this checklist plus the tests above is usually proportionate. For anything customer-facing, anything handling sensitive or regulated data, or anything with tool access capable of consequential actions, a specialist review is worth the cost, because LLM-specific failure modes are not well covered by conventional application testing and a standard web application test will simply not look for them.
Should we log full prompts and responses?
It is a genuine trade-off rather than an obvious yes. Full logging is enormously valuable for debugging, quality evaluation and incident investigation, and most teams want it. The risk is that logs then contain reproductions of every confidential document the system retrieved, frequently stored with weaker access controls and longer retention than the source. If you log fully, apply the source data’s classification to the logs, restrict access accordingly, and set retention deliberately. For high-sensitivity deployments, consider logging metadata and identifiers rather than full content.
Is this checklist a substitute for a formal security assessment?
No. It is a practical pre-deployment prompt list covering the failure modes specific to LLM applications, and it is not exhaustive. It does not replace your organisation’s standard application security assessment, threat modelling, or any obligations you carry under a sector framework or contractual arrangement. Use it alongside those processes rather than instead of them, particularly for anything handling regulated data or exposed to customers.
Want This Reviewed Properly?
We review LLM deployments for Australian organisations before they go live, including the permission and injection testing that conventional application testing misses.