Skip to content
Skip to content

Practical LLM operations guide · By Yes AI

Test who can access information through your LLM

An employee assistant needs a defined answer to who may see each source of information. Before expanding access, test those boundaries through the application people will actually use. This guide helps a project owner prepare permission scenarios and review the results with the people responsible for access controls.

Write the access matrix before the prompts

Choose a small set of representative roles, such as warehouse staff, team supervisors and payroll administrators. For each source, have its owner state which roles may read it and whether any action is permitted. Record shared documents and restricted exceptions separately. Include contractors and people who have changed teams. This matrix becomes the expected result for the test; the language model should not be asked to decide the business permission policy.

Build distinguishable fictional test records

Create approved test documents containing invented information that can be recognised in an answer. Put one record in a broadly available collection and another in a restricted collection. Use separate test accounts with the intended role assignments. Avoid using real salaries, customer details or confidential contract terms as test markers. Record the source identifiers, account roles and expected access so a reviewer can reproduce a failure without receiving sensitive production content.

Check each place information can appear

Test the answer text, source excerpts, citations and any downloadable output. Ask the technical reviewer to inspect the relevant retrieval and application evidence as well as the visible response. Include direct requests for a restricted document and ordinary questions that might draw on it indirectly. If the assistant offers tools or record updates, test those permissions separately. Reading a document and changing a business record require separate expected outcomes.

Include access changes and shared sessions

Plan cases for a revoked permission, a moved document and a user switching accounts on a shared device. Ask the system owner when changes are expected to take effect, then test against that documented behaviour. Check saved conversations and cached results as applicable to the design. If there is an interval before a restriction takes effect, record it as an unresolved operational decision rather than describing the change as immediate.

Rehearse a fictional payroll boundary

In a fictional test, a warehouse account can read shift instructions but must not read an invented payroll adjustment. The tester asks both where to find shift guidance and what adjustment a named fictional employee received. The reviewer checks the visible response and available retrieval evidence against the matrix. They then repeat with an authorised payroll account. A restricted answer passing one role test does not replace testing whether the authorised role can still work.

Record failures before widening the pilot

For each case, save the expected outcome, observed behaviour, application version and evidence location in an access-controlled review record. Assign any unexpected disclosure or denied legitimate access to an owner. Re-run the failed case and related role cases after a correction. Keep broader access on hold while a material boundary remains unresolved. Successful scenarios provide evidence about those scenarios; they do not establish a general security or compliance guarantee.

Before you approve the workflow

  • Source owners approve the role and information matrix.
  • Test documents and accounts contain fictional data.
  • Answers, citations and relevant retrieval evidence are checked.
  • Permission changes and unresolved failures have named owners.

Continue planning

Plan the checks before your next LLM release

Describe the task, source documents and people who will review the answers. We can discuss evaluation, access boundaries and a staged implementation. Keep confidential documents and access credentials out of this enquiry.