Skip to content
Skip to content

Practical LLM operations guide · By Yes AI

Customer chatbot or staff assistant: which first?

Published 5 min read

Both run on the same technology: a model answering from your documents. The difference is who reads the answer before anyone acts on it. This guide compares the two starting points so you can pick the one your organisation can supervise properly.

Quick answer

A staff-facing assistant is usually the lower-risk start, because a trained employee reads each answer before anything reaches a customer, the users are known and signed in, and mistakes are caught inside the business. A customer chatbot answers the public directly, so it needs a tighter scope, clear disclosure that it is AI, a working handover to people and stronger defences against misuse. Start with customers only when their questions are narrow and already well documented.

Customer service team member in a Perth office using an assistant on a laptop while on a headset call

What changes when customers see the answer directly?

With a staff assistant, the answer is a draft for an employee who knows the business. They can spot an outdated fee, an odd tone or a policy that does not apply, and fix it before it reaches anyone outside. With a customer chatbot, nobody stands between the model and the customer. The Australian Government's Guidance for AI Adoption uses a chatbot to illustrate this: one that answers simple questions during business hours while a staff member monitors it is a low-risk use, and the risk expands if it runs around the clock without oversight. The level of supervision is the main variable you control.

Which questions suit each audience?

Customer questions tend to be repetitive and public: opening hours, how to lodge a claim, what documents to bring, where an order is. Staff questions are broader and draw on internal material such as procedures, templates, past decisions and product specifications. Pull a month of enquiries from your inbox, phone notes or ticketing system and sort them by source: could the answer come from a page you already publish, from an internal document, or only from a person's judgement?

  • Customer-facing fits when answers come from published, stable content and a wrong answer is easy to correct.
  • Staff-facing fits when answers need internal documents, judgement or material customers should not see.
  • Neither fits when the answer depends on individual circumstances the system cannot check, such as an eligibility decision.

What does a customer chatbot need that a staff tool does not?

A public chatbot meets people you cannot vet, and some of them will test it on purpose. It needs controls that a signed-in staff tool can often do without, and each control needs an owner and a test before launch. Several of these also appear in Australian government and regulator guidance. Budget staff time to review conversations closely in the first weeks after launch. The list below is the minimum to plan for.

  • Disclosure: the Guidance for AI Adoption says customers receiving services and information from a chatbot should know it is not a human.
  • Handover: a clear route to a person, with the conversation passed along so the customer does not repeat themselves.
  • Narrow scope: a fixed list of topics and approved sources, and a polite refusal outside them.
  • Misuse defences: prompt injection is the first entry in OWASP's Top 10 for LLM applications, including indirect injection through content the model reads.
  • Privacy notice: the OAIC expects privacy policies and notifications to explain how AI is used.

How do the data and privacy questions differ?

A staff assistant mostly reads your documents and employees' questions. A customer chatbot collects personal information from members of the public: names, account numbers, health details or complaints typed into a chat box. For organisations covered by the Privacy Act, the OAIC says privacy obligations apply to personal information input into an AI system and to the output data it generates. Where the conversation is stored and processed also matters. If customer conversations must stay in Australia, check the deployment settings described in our guide to LLMs in Australian cloud regions before you choose a platform.

How do you measure success for each?

Pick measures that show whether the tool helps, and compare them with a baseline taken before launch. For a staff assistant, useful measures include time to find an answer, the share of answers staff accept without edits, and the number of repeated questions sent to subject experts. For a customer chatbot, look at conversations resolved without handover, handovers that reached a person, complaints that mention the chatbot, and repeat contacts within a week. Avoid relying on conversation counts alone, because a chatbot that fails to resolve questions also raises the conversation count.

Can one system serve both audiences later?

Yes, if you plan the separation from the start. A common pattern is one retrieval platform with two collections: a public collection of approved customer content and an internal collection with permissions. The customer chatbot is given the public collection only, enforced in the retrieval layer instead of by instructions to the model. Staff can use both. Starting with the staff assistant lets you test retrieval quality and source maintenance with a forgiving audience, then publish a narrower customer version once the content owners have their update routine working.

Staff assistant and customer chatbot compared

FactorStaff assistantCustomer chatbot
Who reads the answer firstAn employee, before acting on itThe customer, directly
UsersKnown, signed in and trainedAnonymous or lightly verified members of the public
SourcesInternal documents with permissionsPublished, approved content only
Personal informationMostly staff and internal recordsCollected from customers during the chat
Disclosure neededInternal policy and trainingClear notice that it is AI, plus a privacy notice
Misuse exposureLower, with accountable usersHigher, including deliberate testing
FallbackAsk a colleague or open the sourceHandover to a person within service hours

Practical checklist

  • A month of real questions is sorted by whether published content, internal documents or a person can answer them.
  • The first audience has a named owner who reviews a sample of answers each week.
  • For a customer chatbot, the AI disclosure, privacy notice and handover route are written and tested.
  • Retrieval limits each audience to its approved collections in the system itself, as well as in the prompt.
  • Prompt injection tests include text hidden in documents and web pages the model reads.
  • Success measures have a baseline recorded before launch.

Common questions

What rules apply to a customer chatbot in Australia?

Existing obligations apply. The OAIC says privacy obligations apply to personal information input into an AI system and to the output it generates, and it expects privacy policies to explain AI use. The Guidance for AI Adoption recommends telling customers when they are dealing with AI. Sector rules, for example on financial advice or health information, may add more; check with the relevant regulator or a qualified adviser.

Should the chatbot present itself as a person?

It should be clearly presented as automated. Give it a plain name, say that it is an automated assistant, and explain how to reach a person. The Guidance for AI Adoption uses this case as an example: a customer receiving services and information from a chatbot should know it is not a human specialist. Staff who take over a conversation should introduce themselves so the change is clear.

How should the handover to a person work?

Trigger it when the customer asks, when the question falls outside the approved topics, after two failed attempts to answer, or when the topic is sensitive, such as a complaint or financial hardship. Pass the conversation and details already given to the staff member. Outside service hours, say when someone will reply and how to make urgent contact, and make sure that reply happens.

Is a staff assistant still useful if customers never see it?

Yes. Staff often answer the same customer questions by searching shared drives and asking colleagues. An assistant that finds the right procedure with a citation shortens that search, and the employee still decides what to tell the customer. Our document search time calculator helps estimate the time involved for your own team.

Sources and further reading

Continue planning

Plan the checks before your next LLM release

Describe the task, source documents and people who will review the answers. We can discuss evaluation, access boundaries and a staged implementation. Keep confidential documents and access credentials out of this enquiry.