Why does a deployed model need ongoing work?
Hosted models have retirement dates. Under Microsoft's standard lifecycle, a generally available Foundry model retires 18 months after launch, and requests to a retired model return an HTTP 410 error. AWS gives most Bedrock models launched since September 2026 a 6-month Legacy period before end of life, after which requests fail, and migration does not happen automatically. OpenAI commits to at least 6 months' notice before retiring a generally available API model. Separately, your documents and staff list change every week. Without someone checking, an assistant can keep running while its answers fall behind current policy, and nobody notices until a wrong answer causes a problem.
How do you handle a model retirement?
Treat a retirement notice as a planned change with a test. Find which deployments use the model, choose a replacement, run your saved test questions through it, compare the results with the current model, and switch only after the task owner accepts the differences. Watch for automatic upgrades. On Azure, Standard deployments can be upgraded automatically when a version retires unless the deployment's versionUpgradeOption says otherwise, while Provisioned deployments are not upgraded and must be migrated by hand. An automatic upgrade keeps the service running but changes the model underneath it without your test.
- Make sure retirement notices reach a monitored mailbox; Microsoft sends GA retirement notices at least 60 days ahead to subscription owners with active deployments.
- On Bedrock, existing customers of a Legacy model may lose access after 15 days of inactivity.
- Keep the test set and results from the last change so the next comparison is quick.
Who keeps the source documents current?
The assistant answers from what is in its index. When a policy is updated in SharePoint but the sync has failed, staff receive the old answer with a confident citation. Support covers the plumbing: checking that scheduled syncs ran, retrying failed files and confirming that deleted or retired documents leave the index. The content itself belongs to the business. Each source collection needs an owner who approves changes and tells support when a document is withdrawn. Our guides on document version control and retired document removal cover those steps in detail.
Hypothetical example: an organisation with 200 staff has four source collections (HR policies, IT procedures, client templates and board papers). Support reports sync status for each collection every week, and the four owners each review a sample of answers once a month. These numbers are illustrative assumptions.
What should be monitored every week?
Monitoring catches problems before staff stop trusting the tool. A short weekly report works better than a dashboard nobody opens. Agree what goes in it, who reads it and the threshold that triggers action for each item. Microsoft and AWS both expose model lifecycle status through their APIs, so model status can be checked by a script instead of by reading release notes. The items below are a reasonable starting set.
- Sync health: last successful run for each source and the count of failed files.
- Answer quality: a sample of answers checked against their sources, plus staff feedback reports.
- Refusals and escalations: questions the assistant could not answer, grouped by topic.
- Usage and cost: tokens or compute by department, with an alert for unusual rises.
- Access: new, changed and removed users matched against the identity system.
- Model status: the lifecycle state of each model, read from the provider's API.
What happens when something goes wrong?
Write the incident path before you need it. Typical incidents include a sync that exposes a restricted document, an answer that gives wrong instructions, a provider outage and a prompt injection attempt found inside a document. The runbook should name who can switch the assistant off or remove a source, how users are told, and who assesses whether personal information was involved. Under the Notifiable Data Breaches scheme, organisations covered by the Privacy Act must notify affected individuals and the OAIC when a data breach is likely to result in serious harm. Your privacy officer makes that assessment, with technical facts from support.
What should a support agreement spell out?
Support offers vary widely, so compare them on scope instead of labels such as managed or 24x7. Ask each provider to state in writing which tasks they perform, how often, what they report and what needs your approval. Response times and hours of cover are worth stating, but they mean little without a clear list of what is covered. Support is normally quoted after a short scoping conversation; our pricing page explains how we approach it.
- Model lifecycle: who watches notices, tests replacements and schedules the switch.
- Index operations: sync monitoring, failed-file handling and removal of withdrawn documents.
- Security: patching of self-hosted components and review of access logs.
- Reporting: what you receive each week or month, and in what form.
- Exit: what transfers to you or a new provider if the agreement ends.