LLM Fine-Tuning Services for Australian Organisations

The data requirements depend heavily on the fine-tuning technique and the scope of the adaptation. For instruction fine-tuning with LoRA, meaningful improvement is achievable with as few as 500 to 1,000 high-quality instruction-response examples. For broader domain adaptation or complex style alignment, 5,000 to 50,000 examples produce more robust results. For continued pre-training to acquire domain vocabulary, you need a larger corpus of domain text, typically hundreds of megabytes to several gigabytes. The quality of training data matters far more than quantity: 500 carefully curated, accurate examples will outperform 10,000 noisy ones.

Catastrophic forgetting, where fine-tuning on a narrow domain degrades general capability, is a real risk in naive fine-tuning approaches. We mitigate this through several mechanisms: including a proportion of general instruction examples in the training mix, using LoRA or QLoRA which modify only a small fraction of the model's parameters, and running regression tests against general benchmarks before and after fine-tuning to confirm no degradation. For most enterprise fine-tuning tasks, the goal is domain enhancement rather than domain replacement, and correctly configured training preserves general capability while improving domain performance.

Open-source models like Llama 3, Mistral, and Qwen can be fine-tuned with full access to the model weights, deployed on any infrastructure, and the fine-tuned weights are yours entirely. Commercial API models like GPT-4 and Claude offer limited fine-tuning options through their APIs, with restrictions on what can be fine-tuned, training data that passes through their infrastructure, and deployment locked to their API. For Australian organisations with data sovereignty requirements, open-source fine-tuning on sovereign infrastructure is typically the only viable path.

A standard enterprise fine-tuning engagement from data assessment to production deployment takes eight to twelve weeks. The breakdown is: two to three weeks for data assessment and curation (often the most time-consuming phase), one to two weeks for base model selection and baseline evaluation, one to two weeks for fine-tuning runs and hyperparameter search, and two to three weeks for evaluation, refinement, and production deployment. The data curation phase can be accelerated if your organisation already has well-structured instruction-response data, and extended if significant curation work is needed.

When a significantly better base model is released, re-fine-tuning on the new base model is typically worthwhile because the improvement in the underlying model capabilities compounds with your domain fine-tuning. The cost of re-fine-tuning on an existing dataset is significantly lower than the initial fine-tuning engagement, since data curation is already complete. We maintain your training datasets and evaluation test sets so re-fine-tuning on a new base model can be completed quickly. For most deployments, we recommend evaluating re-fine-tuning annually or when major new open-source model generations are released.

Yes, with important caveats. Compliance embedding through fine-tuning means training the model to include required disclosures, avoid non-compliant statements, and follow regulatory guidance in its responses. This works well for consistent compliance patterns, such as always including a required disclaimer or structuring advice according to a prescribed format. It works less well for compliance requirements that are highly context-dependent and require case-by-case judgment. For the latter, RAG retrieval of the relevant compliance guidance combined with a system prompt is more reliable than trying to encode compliance judgment into model weights.