Fine-Tuning Is the Easy Part: Deploying Custom Language Models in Production
0
0
Customizing a large language model (LLM) is now the easy part. A capable engineer can fine-tune an open-weight model over a weekend, or wire up retrieval in an afternoon, and get results in a notebook that look ready to ship. The hard, under-discussed part is everything that happens next: making that customized model run reliably, affordably, and measurably for real users.
I saw exactly this at a mid-sized German equipment rental company. Prompting alone gave them a support chatbot that demoed beautifully and worked about 90% of the time. The other 10% turned out to be the bulk of the work: on the specialized German terms for their equipment it failed badly, and it kept slipping back into English, which infuriated the German customers it served. Prompting gave them no real lever to fix that, only re-wording and hoping. Fine-tuning was ultimately the answer, but only after the cheaper option hit its ceiling.
How much customization do you actually need?
Choosing whether to use fine tuning or context engineering is a crucial decision, because it dictates the operational burden later. Teams routinely reach for fine-tuning when a lighter option would have solved the problem, and then inherit months of serving and maintenance work they never needed.
A useful rule of thumb is to climb the ladder only as far as the problem forces you to:
- Start with prompt engineering. If you can get the behavior you want by changing the prompt, the few-shot examples, or the system instructions, do that first. It is instant to change and needs no training data.
- Add retrieval-augmented generation (RAG) when the model needs live context. If the gap is that the model does not know your documents, policies, or product data, retrieval usually beats fine-tuning. You keep a hosted model and feed it the right context at query time.
- Fine-tune when behavior or format must change. When you need a consistent style, a strict output format, or domain language that prompting cannot reliably produce, fine-tune. Parameter-efficient methods such as low-rank adaptation (LoRA) make this cheap, and a few hundred to a few thousand well-chosen examples is often enough. Quality matters far more than volume.
- Train a custom model only when justified. Training from scratch, or heavy continued pre-training, is rarely the right first move. Reserve it for cases where no base model fits your domain, data residency, or licensing constraints, and where you can fund the compute and the team.
The same four options, compared by effort and biggest downside:
- Prompt engineering: hours to days. Main risk: hitting a ceiling on reliability or format control.
- Retrieval-augmented generation (RAG): days to weeks. Main risk: retrieval quality, not the model, becomes the bottleneck.
- Fine-tuning (LoRA-style): weeks. Main risk: you now own a model artifact to serve and maintain.
- Custom training: months, and the highest cost. Main risk: large, ongoing spend on compute and specialist skills.
The pattern to notice: every rung up the ladder adds operational weight. A prompt change costs nothing to run. A fine-tuned model is an artifact you now have to serve, version, and watch.
Where teams stall: the skills gap
That operational weight is where capable teams stall, and the honest reason is a skills mismatch. Productionizing a custom model is not one discipline, it is three overlapping ones. It needs machine learning skills to implement the training pipeline, machine learning operations (MLOps) to serve, version, and monitor it, and data engineering to build the pipelines that feed both training and retrieval. Few in-house teams hold all three at the depth production demands, and the people who are strong at building a model in a notebook are frequently not the people who keep it healthy under load.
So if your team is deep in one or two of these areas but thin on the rest, the pragmatic move is to close that gap deliberately rather than hope it resolves itself. Because the work spans model customization, serving, and evaluation at once, bringing in specialist LLM development services is often the fastest way to move a fine-tuned prototype into a production-grade, monitored system, without spending two quarters learning MLOps the hard way.
What deployment really involves for a custom model
Calling a hosted API is a solved problem. You send text, you get text, and someone else owns the hardware. A custom or fine-tuned model breaks that simplicity, because now you own the parts the API provider used to hide.
Serving. You have two broad options. A managed endpoint, where you upload your weights and get an autoscaling API back, is the fastest route and hides most of the infrastructure. Self-hosting on your own graphics processing units (GPUs), typically behind an inference server such as vLLM, gives you control, data residency, and better economics at high volume, at the cost of running the stack yourself.
Cost and latency. GPU time is the dominant cost, and it is not cheap. A single modern data-center GPU rents for roughly $1 to $3 per hour on specialist clouds, and several times that on the major hyperscalers. That meter runs whether or not requests are arriving, so idle capacity is pure loss and utilization decides your unit economics. Latency is the other trade-off: you are managing time to first token and tokens per second, and levers such as quantization, batching, and sequence length move cost and speed at the same time.
Versioning and rollback. A fine-tuned model is a versioned artifact, like a software release. You need to know exactly which weights and which prompt template served any given response, and you need a fast path to roll back when a new version misbehaves. “We retrained and pushed it” is not a deployment strategy. Lineage and provenance are requirements, not nice-to-haves.
A fine-tuned model is heavier to operate than a hosted API call, and the cost is ongoing rather than one-off.
Evaluation and monitoring
A customized model also requires an evaluation harness. The ML lifecycle is in part building a dataset to train on, but that doesn’t necessarily cover real world use. You have to measure the behavior you actually care about.
At minimum you need three things:
Artificial Intelligence – The Data Scientist