Training a Model Is Different from Using One

October 13, 2026

3 Minutes

An AI tool responds to you, adapts its wording and remembers something you said a few messages ago. It is easy to imagine that you have just trained it.

Usually, that is the wrong description of what happened. A conversation can supply information for the current response without updating the underlying model's parameters. A product may also store information or later use data in a training process, depending on its settings and policies. These are separate mechanisms.

I think keeping them separate is one of the most useful things a beginner can learn.

What training changes

In supervised learning, developers provide examples with target answers. A model makes predictions, its errors are measured and its parameters are adjusted through the training process. Features are the information available to the model; a label is the target it is learning to predict.

Inference is the use of a trained model on an input to obtain a prediction or generated output. The model can use information supplied with that input without that operation, by itself, constituting a new training run.

A useful question for any supplier is therefore: are we changing the model, providing context, retrieving stored information or doing several of these things?

Two different operations. Training examples: Used to fit a model. Parameter updates: Changes learned parameters. New input: Supplied during use. Inference output: Prediction or generation

Why the test needs unfamiliar examples

A model that performs well on its training examples may perform poorly elsewhere. Overfitting describes learning patterns that suit the training data too closely, without generalising well enough to new data.

Developers commonly separate training, validation and test sets. Training fits the model. Validation helps guide development decisions. A test set is reserved for a later assessment. Repeatedly changing the model in response to the same test results weakens the independence of that test.

Imagine a fictional production company building a document classifier. Its archive contains several copies of the same call sheet. If nearly identical copies appear in both training and evaluation, a strong score may say less than it appears to say.

My proposed test would keep related copies together and include documents from genuinely different productions. I would also include incomplete documents and unexpected formats. That is a suggested evaluation design, not a claim that this alone guarantees generalisation.

The important question is whether the examples resemble the work the tool will encounter after launch, including its less convenient corners.

Keep the final test independent. Train: Fit parameters. Validate: Guide development. Choose: Select the approach. Test: Assess on held out examples

Read a performance claim carefully

When someone shows you an accuracy figure, ask what was measured. Which documents were included? Who checked the answers? Were difficult examples removed? What happened to uncertain predictions? Can you see the errors as well as the successes?

For our fictional archive, an overall score could conceal a painful weakness. The tool might sort most documents correctly while repeatedly misrouting location permissions. I would report performance for that document type separately and keep a manual review step until it was trustworthy enough for the intended use.

Changing a prompt may improve the results you receive. Uploading an updated reference file may improve access to information. Neither should automatically be described as retraining the underlying model.

Before trying a product, write down what information you will provide, what the product says it stores and what you expect to change. Then compare the observed behaviour with that expectation.

The distinction is not academic. It helps you ask clearer questions about an AI service, interpret a demonstration and avoid assuming that a pleasant conversation has taught the system everything you wanted it to know.