Private Company Data Could Make Small Models Much More Useful

Two rounds of an email-triage experiment, an audit that knocked down the first one, and many ways of teaching one business decision to a model, measured on the same test.

A company may have years of emails and support tickets showing how its people make decisions. Who receives a request. Which incident gets immediate attention. When a reply is expected, and when the work can continue without one.

Much of that knowledge is hard to capture in a general instruction such as “classify this email.” It depends on the company’s responsibilities, working habits and exceptions. An email that looks routine to an outsider may need attention today because of a commitment only that company knows about.

That is why I want to test small models on private business data. If those decisions can become reliable training examples, could a small, fast model learn enough of a specific workflow to be useful?

I have not tested this on real company data yet. What I have done is two rounds of the same experiment on synthetic data, with three families of models and every reasonable way of teaching them the decision. The short version:

• Examples are what teach a small model the decision. Four models of 0.3 to 2.2 billion parameters went from 8–57% to 90–93% after fine-tuning on 1,985 labelled emails, and answer in hundredths of a second.
• A model pretrained for decisions needs no examples. The largest Decision 3.0 model, at 26 billion parameters, scored 98.5% straight out of the box. A general-purpose 9-billion-parameter model reached at most 61% with a written policy.
• With enough examples, a larger model goes further. Fine-tuned models of 8–9 billion parameters reached about 99% (Decision 3.0 flash 99.2%, Qwen3.5-9B 99.0%), against 90–95% for the smaller ones, at several times their cost per email.
• Writing a better prompt helps a little; putting examples in the prompt does not. That held for every family.

The data is synthetic, and the labels follow a policy applied without exceptions, so none of this proves how a model would do inside a business. It does show which levers move a model on a decision like this, and what each one costs.

I did not do this alone. Two coding agents, Codex and Claude Code, helped write the dataset-generation, training and evaluation scripts, analyse the results, and manage the Colab and DGX Spark environments. I designed the experiments and reviewed the figures reported here. The code, the datasets and every measurement are public in teaching-one-decision on GitHub.

The useful data explains a decision

Each email needs three answers: which team handles it, how urgent it is, and whether the sender expects a written reply.

Those fields sound simple until someone has to label them consistently. A message about a technical product might belong to sales. A resolved incident might still carry an urgent subject line. A request to update a document might need action but no email response.

A company’s records could help explain those distinctions. The examples worth looking for connect the message, the context available at the time, the decision and any later correction. A ticket moved to another team can reveal a routing mistake. A human override can show where a rule needs an exception.

The training example must contain enough information to support its label. If urgency depends on an internal deadline, the model needs that deadline in its input. If the label depends on information the model cannot see, the model can only guess.

A stable decision rule is also different from a changing fact. Training can teach the model how to treat a missed deadline. The current deadline should come from the business system when the decision is made. No checkpoint will stay current with every contract or staffing change.

Proposed architecture. Reviewed history shapes the checkpoint offline; current business facts arrive with each new email. Nothing in this diagram was deployed in the experiments.

Round one: a result that did not hold up

The first test used 3,200 synthetic emails from 80 topic families. Laya multilingual, a 322-million-parameter model that scores typed decisions instead of generating text, went from 16.75% to 88.25% after fine-tuning, ahead of Qwen3.5-9B at 80.00%.

Then I audited it. A word-count classifier trained on the same emails, with no GPU, scored 84%. Every high-priority email said “in N minutes” or mentioned the live service, so one regular expression found all of them. The distractors, a quoted “URGENT” thread and an injected instruction, were identical in training and test, and most of Laya’s lead came from those emails. Qwen had also run without its optimized kernels.

Every number matched the raw predictions. The test was simply too easy to show what the model had learned.

Round two: a test built to resist shortcuts

I rebuilt the dataset so that surface patterns could not solve it.

Priority depends on context, not words. Each input starts with a header from the triage system: when the email arrived, the customer’s service level (Premium or Standard) and any open incidents. An email is high priority only if it concerns an incident marked “customer impact: yes”, sets a deadline that falls on the day it was received, or comes from a Premium account whose work is blocked.

The same text can carry different labels. 620 emails are counterfactual twins: the same text with a different header. “Thursday at 3 pm” is urgent if the email arrives on Thursday morning and routine if it arrives on Monday. A model that reads only the words can get at most half of each pair right.

Category follows the request, not the topic. Every customer sends emails to all five teams, and half the emails mention another team’s topic in passing.

No templates. I wrote 80 customer accounts, 150 request types and 60 notice types by hand. Accounts and request types used in the test never appear in training. gpt-6.1-sol wrote each email from a structured brief, a second check confirmed the text matched the brief, and labels always came from the brief, never from a model.

The result has 3,176 emails: 1,985 for training, 395 for validation, 398 for calibration and 398 for the test. Before any model ran, I checked that it could not be shortcut:

The last row matters as much as the others. A hard test is useless if its labels are ambiguous; when a strong model given only the written policy recovers 99.5% of them, they follow from the input. Running that check over every email also caught my own mistakes: 11 ambiguous notice topics and 140 requests that cannot sensibly be “done without a reply”, such as “explain why…”. I fixed and regenerated those first.

Three families, five ways to teach

I compared three families of models:

• Laya multilingual, 322 million parameters, a decision model that scores answer options.
• Qwen3.5-9B, a general-purpose model that generates text, run in 4-bit with its JSON output constrained to the schema.
• Decision 3.0, six decision models from 0.59 to 26 billion parameters, built by the vLLM Semantic Router team on Qwen 3.5 and 3.8 and released on 10 October 2026. Like Laya, they score answer options in one pass and take the same question format.

And five ways to teach them the decision, applied to every family where they make sense:

1. The written policy as published. Each field’s definition as an instruction. This is zero-shot: no examples.
2. Prompt v2. The same policy rewritten as an explicit procedure, such as “work out the deadline’s calendar date and compare it with the Received date”. It adds no information.
3. Prompt v2 plus eight examples. The same eight training emails for every model, covering all teams and priorities and two twin pairs.
4. Thinking. Qwen only: up to 1,024 tokens of reasoning before the answer. Decision models do not generate text.
5. Fine-tuning on the 1,985 training emails. Full fine-tuning for the small decision models, QLoRA or LoRA adapters for the larger ones. Details are in the appendix.

Laya and Qwen ran on an NVIDIA T4 in Google Colab. Decision 3.0 ran on an NVIDIA DGX Spark (GB10). Accuracy is comparable across all of them; speed is comparable only on the same hardware.

Without examples: what the model already knows

All three fields right, on the 398 test emails. Qwen with thinking reached 75.0% on a 44-email sample (59.1% for prompt v2 alone on the same emails), at 94 seconds per email.

Three things stand out.

Pretraining for decisions beats size alone. d3-flash and Qwen are about the same size, and d3-flash scored 79–84% against Qwen’s 53–61%. It also used the header: on the 74 twin pairs, it got both priorities right in 80–82% of pairs; Qwen managed 19–35%.

The largest decision model is already at the ceiling. d3 at 26 billion parameters scored 98.5% (95% interval 97.5–99.5%) with no examples. It caught all 122 urgent emails with no false alarms, and got both emails right in every twin pair. Its six errors were all borderline notices, such as a firewall change that one reader would file under technical and another under information. It was released a day before I generated the dataset, so it cannot have seen these emails.

Prompting is a small lever. Rewriting the policy as a procedure helped the mid-sized and large models by two to eight points and did not help the small ones. Eight examples in the prompt made almost every model worse, sometimes by ten points. Small models, in particular, lose track of long instructions.

With examples: what fine-tuning adds

Training times include validation after each epoch for the decision models.

Every small model learned the decision. Laya and the three smallest decision models started between 8% and 57% and ended between 90% and 93%. They learned to use the header, not just the words: d3-edge caught all 122 urgent emails with no false alarms and got both priorities right in every twin pair. Among these small models, size barely mattered once they had the examples: d3-nano, nearly four times the size of d3-edge, ended at the same level.

A bigger model trained on the same data went further. With QLoRA on the full training set, Qwen reached 99.0%, clearly above the small models; the intervals do not overlap. A first QLoRA run that, because of a planning bug, saw only 448 emails already reached 93.2%: a larger pretrained model needs fewer examples to match a small one.

I did not fine-tune the 26B decision model. It already scored 98.5% without examples, at the label-consistency ceiling. On a 398-email test a fine-tune could add at most about 1.5 points, inside the interval, so no improvement could be shown, and it would have cost about five GPU hours. That was a deliberate choice, not a failed run.

Mid-sized models reached the top with one pass of LoRA. d3-flash, at 8.4 billion parameters, reached 99.2% (98.5–100%), level with fine-tuned Qwen at a similar size. d3-mini, at 4.5 billion, reached 94.5% and got every priority and reply decision right; all its remaining errors were the team. The pattern across families is clear: about 90–95% after fine-tuning for models up to 4.5 billion parameters, and about 99% for models of 8–9 billion, whether they started as decision models or as general ones. Where they started mattered without examples, much less with them.

What each option costs to run

Accuracy is close at the top. Running cost is not.

Times are medians per email after loading, one email at a time, with no batching. Batching would help every option, and generative models usually gain more from it, so the gaps are upper bounds for this setup. Loading also differs: under 12 seconds for the small models, about 100 seconds for Qwen and about 320 seconds for the 26B decision model.

On one T4, the small fine-tuned models would get through 1,000 emails in about a minute. Fine-tuned Qwen would take about 80 minutes. The 26B decision model needs about 50 GB of memory, which rules out small GPUs, though it needs no training data at all.

How I would choose

• No labelled examples: a model pretrained for decisions, as large as the hardware allows. On this test, the 26B Decision 3.0 model reached the ceiling with only the written policy, and the 8.4B one reached 79–84% where a general model of the same size managed 61%.
• Labelled examples and high volume: fine-tune a small decision model. It reached about 92% here, answers in hundredths of a second, and runs on hardware a company can keep in-house.
• Labelled examples and the highest accuracy: fine-tune a model of 8–9 billion parameters, ideally one built for decisions. d3-flash with LoRA reached 99.2% here at about half a second per email on the GB10; fine-tuned Qwen reached 99.0% at five seconds on the T4. The 26B decision model reached 98.5% with no training, at about 50 GB of memory.
• Whatever the choice: build a test that a word-count classifier cannot pass, and check that its labels follow from the input before trusting any number.

What a business would need to contribute

A real pilot starts with one narrowly defined decision and the people who already make it. They need to agree on the labels before anyone extracts a large dataset.

From there, the pilot needs:

• A reviewed sample. Each case carries the input available at decision time, a reviewed target decision, and a note on why ambiguous cases got their label. Corrected decisions, unusual cases and examples near category boundaries belong in it. Routine messages stay in proportion to the real workload.
• A critical look at history. A historical outcome is a starting point for review, not ground truth. It may record a mistake, an obsolete policy or a temporary workaround, and training on it unchanged would reproduce those choices. If reviewers disagree, the policy needs clarifying before the model needs more training.
• The context the decision needs, in the input. The second test worked because the header carried the facts priority depends on. In a company, that means the service commitment, the current owner or the state of an incident travel with the message.
• Controlled assets. The dataset and the resulting checkpoint are internal assets and need the same controls as the source data. A small model does not make a system private by itself, and these runs did not test a private enterprise deployment.
• An honest test. Hold back a later period of real cases, keep whole threads together, and check whether customer or project overlap makes the test too easy. Run a word-count baseline before any model, and check label consistency against the written policy.

For a first pilot, decisions would stay under human review, with every correction recorded. That measures the mistakes that matter and produces new examples at the same time.

Proposed pilot workflow. Predictions are saved and any temporary GPU is released before review. At first, every case is reviewed. Corrections go into a reviewed dataset; they never update the live model automatically.

What this does not show

The data is synthetic. The policy is explicit and applied without exceptions, and real decisions are noisier: two experienced people will sometimes disagree about the same email. Real inboxes also have long multi-party threads, missing context and policies that change. The emails were written by a language model from briefs, so they share its style. Each configuration ran once, with one seed. The thinking run covered 44 emails. Laya and Qwen ran on a different GPU from Decision 3.0, so their speeds are not directly comparable.

So the open question stays open: can a company’s own reviewed decisions teach a small, fast model its workflow? These experiments say the method works when the examples carry the right context, that models built for decisions start much closer to the answer, and that a test needs to be checked hard before anyone believes it. The next useful evidence has to come from real, reviewed company data.

Appendix: how each model was set up

Common to all. Same 398 test emails, same header and email text, same written policy. Complete accuracy means all three fields right. Intervals: 95% bootstrap over the ten test accounts (5,000 resamples). No test email was used for training, model selection or calibration.

Laya, full fine-tuning (T4). convaiinnovations/laya-multilingual at revision 1720e3e, Laya’s own trainer with the encoder unfrozen. AdamW with weight decay 0.01, learning rates 2.5e-5 (encoder) and 1e-4 (head), cosine to 1e-6, 32 decision items per update, FP16 mixed precision, gradient checkpointing, four epochs, best validation epoch kept (the fourth), probability temperatures fitted on the calibration split.

Qwen3.5-9B (T4). Revision c202236, 4-bit NF4 weights, FP16 compute, SDPA attention, flash-linear-attention and causal-conv1d kernels confirmed, greedy decoding, JSON constrained with LM Format Enforcer, one email at a time. All answers were valid JSON. QLoRA: rank 16, alpha 32, dropout 0.05 on every linear layer of the language model (about 43 million trainable parameters), loss on the answer tokens only, paged 8-bit AdamW at 2e-4 with 10 warm-up updates and cosine decay to 10%, eight emails per update, prompt v2 as the system prompt. The full run trained on 1,984 of the 1,985 emails (the batch size rounds down) in 4 hours 22 minutes.

Decision 3.0 (GB10). Revisions pinned for all six models. Evaluated with their own runtime (trust_remote_code, reviewed before use): text questions, one pass per question, answer probabilities from a readout over answer codes. The models ship no training code, so I wrote a trainer that reuses their prompt builder and reproduces their scoring path with gradients; a self-check confirmed it matched the runtime’s probabilities (largest difference below 10⁻⁷ for all five models) before any training. Small models (edge, lite, nano): full fine-tuning, AdamW with peak learning rates 1e-5 (backbone) and 5e-5 (readout), 5% linear warm-up then cosine decay, 32 items per update, four epochs, best validation epoch kept. Laya’s higher learning rates without warm-up made d3-edge collapse in a first attempt, with gradient norms up to about 960, so I lowered them. Larger models (mini, flash): LoRA with the same settings as Qwen’s QLoRA, but on BF16 weights because memory allowed, readout frozen, one epoch. The 26B model was not fine-tuned, for the reason given above.

Software. Colab: PyTorch 2.11 (CUDA 13.0), Transformers 5.19, PEFT 0.21.0, bitsandbytes 0.50.2. DGX Spark: PyTorch 2.11 (CUDA 13.0), Transformers 5.17 (required by Decision 3.0), flash-linear-attention 0.5.2, Triton 3.6, PEFT 0.21.0.

Experiment note: two synthetic datasets. The second has 3,176 emails split by customer account, with labels derived deterministically from hand-written scenarios and text written by gpt-6.1-sol. No private company data was used. Codex and Claude Code assisted with dataset generation, training and evaluation scripts, results analysis and environment management (Colab via colab-cli, and a DGX Spark through the operator’s SSH session); the author designed the experiments and reviewed the figures. Model revisions were pinned. Per-row predictions, training reports, timings, hashes and audit scripts are in the public repository.