Blog

Smoke-Testing a Small Language Model with RadiChat

A repeatable smoke test for a small local model: one prompt file, one JSONL result file, and a diff you can read after every change.

The problem

You have deployed or fine-tuned a small model and it answers questions. Then you change something — a new quantization, a longer context window, different sampler settings — and you need to know whether it still behaves. Typing a few questions into a chat window tells you something, but you cannot compare today’s answers with last week’s, and you cannot put the result in a script.

A smoke test needs three things: the same prompts every time, the answers captured in a form you can diff, and a command short enough to actually run.

Start with the cheapest signal

Before testing what a model knows, test whether it is still holding the conversation. Context retention breaks easily when the context window or the trimming rules change, and it is quick to check.

Ask something, then ask a follow-up that only makes sense with the answer in mind. If the second reply shows the model remembered the first, the basics work. It is a signal, not proof: holding the thread says nothing about whether the answer is right.

A working example

RadiChat reads prompts from a file and writes one JSON record per prompt. Point it at a local endpoint:

{
  "endpoint": "http://127.0.0.1:8080/v1/chat/completions",
  "model": "qwen2.5-7b-instruct",
  "context_budget": 8192,
  "generation_reserve": 1024,
  "system_prompt": "You are a concise assistant."
}

The prompts, one per line:

I'm serving a 7B model locally and responses feel slow. Name three likely bottlenecks.
Which of those is cheapest to rule out first?

Then run it:

label=baseline
./radichat -input prompts.txt -output "results-$label.jsonl"

Each line of the output holds one prompt and its complete reply. These records are illustrative:

{"v":1,"line":1,"kind":"turn","status":"ok","input":"I'm serving a 7B model locally and responses feel slow. Name three likely bottlenecks.","output":"1. GPU compute saturation. 2. Memory bandwidth. 3. Prompt prefill cost.","finish_reason":"stop","trimmed_pairs":0}
{"v":1,"line":2,"kind":"turn","status":"ok","input":"Which of those is cheapest to rule out first?","output":"GPU utilization. One command tells you whether the model is compute-bound.","finish_reason":"stop","trimmed_pairs":0}

The second question names no subject. If the reply ranks the GPU bottleneck, the model kept the thread.

The takeaway

Keep the prompt file beside the model and re-run it after every change. RadiChat refuses to write to an existing path, so give each run its own label and yesterday’s results survive. Add more prompts to the same file and the run widens into a regression suite: questions whose answers you have already read once, so a changed answer stands out immediately. A changed answer is not automatically a regression. Local models are not deterministic and a rewording is often an improvement; what you want is a change you cannot explain. Pull the answers from a run to compare:

jq -r 'select(.kind=="turn") | .output' results-baseline.jsonl

Or confirm every turn completed:

jq -r 'select(.kind=="turn") | .status' results-baseline.jsonl | sort | uniq -c

That is the whole workflow: one file of prompts, one file of results, and a diff you can read. RadiChat is a single Go binary with one JSON config file and no runtime dependencies, so it drops into any script and works with any OpenAI-compatible chat-completions endpoint. It stores the replies exactly as the model produced them and does not score them, so the judgment stays yours.

Flags, configuration fields and the JSONL record format are documented in the CLI reference and the batch output reference. The RadiChat product page explains how context is trimmed when a conversation outgrows its budget.