ThinkingBox trains agents on database state, not claims

ThinkingBox-Training rewards agents for the database state they leave behind. Saying the work is finished doesn't count. In the reported Qwen3.8-27B run, tasks that passed every recorded attempt rose from 38 to 107 after 50 GRPO steps. At-least-once task coverage went the other way: it fell. I think this is a promising way to train database-writing agents, but those two results belong together.

Microsoft and Hugging Face announced the available training stack in their joint ThinkingBox post. The release covers data selection, rollouts, rewards and Verl integration. Those are the pieces that turn backend checks into a training signal, rather than leaving the checks as something that only runs during evaluation.

Reward the database state, not the completion message

The training release takes that executable verdict and uses it as a reward for tasks outside ThinkingBox-Bench.

The original benchmark contains 507 tasks. Each starts a real backend and gives the agent MCP tools plus a simulated LLM user with private context; the check then runs against the terminal database state. The motivating failure is worth keeping concrete. An agent closed a customer's ticket as resolved while leaving the underlying carrier exception open. Nine plausible tool calls. Still the wrong database row.

The grader asks whether the required backend outcome happened. It does not demand a prescribed tool-call sequence, so a different path can pass. Wrong, missing or extra side effects fail. For most tasks, no separate scoring step sits between that verdict and the training reward: the correct state earns 1, and an incorrect outcome earns 0.

“Executable” needs a qualification. The large majority of checks are deterministic and require no LLM judge at all. A minority of narrow requirements work differently: they ask the configured judge a binary rubric question, then feed its answer into the executable test. In the reference configuration, that judge is GPT-5 Chat.

For an agent attached to a React and Node.js application, I'd write acceptance criteria against the backend records before evaluating its final chat response. That's the part I'd carry into an evaluation, whether or not I used this training stack.

More repeatable success, less at-least-once coverage

Repeatability improved. So did average success. The fine-tuned model nevertheless solved fewer distinct tasks at least once, which rules out calling this an improvement across every reliability measure.

Each benchmark task was evaluated 20 times: 10,140 attempts per model. Here, pass@1 means average success across those attempts. Pass@20 counts tasks solved at least once. Observed 20/20 is stricter: a task must pass on every recorded attempt, not merely achieve a high average success rate.

I'd preserve those distinctions in the acceptance criteria for a model update.

The hotel-booking example shows why this matters. A requested downgrade required manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved booking and payment state in all 20 attempts, but matched all expected records in 19. That's a meaningful improvement in respecting the approval boundary, in my view. It is not a completely solved task.

All five benchmark domains gained on pass@1 and observed 20/20. Not on pass@20. That measure increased in neobank and consulting but declined in retail, travel and auto insurance. Auto insurance had the largest coverage loss, at 7.0 percentage points; retail had the largest observed 20/20 gain, at 24.5 points.

The authors describe the pattern as consistent with “sharpening”: probability moves toward repeatedly successful trajectories while some low-probability solution paths disappear. That explanation fits what was observed. I'd stop there, though. The pattern alone does not demonstrate what happened to every trajectory that changed, even if “sharpening” is a useful account of the aggregate result.

Give GRPO tasks with outcomes it can distinguish

The training pool keeps tasks the base model sometimes solves and drops the extremes. The reason is concrete: group-relative training needs successful and unsuccessful outcomes to compare.

Eight failed rollouts yield [0, 0, 0, 0, 0, 0, 0, 0]. Eight successes yield [1, 1, 1, 1, 1, 1, 1, 1]. Center either group and no task-reward advantage distinguishes one trajectory from another. A mixed group does.

Producing these groups is expensive. Every rollout provisions an isolated backend, runs a tool session and calls a separate model to simulate the user. Task selection therefore determines how much of that work can produce a useful reward distinction.

The authors call this offline policy-relative difficulty. They probe candidate tasks with the base policy before training, estimate their difficulty, and then hold the selected pool fixed for the entire run.

For a candidate with k successes in n probes, the estimated success probability follows the Jeffreys posterior Beta(k + ½, n − k + ½). The selection rule assigns the bucket containing the greatest posterior probability mass. It is not simply bucketing the raw observed success fraction.

I'd treat these boundaries as reference hyperparameters for this run, not as defaults I could carry over unchecked.

In the worked example, GPT-5.4 first screens a task for solvability. The base model then succeeds on 3 of 8 probes, giving Beta(3.5, 5.5). The Difficult bucket wins: it contains 54.0%, the largest share of posterior mass. Later, during training, a mixed reward group gives successful trajectories above-mean task advantage and failed trajectories below-mean advantage.

The curated pool contains 187 tasks. Difficulty is still relative to the policy; another model could place the same task in another bucket. My concern is the deliberate decision not to reassess difficulty once training starts.

Training and evaluation must read the same agent output

If training and evaluation interpret Qwen output differently, their reward and benchmark measure different agent behaviour. I'd want one shared contract. Two independent implementations are not what I'd choose here, and the release puts the shared interpretation logic in trainer/core/tb_interpret.py, where training and evaluation can use it.

The contract covers reasoning extraction, tool calls, argument typing, rendering and isolated task state. Each rollout is a complete episode, with a fresh backend, an isolated MCP session and GPT-5 Chat acting as the simulated user.

The reference run uses full-parameter fine-tuning with token-mean GRPO. Each update generates 18 groups of 8 rollouts: 144 episodes. Training is synchronous, with both parameter and optimizer offload. For the optimizer, the run uses AdamW at a learning rate of 1e-6 and zero weight decay.

During training, zero-variance groups are replaced DAPO-style. System errors follow a separate path. A backend timeout does not count as an incorrect agent outcome; affected groups are refilled instead. I'd preserve that distinction in an adaptation, because otherwise infrastructure failures would enter the task-outcome signal as if the agent itself had produced the wrong result.

Install the pinned stack, then verify the patched Verl checkout

The validated reference topology is 3 nodes with 8 H100 GPUs each. The environment targets Linux, Python 3.12, Verl 0.9.0 and the pinned package lock. The authors report testing on WSL too. That report is separate from the validated reference topology, not a replacement for it.

Before provisioning, check for NVIDIA GPUs with a compatible CUDA/NCCL stack, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for both the simulated user and the judge.

1. Create the side-by-side repository workspace. Keep the repositories in the supplied layout and preserve the backend pins.

   mkdir -p ~/thinkingbox-workspace
   cd ~/thinkingbox-workspace
   git clone https://github.com/microsoft/thinkingbox.git
   git clone https://github.com/microsoft/thinkingbox-data.git
   git clone --branch main https://github.com/microsoft/thinkingbox-training
   git -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1
   git -C thinkingbox-data checkout thinkingbox-bench-v1.0

2. Install the validated environment and patched Verl checkout.

   cd ~/thinkingbox-workspace/thinkingbox-training
   uv venv --python 3.12
   source .venv/bin/activate
   uv sync --frozen --extra fast --extra tracking --extra eval-proxy
   uv pip install -e ../thinkingbox
   uv pip install --config-settings editable-mode=compat \
     -e ../thinkingbox-data/servers/thinkingbox_tools
   uv pip install --config-settings editable-mode=compat \
     -e ../thinkingbox-data/servers/tb_business_ops_servers_202606
   python scripts/prepare_verl.py --dest .deps/verl --install

3. Verify the installed dependency state. Run both checks below. They should report a verified patched Verl checkout and no broken requirements in the installed environment, not merely an installation that completed without stopping.

   python scripts/verify_verl_install.py
   uv pip check

   Do not run another automatic package sync afterward. It can replace the patched source installation with the vanilla Verl wheel.

4. Supply authorized training tasks outside the benchmark. The public sample list exercises the interface. It is not the private 187-task pool behind the reported result, and that training pool contained no ThinkingBox-Bench tasks. Use docs/data-preparation.md for task hydration, policy probes, benchmark exclusion and train/validation splitting rather than treating the sample list as the reported training data.

5. Inspect the data-preparation interface. I'd start with the released help command below.

   cd ~/thinkingbox-workspace/thinkingbox-training
   source .venv/bin/activate
   python data/prepare_data.py --help

   The preparation script aggregates policy probes, applies difficulty filters and benchmark exclusions, then writes the selected train.yaml and test.yaml files. For the input fields and loader contract, I'd use the data-preparation guide instead of trying to infer them from the output names.

I'd go to docs/reference-q38-run.md for the validated run configuration, then docs/evaluation.md for checkpoint merging and benchmark evaluation.

The benchmark does not establish production reliability

The evaluation tests transfer beyond the training domains. That's useful, but it does not establish general production reliability. Observed 20/20 reports what happened in the recorded attempts; it cannot guarantee the next attempt.

The benchmark's domains, databases, policies and most MCP tools were not encountered during RL training. Along those dimensions, the evaluation is out of distribution. There is still shared machinery: the ThinkingBox framework, simulated users, terminal-state grading, a few common tools and a similar task style. I would account for those shared pieces before applying the transfer result more broadly.

I'd retain existing approval enforcement while evaluating this checkpoint. It does not justify replacing an authorization control with model compliance.

If a team only consumes hosted agents and cannot supply authorized executable tasks or the required training infrastructure, I'd skip this training stack.

The cost table leaves out training and policy probes

The reported dollar values are a comparative serving index, not invoices or cloud bills. Model inference is included. The training run and the policy probes used to curate the dataset are both excluded.

“Cost per dependable task” divides the estimated cost of the full evaluation campaign by the number of tasks that passed every recorded attempt. It's a benchmark-level ratio. The label does not make it a quoted price for completing an arbitrary production workflow, whether that workflow is described as dependable or not.

The estimates use measured token usage. The source also says serving the fine-tuned checkpoint would be the same as serving the base model. I would not read its lower figures as a newly discounted hosting price for the fine-tuned model, rather than estimates based on that measured usage.