Both of the wins were arithmetic I had never done

Third in the series. Part one built a strong player out of a weak model and a good search. Part two shrank the model 70x, put it on Lichess, and concluded the sweet spot was near 5M parameters. This part makes the search 2.4x faster, which demolishes that conclusion in the good direction, then spends a night on cyclic learning rates, weight averaging and puzzle data, measures all three at zero, and wins the day's biggest result from the dullest thing available.

---

This week cost me two numbers, and both of them are divisions I could have done in my head at any point in the previous month.

59% is the fraction of neural network forward passes in my search that evaluated exactly one position, on a GPU that will evaluate sixteen for the same money.

2% is the fraction of one epoch that every fine-tune this project has ever run actually saw.

Between those two facts sit about a dozen ideas I was excited about. Cyclic learning-rate restarts. Weight averaging. Puzzle training. Beam retuning. fp8. A Rust rewrite. Every one of them measured zero or worse. It was a very productive week for long division.

The latency curve is the whole argument

At batch one, a forward pass on this model costs 4.4 milliseconds. At batch sixteen, a forward pass costs 4.4 milliseconds. Turns out there is free lunch.

batch	1	4	8	16	32	64	128
ms/forward	4.9	4.4	4.4	4.4	5.3	7.2	13.1
ms/row	4.9	1.1	0.55	0.28	0.17	0.11	0.10

Flat to sixteen. The model is 10.6MB, so at these sizes it spends its time on kernel launches and fixed overhead rather than arithmetic. Fifteen of those sixteen slots were being paid for and thrown away, six times out of ten. I had been paying for sixteen seats and buying one ticket at a time.

For contrast, the same bench on the 147M model from part one: 5.7 / 7.0 / 9.6 / 16.3 / 27.8 / 51.8. Compute-bound from batch eight. There are no free slots on a big model, so batching is a small-model lever specifically, which is a second reason to be small on top of the one part two found.

I did not spot this myself. It arrived as a question. The search is backloaded, one pass at the root then branching, so could the batches be spread out more evenly? The measurement that followed took twenty minutes and produced the largest single gain in the project's history, which says something uncomfortable about the eight hours I had spent worrying about weight size.

The fix is to stop asking one question at a time

Alpha-beta is depth-first and sequential by design. You need node A's value to decide whether B is worth searching at all, which is what a cutoff is. So the search dribbles out evaluation requests one at a time and the GPU sits there.

But at any given node, the search is about to evaluate the whole beam anyway. So evaluate the beam's children in one forward before descending into them, and do the same for the capture frontier in quiescence. Children that a later cutoff prunes cost nothing, because their slot was free.

	before	after
batch-1 forwards	59%	11%
mean batch	4.9	8.2
nodes at fixed time		+62%
mean depth	2.56	2.87 (+12%)

80 games against the identical model with the identical config: 42W 24L 14D, 61.3%, about +80 Elo. Same weights, same moves considered, same clock. The only thing that changed is when positions get evaluated.

Then the bottleneck moved, four times

Every fix promoted the next constraint, and the obvious next target was wrong more often than right.

Batch-1 forwards went first. Underneath sat 114,582 dtype-copy kernels, about 24 per search node, casting fp32 weights to bf16 on every forward. The weights never change. Casting once at load: +12% nodes.

Underneath that sat launch overhead: 76 CUDA kernels per node at 2.7µs each, which is more CPU time issuing work than the GPU spent doing it. CUDA graphs record a fixed shape and replay it, and since we were now deliberately padding batches, rounding to buckets (4/8/16/32/64/128) made them replayable. 76 kernels per node became 3.4. +37% nodes, +17% depth.

Underneath that, nothing. GPU execution was finally the constraint. Compounded, the search does roughly 2.4x the nodes it did two days ago at the same clock.

For every idea in that chain, several died. Widening the beam looked free now that batching had made width cheap, and the bench agreed enthusiastically at +119% nodes: −64 Elo over 60 games, because the policy's recall@20 is already 97.1% and moves ranked past twenty mostly dilute move ordering. Narrowing the beam to spend the nodes on depth bought +0.5 ply and −58 Elo, so the tuned value of 20 survived attacks from both sides and I have stopped poking it. Deeper quiescence won a five-way screen at +104 Elo over 24 games and was +12 at 84. Speculative padding did nothing at all, because I gated it at depth > 2 and the mean depth is 3.4. Top-K policy readback did nothing either: the cost was per-call synchronisation, not the 1858 floats I was shipping across to read twenty, a thing I misdiagnosed twice in a row. And the Rust rewrite to escape .NET's garbage collector died after I measured GC pause time at 0.06% of search time, making it the fastest Rust project I have ever completed.

Why fp8 did nothing, two parts late

Part two reported, with a shrug where the explanation should have been, that fp8 quantisation gained nothing. Here is the explanation. 5.29M parameters in bf16 is 10.6MB, which streams from memory in 16 microseconds, or 0.36% of a 4.4ms forward. Halving that saves 0.18%.

A 9B model is 18GB of weights and streams in about 27ms, six times its fixed overhead, so it really is memory-bound and quantisation really is the dominant lever. The crossover on this GPU is around 1.5B parameters, and we are 280x below it. The entire published playbook for LLM inference, quantize, stream fewer bytes, compress the KV cache, is aimed at the other side of a threshold we are nowhere near. Below it the lever is operations issued, not bytes moved. I had been reading the manual for somebody else's machine.

Part two's conclusion did not survive

Part two established that at equal time the ordering was 147M < 10.4M < 5.29M. Smaller won, because shrinking bought search depth, and the floor sat near 5M where the forward became all fixed overhead. That floor is a function of per-forward cost. We just cut per-forward cost by a lot.

Re-run at equal time, 40 games each, same recipe at every size:

comparison	score	Elo
10.4M vs 5.29M	65.0%	+108
20M vs 5.29M	65.0%	+108
20M vs 10.4M	51.2%	+9
5.29M vs 147M	56.2%	+43
10.4M vs 147M	53.8%	+26

The ordering is now 20M ≈ 10.4M > 5.29M > 147M. The optimum moved up at least two size classes, purely from making the forward cheaper. The mechanism is unchanged, bigger still costs depth, but bigger costs less depth than it did.

I am not claiming more than that, because three of those five comparisons are mutually inconsistent. Chaining 10.4M−5.29M (+108) with 5.29M−147M (+43) predicts +151 for 10.4M−147M, and the measured value is +26. A second chain misses by 80. Forty-game matches cannot resolve differences of this size, so the honest statement is a plateau from about 10M to 20M, not a peak. I did briefly fit a parabola through three points and announce a maximum at 25M. Three points determine a parabola exactly, so the maximum was a property of my choice of curve and nothing else.

Every Elo number in part two was inflated, again

Part two's Elo figures came from a Stockfish ladder. That ladder gave Stockfish --movetime 50 and let my model search at fixed depth with no time limit at all. Both biases push the same way: a time-starved opponent playing well below its nominal UCI_Elo, against a model thinking as long as it liked. I had scored my engine against a Stockfish that was being timed with an egg timer.

Fixed, at equal 1000ms per move for both sides, the ladder finally triangulates:

opponent	score	implied
UCI_Elo 2250	78%	≈2465
UCI_Elo 2500	68%	≈2627
UCI_Elo 2750	17%	≈2481

Three levels agreeing within 160 points across a 500-point span, which is what a calibrated ladder looks like. So: about 2500 on Stockfish's scale, and that scale is not CCRL's or Lichess's.

The number I actually trust is hardware-independent and calibration-free. Against full-strength Stockfish on a node budget: 89% at 1,000 nodes per move, 21% at 10,000, 4% at 100,000. One second of my GPU is worth roughly three to five thousand Stockfish nodes, which is, at Stockfish's throughput, about three to five milliseconds of one CPU core. A second of laptop GPU, flat out, trades evenly with about four milliseconds of Stockfish. The project was never about beating Stockfish.

The fine-tunes were sampling 2% of an epoch

With the search finally spending its time on arithmetic instead of overhead, the bottleneck moved one more time, out of the engine and into the weights. The second division was waiting there.

The fine-tune recipe had been stable for weeks: mine the model's own mistakes, mix them into the corpus, train 2000 steps at batch 512. It kept producing small gains, so I kept running it.

Then I did the arithmetic I had never done. 2000 steps × 512 samples = 1.02M positions, against a corpus of 44 million. Every fine-tune this project has ever run saw about 2% of one epoch.

It gets better. While checking that, I found an error in my own notes. A comment in the DAgger script warned that a previous run had oversampled its mined rows "~465x" and overfit. The real figure was ~58x. I had multiplied by the gradient-accumulation factor when --batch was already the effective batch. Eight times off, in a comment whose entire job was to stop future me from getting this wrong.

So the plan for the night wrote itself: run 100k steps, which is 51.2M samples, or 1.07 epochs, the first time the model would see its whole corpus once with the mined data folded in.

First, the model needed something to learn from

DAgger needs games. The harness generates them by playing the model against itself from an opening book, and there I hit something embarrassing: the match command played at temperature 0, always. Two identical checkpoints from the same opening produce the same game, move for move. My "3000 self-play games" would have been 800 distinct games and 2200 photocopies, and I would have reported the 3000 in a table with a straight face.

One flag later, sampling the policy instead of taking the argmax, seeded per game so resumes stay reproducible, and 3000 games meant 3000 games. Two temperature streams: 0.5 for play close to what the deployed argmax would actually do, 0.8 for coverage.

Then mine them two ways:

pass	what it keeps	yield
blunder	moves that threw away ≥150cp	1,017 rows (0.7% of moves)
recall	positions where Stockfish's best move is outside the policy's top-4	19,832 rows (17.6%)

The second is the number that keeps this project honest. One model move in six proposes a candidate set that does not contain the best move. Search cannot fix that. A move that never enters the tree is never searched at any depth. Everything in the first half of this post raised the ceiling on searching well, and that 17.6% is the ceiling. It is also why mining recall failures outproduces mining blunders by twenty to one.

Puzzles: a superb evaluation, a terrible teacher

Self-play DAgger has a structural blind spot. It only ever labels positions the model steers itself into, so a weakness it habitually avoids never enters the training set. Lichess publishes 6.1 million puzzles with ratings and themes attached: free ground truth, positions chosen by somebody other than me.

As an evaluation it was excellent from the first run. Accuracy fell monotonically with puzzle rating, 100% at 400 and 10% at 2400+, which is the sanity check that the position and move alignment is right, and it gave a per-theme breakdown no aggregate metric could. veryLong 34.4%, sacrifice 37.1%, quietMove 41.2%, defensiveMove 46.2%.

Then I trained on the failures. 32,224 of them, labeled with Stockfish, mixed at a third of the mined pool.

	puzzles (held-out 20k)	games vs the model it came from
before	68.5%
after	75.7%	−20 Elo (LLR −3.54)

Puzzle accuracy up 7.2 points, first-move accuracy 78.3% → 85.2%, and the thing plays measurably worse chess. A puzzle announces that a tactic exists. Train on 32k of them and the policy learns to expect one everywhere, so in quiet positions it starts hunting for a brilliancy that is not there, like a detective who has decided every death is a murder. The theme table shows it. sacrifice and mateIn4 improved sharply while quietMove moved 41.2% → 42.0% and stayed near the bottom.

Across three checkpoints that night, puzzle score was anti-correlated with playing strength. It is now an eval and nothing else.

The boring thing worked

The overnight run: 100k steps from the DAgger checkpoint, both mined rounds mirrored (139,432 rows) plus 10k puzzle rows kept deliberately small, mixed at 0.08, which is about 29 repeats per mined row and well under the ~58 that had overfit before. Fourteen hours, no crashes, once I had stopped running two GPU jobs at once on a laptop that responds to that by switching itself off mid-sentence.

480 games pooled across two clean opening suites: 58.2%, +58 Elo, SPRT LLR +5.58, H1 accepted.

That is the only decisive training result of the night, and it came from doing nothing more imaginative than letting the model read the whole book once. Combined with the DAgger round before it (+27 Elo over 720 games), roughly +85 Elo in a day, of which the clever half contributed the smaller share.

The clever half, measured

I ran the 100k steps as four cycles with a cosine cooldown each, on the theory that restarting the learning rate would knock the model out of wherever it had settled and let it re-converge somewhere better. Snapshot Ensembles, SGDR, SWA: a well-studied family. Each cooled cycle is a finished, playable checkpoint, so it also promised four models for the price of one.

Everything about it measured zero:

comparison	steps added	Elo	verdict
5 × 4k fast cycles	8k	+5	null
SWA soup of three cooled cycles	0	+1	null
20k more steps, one long cooldown	20k	+3	null

Three different shapes of "more", all indistinguishable from the checkpoint they started from.

The explanation was one small tool away. I wrote a command to report the distance between two checkpoints in weight space:

pair	relative L2	cosine
cycle1 ↔ cycle2	0.30%	0.999998
cycle2 ↔ cycle3	0.28%	0.999999
v16 ↔ cycle3 (+58 Elo apart)	0.56%	0.999988

The restarts never move the model anywhere. There is no basin to escape and no diversity for an ensemble to average. I had told myself two stories, first the textbook one about escaping local minima, then a more sophisticated one about sampling diverse points on a connected low-loss manifold, and the measurement refuted both, because both require the model to actually go somewhere.

It also explains the soup. Weight averaging works when snapshots orbit a basin: the mean lands in the middle, at a flatter point than any of them. Mine do not orbit, they march. cycle1 → cycle2 → cycle3 is a slow drift in one direction, so their average is a point behind the leading edge, and an average of a monotone trajectory cannot beat its endpoint.

The game records had been saying this all along, if I had looked. Genuinely different models drew 30.6% of their games. The cycle snapshots drew 41% against each other.

What a clean measurement costs

Two methodology corrections, both from being caught out.

The first: I built a fresh opening suite to test on, and it came from the same file used to generate the self-play games the model trained on. Testing a model on the openings whose continuations it had memorised. Caught before it reached a conclusion, but only just.

The second is worse, because it makes every small result in this project suspect. The same two models, 240 games on each of two clean, balanced suites:

suite	score	Elo
holdout-bal120	47.9%	−14
openings-120	53.5%	+25
pooled 480	50.7%	+5

A 39-Elo swing from opening selection alone. A single 240-game suite is worth roughly ±25 Elo of noise, which is larger than most of the effects this project has ever chased. Everything now pools both suites at a minimum of 480 games, and the +58 result is trustworthy partly because both suites agreed on it (+76 and +39).

That result also nearly went the other way for a stupid reason. I had declared the fast-cycle arm "destructive" on the strength of validation agreement dropping 3.1 points, and was about to bin it. Playing it out gave +5 Elo, which is no difference at all. Validation mispredicted the winner three times in one night: the DAgger model that won +27 while losing on every metric, the puzzle model that gained 7.2 points of puzzle score and lost 20 Elo, and this one. At some point I have to stop calling that a surprise.

Where this leaves it

v17 is live on Lichess, the search does 2.4x the nodes it did at the start of the week, and the model is roughly 85 Elo stronger than the checkpoint it started from.

The corpus, though, is spent. Going from 50k to 75k steps was worth about 11 Elo. From 75k to 95k, +3. More passes over the same 44M positions have stopped paying, and neither cycling nor averaging nor a longer cooldown recovers it. The recall miss rate says the same thing from the other end: two full rounds of DAgger moved it from 17.6% to about 16%. The mistakes I am mining are not the ones that matter any more.

So the next lever is new information, not more optimisation of the old. The deployed bot logs every move its search played, along with the backed-up value, and search play is a different distribution from the raw-policy self-play I have been mining for two rounds. Those are the positions deployment actually faces.

The lessons, updated

Parts one and two still stand. This part adds seven:

10. Measure the shape of your workload before optimising it. The same codebase is launch-bound at inference and compute-bound in training. Doubling GPU power moved training throughput 6%. Batching moved inference 140%. There is no way to know which regime you are in without a profiler, and the published optimisation playbook is written for the other one. Below ~1.5B parameters, weight streaming is a rounding error and quantization does nothing. What costs you is operations issued.
11. Fix one bottleneck and you have promoted the next. Batch-1 forwards, then dtype copies, then kernel launches, then the GPU itself. Four constraints in one evening, each invisible until the one in front of it was gone. Expect the list not to end, and expect the next item to be somewhere you have already looked and dismissed.
12. A benchmark that measures nodes measures nodes. Beam widening doubled node count and lost 64 Elo. Beam narrowing bought half a ply and lost 58. Validation mispredicted the winner three times in one night. Games decided every question this project has ever asked, and I keep asking the other things first because they answer faster.
13. Compute what fraction of an epoch you are training on. Every fine-tune in this project ran 2000 steps against a 44M corpus, 2% of one pass, and I never checked. One full epoch was worth +58 Elo, more than every architectural and scheduling idea of the past month combined. The arithmetic takes ten seconds, and instead I wrote an eight-times-wrong version of it into a warning comment.
14. If your intervention is supposed to move the model, measure whether it moved the model. Cyclic restarts, by the story I told myself, relocate the weights so they can re-converge somewhere better. They move them 0.3% at cosine 0.999998. And weight averaging needs snapshots that orbit, not snapshots that march: sequential cooldowns along one trajectory give you a line, and the mean of a line sits behind its endpoint. One 40-line diagnostic settled in a minute what three A/B runs had only hinted at.
15. A benchmark that announces the answer teaches the wrong lesson. Puzzles come pre-labeled with "there is a tactic here". Training on them bought 7.2 points of puzzle accuracy and cost 20 Elo, because the model learned the announcement rather than the tactic. Superb as an eval, exactly because it measures something self-play cannot. Poison as training data, for the same reason.
16. One opening suite is one sample. The same pair of models scored −14 Elo on one balanced suite and +25 on another. Anything below ~480 pooled games across independent suites is a coin flip wearing a decimal point, and half the small results in this series were probably read too confidently.

Code and run logs: github.com/Oli-26/ChessLM. Games: lichess.org/@/latentheatlm