GrowLab: Scaling Laws Assume Model Size Is Constant. What If It Grows?

A scaling law answers a narrow, useful question: given a fixed compute budget C, how many parameters N should the model have, and how many tokens D should it see? Chinchilla and its successors answer that well, and the answer shapes how essentially every modern language model is provisioned.

But look at what every one of those laws quietly assumes. N is a single number, fixed at step zero and held there until the run ends. That is not a finding — nobody ever ran the experiment that justified it. It is simply how models are built.

Now notice what compute actually costs. Each optimizer step spends roughly 6N floating-point operations per token, so the bill for a run is the integral of N over the run: training compute is the area under the parameter-count curve. If N is constant, that area is a rectangle — one shape among infinitely many with the same area.

So the question underneath this project is not “how big should the model be?” It is:

Given compute C, what shape should N(t) trace?

The intuition for why the rectangle might be the wrong shape is simple. Early in training a model is learning cheap statistics — which tokens are frequent, that a space usually follows a word — and that knowledge needs almost no capacity. Carrying 5.9M parameters through those steps is paying for a machine you are not yet able to use. Start smaller, buy more steps with the same operations, and add capacity later, when there is something worth spending it on.

The system, which is really the point

The scientific question is old-fashioned. How we attacked it is not.

GrowLab is a small FLOP-budgeted training harness. The one line of code that matters is the loop condition: it accumulates flops_per_token(model) × tokens each step and stops when the budget is exhausted, so step count is an output of the trajectory rather than an input. Budget by wall-clock and you measure how well your hardware packs small matmuls; budget by steps and you have silently handicapped the growing arm, which is smaller for part of the run and therefore spends strictly less compute. Only a FLOP budget asks the question we meant to ask.

The second design decision: the growth schedule is a parameter, not a preset. A string like 200:width:192,400:depth,600:width:256,800:depth,1000:depth means widen to 192 dimensions at step 200, add a layer at step 400, and so on. That string is a search space, and search spaces are things you can hand to an agent.

Which is exactly what happened. Two platforms were involved, doing two completely different jobs.

AutoLab was the researcher. Its control node holds an LLM agent, an objective (“minimise mean validation loss at a fixed 4×10¹⁴ FLOP budget by choosing the trajectory N(t)”), and a set of constraints. The agent writes a schedule into grow/experiment.py, commits it, and queues the job. My laptop — attached as an execution node — picks it up, trains three seeds, and prints one line: OBJECTIVE 4.07450. The agent reads that float, marks the experiment merged or discarded, and writes the next hypothesis. It ran eight experiments autonomously; after the first, no human chose what to try. It reproduced our hand-built schedule exactly, beat it by shifting growth 50 steps earlier, then stopped itself with the status “remaining variation is seed noise.” An agent that declines to keep optimising below the noise floor is doing the right thing.

What AutoLab does not do: it supplies no compute — the laptop is the only silicon in the project — and it never sees a loss curve, a growth event, or a trajectory. Its entire view of a seven-minute run is one number on the last line of stdout.

Maritime was the front door. It builds the repo’s Dockerfile into a serverless micro-VM and serves the results dashboard on a public URL. That is the whole job, and it is the reason the demo survives a laptop lid closing or a conference-room wifi outage. What Maritime does not do: no training, no GPU, no LLM, no AutoLab credentials, no knowledge that an experiment exists.

The structurally important fact is that the two platforms never talk to each other. They are not two halves of a pipeline; they are independent consumers of the same repository, and the only thing crossing between them is a single JSON file the laptop writes. If Maritime is down the science continues. If the agent has finished or the login expired, the dashboard still renders.

What we measured, and what happened when we checked it

The control is a straight A/B. One arm starts at 6 layers, 256 dimensions (5.9M parameters) and stays there; the other starts at 3 layers, 128 dimensions (1.2M parameters) and grows into the identical final architecture across five events. Both get the same FLOP budget, the same WikiText-103 token stream in the same order, the same fixed evaluation slice, and three seeds.

A gap of 0.397 nats with complete seed separation: the worst grown seed (4.172) still beats the best flat seed (4.496), no overlap. The mechanism is visible in the last column — being small early buys 19% more optimizer steps for the same operations.

That is the number we report now. It is not the number we had at first. Our first headline was a 0.913-nat gap and “102× more stable”. Both were artifacts of our own design, found by spending the last hours of the build attacking our result instead of polishing it.

The first artifact was the learning rate. Both arms ran at 1e-3, a value inherited from the growing preset. The grown arm begins at 3 layers and 128 dimensions, where 1e-3 is near optimal; the flat arm is 6 layers and 256 dimensions from step zero, where 1e-3 is roughly twice its stable limit. So the “controlled” learning rate was one arm’s setting imposed on the other. Given its own best rate, 5e-4, the flat arm scores 4.509 ± 0.015 rather than 5.024 ± 0.532 — the gap halves, and the stability claim does not merely shrink, it reverses: at each arm’s own learning rate the flat arm is about 3.6× more stable than the grown one. What we had reported as “growth buys learning-rate robustness” was one flat seed half-diverging at a rate that model could not take.

The second artifact was worse, because it was baked into the question. Both arms were pinned to finish at 5.9M parameters — same finish line, fair race. But for a 4×10¹⁴ FLOP budget compute-optimal is around 0.6M parameters, so the mandated endpoint is roughly ten times larger than the budget wants. Lift the constraint and run a plain flat 1.2M model at the same budget: it scores 3.905, beating our growth trajectory outright, because at that size the same FLOPs buy 11,811 steps instead of 2,908. (A later per-arm learning-rate sweep put the same model at 3.775 ± 0.008, which only widens the margin.)

So the claim that survives is real but much narrower than the one we started the day with:

At a fixed FLOP budget, with both arms required to end at 5.9M parameters and each arm at its own best learning rate, growing into a target size beats starting at that size — 0.397 nats, with complete seed separation.

The conditions it is hostage to belong in the same breath. It is conditional on an endpoint about ten times larger than compute-optimal. It is 5.9M parameters, roughly 2,900 steps, one laptop. It is three seeds against a kernel-nondeterminism floor of about 0.05 nats on Apple Silicon, which means our four-decimal figures carry about two significant figures of real signal. Growth is not shown to be compute-optimal in general — only to be the better way to arrive at an oversized model.

There is a sting in that. The endpoint constraint was enforced in code: any trajectory not ending at exactly 5.9M raises SystemExit before its loss is ever reported. So the configuration that actually wins was structurally unreachable by the agent. However well it searched, it could not have found it. We handed our researcher a search space that excluded the best known answer.

The lesson, which is the most valuable thing here

Both errors are the same species. The instinct to hold everything constant and vary one thing is correct as a default, but fixing a variable only controls it when that variable’s optimum does not interact with the treatment. When the optimum moves with the treatment, fixing it does not neutralise the variable — it silently picks a winner.

Optimal learning rate depends on model size, and our two arms were different sizes for most of training. Sharing a learning rate between them was therefore not a control; it was a handicap applied to whichever arm the shared value fit worse. Fixing the endpoint was the same mistake one level up: the question was what shape should N(t) trace, and pinning the terminal value of N(t) assumed part of the answer before measuring it.

The practical rule: for any claim of the form “X beats Y,” tune each arm independently and compare best to best. Anything else measures “X beats Y at Y’s bad settings.” Safe to fix are the things whose optimum genuinely does not move — dataset, data order, evaluation set, compute budget, seed count. Tuned per arm: learning rate, warmup length, probably batch size.

What is still open

Does growth ever beat the compute-optimal flat model, or only oversized endpoints? This decides whether there is a general result at all. Settling it means sweeping the endpoint and asking whether the grown curve ever dips beneath the flat frontier or merely approaches it from above. Our one probe is inconclusive: a trajectory ending at 2.7M scores 3.787 ± 0.018 against 3.911 ± 0.032 for a flat 2.7M model — a real win at a matched endpoint — but it only ties the best flat model overall (3.775 ± 0.008), inside the noise floor.

Does any of it survive at real scale? Chinchilla predicts where growth should fail: at roughly 20 tokens per parameter, a grown model reaches its final size having seen far fewer than 20·N tokens at that size, so it is structurally undertrained for what it has become. Whether time spent smaller substitutes for those missing tokens is unknown.

Should growth be triggered rather than scheduled? Our step numbers have no justification beyond “they worked,” and the agent’s most productive move was shifting them 50 steps earlier — so timing matters and we did not know the right timing. But “step 200” is not a reason to grow. A model should grow when its current size stops being the thing limiting it: a condition, not a clock. Turning that into a measurable trigger would make growth a policy rather than a hyperparameter you must search.

Beyond those: should the model choose width versus depth for itself, and what about sparsity — growing into a larger but sparser model, so parameter count rises faster than FLOP count? And one a practitioner asks first: at matched FLOPs the grown arm is 6 to 11% slower in wall-clock, because small matmuls underutilise the device. The operations are genuinely saved; the seconds are not. Probably a scale artifact, but we have not shown where the wall-clock curve crosses, and reporting only FLOPs would be choosing the metric that flatters us.

Built at Sundai Hack #133. The claim we can defend is smaller than the one we announced at hour six, and the project is more interesting for it.

Code: github.com/qsimeon/growlab · The agent’s search: app.autolab.ai/projects/qsimeon/growlab · Live dashboard: api.maritime.sh