Writing

GrowLab: Scaling Laws Assume Model Size Is Constant. What If It Grows?

A scaling law answers a narrow and useful question. Given a fixed compute budget C, how many parameters N should a model have, and how many tokens D should it see? Chinchilla and the laws that followed answer this well, and their answer shapes how nearly every modern language model is sized.

Every one of those laws also assumes that N is a single number, set at step zero and held until the run ends. Networks have been grown during training before, but scaling-law work still treats N as fixed, and growth schedules are rarely compared with the best fixed size at matched compute.

Compute itself allows other choices. Each optimizer step costs roughly 6N floating-point operations per token, so the total cost of a run is the integral of N over training. In other words, training compute is the area under the parameter-count curve. A constant N makes that area a rectangle, which is only one of many shapes with the same area.

So this project asks a different question:

Given compute C, what shape should N(t) trace?

The two shaded regions are the same size. Under the 6N approximation the areas of these two real trajectories agree to within 0.001%: both runs spent 4×10¹⁴ FLOPs. Staying small early buys the growing run 2,908 optimizer steps instead of 2,433.

The intuition for why a rectangle might be the wrong shape is simple. Early in training, a model is learning cheap statistics, such as which tokens are frequent and that a space usually follows a word. That knowledge needs almost no capacity, so carrying 5.9M parameters through those steps pays for a machine the model cannot use yet. A better plan may be to start small, spend the saved operations on more steps, and add capacity later, once there is something worth spending it on.

The harness and the agent

GrowLab is a small training harness with a FLOP budget. The line of code that matters most is the loop condition. Each step adds flops_per_token(model) × tokens to a running total, and training stops when the total reaches the budget. The number of steps is therefore an output of the trajectory. A wall-clock budget would measure how well the hardware handles small matrix multiplications. A step budget would quietly handicap the growing arm, because that arm is smaller for part of the run and so spends less compute. A FLOP budget is the only one that asks the question we meant to ask.

The second design decision is that the growth schedule is a parameter that can be searched. A string like 200:width:192,400:depth,600:width:256,800:depth,1000:depth means widen to 192 dimensions at step 200, add a layer at step 400, and so on. That string defines a search space, and a search space can be handed to an agent.

We did exactly that, using two platforms with two separate jobs.

AutoLab was the researcher. Its control node holds an LLM agent, an objective (“minimise mean validation loss at a fixed 4×10¹⁴ FLOP budget by choosing the trajectory N(t)”), and a set of constraints. The agent writes a schedule into grow/experiment.py, commits it, and queues the job. My laptop, attached as an execution node, picks up the job, trains three seeds, and prints one line: OBJECTIVE 4.07450. The agent reads that number, marks the experiment merged or discarded, and writes its next hypothesis. It ran eight experiments on its own. After the first one, no person chose what to try. It reproduced our hand-built schedule exactly and then beat it by moving growth 50 steps earlier. Then it stopped itself with the status “remaining variation is seed noise.” Stopping at the noise floor was the right call.

AutoLab supplies no compute, since the laptop is the only hardware in the project. It also never sees a loss curve, a growth event or a trajectory. Its whole view of a seven-minute run is one number on the last line of the output.

Maritime was the front door. It built the repository’s Dockerfile into a serverless micro-VM and served the results dashboard at a public URL during the hackathon. A static copy now lives on this site at /growlab/. That was its only job. Because the dashboard ran there, the demo kept working when the laptop lid closed or the conference Wi-Fi dropped. Maritime did no training and held no GPU, no LLM, no AutoLab credentials and no record of the experiments.

The most important structural fact is that the two platforms never talk to each other. Each one reads the same repository on its own. The only thing that passes between them is a single JSON file that the laptop writes. If Maritime is down, the science continues. If the agent has finished or its login has expired, the dashboard still renders.

Three machines do three jobs. A cloud agent decides what to run, a laptop runs it, and a micro-VM shows the result to strangers. The science runs without the third machine, and the demo runs without the first.

Results

The control is a straight A/B comparison. One arm starts at 6 layers and 256 dimensions (5.9M parameters) and stays there. The other starts at 3 layers and 128 dimensions (1.2M parameters) and grows into the same final architecture over five growth events. Both arms get the same FLOP budget, the same WikiText-103 tokens in the same order, the same fixed evaluation slice, and three seeds.

arm	val loss	seeds	steps bought
flat (constant N)	4.509 ± 0.015	4.525, 4.496, 4.506	2,433
grown (1.2M → 5.9M)	4.111 ± 0.053	4.080, 4.172, 4.082	2,908

The grown arm wins by 0.397 nats, and the seeds separate completely: the worst grown seed (4.172) still beats the best flat seed (4.496). The last column shows the mechanism. Staying small early buys 19% more optimizer steps for the same number of operations.

Checking our own result

That is the number we report now, but our first headline was different. At first we claimed a 0.913-nat gap and “102× more stable” training. Both came from flaws in our own design, and we found them by spending the last hours of the build attacking the result instead of polishing it.

The first flaw was the learning rate. Both arms ran at 1e-3, a value carried over from the growing preset. That rate is close to optimal for the grown arm, which begins at 3 layers and 128 dimensions. The flat arm has 6 layers and 256 dimensions from step zero, and for that model 1e-3 is about twice the stable limit. Our “controlled” learning rate was really one arm’s setting imposed on the other. At its own best rate of 5e-4, the flat arm scores 4.509 ± 0.015 instead of 5.024 ± 0.532. That halves the gap and reverses the stability claim. When each arm uses its own learning rate, the flat arm is about 3.6× more stable than the grown one. The learning-rate robustness we had credited to growth was one flat seed half-diverging at a rate that model could not handle.

The second flaw was worse, because it was built into the question. We required both arms to finish at 5.9M parameters so that they would run the same race. For a 4×10¹⁴ FLOP budget, though, the best fixed size is far smaller. Of the flat sizes we trained, 1.2M did best, and a rough estimate puts the optimum near 0.6M, a size we have not trained yet. So the required endpoint is about five times the best size we trained, and roughly ten times that estimate. When we removed that constraint and trained a plain flat 1.2M model on the same budget, it scored 3.905 (one seed, LR 1e-3) and beat our growth trajectory outright. At that size the same FLOPs buy 11,811 steps instead of 2,908. (A later learning-rate sweep for each arm put the same model at 3.775 ± 0.008, which widens the margin.)

Growth beats the flat model at the same 5.9M endpoint (orange below blue). Both lose to a flat 1.2M model that was never told where to finish (teal). Curves are the in-run evaluations logged every 50 steps over 20 batches. The quoted final numbers use a 50-batch evaluation and run about 0.05 higher. Seed 0 shown for each arm.

The claim that survives is real, but it is much narrower than the one we started the day with:

At a fixed FLOP budget, with both arms required to end at 5.9M parameters and each arm at its own best learning rate, growing into a target size beats starting at that size by 0.397 nats, with complete seed separation.

That claim comes with conditions. It depends on an endpoint about five times the best fixed size we trained, and roughly ten times a rough estimate of the optimum. It covers 5.9M parameters, roughly 2,900 steps and one laptop. It rests on three seeds, against a floor of about 0.05 nats of kernel nondeterminism on Apple Silicon, so our four-decimal figures carry only about two significant figures of real signal. We have shown that growth is the better way to arrive at an oversized model. We have not shown that growth is compute-optimal in general.

The endpoint constraint also hurt the agent. Our code enforced it: any trajectory that did not end at exactly 5.9M raised SystemExit before its loss was reported. The configuration that actually wins was therefore structurally unreachable by the agent. We had handed our researcher a search space that excluded the best known answer, so even a perfect search would have missed it.

The lesson

Both mistakes have the same cause. Holding everything constant and varying one thing is a good default, but fixing a variable only controls for it when that variable’s optimum does not depend on the treatment. When the optimum moves with the treatment, fixing the variable silently picks a winner.

The optimal learning rate depends on model size, and our two arms were different sizes for most of training. A shared learning rate handicapped whichever arm it fit worse. Fixing the endpoint was the same mistake one level up. The question was what shape should N(t) trace, and pinning the final value of N(t) assumed part of the answer before we measured it.

The practical rule is this: for any claim of the form “X beats Y,” tune each arm independently and compare best to best. Any other comparison measures whether X beats Y at Y’s bad settings. It is safe to fix the things whose optimum really does not move: the dataset, data order, evaluation set, compute budget and number of seeds. The learning rate, the warmup length and probably the batch size should be tuned for each arm.

Open questions

Does growth ever beat the compute-optimal flat model, or only oversized endpoints? The answer decides whether there is a general result at all. Settling it means sweeping the endpoint to see whether the grown curve ever drops below the flat frontier or only approaches it from above. Our one probe is inconclusive. A trajectory ending at 2.7M scores 3.787 ± 0.018, against 3.911 ± 0.032 for a flat 2.7M model, which is a real win at a matched endpoint. It only ties the best flat model overall (3.775 ± 0.008), within the noise floor.

Does any of it survive at real scale? Chinchilla predicts where growth should fail. At roughly 20 tokens per parameter, a grown model reaches its final size having seen far fewer than 20·N tokens at that size, so it is undertrained for the model it has become. We do not know whether the time it spent smaller makes up for those missing tokens.

Should a condition trigger growth instead of a schedule? Our step numbers have no justification beyond the fact that they worked. The agent’s most productive move was shifting them 50 steps earlier, which shows that timing matters and that we did not know the right timing. “Step 200” gives no reason to grow. A model should grow when its current size becomes the thing limiting it. Turning that idea into a measurable trigger would make growth a policy, so nobody would have to search for it as a hyperparameter.

A few more questions follow from these. Should the model choose between width and depth for itself? What about sparsity, growing into a larger but sparser model so that the parameter count rises faster than the FLOP count? A practitioner will also ask about speed. At matched FLOPs the grown arm is 6 to 11% slower in wall-clock time, because small matrix multiplications underuse the device. Growth saves operations but not seconds at this scale. That is probably an artifact of the small scale, but we have not shown where the wall-clock curves cross, and reporting only FLOPs would mean choosing the metric that flatters us.

We built GrowLab at Sundai Hack #133. The claim we can defend is smaller than the one we announced at hour six, which makes the project more interesting.

Code: github.com/qsimeon/growlab · Dashboard with the final results and the agent’s eight experiments: qsimeon.github.io/growlab