Can a model learn the update that backpropagation finds?
Here is the idea in one paragraph. We trained a meta-model: a dedicated network, far smaller than any frontier model, built for one job, which is to read a description of a task and write the weight change that makes a base language model (Qwen3.5-2B throughout this post) do it. The underlying concept is straightforward. The meta-model interprets the task, identifies the tendencies of the base model that get in the way, such as a preferred option, a preferred label, or an extended response, and adjusts the weights to mitigate them. That turns out to be enough on a lot of tasks: on 75 public benchmarks, the meta-update written from a one-paragraph description raises the score of the base model on 59 and lowers it on 8, with the biggest jumps on BIG-Bench Hard and on commonsense sentence pairs (ComVE, 50.0 to 91.1 percent). Given the same 16 labeled rows, the meta-update is ahead of a LoRA trained on them on 24 of the 75 benchmarks and behind on 14. The rest of this post explains why we think such a map should exist at all, how we trained it, and where it works and where it doesn't yet.
On November 2, Takehi et al. had a frontier model read the weights of Qwen3.5-0.8B and change 131,072 of them by hand, about 0.016 percent, all in two MLP output projections. MATH-500 went from 7.9 to 48.1. The edit didn't teach the model mathematics. It stopped a loop: 93 percent of the base model's answers had run into the token limit without a final answer, and after the edit 33 percent did. A small, well-placed change in weight space moved the model a long way.
We have been circling a neighboring question. Backpropagation takes a loss and a dataset and adjusts the weights over thousands of steps. Can a model learn that whole mapping, so that a new task costs one short run of a meta-model, a second network that reads the task and writes the weight update, and no gradient step at all? Some pieces of an answer exist: a transformer can run gradient descent inside its own forward pass on simple problems (von Oswald et al., 2023); HyperTuning wrote adapters for a frozen T5 from a few examples (Phang et al., 2023); Text-to-LoRA generates it from a task description (Charakorn et al., 2025). Takehi’s team got to the weights by reading them directly. Our path was a bit different: we trained a meta-model on thousands of tasks, each matched with the adapter that gradient descent discovered for it, and then ran those changes through the base model. Cost is the other big difference. Their edit needs a frontier model to read the weights and try edits round after round. Our meta-model is a dedicated model, trained for this one job and far smaller than any frontier model; once it is trained, writing the meta-update for a new task is one short run that takes a few seconds on a single small GPU, with no gradient step and no frontier model in the loop. Figure 1 shows how the three approaches line up.
Figure 1. With gradient descent, the model slowly learns by making lots of small changes, one step at a time, using the task data. In weight-space programming (Takehi et al., 2026), a cutting-edge model takes a close look at its own weights and tries out edits, but only gets a handful of chances. The third approach uses a learned map: a meta-model quickly writes the meta-update in a single go, no need for gradient steps on the base model, but you do need to train that meta-model first. Once it is trained, each new meta-update takes seconds on one small GPU.
Seventy-five public benchmarks
The setup
The procedure was this: for each public benchmark we wrote a single paragraph describing the task and gave it to the meta-model. The meta-update it produced was applied to the base model, and that model was evaluated on the benchmark's official test split. We compared it against three other conditions: the base model with the same paragraph in its prompt as an instruction; the base model with as many training examples as would fit in its prompt (between 47 and 84); and the base model with a meta-update that the meta-model produced from 100 training examples. The same decoding and answer rules were applied in all four cases, and any test items that were present in the training data were stripped out. 75 benchmarks across 7 task types were used.
Across the seventy-five
Compared with the base model given the same description in its prompt, the meta-update raises the score on 59 of the 75 benchmarks and lowers it on 8; on the remaining 8 it lands within a few points either way. On 44 of the 59 the gain is large enough to clear a 95% confidence interval (see Figure 2). The improvements are the following: 11 of the 18 multiple choice sets improve and 1 declines, 8 of the 12 classification sets improve and 2 decline, and 10 of the 18 BIG-Bench Hard tasks improve and none decline (every BIG-Bench Hard task is in Table 4 in the appendix). For the 3 object logical deduction task performance goes from 46.4 to 82.0. In the ComVE sentence pair task where the model is asked to pick which of the two sentences goes against common sense the accuracy goes from 50.0 to 91.1%. Four of the BLiMP paradigms go from 70.1 to 98.1. Among the short answer tasks DROP improves while NQ-open and WebQuestions decline; the rest barely move. The losses are in one of the structured output tasks (named entities as JSON), with further losses seen in SciTail, the ambiguous questions in BBQ, ToxiGen, TruthfulQA, Banking77, NQ-open and WebQuestions.
Figure 2. 75 public benchmarks across 7 categories: the meta-update written from the description (minus the base model with the description as its instruction), with 95% intervals. Black: better. Light: the interval contains zero. Open: worse. Grey names: a training task was constructed from that dataset. The prediction for each category is on the right hand side; hover over a point to see all four arms.
| Benchmark | Items | Base + description | Meta-update from description | Description minus base [95% CI] | Examples in prompt | Meta-update from examples |
|---|---|---|---|---|---|---|
| OpenBookQA | 495 | 71.5 | 77.0 | +5.5 [+1.4, +9.7] | 75.6 | 74.5 |
| ARC-Easy | 2,338 | 90.9 | 91.4 | +0.5 [-0.7, +1.7] | 92.9 | 90.8 |
| ARC-Challenge | 1,161 | 77.9 | 82.1 | +4.2 [+1.9, +6.6] | 82.9 | 81.1 |
| CommonsenseQA | 1,219 | 63.3 | 70.5 | +7.1 [+4.5, +9.9] | 70.4 | 69.2 |
| WinoGrande | 1,267 | 50.5 | 57.4 | +6.9 [+2.8, +10.9] | 54.6 | 57.5 |
| ComVE | 990 | 50.0 | 91.1 | +41.1 [+37.7, +44.4] | 51.2 | 91.4 |
| BLiMP, 4 held-out paradigms | 4,000 | 70.1 | 98.1 | +28.0 [+26.6, +29.4] | 60.3 | 97.1 |
Table 1. Seven of the benchmarks, in percent, after the contamination scan: the base model with the description as its instruction against the meta-update produced from the description, with the difference and its 95% interval. The two example-based arms, examples in the prompt and the meta-update from 100 examples, are at right for reference.
A paragraph against a training run
How many labeled rows is a paragraph worth? For each benchmark we also trained a LoRA directly on the first 16, 64, 100 and, where the benchmark has them, 800 of the same rows the meta-update reads, with the recipe that produces the adapters the meta-model learns from in Section 5 (the epoch count per row budget was fixed before the run, with three seeds at 16 and 64 rows). Given the same 16 rows, the meta-update is ahead of the trained adapter on 24 of the 75 benchmarks and behind on 14 (Table 2), and the meta-update written from the description alone is ahead on 29 and behind on 15. At 64 rows the training run has caught up, 12 ahead to 27 behind, and from 100 rows on it pulls away, 68.5 against 62.2 on average. The two stay close even where the training run leads: at 64 rows the gap is 3.8 points on average, and at 100 rows the meta-update and the adapter are within 5 points of each other on 38 of the 75 benchmarks. The larger gaps sit in a handful of sets where there is a label set or an output format to learn from the rows, led by the ambiguous questions in BBQ and named entities as JSON.
| Rows | LoRA trained on the rows | Meta-update from the rows | Meta-update from rows vs LoRA (better / level / worse) |
|---|---|---|---|
| 16 | 61.5 | 62.2 | 24 / 37 / 14 |
| 64 | 66.0 | 62.3 | 12 / 36 / 27 |
| 100 | 68.5 | 62.2 | 12 / 29 / 34 |
| 800 (47 benchmarks) | 76.0 | 67.9 | 4 / 8 / 35 |
Table 2. A LoRA trained on the first k rows of each benchmark's training set, against the meta-update written from the same rows: mean accuracy over the 75 benchmarks (47 at 800 rows, the ones with an 800-row set), and the counts better / level / worse by the paired 95% interval. The LoRA at 16 and 64 rows is the mean of three seeds.
Figure 3. For each benchmark, the smallest number of training rows at which a LoRA trained on them beats the meta-update written from the description (interval above 0). The last column holds the benchmarks where no training run on up to 800 rows did. Colors give the task type.
Math
In this experimental setup, the meta-model was prompted to generate its meta-update without incorporating any reasoning. To ensure a fair comparison, all conditions were evaluated without reasoning. All models received the instruction "the final number only, with no working" (16 token limit) and were provided with answer-only examples. Across the five datasets, the meta-update outperformed the base model: GSM8K increased from 5.7 to 18.6, SVAMP from 45.7 to 54.0, MultiArith from 6.3 to 54.9, ASDiv from 48.7 to 69.6, and MATH-500 from 4.0 to 14.3 (all with confidence intervals above zero). The meta-update also surpassed the prompt-based examples on all five datasets (see Table 3). Although the absolute scores remain low, which is expected for a multi-step reasoning task, the base model performs significantly better when permitted to show its working (for example, 70.9 on GSM8K).
| Benchmark | Base + description | Examples in prompt | LoRA, 16 rows | LoRA, 100 rows | LoRA, up to 1,400 rows | Meta-update from examples | Meta-update from description | Description minus base [95% CI] | Description minus 1,400-row LoRA [95% CI] |
|---|---|---|---|---|---|---|---|---|---|
| GSM8K | 5.7 | 7.0 | 5.5 | 6.7 | 12.7 | 21.7 | 18.6 | +12.9 [+10.6, +15.3] | +5.9 [+3.6, +8.2] |
| SVAMP | 45.7 | 41.3 | 46.4 | 53.1 | 61.0 | 48.0 | 54.0 | +8.3 [+2.3, +14.3] | -7.0 [-13.3, -0.7] |
| MultiArith | 6.3 | 7.4 | 7.8 | 35.1 | 72.0 | 74.9 | 54.9 | +48.6 [+40.6, +56.0] | -17.1 [-25.7, -9.1] |
| ASDiv | 48.7 | 53.6 | 51.4 | 62.8 | 69.0 | 66.3 | 69.6 | +20.9 [+18.0, +23.9] | +0.6 [-1.9, +3.2] |
| MATH-500 | 4.0 | 8.4 | 6.8 | 11.3 | 11.0 | 13.3 | 14.3 | +10.3 [+7.2, +13.5] | +3.4 [+0.0, +6.5] |
Table 3. Results for five mathematics datasets with working steps omitted (in percent); each condition reports only the final number. The LoRA columns are adapters trained on the first 16 and 100 of the rows the meta-update reads (mean of seeds) and on up to 1,400 rows (trained to a plateau). The last two columns present the difference between the meta-update as written from the description and the base model, and between the meta-update and the 1,400-row LoRA, each with its 95% confidence interval.
When reasoning is reintroduced, the meta-update does not help. On MATH-500, the meta-update written from the description lowers the base model's score from 22.8 to 10.5 percent (95% interval -16.5 to -8.0); on the same benchmark, Takehi et al.'s hand edit resulted in improvement. The meta-update still provides only the final number where the base model works the problem out. For GSM8K, the meta-update written from 100 examples does not surpass the base model with reasoning enabled, 69.3 compared to 70.9. StrategyQA is level, 57.6 versus 58.9.
What the meta-update changes
One effect of the meta-update, for instance, can be seen on benchmarks with a fixed set of options by looking at the distribution of choices made by each condition (see Table 1 for the seven benchmarks). Take ComVE as an example: the base model picks "second" in 984 of the 990 cases and with the examples in the prompt it picks "second" in 964 cases. The meta-update, by contrast, spreads out its choices more evenly, picking "first" 471 times and "second" 519 times (91.1% accuracy). On WinoGrande the base model picks the same position on 99.8% of the items, and with the meta-update that falls to 50.7%. For the 3 object logical deduction task the base model picks "C" in 213 of 250 cases whereas the meta-update spreads its choices out. The base model clearly has a strong positional bias and the meta-update removes that. This result is very much as described by Takehi et al. in their hand edit where they broke a loop that was just repeating itself without adding any new functionality.
Figure 4. How often each arm gives its most picked answer on three benchmarks where the base model has a position bias and on the held out task families with a fixed label set. Open circle: base model with the description as its instruction. Grey: with examples in the prompt. Black: with the written meta-update. Lower is better: a model that gives the same answer on every item sits at 100, a balanced one at 50. Accuracy of the base and the meta-update at right.
The failures we see are due to the same fundamental mechanism: a model which can remove a learned response can also learn to introduce a new one. On SciTail the meta-update picks 'yes' for 74% of the items (the correct proportion is 39%). On ToxiGen the base model responses are almost perfectly balanced, but the meta-update picks 'no' for 780 of the 939 items. In the shuffled object tracking task we see the positional bias shift from 'C' to 'A' with no clear improvement in accuracy (the task is one that requires multi-step reasoning). So both removing and introducing bias are just different manifestations of the same process in opposite directions. That's what the reward stage of Section 5 is trying to achieve by directing the meta-model.
The underlying concept is straightforward. A paragraph that describes a task does not introduce new information about the world. Therefore, the model cannot acquire such information through a prompt, fine-tuning, or a meta-model. Instead, the task can only require the model to utilize knowledge already present in the base model. This process appears to be what the meta-model is learning: it interprets the task, identifies base model tendencies that interfere with performance (such as a preferred option, label, or extended response), and adjusts the weights to mitigate these tendencies. Fine-tuning on the same task achieves a similar effect (Section 4). If the discrepancy between the base model and the task is habitual, weight adjustments typically resolve it, sometimes substantially. However, if the discrepancy involves knowledge absent from the model or an output style not demonstrated in the paragraph, the model will still lack these attributes. This presents the next challenge to address.
Why write weights and not code?
Before we dive into the results, let’s talk about why this idea even matters. We already have models that can write code from a description (Chen et al., 2021), and code is something we can read, test, and fix. But if you take a step back, you’ll see a bigger pattern: with each generation of computing, people end up writing less and letting tools do more of the heavy lifting. Once, we wrote machine code by hand. Then compilers took over, turning high-level languages into machine instructions. Karpathy (2017) called the next leap Software 2.0: where the dataset is the source code, and gradient descent is the compiler filling in the weights. Models that turn a description into code are one step along this path. The next step, and a longstanding dream, is a model that goes straight from description to the weights themselves. This isn’t a new wish: Schmidhuber (1992) trained a network to produce weight changes for another network. Hypernetworks did it for whole layers (Ha et al., 2017), learned optimizers swapped hand-crafted update rules for trained networks (Andrychowicz et al., 2016), and generative models of checkpoints spit out parameters from a prompted loss (Peebles et al., 2022) or even just noise (Wang et al., 2024). In program-learning research, there’s a classic distinction: synthesis means a network writes a program, induction means it becomes one (Devlin et al., 2017). Writing the weight update is the induction route: the “program” is merged into the base model and runs with all its abilities, with no extra cost at inference time.
Weights themselves have some quirks that code just doesn’t. They’re continuous, so you can nudge them in small ways, and updates for related tasks can even add together or form analogies (Ilharco et al., 2023). Even cooler, a single mapping can write lots of different weights. Sutton’s “bitter lesson” (2019) reminds us that, in the long run, broad methods that learn from data tend to beat hand-crafted solutions. So, if you have a learned mapping from a task description to weights, that’s basically a smarter, learned version of a compiler. And the next step, having a map that writes both the architecture and the weights, has already started to take shape with work like graph hypernetworks (Knyazev et al., 2021). Figure 5 shows all these levels at a glance.
Figure 5. There are six levels to how much work a human has to do when specifying a task. With each new level, people hand off more of the job to the tool. The top row shows what a person writes; the bottom row shows what the tool produces. That last step? It’s still an open question.
Why would such a map exist?
Before we can ask if a model can learn the map from a task to its weight update, it helps to ask if there’s actually something simple enough to learn. As it turns out, there might be. When you fine-tune a pretrained language model, the update usually lives in a tiny corner of its huge weight space. For example, Aghajanyan et al. (2021) tweaked 200 random directions in a RoBERTa model and got 90 percent of the way to full fine-tuning on a paraphrase task. They linked a task’s “intrinsic dimension” to how long its description is within the pretrained model, building on Li et al. (2018). Updates for related tasks behave like vectors: they add and subtract (Ilharco et al., 2023). Even alignment tuning, though it seems like a big shift, mostly changes the next-token prediction in a few places, and usually just on stylistic tokens. The underlying knowledge was already in the base model (Zhou et al., 2023; Lin et al., 2024). So if a fine-tune is that compact and regular, a one-paragraph description could, in theory, capture it. A map from paragraph to update could have something small and learnable to work with. The results we share in this post say much the same thing from another angle: the meta-update that worked didn’t add new knowledge, it just fixed a bias.
In fact, something like this mapping already happens inside every language model. In-context learning, the way a model picks up a task just from examples in the prompt, has been described as the model doing a sort of implicit gradient step during its forward pass. Attention can act like a stand-in for gradient descent on those examples (Dai et al., 2023; von Oswald et al., 2023), and small transformers trained on regression problems have been shown to carry out actual learning algorithms (Akyürek et al., 2023). So, a model already maps a specification to a temporary tweak in how it thinks. If a meta-model could write that change directly into the weights, you’d make this mapping explicit, lasting, and easy to share. That’s the idea powering fast-weight programmers, from Schmidhuber (1992) to Schlag et al. (2021).
Brains have their own way of handling new tasks. Think about how you can follow verbal instructions and do something new on your first try, no practice runs needed. Neuroscientists call this rapid instructed task learning. Your brain quickly pieces together a solution from things you already know (Cole et al., 2013). There’s a famous study with monkeys: after lots of training, they could tackle new problems on their very first attempt, they’d basically learned how to learn (Harlow, 1949). Some researchers think this happens in the prefrontal cortex, where slow chemical changes set the stage, and then the brain can quickly adapt to whatever comes next, no rewiring required (Wang et al., 2018). Even at the level of brain circuits, there are cool examples. In crabs, a single set of neurons can do lots of different things, just by changing the brain’s chemistry, the wiring stays the same, but different signals bring out new behaviors (Marder, 2012). In a way, a frozen model with a small update is like a fixed circuit that suddenly acts differently with a new modulator. These examples aren’t a perfect match for how meta-models work, and they don’t prove that mapping language to weights will always succeed. But they do show that nature has figured out clever ways to set up systems that can switch gears fast when needed.
Training Procedure
The straightforward approach is nearly effective, and its limitations provide valuable insights. Each training task is paired with the corresponding adapter gradient descent, and the meta-model is trained to reproduce that adapter. While this approach provides a useful target for the meta-model, it is insufficient in isolation. This limitation arises because multiple updates can yield identical behavior on a given task, and optimizing toward a single adapter may cause the meta-model to converge on an arbitrary solution.
A significant improvement was observed when the meta-model was trained to optimize the performance of the base model, rather than to replicate the adapter. The meta-update is applied to the frozen base model, which is then evaluated on the task-specific data. The resulting loss propagates back through the base model into the meta-model. In a third approach, this loss is replaced by the task's intrinsic reward, such as exact-match accuracy for mathematical tasks, test results for code, or a verifier when available. Since the reward is not differentiable, it is optimized directly as a reward signal. Figure 6 illustrates this process. The order in which these three signals contributed to performance improvements is based on empirical observations from training runs, rather than controlled ablation studies.
Figure 6. The meta-model generates a meta-update ΔW, which is applied to the frozen base model. The training signal then propagates back through the base model into the meta-model. Selecting a signal reveals its entry point in the process.