Dispatch 01 · November 2026

Can we compressbackpropagationinto a model?

Most of us can dive into a new board game just by skimming the rules, or whip up dinner after glancing at a recipe. But for language models, things aren’t so simple. A prompt can carry a new task, but only for as long as it stays in the context; to really make the task stick, they often require large amounts of data and multiple rounds of fine-tuning. That got us thinking: what if a model could just read a short description or a couple of examples once, and instantly update itself, like fine-tuning, but without all the waiting? So we gave it a shot. The results? kinda surprising! Our model improved its score on 59 out of 75 public benchmarks with just a one-paragraph update, which took only a few seconds on a single small GPU and did not require a training run. On one benchmark, the score jumped from 50.0 to 91.1 percent. Given the same 16 labeled examples, our update beat a LoRA trained on them on 24 of the benchmarks and lost on 14. There’s still a lot to figure out, but the idea of turning words straight into working solutions is just too exciting to pass up.

Task description, as the meta-model reads itmath
meta-modelweight update
rows: the 12 projection types of the basecolumns: layer 1 to 24
update size per weight matrixsmall → large
1 / 4step through four held-out tasks
Measured. How much the meta-update changes each weight matrix of Qwen3.5-2B, on four held-out tasks.no gradient step

Can a model learn the update that backpropagation finds?

Here is the idea in one paragraph. We trained a meta-model: a dedicated network, far smaller than any frontier model, built for one job, which is to read a description of a task and write the weight change that makes a base language model (Qwen3.5-2B throughout this post) do it. The underlying concept is straightforward. The meta-model interprets the task, identifies the tendencies of the base model that get in the way, such as a preferred option, a preferred label, or an extended response, and adjusts the weights to mitigate them. That turns out to be enough on a lot of tasks: on 75 public benchmarks, the meta-update written from a one-paragraph description raises the score of the base model on 59 and lowers it on 8, with the biggest jumps on BIG-Bench Hard and on commonsense sentence pairs (ComVE, 50.0 to 91.1 percent). Given the same 16 labeled rows, the meta-update is ahead of a LoRA trained on them on 24 of the 75 benchmarks and behind on 14. The rest of this post explains why we think such a map should exist at all, how we trained it, and where it works and where it doesn't yet.

On November 2, Takehi et al. had a frontier model read the weights of Qwen3.5-0.8B and change 131,072 of them by hand, about 0.016 percent, all in two MLP output projections. MATH-500 went from 7.9 to 48.1. The edit didn't teach the model mathematics. It stopped a loop: 93 percent of the base model's answers had run into the token limit without a final answer, and after the edit 33 percent did. A small, well-placed change in weight space moved the model a long way.

We have been circling a neighboring question. Backpropagation takes a loss and a dataset and adjusts the weights over thousands of steps. Can a model learn that whole mapping, so that a new task costs one short run of a meta-model, a second network that reads the task and writes the weight update, and no gradient step at all? Some pieces of an answer exist: a transformer can run gradient descent inside its own forward pass on simple problems (von Oswald et al., 2023); HyperTuning wrote adapters for a frozen T5 from a few examples (Phang et al., 2023); Text-to-LoRA generates it from a task description (Charakorn et al., 2025). Takehi’s team got to the weights by reading them directly. Our path was a bit different: we trained a meta-model on thousands of tasks, each matched with the adapter that gradient descent discovered for it, and then ran those changes through the base model. Cost is the other big difference. Their edit needs a frontier model to read the weights and try edits round after round. Our meta-model is a dedicated model, trained for this one job and far smaller than any frontier model; once it is trained, writing the meta-update for a new task is one short run that takes a few seconds on a single small GPU, with no gradient step and no frontier model in the loop. Figure 1 shows how the three approaches line up.

Three ways to turn a task into a weight update
Select a route to see how it works.

Figure 1. With gradient descent, the model slowly learns by making lots of small changes, one step at a time, using the task data. In weight-space programming (Takehi et al., 2026), a cutting-edge model takes a close look at its own weights and tries out edits, but only gets a handful of chances. The third approach uses a learned map: a meta-model quickly writes the meta-update in a single go, no need for gradient steps on the base model, but you do need to train that meta-model first. Once it is trained, each new meta-update takes seconds on one small GPU.

Seventy-five public benchmarks

The setup

The procedure was this: for each public benchmark we wrote a single paragraph describing the task and gave it to the meta-model. The meta-update it produced was applied to the base model, and that model was evaluated on the benchmark's official test split. We compared it against three other conditions: the base model with the same paragraph in its prompt as an instruction; the base model with as many training examples as would fit in its prompt (between 47 and 84); and the base model with a meta-update that the meta-model produced from 100 training examples. The same decoding and answer rules were applied in all four cases, and any test items that were present in the training data were stripped out. 75 benchmarks across 7 task types were used.

Across the seventy-five

Compared with the base model given the same description in its prompt, the meta-update raises the score on 59 of the 75 benchmarks and lowers it on 8; on the remaining 8 it lands within a few points either way. On 44 of the 59 the gain is large enough to clear a 95% confidence interval (see Figure 2). The improvements are the following: 11 of the 18 multiple choice sets improve and 1 declines, 8 of the 12 classification sets improve and 2 decline, and 10 of the 18 BIG-Bench Hard tasks improve and none decline (every BIG-Bench Hard task is in Table 4 in the appendix). For the 3 object logical deduction task performance goes from 46.4 to 82.0. In the ComVE sentence pair task where the model is asked to pick which of the two sentences goes against common sense the accuracy goes from 50.0 to 91.1%. Four of the BLiMP paradigms go from 70.1 to 98.1. Among the short answer tasks DROP improves while NQ-open and WebQuestions decline; the rest barely move. The losses are in one of the structured output tasks (named entities as JSON), with further losses seen in SciTail, the ambiguous questions in BBQ, ToxiGen, TruthfulQA, Banking77, NQ-open and WebQuestions.

Seventy-five public benchmarks, by category
measured · official splits · November 2026

Figure 2. 75 public benchmarks across 7 categories: the meta-update written from the description (minus the base model with the description as its instruction), with 95% intervals. Black: better. Light: the interval contains zero. Open: worse. Grey names: a training task was constructed from that dataset. The prediction for each category is on the right hand side; hover over a point to see all four arms.

BenchmarkItemsBase + descriptionMeta-update from descriptionDescription minus base [95% CI]Examples in promptMeta-update from examples
OpenBookQA49571.577.0+5.5 [+1.4, +9.7]75.674.5
ARC-Easy2,33890.991.4+0.5 [-0.7, +1.7]92.990.8
ARC-Challenge1,16177.982.1+4.2 [+1.9, +6.6]82.981.1
CommonsenseQA1,21963.370.5+7.1 [+4.5, +9.9]70.469.2
WinoGrande1,26750.557.4+6.9 [+2.8, +10.9]54.657.5
ComVE99050.091.1+41.1 [+37.7, +44.4]51.291.4
BLiMP, 4 held-out paradigms4,00070.198.1+28.0 [+26.6, +29.4]60.397.1

Table 1. Seven of the benchmarks, in percent, after the contamination scan: the base model with the description as its instruction against the meta-update produced from the description, with the difference and its 95% interval. The two example-based arms, examples in the prompt and the meta-update from 100 examples, are at right for reference.

A paragraph against a training run

How many labeled rows is a paragraph worth? For each benchmark we also trained a LoRA directly on the first 16, 64, 100 and, where the benchmark has them, 800 of the same rows the meta-update reads, with the recipe that produces the adapters the meta-model learns from in Section 5 (the epoch count per row budget was fixed before the run, with three seeds at 16 and 64 rows). Given the same 16 rows, the meta-update is ahead of the trained adapter on 24 of the 75 benchmarks and behind on 14 (Table 2), and the meta-update written from the description alone is ahead on 29 and behind on 15. At 64 rows the training run has caught up, 12 ahead to 27 behind, and from 100 rows on it pulls away, 68.5 against 62.2 on average. The two stay close even where the training run leads: at 64 rows the gap is 3.8 points on average, and at 100 rows the meta-update and the adapter are within 5 points of each other on 38 of the 75 benchmarks. The larger gaps sit in a handful of sets where there is a label set or an output format to learn from the rows, led by the ambiguous questions in BBQ and named entities as JSON.

RowsLoRA trained on the rowsMeta-update from the rowsMeta-update from rows vs LoRA (better / level / worse)
1661.562.224 / 37 / 14
6466.062.312 / 36 / 27
10068.562.212 / 29 / 34
800 (47 benchmarks)76.067.94 / 8 / 35

Table 2. A LoRA trained on the first k rows of each benchmark's training set, against the meta-update written from the same rows: mean accuracy over the 75 benchmarks (47 at 800 rows, the ones with an 800-row set), and the counts better / level / worse by the paired 95% interval. The LoRA at 16 and 64 rows is the mean of three seeds.

How many labeled rows does a training run need to beat the paragraph?
measured · a LoRA on the same rows · November 2026

Figure 3. For each benchmark, the smallest number of training rows at which a LoRA trained on them beats the meta-update written from the description (interval above 0). The last column holds the benchmarks where no training run on up to 800 rows did. Colors give the task type.

Math

In this experimental setup, the meta-model was prompted to generate its meta-update without incorporating any reasoning. To ensure a fair comparison, all conditions were evaluated without reasoning. All models received the instruction "the final number only, with no working" (16 token limit) and were provided with answer-only examples. Across the five datasets, the meta-update outperformed the base model: GSM8K increased from 5.7 to 18.6, SVAMP from 45.7 to 54.0, MultiArith from 6.3 to 54.9, ASDiv from 48.7 to 69.6, and MATH-500 from 4.0 to 14.3 (all with confidence intervals above zero). The meta-update also surpassed the prompt-based examples on all five datasets (see Table 3). Although the absolute scores remain low, which is expected for a multi-step reasoning task, the base model performs significantly better when permitted to show its working (for example, 70.9 on GSM8K).

BenchmarkBase + descriptionExamples in promptLoRA, 16 rowsLoRA, 100 rowsLoRA, up to 1,400 rowsMeta-update from examplesMeta-update from descriptionDescription minus base [95% CI]Description minus 1,400-row LoRA [95% CI]
GSM8K5.77.05.56.712.721.718.6+12.9 [+10.6, +15.3]+5.9 [+3.6, +8.2]
SVAMP45.741.346.453.161.048.054.0+8.3 [+2.3, +14.3]-7.0 [-13.3, -0.7]
MultiArith6.37.47.835.172.074.954.9+48.6 [+40.6, +56.0]-17.1 [-25.7, -9.1]
ASDiv48.753.651.462.869.066.369.6+20.9 [+18.0, +23.9]+0.6 [-1.9, +3.2]
MATH-5004.08.46.811.311.013.314.3+10.3 [+7.2, +13.5]+3.4 [+0.0, +6.5]

Table 3. Results for five mathematics datasets with working steps omitted (in percent); each condition reports only the final number. The LoRA columns are adapters trained on the first 16 and 100 of the rows the meta-update reads (mean of seeds) and on up to 1,400 rows (trained to a plateau). The last two columns present the difference between the meta-update as written from the description and the base model, and between the meta-update and the 1,400-row LoRA, each with its 95% confidence interval.

When reasoning is reintroduced, the meta-update does not help. On MATH-500, the meta-update written from the description lowers the base model's score from 22.8 to 10.5 percent (95% interval -16.5 to -8.0); on the same benchmark, Takehi et al.'s hand edit resulted in improvement. The meta-update still provides only the final number where the base model works the problem out. For GSM8K, the meta-update written from 100 examples does not surpass the base model with reasoning enabled, 69.3 compared to 70.9. StrategyQA is level, 57.6 versus 58.9.

What the meta-update changes

One effect of the meta-update, for instance, can be seen on benchmarks with a fixed set of options by looking at the distribution of choices made by each condition (see Table 1 for the seven benchmarks). Take ComVE as an example: the base model picks "second" in 984 of the 990 cases and with the examples in the prompt it picks "second" in 964 cases. The meta-update, by contrast, spreads out its choices more evenly, picking "first" 471 times and "second" 519 times (91.1% accuracy). On WinoGrande the base model picks the same position on 99.8% of the items, and with the meta-update that falls to 50.7%. For the 3 object logical deduction task the base model picks "C" in 213 of 250 cases whereas the meta-update spreads its choices out. The base model clearly has a strong positional bias and the meta-update removes that. This result is very much as described by Takehi et al. in their hand edit where they broke a loop that was just repeating itself without adding any new functionality.

What the meta-update changed
measured · every reply counted · November 2026

Figure 4. How often each arm gives its most picked answer on three benchmarks where the base model has a position bias and on the held out task families with a fixed label set. Open circle: base model with the description as its instruction. Grey: with examples in the prompt. Black: with the written meta-update. Lower is better: a model that gives the same answer on every item sits at 100, a balanced one at 50. Accuracy of the base and the meta-update at right.

The failures we see are due to the same fundamental mechanism: a model which can remove a learned response can also learn to introduce a new one. On SciTail the meta-update picks 'yes' for 74% of the items (the correct proportion is 39%). On ToxiGen the base model responses are almost perfectly balanced, but the meta-update picks 'no' for 780 of the 939 items. In the shuffled object tracking task we see the positional bias shift from 'C' to 'A' with no clear improvement in accuracy (the task is one that requires multi-step reasoning). So both removing and introducing bias are just different manifestations of the same process in opposite directions. That's what the reward stage of Section 5 is trying to achieve by directing the meta-model.

The underlying concept is straightforward. A paragraph that describes a task does not introduce new information about the world. Therefore, the model cannot acquire such information through a prompt, fine-tuning, or a meta-model. Instead, the task can only require the model to utilize knowledge already present in the base model. This process appears to be what the meta-model is learning: it interprets the task, identifies base model tendencies that interfere with performance (such as a preferred option, label, or extended response), and adjusts the weights to mitigate these tendencies. Fine-tuning on the same task achieves a similar effect (Section 4). If the discrepancy between the base model and the task is habitual, weight adjustments typically resolve it, sometimes substantially. However, if the discrepancy involves knowledge absent from the model or an output style not demonstrated in the paragraph, the model will still lack these attributes. This presents the next challenge to address.

Why write weights and not code?

Before we dive into the results, let’s talk about why this idea even matters. We already have models that can write code from a description (Chen et al., 2021), and code is something we can read, test, and fix. But if you take a step back, you’ll see a bigger pattern: with each generation of computing, people end up writing less and letting tools do more of the heavy lifting. Once, we wrote machine code by hand. Then compilers took over, turning high-level languages into machine instructions. Karpathy (2017) called the next leap Software 2.0: where the dataset is the source code, and gradient descent is the compiler filling in the weights. Models that turn a description into code are one step along this path. The next step, and a longstanding dream, is a model that goes straight from description to the weights themselves. This isn’t a new wish: Schmidhuber (1992) trained a network to produce weight changes for another network. Hypernetworks did it for whole layers (Ha et al., 2017), learned optimizers swapped hand-crafted update rules for trained networks (Andrychowicz et al., 2016), and generative models of checkpoints spit out parameters from a prompted loss (Peebles et al., 2022) or even just noise (Wang et al., 2024). In program-learning research, there’s a classic distinction: synthesis means a network writes a program, induction means it becomes one (Devlin et al., 2017). Writing the weight update is the induction route: the “program” is merged into the base model and runs with all its abilities, with no extra cost at inference time.

Weights themselves have some quirks that code just doesn’t. They’re continuous, so you can nudge them in small ways, and updates for related tasks can even add together or form analogies (Ilharco et al., 2023). Even cooler, a single mapping can write lots of different weights. Sutton’s “bitter lesson” (2019) reminds us that, in the long run, broad methods that learn from data tend to beat hand-crafted solutions. So, if you have a learned mapping from a task description to weights, that’s basically a smarter, learned version of a compiler. And the next step, having a map that writes both the architecture and the weights, has already started to take shape with work like graph hypernetworks (Knyazev et al., 2021). Figure 5 shows all these levels at a glance.

Each level moved the specification up
hover a step for the reference

Figure 5. There are six levels to how much work a human has to do when specifying a task. With each new level, people hand off more of the job to the tool. The top row shows what a person writes; the bottom row shows what the tool produces. That last step? It’s still an open question.

Why would such a map exist?

Before we can ask if a model can learn the map from a task to its weight update, it helps to ask if there’s actually something simple enough to learn. As it turns out, there might be. When you fine-tune a pretrained language model, the update usually lives in a tiny corner of its huge weight space. For example, Aghajanyan et al. (2021) tweaked 200 random directions in a RoBERTa model and got 90 percent of the way to full fine-tuning on a paraphrase task. They linked a task’s “intrinsic dimension” to how long its description is within the pretrained model, building on Li et al. (2018). Updates for related tasks behave like vectors: they add and subtract (Ilharco et al., 2023). Even alignment tuning, though it seems like a big shift, mostly changes the next-token prediction in a few places, and usually just on stylistic tokens. The underlying knowledge was already in the base model (Zhou et al., 2023; Lin et al., 2024). So if a fine-tune is that compact and regular, a one-paragraph description could, in theory, capture it. A map from paragraph to update could have something small and learnable to work with. The results we share in this post say much the same thing from another angle: the meta-update that worked didn’t add new knowledge, it just fixed a bias.

In fact, something like this mapping already happens inside every language model. In-context learning, the way a model picks up a task just from examples in the prompt, has been described as the model doing a sort of implicit gradient step during its forward pass. Attention can act like a stand-in for gradient descent on those examples (Dai et al., 2023; von Oswald et al., 2023), and small transformers trained on regression problems have been shown to carry out actual learning algorithms (Akyürek et al., 2023). So, a model already maps a specification to a temporary tweak in how it thinks. If a meta-model could write that change directly into the weights, you’d make this mapping explicit, lasting, and easy to share. That’s the idea powering fast-weight programmers, from Schmidhuber (1992) to Schlag et al. (2021).

Brains have their own way of handling new tasks. Think about how you can follow verbal instructions and do something new on your first try, no practice runs needed. Neuroscientists call this rapid instructed task learning. Your brain quickly pieces together a solution from things you already know (Cole et al., 2013). There’s a famous study with monkeys: after lots of training, they could tackle new problems on their very first attempt, they’d basically learned how to learn (Harlow, 1949). Some researchers think this happens in the prefrontal cortex, where slow chemical changes set the stage, and then the brain can quickly adapt to whatever comes next, no rewiring required (Wang et al., 2018). Even at the level of brain circuits, there are cool examples. In crabs, a single set of neurons can do lots of different things, just by changing the brain’s chemistry, the wiring stays the same, but different signals bring out new behaviors (Marder, 2012). In a way, a frozen model with a small update is like a fixed circuit that suddenly acts differently with a new modulator. These examples aren’t a perfect match for how meta-models work, and they don’t prove that mapping language to weights will always succeed. But they do show that nature has figured out clever ways to set up systems that can switch gears fast when needed.

Training Procedure

The straightforward approach is nearly effective, and its limitations provide valuable insights. Each training task is paired with the corresponding adapter gradient descent, and the meta-model is trained to reproduce that adapter. While this approach provides a useful target for the meta-model, it is insufficient in isolation. This limitation arises because multiple updates can yield identical behavior on a given task, and optimizing toward a single adapter may cause the meta-model to converge on an arbitrary solution.

A significant improvement was observed when the meta-model was trained to optimize the performance of the base model, rather than to replicate the adapter. The meta-update is applied to the frozen base model, which is then evaluated on the task-specific data. The resulting loss propagates back through the base model into the meta-model. In a third approach, this loss is replaced by the task's intrinsic reward, such as exact-match accuracy for mathematical tasks, test results for code, or a verifier when available. Since the reward is not differentiable, it is optimized directly as a reward signal. Figure 6 illustrates this process. The order in which these three signals contributed to performance improvements is based on empirical observations from training runs, rather than controlled ablation studies.

Train the meta-model through the base model

Figure 6. The meta-model generates a meta-update ΔW, which is applied to the frozen base model. The training signal then propagates back through the base model into the meta-model. Selecting a signal reveals its entry point in the process.

Trying it yourself

We plan to launch a hosted demo of the meta-model, allowing you to submit a description or examples and receive the meta-updated base model within seconds. In the meantime, if you would like to discuss this work or suggest a use case, please contact us.

Appendix: BIG-Bench Hard, task by task

BIG-Bench Hard is where the meta-update moves the base model the most outside the sentence pairs, so here is every one of its 18 tasks with all four arms. The meta-update written from the description is above the base model on 10 of the 18 and below it on none; the biggest jumps are temporal sequences, the logical deduction tasks, geometric shapes and date understanding. The tasks it does not clearly move include tracking shuffled objects, which needs a chain of steps, snarks, sports understanding and ruin names.

TaskItemsBase + descriptionExamples in promptMeta-update from examplesMeta-update from descriptionDescription minus base [95% CI]
temporal sequences25047.640.0100.0100.0+52.4 [+46.0, +58.4]
logical deduction, 3 objects25046.445.681.282.0+35.6 [+27.6, +43.6]
geometric shapes2509.645.629.244.4+34.8 [+26.8, +42.4]
date understanding25038.844.474.867.2+28.4 [+21.2, +36.0]
reasoning about colored objects25048.050.878.874.8+26.8 [+20.0, +34.0]
logical deduction, 7 objects25027.633.252.052.0+24.4 [+17.2, +31.6]
logical deduction five objects25033.638.457.657.2+23.6 [+16.0, +30.8]
movie recommendation21139.383.452.155.5+16.1 [+9.0, +23.2]
hyperbaton25057.651.272.873.6+16.0 [+8.0, +24.0]
penguins in a table14636.336.342.544.5+8.2 [-1.4, +17.1]
salient translation error detection25045.241.251.252.8+7.6 [+2.0, +13.2]
tracking shuffled objects, 521918.713.718.323.7+5.0 [-3.2, +13.2]
snarks17852.253.954.556.2+3.9 [-3.9, +11.8]
disambiguation qa20846.656.247.149.5+2.9 [-2.9, +8.7]
tracking shuffled objects, 721215.17.515.617.0+1.9 [-5.2, +9.4]
sports understanding25050.847.649.651.6+0.8 [-6.0, +7.2]
tracking shuffled objects three objects23432.932.530.833.3+0.4 [-7.7, +8.1]
ruin names24640.237.447.237.0-3.3 [-10.2, +3.7]

Table 4. The 18 BIG-Bench Hard tasks, in percent. Base + description is the base model with the task description as its instruction; examples in prompt is the base model with 100 training examples in its prompt. Last column: the meta-update written from the description minus the base model, with its 95% interval.

Appendix: the task descriptions

The meta-model was provided with concise task descriptions. For the broad sweep and short-answer math sets, each benchmark was described in a single sentence (14 to 31 words) specifying the input and the required answer format. For the seven benchmarks in the initial run, each description comprised three brief sections detailing the task, the input, and the output (81 to 98 words). No additional information was included: there were no examples, label statistics, or indications of common base model errors. Table 5 presents expanded versions of nine of them. For BIG-Bench Hard, each sentence identifies the task with minimal detail. All seventy-five descriptions are available in the data files.

BenchmarkFormThe description
ComVEfirst run, three sectionsTask: Given two statements that differ in a few words, decide which one goes against common sense: the one describing something that could not happen, would not be done, or contradicts everyday knowledge of how the world works. The answer is "first" or "second".

Input: Fixed instruction lines, a blank line, then the two statements on separate lines. They are short, share most of their wording, and differ in the element that makes one of them nonsensical.

Output: One line holding exactly one of the labels "first" or "second", naming the statement that makes less sense. No explanation and no other text.
OpenBookQAfirst run, three sectionsTask: Answer an elementary-science multiple-choice question. Each question applies a core science fact (about materials, energy, living things, the Earth or the sky) to a new situation, so the right option follows from combining that fact with everyday reasoning rather than from recalling a passage.

Input: A question, then four options labeled A, B, C and D on their own lines, a blank line, and the fixed instruction "Answer with the letter." Some questions are incomplete sentences that the option completes.

Output: A single line with one uppercase letter, the label of the chosen option, and nothing else.
MMLU (1,000 seeded items)paragraphEach input is a multiple-choice exam question from one of 57 subjects spanning the humanities, the social sciences, the natural sciences, mathematics, medicine, law and professional fields, at levels from high school to graduate and professional exams. The question is followed by its options, each labeled with a letter. The model answers with the letter of the correct option alone, with no explanation and no restatement of the option.
SciTailparagraphEach input is a premise and a hypothesis drawn from science questions and their answers. The model decides whether the premise entails the hypothesis, that is, whether someone who accepts the premise must also accept the hypothesis: if so the answer is yes, and if the premise is neutral toward or contradicts the hypothesis the answer is no. The reply is the single word yes or no, with nothing else.
SQuAD v1.1paragraphEach input is a passage of a few sentences from an encyclopedia article followed by a question whose answer is stated in the passage as a contiguous span of words. The model answers with that span alone, copied as it appears in the passage: a name, a date, a number or a short phrase, with no sentence around it, no explanation, and no words that are not in the passage.
GSM8Kparagraph, working allowedEach input is a grade-school math word problem that takes several steps of arithmetic to solve, with quantities, rates and simple relations stated in everyday language. The model works through the problem, showing the intermediate calculations, and ends its reply with the final numerical answer.
GSM8Kparagraph, final number onlyEach input is a grade-school math word problem in everyday language, solved by a few steps of arithmetic on the quantities it states. The model works it out internally and replies with the final number only, on a single line: no steps, no units, no currency sign, no explanation.
MATH-500paragraph, final number onlyEach input is a competition mathematics problem (algebra, counting and probability, geometry, number theory, precalculus or intermediate algebra) written in LaTeX. The model replies with the final answer only, on one line, in the form the problem calls for: an integer, a fraction, a radical or a short LaTeX expression, simplified, with no working, no explanation and no surrounding sentence.
CoNLL-2003 NER as JSONparagraphEach input is one sentence of English news text. The model finds every named entity in it and sorts them into four types: PER for people, ORG for organizations (companies, institutions, teams, governments), LOC for geographic and political places, and MISC for other proper names such as nationalities, events, languages and titles of works. It replies with a single JSON object whose keys are PER, ORG, LOC and MISC, each holding a list of the entity strings exactly as they appear in the sentence (an empty list when there are none), and nothing else.

Table 5. Nine task descriptions, one per kind of task, in expanded form. The first-run descriptions follow a three-section template. For the other benchmarks, the runs in this post used a single manually written sentence per benchmark; the versions shown here restate those sentences at greater length, and the sentences themselves are in the data files.

References

  1. Takehi, R., Teoh, J., Zhang, A., Khattab, O. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. Blog post, MIT CSAIL, November 2, 2026. rikiyat.github.io/blog/weight-space-programming
  2. Phang, J., Mao, Y., He, P., Chen, W. HyperTuning: Toward Adapting Large Language Models without Back-propagation. ICML 2023, PMLR 202:27854-27875. arXiv:2211.12485.
  3. Charakorn, R., Cetin, E., Tang, Y., Lange, R. T. Text-to-LoRA: Instant Transformer Adaption. ICML 2025, PMLR 267. arXiv:2506.06105.
  4. Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., Farhadi, A. Editing Models with Task Arithmetic. ICLR 2023. arXiv:2212.04089.
  5. von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., Vladymyrov, M. Transformers learn in-context by gradient descent. ICML 2023, PMLR 202:35151-35174. arXiv:2212.07677.
  6. Ha, D., Dai, A., Le, Q. V. HyperNetworks. ICLR 2017. arXiv:1609.09106.
  7. Karpathy, A. Software 2.0. Blog post, November 2017. karpathy.medium.com/software-2-0-a64152b37c35
  8. Chen, M. et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021.
  9. Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A., Kohli, P. RobustFill: Neural Program Learning under Noisy I/O. ICML 2017, PMLR 70:990-998. arXiv:1703.07469.
  10. Schmidhuber, J. Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation 4(1):131-139, 1992.
  11. Andrychowicz, M., Denil, M., Gómez Colmenarejo, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., de Freitas, N. Learning to learn by gradient descent by gradient descent. NeurIPS 2016. arXiv:1606.04474.
  12. Peebles, W., Radosavovic, I., Brooks, T., Efros, A. A., Malik, J. Learning to Learn with Generative Models of Neural Network Checkpoints. arXiv:2209.12892, 2022.
  13. Wang, K. et al. Neural Network Diffusion. arXiv:2402.13144, 2024.
  14. Knyazev, B., Drozdzal, M., Taylor, G. W., Romero Soriano, A. Parameter Prediction for Unseen Deep Architectures. NeurIPS 2021. arXiv:2110.13100.
  15. Sutton, R. S. The Bitter Lesson. Blog post, March 2019. incompleteideas.net/IncIdeas/BitterLesson.html
  16. Aghajanyan, A., Gupta, S., Zettlemoyer, L. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL-IJCNLP 2021, pages 7319-7328. arXiv:2012.13255.
  17. Li, C., Farkhoor, H., Liu, R., Yosinski, J. Measuring the Intrinsic Dimension of Objective Landscapes. ICLR 2018. arXiv:1804.08838.
  18. Zhou, C. et al. LIMA: Less Is More for Alignment. NeurIPS 2023. arXiv:2305.11206.
  19. Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., Choi, Y. The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. ICLR 2024. arXiv:2312.01552.
  20. Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., Wei, F. Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers. Findings of ACL 2023, pages 4005-4019.
  21. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., Zhou, D. What Learning Algorithm Is In-Context Learning? Investigations with Linear Models. ICLR 2023. arXiv:2211.15661.
  22. Schlag, I., Irie, K., Schmidhuber, J. Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021, PMLR 139:9355-9366. arXiv:2102.11174.
  23. Cole, M. W., Laurent, P., Stocco, A. Rapid instructed task learning: A new window into the human brain's unique capacity for flexible cognitive control. Cognitive, Affective, & Behavioral Neuroscience 13(1):1-22, 2013.
  24. Harlow, H. F. The formation of learning sets. Psychological Review 56(1):51-65, 1949.
  25. Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., Botvinick, M. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience 21(6):860-868, 2018.
  26. Marder, E. Neuromodulation of neuronal circuits: back to the future. Neuron 76(1):1-11, 2012.

How to cite this post

@misc{cohen2026compress,
  title  = {Can we compress backpropagation into a model?},
  author = {Cohen, Roi and Shapira, Ben},
  year   = {2026},
  month  = {November},
  url    = {https://roicohen.org/blog/compress-backpropagation/}
}