Visual Seeding Only Moves the Words You Seed
A DeBERTa trained on 10M words picks up image-derived embeddings as a starting point, and the gain shows up only on object-property questions and only for the tokens that were seeded.
Give a small language model embeddings derived from images before training starts, and the imprint survives to the end of training. It is invisible to most BabyLM benchmarks. On the corpus-tailored Visual-Property Swap follow-up the author builds, which tests color, material, size and shape, the gain stays confined to the words that were seeded.
The paper is "Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model". The model is a DeBERTa trained on 10M words. The intervention happens before the first gradient step.
The paper implements St. Augustine's picture of word learning, meaning by ostension, as an initialization scheme. Tokens with a visual referent get embeddings computed from the image regions they label. Every other token starts at random, which is what all of them would have done by default.
In a normal run the embedding for banana is noise, and whatever the model ends up knowing about bananas it extracts from text. The paper asks whether an image-derived vector can be handed over in advance, and whether it sticks past training.
| Random init | Visual (ostensive) init | |
|---|---|---|
| Concrete nouns | noise | embedding from labelled image regions |
banana at step zero |
random vector | image-derived vector |
| Most BabyLM grammar benchmarks | baseline | no measurable change |
| COMPS, zero-shot object properties | baseline | helps in every configuration |
| Corpus-tailored Visual-Property Swap (color, material, size, shape) | baseline | persistent, seed-replicated, confined to seeded words |
| Function and abstract words, held-out mask loss | baseline | falls in every seed |
| Any benchmark registering that fall | none | none reported |
What the benchmarks did and didn't see
Most BabyLM benchmarks probe abstract grammatical knowledge, and on those the seeding does nothing measurable. That says the intervention is not making the model better at language in general.
COMPS (Misra et al. 2023) is the exception, a zero-shot test of object-property knowledge, and there seeding helps in every configuration the author ran. To follow that up, the author builds a corpus-tailored version of the Visual-Property Swap benchmark (Lin et al., 2026), which tests color, material, size and shape, and which carries per-item training frequency and seeded status. There the advantage is persistent, it replicates across seeds, and it is confined to the seeded words.
Seeding is correlational until you can move words in and out of the seeded set, so the author does. Synthetic grounding of words that were previously unseeded transfers the advantage to exactly those words. That is a clean causal result and it is also the narrowest one available. The effect moves with the word. Nothing in the abstract suggests it spreads to related vocabulary, and the finding that the advantage is "confined to the seeded words" says it does not.
Read that as an initializer rather than as learning. A seeded embedding is a better starting point for a term. That is worth something, and it is not a theory of meaning. It is closer to a lookup table installed before training than to a model that has been shown a banana.
The result nobody can score
Function words and abstract vocabulary also receive strong visual seeds and retain them throughout training, and the training objective draws on them: held-out mask-prediction loss falls for these words in every seed. No benchmark in the paper registers it. The abstract does not say what image region a function word like the could be derived from. A loss drop with no downstream movement is a gap the author names directly: the paper states that no benchmark it runs picks the effect up, and that what evaluation would pick it up remains an open question.
Why this lands for anyone building
In my own work, embedding initialization is a config line I have never touched, and the default has been fine. Here that line is the embedding table's initialization: seeded tokens start from image-derived vectors instead of random ones, on a 10M-word corpus with a DeBERTa encoder. Moving that one line onto a seeded vocabulary is the whole method, and reproducing it needs labelled image regions that line up with the tokens you care about.
Concretely: if you have those regions and a narrow vocabulary you care about, seeding those embeddings is a cheap intervention with a documented effect on object-property questions. It will not make your grammar better, and it will not spread to words you did not seed.
Held-out mask loss on function and abstract words falls in every seed while every benchmark in the paper sits flat. If you want to catch that in your own fine-tunes, log held-out loss on vocabulary your benchmarks do not cover.
What would change the picture
Everything here is one model, one objective, one corpus size. DeBERTa is an encoder with a masked objective at 10M words. Whether any of this survives at 100M words, or in a decoder-only model trained to predict the next token, is untested in the abstract. The scaling direction is not obvious either way: more data could wash the seed out, or the seed could matter less as the text alone does the work.
The other gap is where the seeds came from. If the regions were labelled by hand, the method is capped by how much of a vocabulary anyone has bothered to annotate, which is a real constraint for anyone thinking about applying it. If they were extracted automatically, the cap moves, and the abstract does not say which it is.
The method needs two things you have to have in advance: a vocabulary you already know, and labelled regions for it. A fixed 10M-word training corpus is a setting where you can plausibly have both. Open-domain training is not, and that is roughly where this initializer stops being useful.