Specification Oracles: Can a Model Become a Living Specification?

A specification has to answer questions. What can this system do? How do its parts relate? What happens under a particular set of conditions? Write too little, and the answers are missing. Write everything down separately, and the document can become filled with redundancy and even contradiction.

Our first research paper, Specification Oracles, asks whether a language model could serve as a living specification: learning facts about a target, then answering questions about it directly.

Two ways to remember

We compare two ways of giving a model the knowledge it needs. A note-sheet oracle stores facts in an external text file, which the model reads when answering questions. A weight-only oracle stores what it learns in trained adapter weights, leaving the base model frozen.

The question is how each approach responds to structure. If many facts follow from a smaller set of relationships, can an oracle take advantage of that regularity? Or does it still need to remember each fact individually?

To investigate, we use generated worlds containing relationships among rooms, regions, items, and factions. The experiments span four families of 596-fact worlds and two Qwen2.5-Instruct model sizes (7B and 14B), with different budgets for notes and adapter storage.

What we found

The weight-only oracles benefited more from structured worlds. With the 7B model, accuracy integrated across storage capacities was 18.5 percentage points higher on structured than unstructured worlds, compared with 1.1 points for note-sheet oracles.

That advantage came with a substantial storage cost. The smallest adapter used approximately 175 KiB, while the largest note budget was 16 KiB. Learned weights made better use of latent structure in these experiments; external notes used substantially less storage specific to each world.

These results concern controlled, generated worlds. They give us a way to study how the format of a specification affects what it can capture, and at what cost.

Further work on specification

This paper begins an ongoing research series from Diffusion. We are interested in how systems acquire, represent, and use the knowledge needed to build software. Specification is one place to start: before a system can do the right thing, it needs a useful account of what the right thing is.

Read Specification Oracles for the methods, experiments, and limitations.