Recent Posts
How Much Data Does a Transformer Need to Learn Repetition?
We systematically degrade the repetition signal in the training data — token by token, and row by row — and find a critical threshold below which induction heads cease to form. Even 10% of tokens in repeated sequences is enough.
Do Transformers Memorise or Generalise? The Illusion of Induction
We probe whether the repetition capability of our toy transformer reflects genuine generalisation or memorisation of the training distribution. A single-token experiment reveals an 'illusion of induction' in deeper models — a cautionary finding for evaluations of larger LLMs.
Token Distribution Drives Repetition Learning
We surgically replace the tokens inside repeated sequences with random tokens, while keeping the sequence structure fixed.
Is Natural Language Special for Learning Repetition?
We reverse all tokens in the Pile dataset and find that a transformer trained on completely unnatural data still learns to repeat sequences — suggesting linguistic structure is not required for induction head formation.
Repetition is surprisingly ubiquitous in tokenized natural language
55% of tokens in the tokenized Pile dataset are part of repeated sequences, defined as either A or B in ...AB...AB, and we characterise the structure of those repetitions in detail.
An introduction to our investigation into repetition capability in toy transformer models
Why we want to study repetition in toy transformer models and what we aim to investigate