# Talkie Talkie (formally **talkie-1930-13b**) is a 13-billion-parameter [[Large Language Model]] trained exclusively on 260 billion tokens of English text published before January 1, 1931. Released in April 2026 by Nick Levine, David Duvenaud, and Alec Radford — the latter a co-creator of the GPT architecture at [[OpenAI]] — the model is the largest "vintage language model" built to date. Both checkpoints ship under the Apache 2.0 license on [[Hugging Face]], making Talkie a notable entry in the [[Open Source AI]] ecosystem. ## Why pre-1931 The December 31, 1930 cutoff is not arbitrary — it tracks U.S. copyright law. Works published in the United States in 1930 entered the public domain on January 1, 2026, making every token in Talkie's training corpus legally free to use, distribute, and train on without license. As copyright lawsuits pile up against labs like [[OpenAI]] and Meta for training on copyrighted web text, Talkie sidesteps the entire [[Intellectual Property]] question by design. The project thus doubles as a proof of concept that consequential [[Large Language Model]] research can be conducted without modern copyrighted material. ## Architecture and training Talkie uses a standard [[Transformer]] architecture at the 13B-parameter scale. The team released two checkpoints: **talkie-1930-13b-base** (raw pretrained model) and **talkie-1930-13b-it** (instruction-tuned for conversation). A parallel "modern twin," **talkie-web-13b-base**, was trained on FineWeb with identical architecture and compute budget, allowing controlled apples-to-apples comparisons between historical and modern training distributions. ### Training corpus The 260B-token corpus draws from books, newspapers, periodicals, scientific journals, patents, and case law, sourced primarily from the Harvard Law School Library's Institutional Data Initiative (IDI) — nearly one million public-domain volumes digitized from Harvard Library collections — and the [[Internet Archive]]. All training data sources, OCR pipelines, and filtering methods are open-sourced. ### Post-training pipeline Standard instruction-tuning datasets would inject modern knowledge and style, defeating the purpose. The team instead built a bespoke pipeline from scratch using pre-1931 reference works — etiquette manuals, encyclopedias, letter-writing guides, cookbooks, poetry collections — to generate instruction-response pairs. They then ran online direct preference optimization (DPO) using [[Claude Sonnet]] as a judge, followed by a final round of supervised [[Fine-Tuning]] on rejection-sampled multi-turn synthetic chats generated between [[Claude Opus]] and Talkie. The team acknowledges this introduces some anachronistic behavioral shaping through [[RLHF]]-style feedback, and plans to eventually use vintage base models themselves as judges for a fully bootstrapped, era-appropriate pipeline. ## Research significance ### Contamination-free benchmarking Because Talkie has never seen modern data, it is contamination-free by construction with respect to any modern [[Benchmark]]. This is a persistent problem in AI evaluation — models trained on web data may have memorized test questions rather than genuinely reasoning about them. Talkie provides a uniquely clean testbed for generalization experiments. ### Can a model from 1930 write Python? One headline experiment asked whether a model with no knowledge of digital computers could learn to write Python code from in-context examples alone. Using OpenAI's HumanEval test, Talkie dramatically underperformed web-trained models but did produce correct solutions — mostly simple one-line programs or small modifications to provided examples. In one case it implemented the decoding function of a rotation cipher when given the encoding function, suggesting an understanding of inverse functions despite having no knowledge of programming. Performance improved steadily with scale, hinting that larger vintage models could push this boundary further. ### Predicting the future Using nearly 5,000 historical event descriptions from the New York Times's "On This Day" feature, the team measured how "surprised" Talkie was by events from different decades. Surprise increased after the 1930 knowledge cutoff, peaked sharply in the 1950s and 1960s, then plateaued — suggesting the model had internalized enough pre-1931 trends to partially anticipate near-term history but was increasingly blindsided by developments further out. ## Technical challenges **Temporal leakage.** Despite filtering, some post-1930 knowledge slipped through — the model knows about the Roosevelt presidency, New Deal legislation, and some World War II details (the United Nations, the division of Germany). The team developed an n-gram-based anachronism classifier but acknowledges it remains imperfect and is building more advanced detection methods. **OCR noise.** All pre-1931 text had to be transcribed from physical sources, and conventional OCR systems struggle with historical layouts and fonts. The team found that training on OCR-transcribed text achieves only 30% of the learning efficiency of human-transcribed text. Simple regex cleaning recovers performance to about 70%, but a significant gap remains. Modern VLM-based OCR systems are more accurate but prone to [[Hallucination]] — hallucinating modern facts into historical text, which poisons the exercise. The team is building a custom "vintage OCR" system to close this gap. **Data quality vs. compute.** On standard evaluations Talkie underperforms its modern twin even after filtering out anachronistic questions, likely due to the combination of OCR noise and differences in corpus subject-matter distribution rather than any fundamental limitation of historical text. ## Why it matters here Talkie is a controlled experiment in what happens when an AI model absorbs only the worldview of pre-[[Civil Rights Movement]] America and the colonial-era English-speaking world. The implications cut directly into the vault's core concerns. ### A mirror of pre-civil-rights bias The model's training data — 260 billion tokens of text from an era of [[Jim Crow Era]] laws, [[Eugenics]] movements, colonial rule, and [[Scientific Racism]] — reflects the prejudices baked into the English-language written record before 1931. The output mirrors colonialist and social prejudices as a feature, not a bug: this is what the historical record contained. Talkie thus serves as a vivid demonstration of how [[AI Ethics]] concerns about bias are not merely about flawed algorithms but about the text humans produced and preserved. ### The copyright-free frontier Talkie challenges the assumption that building capable [[Large Language Model]] models requires modern copyrighted web data. If a public-domain-only corpus can produce a model that generalizes, reasons, and even learns unfamiliar programming languages from examples, the [[Intellectual Property]] arguments undergirding major AI copyright lawsuits look different. The project implicitly asks: if you *can* build on public domain text, should that become a norm — or at minimum a defensible alternative? ### Data shapes disposition By training a "modern twin" on web data with identical architecture and compute, Talkie enables direct measurement of how training-data era shapes a model's reasoning, knowledge, and biases. The team's broader question — how much of what we think we know about [[Large Language Model]] behavior is about human language in general versus the specific dataset (the web) that all modern models share — has implications for [[AI Safety]], [[Alignment]], and the monoculture risk in the current [[AI Arms Race]]. ## Scaling plans The team plans rapid scaling: a GPT-3-level model targeted for summer 2026, with a preliminary estimate that they can grow the corpus to over a trillion tokens of historical text — enough for a GPT-3.5-level model comparable to the original ChatGPT. They are also expanding beyond English to increase both corpus size and diversity of perspectives. ## See also - [[Open Source AI]] — the open-weights movement Talkie participates in - [[Foundation Model]] — the model class - [[AI Ethics]] — bias implications of training data choices - [[Benchmark]] — contamination problems Talkie sidesteps - [[Scaling Laws]] — whether vintage models follow the same curves - [[Scientific Racism]] · [[Eugenics]] · [[Jim Crow Era]] — ideologies embedded in Talkie's training era - [[Natural Language Processing]] — the subfield - [[Deep Learning]] · [[Machine Learning]] — the technical foundations