Show HN: I Turned DOOM into a Transformer

Neural networks computed by voodoo dolls inside DOOM maps. Every value is read back and checked against NumPy. Tolerance 0. Quick start · Walk a digit · The museum · How it works · What is checked · Films · Docs 10,000 / 10,000 1,355 0 4 MNIST test digits exact voodoo dolls in one MNIST map mismatches in 20,000 fuzzed nets other DOOM engines run the museum exactly Quick start pip install git+https://github.com/mottopanikeiku/doomformer # Python 3.11+, Linux or macOS doomformer hello # 2 s: a small MLP, every value checked doomformer draw --digit 7 # 15 s: the marine walks a 7 and the level reads it doomformer doctor checks your setup. It builds ACC, the ACS compiler, if it is missing. That needs git, cmake and a C compiler. On Fedora: sudo dnf install git cmake gcc gcc-c++ make. Tip No install needed for the Colab notebook. Draw a digit in your browser and DOOM reads it. Walk a digit, DOOM reads it The marine holds FIRE and inks the floor. Then he presses READ. 1,355 voodoo dolls in the same map run an MNIST network. The winning digit's doll crosses a line. That line lights the digit on the display. ▄▄▄▄▄▄▄▄▄▄▄▄▄▄ ▄████████████████ ████▀▀ ▀████▀ ▀▀▀ ▄███▀ ████ ███ ███ ███ ████ ███▀ ████ ▀███ DOOM reads: 7 (right) -- the winner's doll lit it on the 7-segment display EXACT: all 248 values DOOM computed [...] equal the NumPy integer reference (tolerance 0) READ pass: 273 game tics = 8 s of real-time DOOM at 35 tics/s, 3 s wall [...] Python reads back 248 values and checks every one. With --digit, Python presses the keys for you. Without it, you walk and draw yourself. Full output Note EXACT means DOOM got the same numbers as the integer network. It does not mean the digit is right. run right exact first 100 MNIST test digits 99 100 1,000 sampled test digits, one map each 966 1,000 all 10,000 test digits, tile maps 9,716 (97.16 %) 10,000 The float model scores 97.58 %. The network also trained on the strokes the scripted marine walks. Details · Film The museum: rewire one connection Tip Play it without Python. Download museum.wad (sha256) and the free Freedoom IWAD. Then run gzdoom -iwad freedoom2.wad -file museum.wad. Step on seven floor pads to light a digit. Press READ. 136 voodoo dolls run a tiny network and light the answer. Press MUTATE and the level rewires one connection while it runs. The same 7 now reads as 1. You can watch it happen: one doll stops at its gate. $ doomformer museum verify 7 pads abc w_7a +1 DOOM reads 7 scores [4, 6, 3, 5, 4, 3, 2, 7, 3, 4] EXACT (148 values; 744 tics; [...]) 7 after MUTATE pads abc w_7a -1 DOOM reads 1 scores [4, 6, 3, 5, 4, 3, 2, 5, 3, 4] EXACT (148 values; 744 tics; [...]) 7 after RESTORE pads abc w_7a +1 DOOM reads 7 scores [4, 6, 3, 5, 4, 3, 2, 7, 3, 4] EXACT (148 values; 744 tics; [...]) A self-test version runs exactly in GZDoom, LZDoom, UZDoom and ZDoom 2.8.1. Details How it works Every part of a network becomes a part of the map: in a neural network in the DOOM map a value of 5 5 voodoo dolls a weight a line on the floor; each doll that crosses it adds the weight a sum a sector's light level or floor height an activation (ReLU) a staircase of gates the dolls must climb attention dolls squeezing under ceilings that another value lowered a lesion a closed door One layer, from dolls to dolls: flowchart LR A["input value<br/>= a crowd of dolls"] --> B["weight lines<br/>each crossing adds w"] B --> C["the sum<br/>= a light level or height"] C --> D["staircase of gates<br/>= the activation"] D --> E["next layer's dolls"] Loading The engine has sharp edges, and the compiler works around them: A floor mover ignores a second request while it is still moving. A teleport cancels lines a doll crossed earlier in the same tic. Things far from a BSP node can land in the wrong room, so every map fits in 32,000 units. doomformer facts checks these in about 10 s. Engine facts · Engine features it needs · Other ports Computers in DOOM maps are not new. counter.wad counts with voodoo dolls, conway.wad runs Game of Life, and Danny Spencer's calculator adds numbers. New here: a compiler for neural networks, and an exact check on every value. Related work What is checked Every run compares each value the map exposes with NumPy. That proves the inputs that ran, nothing more. So the repo also attacks its own check: attack result Tamper with one weight line the check fails on exactly the inputs that cross it Fuzz 20,000 random nets at the range limits 0 mismatches; CI fuzzes a fresh seed on every push Mutate one line of a map, 1,004 times 297 caught; no survivor is proven harmless Extract the weights back out of the map, without DOOM equal to the source net Guard every readout page with a poison value a page that never runs cannot pass old values The MNIST map exposes 248 values. It does not expose its 903 staircase counts and 1,351 gate comparisons. Those are checked only through the sums they feed. Fuzzing and tests found four bugs in the compiler. All are fixed. Verification Who computes what run Python voodoo dolls mnist, df.compile writes the inputs, waits, reads every sum, activation, comparison and the choice demo, ask (tile maps) writes tokens, copies values between maps, samples letters every sum and activation draw, --loop, the marine's brain, the fly only advances the clock and reads everything, including the choice machine loads the program, then advances the clock and reads program counter and registers museum, played in GZDoom nothing every sum, comparison and the choice ACS level scripts move gates, rewind the dolls, sense the world and copy results out. Two do more. df.Agent takes an argmax in ACS. --opt 3 copies each layer's values into the next layer with a script. Per demo The maps need ZDoom (UDMF and ACS). This repo runs ViZDoom 1.3.2 with Freedoom assets. Commands Command What DOOM does Time Skill level doomformer hello a small MLP, 5 random inputs 2 s I'm too young to die doomformer draw --digit 7 the marine walks a 7, the level reads it 15 s Hey, not too rough doomformer museum build a playable map that reads a digit 1 s Hey, not too rough doomformer mnist reads 4 test digits, or your own image 1.5 to 3 min Hurt me plenty doomformer demo a transformer writes a level name 15 to 60 s Hurt me plenty doomformer machine subleq all a tiny computer runs three programs about 5 min Ultra-Violence doomformer learn digits --classes 4 --steps 300 a blank brain learns digits inside the map about 5 min Ultra-Violence doomformer ask "what does iddqd do" answers from 189 DOOM facts about 11 min Ultra-Violence doomformer demo --nightmare the 2-layer model, 106 tile maps 3.5 to 7 min Nightmare! Times are from a 16-core arm64 Linux machine. Every command Some outputs are picked. DEMON CANYON, the name --nightmare writes, is one hand-picked sample. ask gets 94.7 % of its training questions right, and 35.4 % of new wordings. Bigger experiments His brain is the level. The level senses the nearest demon. 1,345 voodoo dolls of the marine run his brain and pick an action. The level turns him and fires. Python only steps the clock and reads. 50 of 50 decisions are exact. Film · Code Warning He is not playable. One decision takes 22 s of game time, and the demons are frozen while he thinks. With the demons awake (--live), he dies in 5.5 s without a kill. Live demons run it from a clone what you see python -m demos.brainlevel his brain is the level python -m demos.brainlevel --live the same brain against awake demons python -m demos.brainlevel --lesion-when near the level closes neuron doors when a demon gets close python -m demos.aimlearn a perceptron learns to aim inside one game python -m demos.flyloop a fruit-fly circuit drives the marine python -m demos.lobotomy lesions that turn him into a pacifist All demos, numbers and picked seeds Build your own Compile a network into a map. Let its outputs open doors, flash lights or shove the marine. This runs as pasted: import numpy as np import doomformer as df # 2 inputs -> [a AND b, a OR b] -> 2 outputs, all integers net = df.MLP( weights=[np.array([[1, 1], [1, 1]]), np.array([[1, 0], [0, 1]])], biases=[np.array([-1, 0]), np.array([0, 0])], thresholds=[np.array([1]), np.array([1])], signed_input=False, ) with df.compile(net) as brain: for x in ([0, 0], [1, 0], [1, 1]): # computed by voodoo dolls; raises df.MismatchError on any difference from net.reference print(x, "->", brain(np.array(x)).tolist()) brain.record([np.array([0, 1]), np.array([1, 1])], "dolls.gif") # film the dolls doing it df.MLP.from_torch converts a PyTorch model (example 02). The world and actions hooks wire outputs to doors, lights and thrusts (example 03). Guide · 12 examples · Show and tell Many games at once One map runs in one game, on one core. df.run_many spreads inputs over many games, or over a Ray cluster. A network can also be split into tile maps, one per layer. Then Python copies values between them. run digits right exact time one map, 14 games 1,000 966 1,000 2 h 4 tile maps, 6 games 10,000 9,716 10,000 23 min doomformer mnist --tiles --all --workers 6 # all 10,000 digits as tile maps, about 23 min pip install "doomformer[ray]" ray start --head # other machines: ray start --address=HEAD:6379 doomformer mnist --all --ray --record mnist.json # all 10,000 digits, one map each, on a cluster Budget one core and 1.3 GB of RAM per game. Scaling More what try read Faster schedules: MNIST in one map from 5,696 to 272 tics --opt 3 scheduling A counter machine: 7 × 9 = 63 in 464 checked steps doomformer machine multiply 7 9 machine A SUBLEQ machine with self-modifying code doomformer machine subleq all machine A blank brain learns digits in the map: 71 % held-out doomformer learn digits learning PyTorch import with BatchNorm, pooling and QAT MLP.from_torch interop A transformer that learned induction, exact in DOOM python -m demos.induction induction Patching, lesions and steering, checked in DOOM intref.forward(interventions=...) interventions Speed per pass and throughput doomformer bench scaling Limits Important Slow. A small MLP runs about 23,000 times slower than NumPy. Most of the time goes to idle conveyor belts. Speed Only the runs are checked. Inputs that never ran are not covered. Not covered Integer models. Attention is integer attention. No model has LayerNorm. ViZDoom first. Other DOOM engines ran only small maps. No reinforcement learning. The marine copies a scripted aimbot. Finite machines. Memory is bounded. Nothing here claims DOOM is Turing complete. Picked samples. Showcased seeds and names are picked. The list What's in the box Path What it holds doomformer/ the toolkit: map compiler, DOOM runtime, exact checks, CLI demos/ research code: the marine's brain, the level generator, the fly circuit, the learning arena training/ training scripts for every model, with the data and sources models/ every trained model with its training log and evaluation examples/ twelve one-file examples, most under 15 s tests/ 496 tests; CI runs them against real headless DOOM on Linux and macOS docs/ how it works, every demo with its numbers, the engine facts, the guide, every command CHANGELOG.md what each release added Licensing Code and docs: MIT. Data keeps its own licence (THIRD_PARTY.md). The level-name lists come from the Doom Wiki (CC BY-SA 4.0) and id Software's source (GPL-2.0). The 100 sample MNIST digits are CC BY-SA 3.0. The fruit-fly circuits in models/fly come from the FlyWire connectome (CC BY-NC 4.0, non-commercial). Freedoom graphics are BSD-licensed. DOOM is a trademark of id Software / ZeniMax. This project is not affiliated with them.