{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-31-memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. A script that runs the samples decides success, not a model. It is the method of [GVS5H](https://github.com/slee-persis/GVS5H) (Lee, 2026), written as a [Memdoor](https://github.com/guregodevo/memdoor/blob/main/README.md) workflow: eight YAML files, no code, no model named.\n\n## The result\n\nThe authors' harness rebuilt on the same model, Qwen3.8-27B, with their prompts, loop and 128K reply cap. The first ten problems of their LiveCodeBench Hard list, five passes each, graded on every hidden test with their corrected grader:\n\n| pass | this workflow | authors' harness | Fable 5, one call |\n| ---- | ------------- | ---------------- | ----------------- |\n| 1 | **10** / 10 | 9 | 7 |\n| 2 | **10** / 10 | 10 | 8 |\n| 3 | **10** / 10 | 10 | 10 |\n| 4 | **10** / 10 | 9 | 8 |\n| 5 | **9** / 10 | 10 | 7 |\n| mean | **9.8** | 9.6 | 8.0 |\n\nThe authors' two columns are the regraded result files in their repository, the numbers in their paper. Our column is [`bench/results/parity-5pass/`](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/bench/results/parity-5pass): all 50 runs, each with every request and reply, the program, the grader's output and the cost. A run took a median of 40 minutes and $0.39; the 50 cost $35.79.\n\nWhat this does and does not show:\n\n- Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.\n- The contests date from spring 2025 and the model may have seen them in training. Contamination cannot be ruled out, for any column.\n- Fable 5's column is one call per problem, from the authors' files. It was not run in this harness.\n- The loop sees only the public samples; the score is the hidden tests.\n- The model ran on OpenRouter, not a local machine.\n- The one lost problem: the manager invented a wrong counterexample and the team rewrote a correct program. Twice elsewhere a correct program was overwritten and then recovered. Both gaps are the authors' loop, kept as is.\n\nThe parity check (every request byte for byte against their code), the hosts, the changes it took and the first run's findings are in [NOTES.md](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/NOTES.md).\n\n## Run it\n\n```\ncurl -fsSL https://memdoor.ai/install.sh | bash\nmemdoor connect openrouter # or anthropic, openai, gemini, chatgpt\n# or no key: memdoor connect local (Ollama, llama.cpp, vLLM, LM Studio; Ollama needs a 32K window, not its 4K default)\ngit clone --depth 1 https://github.com/guregodevo/memdoor\n\nmkdir -p ~/.memdoor/workflows\ncp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/\nmkdir my-problem && cd my-problem\ncp -r ../memdoor/examples/workflows/team-solve/sample/* . # task.md + tests/1.in … 5.out\nmemdoor workflow run team-solve --wait # exit 0 when every sample passes\n```\n\nFor your own problem, replace `task.md` and put paired `tests/N.in` / `tests/N.out` beside it. The program is `solution.py`, read from stdin, compared to the samples with `diff`; it needs Python 3 and bash. `/workflow` in `memdoor tui` draws the run live.\n\nThe steps name no model: whatever `/model` is pinned to answers, a local one included ([how to connect one](https://memdoor.ai/docs/providers#a-local-runtime-no-key)). To change a role, change its prompt in `steps/`.\n\n## How it works\n\n```\nplan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify\n └── ideate-b ──┘\n```\n\n| step | kind | reads | writes |\n| ---------------------- | ------------------ | ---------------- | -------------------------------------------------------------------------------- |\n| `plan` | agent | `task.md` | `plan.md`, `notes.md` |\n| `ideate-a`, `ideate-b` | agent, in parallel | the plan | a proposal each: the most direct provable algorithm, and one from another family |\n| `solve` | agent | both proposals | `solution.py`, the decision in `notes.md` |\n| `test` | command | `tests/` | `test-output.log` (never fails the run) |\n| `review` | agent | the test output | repair instructions in `notes.md` |\n| `repair` | agent | the instructions | a fixed `solution.py`, or nothing |\n| `verify` | command | `tests/` | the verdict: a nonzero exit fails the run |\n\nEvery agent prompt says \"fresh context, read the files, do not rely on chat\". The paper credits that ledger discipline for the gain. Each step is one YAML file in `steps/`, in the format of [mario](https://github.com/guregodevo/mario), the engine underneath; `requires:` is the DAG.\n", "authors": [ "guregodevo" ], "captured_at": "2026-11-01T09:08:30.968650873+00:00", "category": "news", "content_html": "

Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. A script that runs the samples decides success, not a model. It is the method of GVS5H (Lee, 2026), written as a Memdoor workflow: eight YAML files, no code, no model named.

\n

The result

\n

The authors' harness rebuilt on the same model, Qwen3.8-27B, with their prompts, loop and 128K reply cap. The first ten problems of their LiveCodeBench Hard list, five passes each, graded on every hidden test with their corrected grader:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
passthis workflowauthors' harnessFable 5, one call
110 / 1097
210 / 10108
310 / 101010
410 / 1098
59 / 10107
mean9.89.68.0
\n

The authors' two columns are the regraded result files in their repository, the numbers in their paper. Our column is bench/results/parity-5pass/: all 50 runs, each with every request and reply, the program, the grader's output and the cost. A run took a median of 40 minutes and $0.39; the 50 cost $35.79.

\n

What this does and does not show:

\n\n

The parity check (every request byte for byte against their code), the hosts, the changes it took and the first run's findings are in NOTES.md.

\n

Run it

\n
curl -fsSL https://memdoor.ai/install.sh | bash\nmemdoor connect openrouter        # or anthropic, openai, gemini, chatgpt\n# or no key: memdoor connect local   (Ollama, llama.cpp, vLLM, LM Studio; Ollama needs a 32K window, not its 4K default)\ngit clone --depth 1 https://github.com/guregodevo/memdoor\n\nmkdir -p ~/.memdoor/workflows\ncp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/\nmkdir my-problem && cd my-problem\ncp -r ../memdoor/examples/workflows/team-solve/sample/* .    # task.md + tests/1.in … 5.out\nmemdoor workflow run team-solve --wait   # exit 0 when every sample passes\n
\n

For your own problem, replace task.md and put paired tests/N.in / tests/N.out beside it. The program is solution.py, read from stdin, compared to the samples with diff; it needs Python 3 and bash. /workflow in memdoor tui draws the run live.

\n

The steps name no model: whatever /model is pinned to answers, a local one included (how to connect one). To change a role, change its prompt in steps/.

\n

How it works

\n
plan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify\n       └── ideate-b ──┘\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
stepkindreadswrites
planagenttask.mdplan.md, notes.md
ideate-a, ideate-bagent, in parallelthe plana proposal each: the most direct provable algorithm, and one from another family
solveagentboth proposalssolution.py, the decision in notes.md
testcommandtests/test-output.log (never fails the run)
reviewagentthe test outputrepair instructions in notes.md
repairagentthe instructionsa fixed solution.py, or nothing
verifycommandtests/the verdict: a nonzero exit fails the run
\n

Every agent prompt says "fresh context, read the files, do not rely on chat". The paper credits that ledger discipline for the gain. Each step is one YAML file in steps/, in the format of mario, the engine underneath; requires: is the DAG.

\n", "content_markdown": "Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. A script that runs the samples decides success, not a model. It is the method of [GVS5H](https://github.com/slee-persis/GVS5H) (Lee, 2026), written as a [Memdoor](https://github.com/guregodevo/memdoor/blob/main/README.md) workflow: eight YAML files, no code, no model named.\n\n## The result\n\nThe authors' harness rebuilt on the same model, Qwen3.8-27B, with their prompts, loop and 128K reply cap. The first ten problems of their LiveCodeBench Hard list, five passes each, graded on every hidden test with their corrected grader:\n\n| pass | this workflow | authors' harness | Fable 5, one call |\n| ---- | ------------- | ---------------- | ----------------- |\n| 1 | **10** / 10 | 9 | 7 |\n| 2 | **10** / 10 | 10 | 8 |\n| 3 | **10** / 10 | 10 | 10 |\n| 4 | **10** / 10 | 9 | 8 |\n| 5 | **9** / 10 | 10 | 7 |\n| mean | **9.8** | 9.6 | 8.0 |\n\nThe authors' two columns are the regraded result files in their repository, the numbers in their paper. Our column is [`bench/results/parity-5pass/`](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/bench/results/parity-5pass): all 50 runs, each with every request and reply, the program, the grader's output and the cost. A run took a median of 40 minutes and $0.39; the 50 cost $35.79.\n\nWhat this does and does not show:\n\n- Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.\n- The contests date from spring 2025 and the model may have seen them in training. Contamination cannot be ruled out, for any column.\n- Fable 5's column is one call per problem, from the authors' files. It was not run in this harness.\n- The loop sees only the public samples; the score is the hidden tests.\n- The model ran on OpenRouter, not a local machine.\n- The one lost problem: the manager invented a wrong counterexample and the team rewrote a correct program. Twice elsewhere a correct program was overwritten and then recovered. Both gaps are the authors' loop, kept as is.\n\nThe parity check (every request byte for byte against their code), the hosts, the changes it took and the first run's findings are in [NOTES.md](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/NOTES.md).\n\n## Run it\n\n```\ncurl -fsSL https://memdoor.ai/install.sh | bash\nmemdoor connect openrouter # or anthropic, openai, gemini, chatgpt\n# or no key: memdoor connect local (Ollama, llama.cpp, vLLM, LM Studio; Ollama needs a 32K window, not its 4K default)\ngit clone --depth 1 https://github.com/guregodevo/memdoor\n\nmkdir -p ~/.memdoor/workflows\ncp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/\nmkdir my-problem && cd my-problem\ncp -r ../memdoor/examples/workflows/team-solve/sample/* . # task.md + tests/1.in … 5.out\nmemdoor workflow run team-solve --wait # exit 0 when every sample passes\n```\n\nFor your own problem, replace `task.md` and put paired `tests/N.in` / `tests/N.out` beside it. The program is `solution.py`, read from stdin, compared to the samples with `diff`; it needs Python 3 and bash. `/workflow` in `memdoor tui` draws the run live.\n\nThe steps name no model: whatever `/model` is pinned to answers, a local one included ([how to connect one](https://memdoor.ai/docs/providers#a-local-runtime-no-key)). To change a role, change its prompt in `steps/`.\n\n## How it works\n\n```\nplan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify\n └── ideate-b ──┘\n```\n\n| step | kind | reads | writes |\n| ---------------------- | ------------------ | ---------------- | -------------------------------------------------------------------------------- |\n| `plan` | agent | `task.md` | `plan.md`, `notes.md` |\n| `ideate-a`, `ideate-b` | agent, in parallel | the plan | a proposal each: the most direct provable algorithm, and one from another family |\n| `solve` | agent | both proposals | `solution.py`, the decision in `notes.md` |\n| `test` | command | `tests/` | `test-output.log` (never fails the run) |\n| `review` | agent | the test output | repair instructions in `notes.md` |\n| `repair` | agent | the instructions | a fixed `solution.py`, or nothing |\n| `verify` | command | `tests/` | the verdict: a nonzero exit fails the run |\n\nEvery agent prompt says \"fresh context, read the files, do not rely on chat\". The paper credits that ledger discipline for the gain. Each step is one YAML file in `steps/`, in the format of [mario](https://github.com/guregodevo/mario), the engine underneath; `requires:` is the DAG.\n", "dateCreated": "2026-11-01T09:08:30.968650873+00:00", "date_created": "2026-10-31T22:46:12+00:00", "date_modified": "2026-10-31T22:46:12+00:00", "date_published": "2026-10-31T22:46:12+00:00", "description": "Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. A script that runs the samples decides…", "external_url": "https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve", "git": { "commit": "20068632b65d2830a2d54f25ca1d90d697bea2c0", "path": "items/hnrss-org/2026/10/2026-10-31-memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/20068632b65d2830a2d54f25ca1d90d697bea2c0/items/hnrss-org/2026/10/2026-10-31-memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on.md" }, "id": "urn:aggr:item:6aa27ec0f6cf5042b8eea6734f8308efd396610e", "isBasedOn": { "@id": "https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve" }, "mainEntity": { "@id": "https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-31-memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on/#webpage" }, "author": [ { "@type": "Person", "name": "guregodevo" } ], "dateModified": "2026-10-31T22:46:12+00:00", "datePublished": "2026-10-31T22:46:12+00:00", "headline": "Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard", "image": { "@type": "ImageObject", "caption": "A lightweight coding and automation agent in one CLI binary. Run locally or over SSH, optimize token use, and automate CI and cron with durable workflows. - guregodevo/memdoor", "height": 90, "url": "https://opengraph.githubassets.com/31e7563ac084244a417ccae15e55c468113b8911a2d9fea5a0472d4ebd45d06e/guregodevo/memdoor", "width": 160 }, "timeRequired": "PT3M", "url": "https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve", "wordCount": 658 }, "name": "Archived snapshot: Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard", "preview": { "alt": "A lightweight coding and automation agent in one CLI binary. Run locally or over SSH, optimize token use, and automate CI and cron with durable workflows. - guregodevo/memdoor", "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://opengraph.githubassets.com/31e7563ac084244a417ccae15e55c468113b8911a2d9fea5a0472d4ebd45d06e/guregodevo/memdoor", "width": 160 }, "reading_minutes": 3, "replicated_at": null, "source": "hnrss-org", "source_name": "Hacker News: Front Page", "summary": null, "tags": [], "temporalCoverage": "2026-11-01T09:08:30.968650873+00:00", "title": "Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard", "url": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-31-memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on/", "word_count": 658 }