{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-09-07-creepy-crawlies/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "**[Creepy crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies)** ([via](https://news.ycombinator.com/item?id=49491791 \"Hacker News\")) Konstantin Ryabitsev discusses how bad the \"background radiation\" of abusive crawlers has become from the perspective of [git.kernel.org](https://git.kernel.org/), the official Git repository for the Linux kernel:\n\n> TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.\n\nI worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.\n",
  "authors": [],
  "captured_at": "2026-09-08T10:17:10.166315+00:00",
  "category": "programming",
  "content_html": "<p><strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://people.kernel.org/monsieuricon/creepy-crawlies\">Creepy crawlies</a></strong> (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://news.ycombinator.com/item?id=49491791\" title=\"Hacker News\">via</a>) Konstantin Ryabitsev discusses how bad the &quot;background radiation&quot; of abusive crawlers has become from the perspective of <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://git.kernel.org/\">git.kernel.org</a>, the official Git repository for the Linux kernel:</p>\n<blockquote>\n<p>TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.</p>\n</blockquote>\n<p>I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.</p>\n",
  "content_markdown": "**[Creepy crawlies](https://people.kernel.org/monsieuricon/creepy-crawlies)** ([via](https://news.ycombinator.com/item?id=49491791 \"Hacker News\")) Konstantin Ryabitsev discusses how bad the \"background radiation\" of abusive crawlers has become from the perspective of [git.kernel.org](https://git.kernel.org/), the official Git repository for the Linux kernel:\n\n> TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.\n\nI worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.\n",
  "dateCreated": "2026-09-08T10:17:10.166315+00:00",
  "date_created": "2026-09-07T23:08:58+00:00",
  "date_modified": "2026-09-07T23:08:58+00:00",
  "date_published": "2026-09-07T23:08:58+00:00",
  "description": "Creepy crawlies Konstantin Ryabitsev discusses how bad the \"background radiation\" of abusive crawlers has become from the perspective of git.kernel.org, the official Git repository for the Linux kernel: TL;DR: we spend more CPU cycles…",
  "external_url": "https://simonwillison.net/2026/Sep/7/creepy-crawlies/",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/simonwillison-net/2026/09/2026-09-07-creepy-crawlies.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/simonwillison-net/2026/09/2026-09-07-creepy-crawlies.md"
  },
  "id": "urn:aggr:item:dd32d656379b694ab36eac74a4ac121815ebd375",
  "isBasedOn": {
    "@id": "https://simonwillison.net/2026/Sep/7/creepy-crawlies/"
  },
  "mainEntity": {
    "@id": "https://simonwillison.net/2026/Sep/7/creepy-crawlies/",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-09-07-creepy-crawlies/#webpage"
    },
    "dateModified": "2026-09-07T23:08:58+00:00",
    "datePublished": "2026-09-07T23:08:58+00:00",
    "headline": "Creepy crawlies",
    "timeRequired": "PT1M",
    "url": "https://simonwillison.net/2026/Sep/7/creepy-crawlies/",
    "wordCount": 95
  },
  "name": "Archived snapshot: Creepy crawlies",
  "preview": null,
  "reading_minutes": 1,
  "replicated_at": null,
  "source": "simonwillison-net",
  "source_name": "Simon Willison's Weblog",
  "summary": "Creepy crawlies Konstantin Ryabitsev discusses how bad the \"background radiation\" of abusive crawlers has become from the perspective of git.kernel.org, the official Git repository for the Linux kernel: TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages. Via Hacker News Tags: crawling, git, linux, datasette, ai-ethics",
  "tags": [
    "ai-ethics",
    "crawling",
    "datasette",
    "git",
    "linux"
  ],
  "temporalCoverage": "2026-09-08T10:17:10.166315+00:00",
  "title": "Creepy crawlies",
  "url": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-09-07-creepy-crawlies/",
  "word_count": 95
}