A Decade of Kill Bugs

2026-11-02T21:23:24+00:00

On the morning of September 24th 2026, Endcrawl had a brief outage. I'd just typed:

sudo kill -TERM -1490947

And instantly all services went down.

Right away I blamed myself. Maybe I mistyped that number! Maybe my code had a signal handling bug. Maybe I don't understand Unix as well as I think I do. Maybe I'm an idiot!

As it turned out, the outage was caused by a bug in kill(1).

This is really surprising. The kill program is 50 years old. Its spec is simple and stable. It's probably exercised a billion times a day.

A bug in kill was so surprising that, once I'd found it, I couldn't stop there. I needed to know how this bug came to be, and what had led the programmers so badly astray. When I eventually found the root cause, there was an even bigger surprise. It had led to not just one bug, but a whole series of kill bugs that shipped in Ubuntu 14, 16, 18, 22 and 24 LTS.

This post documents a decade of kill bugs, then makes some recommendations.

An Outage Not Understood Happens Again

Back to the outage. I scrambled to put the fire out. After that came a surprise second fire. By the end of all this I was pretty demoralized. I was ready to accept that I'd made a dumb mistake and ready to get on with the work day. There was this nagging feeling though…

Remember that final scene in Burn After Reading?

"What did we learn Palmer?"

"I don't know sir."

"I don't f**king know either. I guess we learned not to do it again."

"Yes sir."

"Although I'm f**ked if I know what we did..."

What had just happened? Was I about to be hit with a third outage? How could I be sure?

An outage not understood is an outage that happens again. That nagging feeling I had was a call to understand. With a sigh, I put all my other work on hold and started a proper post mortem.

The Proximate Cause

Let's examine the command that caused the outage:

sudo kill -TERM -1490947

The negative number here is a process group identifier, or pgid for short. A pgid is like a pid, but identifies a group of related processes. By convention, killing a positive number sends a signal to a single process, while killing a negative number signals all processes in that process group.

I was attempting to cancel a big Endcrawl 7k test render. A normal person would have killed one of the processes involved and let failures cascade to the others. But killing the whole group simultaneously strikes me as cleaner.

Process group signaling isn't some exotic operation. You use it all the time. Every time you ^C a command, the tty sends SIGINT to the foreground process group. Interactive shells run all but the simplest commands in a new process group, which among other things makes job control possible.

Process group signaling is in fact an awesome Unix innovation. I love it so much I added it to daemontools, where svc -+h now avoids orphans in a multiprocess service bounce.

At any rate, imagine my surprise when the command didn't just kill those render processes, but killed all processes on the machine save one!

The Bug

Here's the relevant snippet from procps 4.0.4, the provider of /bin/kill in Ubuntu 24.04 LTS. Can you spot the bug?

opterr=0; /* suppress errors on -123 */
while ((i = getopt_long(argc, argv, "l::Ls:hVq:", longopts, &optindex)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        print_usage(stderr);
      } else {
        /* Special case for signal digit negative
         * PIDs */
        pid = (long)('0' - optopt);
        if (!execute_kill((pid_t) pid, signo, use_sigqueue, sigval))
          exitvalue = EXIT_FAILURE;
        exit(exitvalue);
      }
    /* ... */
  }

My initial guess was wrong. There's no underflow or overflow bug with -1490947.

Notice we're in the unrecognized option switch case. Here's what getopt(3) says about unrecognized options:

By default, getopt() prints an error message on standard error, places the erroneous option character in optopt, and returns '?' as the function result.

Aha. The optopt variable only holds the first character after the dash, not the whole negative pid. When character 1 is subtracted from character 0 we get a pid of -1. What happens when you kill pid -1? Bad things:

If pid equals -1, then sig is sent to every process for which the calling process has permission to send signals, except for process 1 (init).

This ran as root, so the misparsing of -1490947 as -1 killed every process but init. Bug found, mystery solved.

Wait But Why

Except for me this just raised more questions. Why is the code assuming the unrecognized option is a "signal [sic] digit negative PID"? Why is the single digit negative pid case special, and what handles the general case of a multidigit negative pid?

Moreover, how could a bug like this exist in 2026, and in a Ubuntu LTS release no less? Has it always been there?

No, that couldn't be. I've killed process groups in the past. And the kill -TERM -12345 thing is an ancient Unix incantation. This used to work dammit!

Thinking back, I could vaguely remember some negative pid kills gone awry. Never in an "oops this killed everything in sight" way. More in a "weird, nothing happened" way, or a "poof there goes my shell" way. Minor ways more easily shrugged off as user error.

It turns out I wasn't imagining these. The last decade has been a rough one for negative pid killers. There have been no less than FOUR distinct bugs around negative pids. And they all shipped in Ubuntu LTS versions.

The History

Before I tell the tale of four bugs, let's go even farther back in Unix history.

In the beginning Unix had processes. By 1973 it had the kill(1) program. But it took a decade to evolve process groups as we now know them. It wasn't until Unix V9 in 1986 that you could kill a process group via negative pid. Near as I can tell, negative pid killing has worked fine in the BSDs all these years.

In the early 90s this thing called Linux came along. Although Linux initially borrowed a lot of BSD code for its userland, by the late 90s this approach was in legal jeopardy. Everything had to be rewritten, every wheel reinvented. And quickly. It was an exciting time to be a unix programmer.

And so it came to pass that Linux's kill was reinvented written by Albert Calahan. After Albert DoSed himself by forking 5 children, Debian downstream maintainer Craig Small took over poor old neglected procps.

Negative pid killing seems to have worked fine until 2013. Then the trouble began.

Bug 1: Ubuntu 14 Can't Kill -1

Ubuntu 14.04 LTS included procps 3.3.9. Prior to this, the kill code did what the BSDs do and used pointer arithmetic to walk and parse argv.

Personally I'm fond of that approach. Stallman isn't so fond of it though. In GNU world, you're supposed to standardize command line parsing via getopt. This is probably a good idea in the general case, but as we'll see, it was probably the wrong choice in the specific case of kill(1).

Here's the relevant code after the conversion to getopt:

opterr=0; /* suppress errors on -123 */
while (loop == 1 && (i = getopt_long(argc, argv, "l::Ls:hV", longopts, NULL)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        kill_usage(stderr);
      }
      loop=0;
      break;
    /* ... */
  }

argc -= optind + sigopt;
argv += optind;

for (i = 0; i < argc; i++) {
  pid = strtol_or_err(argv[i], _("failed to parse argument"));
  if (!kill((pid_t) pid, signo))
    continue;
  exitvalue = EXIT_FAILURE;
  continue;
}

The code worked fine for multidigit negative pids. However, if you were one of those freaks creative sysadmins who delighted in a kill -HUP -1 now and again to restart all your services, you noticed a bug: nothing happened.

The bug is due to option bundling.

Multidigit case: With kill -HUP -123, getopt acts as if -1, -2, and -3 were all separately passed. The unrecognized option case will be hit three times, and only at -3 will the -123 argument be consumed by incrementing optind. Because the code breaks out of the getopt loop early at -1, an unconsumed -123 argument correctly makes it to the pid killer.
Single digit case: With kill -HUP -1, however, getopt has fully consumed that argument and incremented optind before we break out of the loop. The pid killer sees no arguments and nothing happens.

Bug 2: Ubuntu 16 Accidentally Kills -1

A cure was in order. Unfortunately it was worse than the disease.

Ubuntu 16.04 LTS shipped with procps 3.3.10. The kill -HUP -1 bug was fixed at the expense of multidigit negative pid kills: a kill -TERM -12345 now killed everything.

This is the same bug that caused the outage under Ubuntu 24. We reviewed this above. The only mystery here is why procps circa Ubuntu 24 resurrected the known-buggy code circa Ubuntu 16. My best guess is that the following bugs were deemed even worse.

Bug 3: Ubuntu 18 Segfaults Killing -1

Ubuntu 18.04 LTS shipped with procps 3.3.12.

This attempted to fix multidigit negative pid killing. Instead of parsing optopt, which only corresponds to a single bundled character like -1 when -123 is passed, the new code stops and tries to parse the whole argument:

opterr=0; /* suppress errors on -123 */
while (loop == 1 && (i = getopt_long(argc, argv, "l::Ls:hV", longopts, NULL)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        kill_usage(stderr);
      } else {
        /* Special case for signal digit negative
         * PIDs */
        pid = atoi(argv[optind]);
        if (kill((pid_t)pid, signo) != 0)
        exitvalue = EXIT_FAILURE;
        exit(exitvalue);
      }
      loop=0;
      break;
    /* ... */
  }

This works correctly for multidigit negative pids. But if you passed a single digit negative pid, as in a kill -TERM -1, the pid argument was fully consumed, and optind would run off the end of argv[]. Segfault!

Bug 4: Ubuntu 22 Kills Your Shell

Ubuntu 22.04 LTS shipped with procps 3.3.17.

This fixed the segfault by ensuring optind never ran off the end of argv[]:

opterr=0; /* suppress errors on -123 */
while ((i = getopt_long(argc, argv, "l::Ls:hVq:", longopts, NULL)) != -1)
  switch (i) {
    /* ... */
    case '?':
      if (!isdigit(optopt)) {
        xwarnx(_("invalid argument %c"), optopt);
        kill_usage(stderr);
      } else {
        /* Special case for signal digit negative
         * PIDs */
        pid = atoi(argv[optind-1]);
        if (kill((pid_t)pid, signo) != 0)
        exitvalue = EXIT_FAILURE;
        exit(exitvalue);
      }
      xerrx(EXIT_FAILURE, _("internal error"));
    /* ... */
  }

Once again, it seems the contributor didn't understood how getopt option bundling interacts with optind, and hopefully by this point you're developing some sympathy for them. With this patch a kill -TERM -1 works correctly and no longer segfaults. But multidigit negative pid killing was broken for a second time, in a brand new way.

Suppose we try to kill -TERM -123. Thanks to option bundling, optind isn't incremented yet when we hit the unrecognized option case. Per some quirky behavior noted below, argv[optind-1] is now the string kill.

How does atoi(3) handle a non-numeric string like "kill"? It returns 0.

And what does the kill system call do with zero?

If pid equals 0, then sig is sent to every process in the process group of the calling process.

Because it's a simple command, your shell won't isolate kill -TERM -123 in a new process group. So this bug will unexpectedly kill your shell. Poof!

What Did We Learn, Palmer?

There are several takeaways.

The Narrow Learning

For one, the way getopt only advances optind with the final bundled option is deeply non-obvious. I didn't find evidence that any contributor understood this. I didn't understand it myself at first. That took careful study.

It's a real surprise. Surprising behavior like this should be avoided.

I feel this learning is too myopic though. Can one even avoid this surprising getopt behavior? You can't disable option bundling.

The Broader Learning

Getopt Considered Harmful.

Well okay, at least in this specific case. The kill(1) program is just an unusual beast. It accepts options that are actually signal names, like -KILL. You can also specify that same signal by number, like -9. And pgids are negative numbers that look deceptively like options.

How should a command line like kill -1 -9 be interpreted?

Should it send SIGHUP to pgid 9?
Should it send SIGKILL to pgid 1, aka everything?
Should it send SIGTERM, the default signal, to pgids 1 and 9?

All of these interpretations are potentially valid if any argv element can be an option or a pid.

The solution is to instead parse argv positionally. Don't use getopt here. Abandon capabilities like option bundling and option reordering that are more footgun than feature in this context. The POSIX spec says that only argv[1] can specify the signal to send, and arguments beyond that are pids and pgids. These are nice simplifying assumptions.

I think the old-school Unix and BSD approach is the right one. Have a look at FreeBSD's kill.c. Like ancestral Unix did back in the 70s, it parses argv positionally. The approach is simple and the code is simple. By contrast with Linux's kill, FreeBSD's kill has had no command line parsing bugs.

I do regard getopt dogma as the real root cause of the outage.

Oh, We're Halfway There

If abandoning getopt for "primitive" argv parsing still feels sinful, just look at what procps does before even calling getopt_long:

if (argc < 2)
  print_usage(stderr);

signo = skill_sig_option(&argc, argv);
if (signo < 0)
  signo = SIGTERM;

opterr=0; /* suppress errors on -123 */
while ((i = getopt_long(argc, argv, "l::Ls:hVq:", longopts, &optindex)) != -1)
/* ... */

What does skill_sig_option do?

int skill_sig_option(int *argc, char **argv)
{
  int i;
  int signo = -1;
  for (i = 1; i < *argc; i++) {
    if (argv[i][0] == '-') {
      signo = signal_name_to_number(argv[i] + 1);
      if (-1 < signo) {
        memmove(argv + i, argv + i + 1,
            sizeof(char *) * (*argc - i));
        (*argc)--;
        return signo;
      }
    }
  }
  return signo;
}

It scans argv looking for the first signal spec. If it finds a -KILL, a -9, or similar, it modifies argv in place (!) so getopt doesn't have to deal with that.

This is already "primitive" argv parsing! We're halfway there.

Some Final Practical Advice

The previous takeaways were aimed at software developers. For users of kill, I'd recommend the following.

First, use -- to separate options from pids and pgids:

kill -TERM -1490947     # no
kill -TERM -- -1490947  # yes

The second form is parsed correctly even by the buggy procps versions listed above.

Second, kill with least privilege. If I'd done a sudo -u render kill instead of a sudo kill, it would have only killed in-flight renders instead of all processes on the box. I feel a little insane recommending this kind of defensive tool usage — let's be clear, we're defending against bugs in kill, a small, mature, widely-used and very important tool — but here we are.

Finally, the good news is that a bugfix for Ubuntu 24 is already on its way at the time of publication. I still favor a simplifying rewrite of procps's kill though.

alangrow.com

A Decade of Kill Bugs