37signals Dev

Parsing JSON-structured Docker logs with the OpenTelemetry Collector

Lewis Buckley — Mon, 19 Oct 2026 17:00:00 +0000

How we use the filelog receiver’s operator chain to parse, recombine, and route JSON-structured log lines from Docker containers.

We deploy our Rails apps with Kamal, which runs them as Docker containers. Alongside the Prometheus metrics collection we set up earlier, we run the OpenTelemetry Collector on each host to ship logs to Loki. Our Rails apps emit one JSON object per request to stdout: controller, status code, duration and the full Rails log output, all in a single line. Docker’s json-file logging driver then wraps each of those lines in another layer of JSON. Here’s how we use the collector’s filelog receiver to unwrap and parse them.

No built-in Docker log receiver

The OpenTelemetry Collector doesn’t have a dedicated Docker log receiver. There’s a dockerstats receiver for container metrics, but nothing equivalent for logs. The filelog receiver’s container operator can parse Docker’s json-file format, but it only unwraps the outer JSON envelope. It recombines split lines for the CRI format used by containerd and CRI-O, but not for Docker’s format, and it doesn’t handle non-JSON content mixed into your log stream. Most Rails apps write some plain text to stdout too.

So collecting Docker container logs with the OTel Collector means putting together a chain of operators yourself. It’s not hard once you know the pieces, but it’s not obvious either.

The problem: JSON inside JSON

Docker’s json-file driver wraps every line your app emits in a JSON envelope:

{"log":"{\"@timestamp\":\"2026-03-14T12:00:00Z\",\"rails\":{\"controller\":\"PostsController\"},\"http\":{\"response\":{\"status_code\":200}},\"original\":\"Started GET /posts...\"}\n","stream":"stdout","time":"2026-03-14T12:00:00.123Z"}

The log field contains your app’s JSON as an escaped string. And if your app’s line is longer than 16KB, Docker splits it across multiple records, each one a separate JSON object with part of the log value. Only the last part ends with a newline. That newline is the only way to tell a partial record from a complete one.

The operator chain

The filelog receiver lets you define a chain of operators that process log entries in sequence. Here’s our setup:

receivers:
  filelog:
    include: [/var/lib/docker/containers/*/*-json.log]
    start_at: end
    operators:
      - type: container
        add_metadata_from_filepath: false

      - type: filter
        expr: 'body matches "^[^{]" and body matches "[^}]\\n$"'

      - type: recombine
        combine_field: body
        source_identifier: attributes["log.file.name"]
        combine_with: ""
        is_last_entry: body matches "\n$"
        max_unmatched_batch_size: 0

      - type: json_parser
        if: body matches "^\\{"
        parse_from: body
        parse_to: attributes
        on_error: send

Let’s walk through each step.

1. `container`: unwrap the Docker envelope

The container operator parses Docker’s json-file format (and CRI format too). It extracts the log field as the body, stream as an attribute, and time as the timestamp. After this step, the body is your app’s raw log line, or part of it, but still a string, not yet parsed.

add_metadata_from_filepath: false turns off the Kubernetes metadata the operator reads from a pod’s log file path. Docker’s log paths don’t contain that metadata, and with the option on, the operator logs an error for every record unless the receiver also adds the file path (include_file_path: true).

2. `filter`: drop non-JSON lines

Not everything your app writes to stdout is JSON. Puma access logs, Ruby warnings, and middleware timing lines (like Rack handled in 18.27ms. 0 GCs.) are plain text. We don’t want these in Loki, and some of our pipelines pick up lines by container name alone, so we drop them here.

We need to be careful because of how Docker splits long lines. The first chunk of a split line starts with { but the middle chunks won’t, and only the final chunk ends with a newline. We can’t just drop everything that doesn’t look like JSON, or we’d lose pieces of split log lines.

The filter drops lines that start with a non-{ character and don’t end with }\n:

body matches "^[^{]" and body matches "[^}]\\n$"

A complete JSON line starts with {, so the first condition is false and it passes through. A middle chunk of a split JSON line has no newline at the end, so the second condition is false and it also passes through. Only standalone plain-text lines match both conditions and get dropped.

3. `recombine`: reassemble split JSON

Docker’s json-file driver splits log lines at 16KB boundaries. A single JSON log from your app might become two or three Docker log entries. The container operator handles this for CRI format but not for Docker’s json-file format, so we do it ourselves.

The recombine operator concatenates consecutive entries until it sees one whose body ends with a newline, which is how Docker marks the end of a line:

- type: recombine
  combine_field: body
  source_identifier: attributes["log.file.name"]
  combine_with: ""
  is_last_entry: body matches "\n$"
  max_unmatched_batch_size: 0

source_identifier ensures we only recombine entries from the same log file, so logs from different containers don’t get mixed up. That’s important when Kamal runs multiple containers on the same host during deploys. combine_with: "" concatenates without adding any separator, since Docker’s splits are at arbitrary byte boundaries.

max_unmatched_batch_size: 0 turns off a default limit of 100 entries. Without it, recombine sends on what it has after 100 partial records, so any line over 1.6MB gets cut in two.

4. `json_parser`: parse the app JSON

Now the body contains the complete, reassembled JSON string from your app. The json_parser operator parses it and writes the resulting fields into the log record’s attributes:

- type: json_parser
  if: body matches "^\\{"
  parse_from: body
  parse_to: attributes
  on_error: send

After this, attributes["rails"]["controller"] is "PostsController", attributes["http"]["response"]["status_code"] is 200, and so on. The log record is now fully structured and ready for routing, transformation, and export.

The if skips anything that doesn’t start with {, so the parser doesn’t log an error for every piece of plain text that reaches it. on_error: send passes a record that fails to parse on with its raw body, and the collector logs the error. We count those errors in the collector’s own logs, and that’s how we found the two problems below.

Two things we got wrong

Our first version of this chain ended a line on }\n rather than \n, and left max_unmatched_batch_size at its default. Both lost log lines.

When a line is an exact multiple of 16KB before its newline, Docker’s split falls between the closing } and the \n. The last record is then just "\n". Neither record matched }\n$, so recombine kept going and joined the next log line on as well. That gave us two JSON objects in one body, invalid character '{' after top-level value from the parser, and two lost lines. It only happens to about 1 in 16,384 long lines, but on Basecamp that was around 320 times an hour.

The 100-record limit cut lines over 1.6MB in two. The first part failed to parse with unexpected end of JSON input and the rest was dropped. That was about 110 lines every 6 hours on Basecamp.

We now test the receiver config by running the collector against sample Docker log files, including a line split just before its newline and a line in more than 100 records, and checking which records come out.

What comes next

With structured attributes on each record, you can use the routing connector to send different log formats to different pipelines, and the transform processor to normalize fields, set severity levels, and enrich logs before exporting them.

The operator chain runs inside the receiver, before logs enter the pipeline. Getting the unwrap → filter → recombine → parse sequence right here means your processors only ever see clean, fully-parsed log records.

Other posts in the “On-prem observability” series

Hot Cell v1.0: Securing Active Storage in the age of AI

Mike Dalessio — Mon, 5 Oct 2026 17:00:00 +0000

Today we are releasing Hot Cell v1.0, a suite of gems that moves Active Storage’s attachment processing out of your Rails application and into an unprivileged sidecar container with no network, no credentials, and nothing on its filesystem worth stealing. Adopting it is a configuration change, not a code change. It is already running in production at 37signals, in Basecamp, HEY, and Fizzy.

I introduced Hot Cell at Rails World 2026 in a talk titled “Hot Cell: Securing Active Storage in the age of AI.” What follows is basically that talk, written down, plus a couple of things that have changed since then. (If you’d rather watch the video or flip through the slides, go for it.)

Where we are

In the beginning, the earth cooled. Dinosaurs roamed the earth. Then humans started writing software, and a lot of that software was very trusting in nature: corner cases weren’t explored, and the design assumed users were friendly and input could be trusted.

Now we find ourselves in a very chaotic moment where AI is very good at finding these bugs and security problems. I don’t know what’s coming, but at this moment I’m very worried, and I think you should be too. So first I’m going to scare you, and then I’m going to give you the tools to do something about it.

Part 1: You are not worried enough (probably)

My summer started with CVE-2026-66066, nicknamed “KindaRails2Shell.” I’m a member of the Rails security team, and I happened to catch this report and ended up working on it. We scored it 9.5 on CVSS, which is about as bad as it gets. Here’s the description we published:

In versions prior to 7.2.3.2, 8.0.5.1 and 8.1.3.1, Active Storage does not disable libvips operations marked unsafe for untrusted content, allowing a crafted upload to invoke such an operation. Consuming applications are affected when configured to use libvips and accept image uploads from untrusted users. An unauthenticated attacker may exploit this behavior to read arbitrary files accessible to the Rails process, including environment variables and application secrets. Exposure of credentials such as secret_key_base or external-service tokens may enable remote code execution or lateral movement.

I wrote this CVE description to be intentionally opaque. We didn’t disclose any details about the attack, because we didn’t want attackers to start exploiting live apps before anyone had a chance to upgrade. We were trying to buy you some time. Unfortunately, AI has gotten so good that multiple researchers were still able to derive this attack within hours of the announcement, quickly making the embargo meaningless.

You can watch the talk for a more in-depth explanation, but the root of the problem is that the image processing libraries are running in a container with access to your secrets and to your network.

And there was no single big vulnerability here that AI agents could find. The attack requires chaining together several low-severity vulnerabilities along with some Rails design decisions that helped enable the attack.

Active Storage direct uploads were enabled by default, so the route was open even in apps that didn’t use them. Niklas Häusele fixed that in rails/rails#58369.
Direct uploads never examine the bytes. The file goes from the browser to the blob store without passing through Rails, so Rails trusts the reported content type. Lying about the content type is the way in. Fixing this would cost performance.
Signed variation keys do not include the blob ID, which makes the attack easier to pull off. It’s not a security problem on its own, and changing it would be a breaking change.
Rails didn’t call Vips.block_untrusted. libvips added it in 8.13 as an opt-in, buried in a changelog. Calling it is the fix for this CVE.
libvips was fooled by a spoofed MATLAB header. This is the actual flaw. It’s fixed in libvips 8.18.5, and no CVE was published.
libmatio trusts every file it is handed, as designed. The maintainers pointed back at libvips: you shouldn’t be handing untrusted files to libmatio. Won’t fix.
Known gadgets turn secret_key_base into RCE. The Rails security team knows about these, but closing them is a breaking change.

Agents are able to chain all seven together from very little information. They are getting very good at this! In the talk I showed a recording of the exploit script pulling every secret out of a development app in real time. Once you know how to execute the attack, it happens very fast.

It should also scare you that 378 vulnerabilities were published in libvips and its dependencies in the first nine months of 2026, and more are reported every day.

This chart is 37signals’ HackerOne data: valid reports received each month. There has been a steady increase since models turned the corner back in November 2025.

So there’s a whole category of problems here. I can fix this one CVE, but we’ll have to go through all of this again when someone finds a zero-day in the next image processing library. Hardening those libraries is not under our control, and it’s not something we can fix within Rails.

When I think about addressing the bigger issues here, I think about risk:

The probability of another vulnerability in these libraries is very high, and it’s not going down any time soon. That leaves impact. How do we shrink the blast radius, so that the next vulnerability does as little damage as possible?

Part 2: Reducing the blast radius

The question I want to answer is: How can we run Active Storage so that it doesn’t matter if the libraries are vulnerable or if the input is maliciously crafted?

You’re probably thinking “sandbox!” and so was I. But I wanted some very specific attributes in this sandbox.

Step 1: Define the requirements

This list became the prompt I gave my agent as we built the system together:

Implement existing Rails behavior. A drop-in replacement for Active Storage’s analyzers, transformers, and previewers. A configuration change, not a code change.
Least privilege. No capabilities, no privilege escalation, no setuid, an unprivileged user.
Hard limits. Wall-clock deadline, memory, file size, open files, and process count.
No access to the app filesystem. No app code, no credentials. The filesystem is read-only and noexec.
No access to the app environment. No secrets to exfiltrate.
No network interface. No calling home, and no lateral movement.
Disposable. A process per request, killed and reaped when it’s done, so remote code execution has a short lifetime.
Familiar lifecycle. Deployable with Kamal or Kubernetes.
Extensible behavior. If you want a secure place to unzip files, you should be able to build that too.

Step 2: An obscure, too-clever name 🤣

Image: a frame from an Oak Ridge National Laboratory video.

This is a hot cell at Oak Ridge National Laboratory. It’s a shielded chamber for handling radioactive material: a big lead box with a window that radiation can’t pass through. The operator stays outside and does everything with manipulator arms, and nothing enters or leaves except by controlled transfer.

That’s exactly what I want for Active Storage.

Step 3: Jam it into Rails

Thankfully, Rails already has configuration for this. Analyzers extract metadata, like the number of pages in a PDF. The variant processor does transformations: rotating, cropping, resizing. Previewers generate thumbnails for files that aren’t images, like PDFs and videos. Here are the defaults, trimmed to the ones we care about:

# config/application.rb
config.active_storage.variant_processor = :vips
config.active_storage.analyzers = [ ActiveStorage::Analyzer::ImageAnalyzer::Vips,
                                    ActiveStorage::Analyzer::VideoAnalyzer ]
config.active_storage.previewers = [ ActiveStorage::Previewer::MuPDFPreviewer ]

And here’s the target API, with no application code changes:

# config/application.rb
config.active_storage.variant_processor = ActiveStorage::HotCell::Client::Transformers::Image::Vips
config.active_storage.analyzers = [ ActiveStorage::HotCell::Client::Analyzers::Image::Vips,
                                    ActiveStorage::HotCell::Client::Analyzers::Video::FFprobe ]
config.active_storage.previewers = [ ActiveStorage::HotCell::Client::Previewers::Pdf::Mutool ]

One caveat: passing a class as the variant_processor became possible with rails/rails#58384, so the Active Storage gems need Rails 8.2.

Hot Cell is a family of gems. hotcell-core holds the common bits: the wire protocol, descriptor passing, and the error codes. hotcell-client runs in your app and hotcell-server runs in the cell. The activestorage-hotcell-client and activestorage-hotcell-server gems subclass those to provide the drop-in replacements for Rails. Since the talk, a sixth gem has joined them: yabeda-hotcell, which I’ll get to below.

Step 4: Inputs and outputs

I’ve hand-waved over how the client and the server communicate, given that the cell has no network interface.

The answer is UNIX sockets. A UNIX socket is a file on disk that two processes on the same host can talk over, and nothing about it is routable on a network. A cell exposes two of them in a shared directory: one for work, and one for metadata and control.

And the magic bit is that sendmsg() can pass open file descriptors over a UNIX socket. The Hot Cell client in your app opens the upload and the output file, and hands those descriptors to the cell. The cell reads and writes through them but never sees a path, and has no access to the rest of your filesystem. Inputs are read-only and outputs are write-only, and the kernel enforces it. Passing descriptors also closes off the class of attacks that use symbolic links to traverse paths.

Step 5: Doing the work

This part is a supervisor and workers, like Puma or Solid Queue. The request lifecycle walks through each step.

The supervisor is pid 1 in the cell. It accepts connections, queues them, hands each to a worker, enforces the wall-clock deadline, kills the process group, and reaps. It never reads a request and never evaluates a byte of image data.

Each request is handled by a forked worker. A successful attack only compromises that worker, which gets reaped.

That’s the finished system, and it’s what is running in production at 37signals today. Now that I’ve scared you, I’m telling you that you should be running Hot Cell in your apps, and here’s how.

Part 3: Rolling it out

Add activestorage-hotcell-client to your app’s Gemfile, and run the installer:

$ bin/rails hotcell:install
$ find hotcell
hotcell/Dockerfile
hotcell/Gemfile
hotcell/config.rb
hotcell/operations/
hotcell/operations/.keep

Everything about the cell lives in that hotcell/ directory. The cell has its own Gemfile, which you should keep short, because every gem in it is inside the blast radius. config.rb is plain Ruby, not Rails, and sets the cell’s limits. Each operation’s own limits are clamped to these:

# hotcell/Gemfile
source "https://rubygems.org"

gem "hotcell-server", "~> 1.0"
gem "activestorage-hotcell-server", "~> 1.0"

# hotcell/config.rb
HotCell.limits concurrency: 4,
               queue_size: 8,
               queue_wait: 10,
               deadline: 120,
               memory: 1536 * 1024**2,
               file_size: 256 * 1024**2

The only thing the app and the cell share is a volume holding the two sockets. In the app’s Kamal config, you join the cell’s group and mount the volume:

# config/deploy.yml -- the app
servers:
  web:
    options:
      group-add: 10001

volumes:
  - hotcell-sockets:/run/hotcell/active_storage

env:
  clear:
    HOTCELL_ROOT: /run/hotcell
    HOTCELL_GROUP: 10001

The cell is a Kamal accessory, because Kamal hard-codes the network for roles and network: none is the point:

# config/deploy.yml -- the cell
accessories:
  active_storage:
    image: your.registry.com/your-image:latest
    roles: [ web, jobs ]
    network: none
    volumes:
      - hotcell-sockets:/run/hotcell/cell
    options:
      cpus: 2
      memory: 2g
      memory-swap: 2g
      pids-limit: 512
      read-only: true
      cap-drop: ALL
      security-opt: no-new-privileges:true
      user: 10001:10001
      tmpfs: /tmp:rw,nosuid,nodev,noexec,size=512m
    env:
      clear:
        HOTCELL_DIR: /run/hotcell/cell

The volume matches the app’s. The resource limits cap CPU, memory (with swap pinned equal so the limit holds), and processes, so a fork bomb dies in the cell. And every one of network: none, read-only, cap-drop, and no-new-privileges is a security property. If you omit one, the protection is gone and the cell keeps serving requests exactly as before.

Then pick the Hot Cell twin of each Active Storage class you use:

`ActiveStorage`	`ActiveStorage::HotCell::Client`
`Transformers::Vips`	`Transformers::Image::Vips`
`Transformers::ImageMagick`	`Transformers::Image::Magick`
`Analyzer::ImageAnalyzer::Vips`	`Analyzers::Image::Vips`
`Analyzer::ImageAnalyzer::ImageMagick`	`Analyzers::Image::Magick`
`Analyzer::VideoAnalyzer`	`Analyzers::Video::FFprobe`
`Analyzer::AudioAnalyzer`	`Analyzers::Audio::FFprobe`
`Previewer::MuPDFPreviewer`	`Previewers::Pdf::Mutool`
`Previewer::PopplerPDFPreviewer`	`Previewers::Pdf::Poppler`
`Previewer::VideoPreviewer`	`Previewers::Video::FFmpeg`

Set the three Active Storage configs as above, and point Hot Cell at the sockets:

# config/initializers/hotcell.rb
HotCell.root  = ENV["HOTCELL_ROOT"]
HotCell.group = ENV["HOTCELL_GROUP"]

On the cell side, require the operations that match. The cell isn’t a Rails app, so there’s no Zeitwerk; requiring a file is what serves its operation. This is also where you set per-operation limits:

# hotcell/operations/active_storage.rb
require "active_storage/hot_cell/server/transformers/image/vips"
require "active_storage/hot_cell/server/analyzers/image/vips"
require "active_storage/hot_cell/server/analyzers/media/ffprobe"
require "active_storage/hot_cell/server/previewers/pdf/mutool"
require "active_storage/hot_cell/server/previewers/video/ffmpeg"

ActiveStorage::HotCell::Server::Transformers::Image::Vips
  .limits file_size: 256 * 1024**2

ActiveStorage::HotCell::Server::Previewers::Pdf::Mutool
  .limits deadline: 30

That’s all it takes to port a Rails app that’s using vanilla Active Storage!

Configure Active Storage.
Configure the cell’s Gemfile and config.rb.
Extend the Kamal (or Kubernetes) config.

Once a file type is handled by the cell, consider removing its packages (libvips, ffmpeg, and so on) from your application image to reduce the attack surface.

The Hot Cell documentation covers all of this, including every container flag and how to size the limits.

Observability

We can’t trust the workers, since any one of them may have been compromised. But we can trust the supervisor, and we can trust your Rails app, and between them they can give us fantastic visibility into image processing:

Events in the app. Every call publishes a perform.hot_cell Active Support notification with the operation, outcome, cause, bytes in and out, and timing. A dead cell shows up here too, as unavailable.
Metrics from the supervisor. The app asks the cell’s control socket for metrics like queue depth, running workers, and kills by cause.
Logs from the supervisor. One JSON object per event on stdout, so you can see when a worker was killed and why.

In the talk, wiring those into your app was left as an exercise using our examples. In v1.0 they ship in the gems:

HotCell::LogSubscriber, in hotcell-client, writes one line to the Rails log for every call.
The yabeda-hotcell gem records Yabeda metrics for every call, plus gauges scraped from each cell’s control socket. Add the gem and call Yabeda::HotCell.install!.
HotCell::HealthController and HotCell::DiagnosticsController, in hotcell-client, give you a public health check and an authenticated diagnostic endpoint that does a real round trip through the work socket.

The Hot Cell docs list the alerts we recommend.

This is the dashboard we use for Basecamp.

The cost

In the talk, I said Hot Cell costs about 8ms per call in our environment. Since then, we’ve brought that down: in Basecamp’s production environment, the overhead is now about 3.4ms per call.

That’s a cheap price to know you won’t be owned the next time a zero-day is found in an image library.

Coloring outside the box

Hot Cell isn’t only for Active Storage:

You can run multiple cells per host, which gives you multiple queues with different depths, timeouts, and deadlines.
An operation can take multiple input and output files.
You can bring your own container. A conformance test tells you whether it’s configured properly.
Every operation’s limits are configurable.

A custom operation is shaped like an Active Job. It lives in hotcell/operations/, has a routing name and its own default limits, and does its work in perform:

# hotcell/operations/extract_text.rb
class ExtractTextOperation < HotCell::Operation
  operation "extract_text"
  limits deadline: 30.seconds, memory: 1280.megabytes

  def perform(inputs, outputs, format:, pages: [])
    source, = inputs
    destination, = outputs

    complicated_image_manipulation(source, destination, format:, pages:)
  end
end

In the app, a thin client class names the cell and the operation. Where a job has perform_later, a client has perform_in_hotcell, a blocking call that serializes the arguments, passes the descriptors, and returns the operation’s result:

class ExtractText < HotCell::Client
  hotcell "documents"
  operation "extract_text"
end

File.open(pdf_path, "rb") do |source|
  File.open(text_path, "wb") do |destination|
    @result = ExtractText.perform_in_hotcell(source, destination, format: "txt", pages: [1, 2])
  end
end

Opening the files yourself is the only hoop to jump through, and usually the class you write hides it.

What production taught us

HEY and Fizzy are relatively modern apps that use vanilla Active Storage, and porting them was no sweat.

Basecamp was harder. Basecamp predates Active Storage (Active Storage was extracted from it), and a lot of its attachment code never moved over, so I had to write custom operations. Basecamp has 12 of them in its cell so far. Basecamp also uses attachments heavily; for some reason our customers love sending each other memes.

We had outages along the way, and I rolled the lessons back into the library:

The OpenMP thread pool. ImageMagick, and the libraries libvips delegates to, size their OpenMP thread pools from the host’s core count, not the container’s cpus quota. On a 98-core production host that’s 98 threads at 8MB of stack each, which blew through the worker’s memory limit, and 285 workers died. The fix is to set OMP_NUM_THREADS and OMP_THREAD_LIMIT in the image, forwarded to every tool the cell runs.
The scratch disk filled. ImageMagick wrote its pixel cache to /tmp, outside the per-request directory, and cleans it up only on a clean exit. Every killed worker left its cache behind, until the 4GB scratch filled on six hosts. While it was full, 3,393 files were marked permanently unreadable. The fixes were to point TMPDIR and MAGICK_TMPDIR at the request’s directory (hotcell#51), empty the scratch at boot (hotcell#52), and size ImageMagick’s limits to a worker’s share of the scratch.

Allocate some time for tuning. Size file_size and the deadlines from what your real uploads take, then watch killed by cause. The tuning guide covers how.

What’s next

At Rails World I said Hot Cell wasn’t 1.0 yet because we were waiting for Rails 8.2, and because I wanted to ship more of the observability features. The observability features are now in the gems.

I’m still interested in Linux Landlock. Landlock lets a process ratchet down its own permissions so that it can never regain them, even if an attacker takes it over. That’s a nice belt-and-suspenders addition, and if you know Landlock, I’d love to talk.

Part 4: The moment we are in

Is this AI’s fault? Well, it’s easy to assign blame, and you’d be forgiven for doing so. AI chained several low-severity flaws into a critical-severity attack, and it made an effective embargo impossible.

But I want to suggest a more nuanced view. I’m seeing things trend in a good direction, and I want to be optimistic. I don’t want Terminator, I want Star Trek. The alternative is that we all quit our jobs, turn off our computers, and become sheep farmers (I guess!?).

That HackerOne chart I showed you to scare you is the system working as designed. People are finding vulnerabilities and reporting them responsibly, which is exactly what we want. It’s frustrating to deal with, but there are a finite number of vulnerabilities, and the number has to come back down at some point.

And when I said 378 vulnerabilities were found this year, what I should have said is that 378 vulnerabilities were fixed. Responsible maintainers are fixing the problems and shipping updates, and those libraries are trending in the right direction.

Here’s what you can do to help us get through this faster:

Vulnerability reports should come with their own agent skills. After the CVE, I published rails/rails-forensics-CVE-2026-66066, extracted from my investigation of our own apps at 37signals. Point your agent at it and at your application, and ask whether you were vulnerable (probably) and whether you were exploited. It will go through your Active Storage records and tell you whether you need to rotate your secrets.
Attack your own work. While building Hot Cell, I ran an adversarial review on almost every commit, and Jeremy helped me red-team it by setting agents loose to break in. They found real gaps. That doesn’t make Hot Cell perfect, but it has fewer problems than it would have.
Improve or replace existing systems. The only way through this period is for software to get better, or to be replaced. Hot Cell, baby!

Don’t be too scared. Be proactive and constructive, and the future will be Star Trek, not Terminator.

Live long and prosper, and go try Hot Cell.

Warming up the Puma master before it forks

Lewis Buckley — Thu, 1 Oct 2026 17:00:00 +0000

Basecamp 5 runs on Puma in cluster mode: one master process with preload_app! and 63 single-threaded workers per host, deployed as a Docker container with Kamal.

We serve Basecamp from several sites. Each site has its own web hosts and a read replica of the database, and writes go to a single primary database in one of them.

On our busiest hosts, each deploy left up to 2,000 requests waiting while the new workers warmed up. We reduced those queues by running signed-in requests through the app in the Puma master, before it forked the workers.

Why 63 single-threaded workers?

Basecamp has always served web requests from processes rather than threads. It ran on Unicorn, which only does processes, until we moved to Puma in January 2025, and we kept the same setup:

workers (Concurrent.physical_processor_count * 1.3).ceil
threads 1, 1
preload_app!

On a 48-core host that’s 63 workers, each handling one request at a time.

We chose 1.3 after benchmarking HEY in 2023, when we moved our apps out of the cloud and onto our own hardware. We tested several combinations of workers and threads with a mix of GET and POST requests on a 32-vCPU VM. Every multithreaded configuration we tested was slower and handled fewer requests than single-threaded workers. Adding workers beyond about 1.2 to 1.3 per vCPU brought little benefit.

The threaded workers spent a lot of their time waiting for Ruby’s global VM lock. That made single-threaded workers a good fit for this workload, and we use the same setup for Basecamp. An app that spends more time waiting on its database or other services may benefit from more threads, so benchmark your own app.

The other reason is the app itself. Basecamp has class-level state in places and has never needed to be thread-safe. With one request per process, it still doesn’t.

Processes do use more memory than threads, and preload_app! reduces the difference. The master loads the app once and the workers share its memory through copy-on-write until they write to it. Shopify’s comparison of Ruby execution models explains the trade-off well. In the HEY benchmark the best setup came to about 260 MB of PSS per core, where PSS counts each shared page once, split between the processes using it, and the gap to a threaded setup was smaller than we’d expected.

Two things about this setup matter for the rest of the post. A worker that’s compiling or loading something is fully blocked — there’s no other thread to pick up the next request. And whatever the master has in memory before it forks, all 63 workers share. Whatever they build after the fork, they build 63 times.

What happens when we deploy

Kamal starts the new container alongside the old one, and kamal-proxy moves the host’s traffic across as soon as the health check passes. At that moment, the new workers have handled health checks but no customer requests.

preload_app! means the master loads the app once and the workers inherit it through fork. That covers the code. It doesn’t cover anything Ruby and Rails set up on first use:

YJIT compiled code. YJIT compiles a method once it’s been called a certain number of times. The master calls very little during boot, so every worker compiles the same methods again on its own first requests.
Compiled templates. Action View turns each ERB template into a Ruby method the first time it’s rendered.
The schema cache. Active Record reads each model’s columns from the database the first time that model is used.
Inline caches and memoized values throughout Ruby, Rails and the app.

All 63 workers did all of this at once, while serving the traffic the old container had been handling a second earlier. In the test environment with YJIT on, the first request to a project page on a cold process took 652 ms, 151 ms of it YJIT compiling. The same request to a warm process took 28 ms. In production, CPU time per request peaked at around 200 ms while kamal-proxy moved traffic to the new container, against about 30 ms once the workers had warmed up.

A host with spare CPU absorbs this. Every one of our web hosts has 48 cores and 63 workers, but each Amsterdam host serves around 250 requests per second, against 25 to 60 at our other sites. In Amsterdam the slow first requests turned into a queue. At a peak-hour deploy, the Puma backlog on an Amsterdam host reached anywhere from 250 to 2,238 requests, and kamal-proxy’s p99 response time hit about 10 seconds.

Eron, our Director of Operations, had been tracking this since June. Another server in Amsterdam would help, but it would take weeks to arrive, so we also wanted to make deploys cheaper on the hardware we already had.

What didn’t work

We tried a few things first. In June, Donal tested the first two on a single Amsterdam host, comparing it with its neighbors, and they ruled out two likely causes.

Warming each worker’s database connections. Puma’s before_fork hook clears the master’s connections, and each worker opened its own on its first request. Opening them in before_worker_boot instead made no difference. Queries on a freshly booted production host were already under a millisecond, so connections weren’t the problem.

A synthetic request in each worker. Next, each worker made a few requests in before_worker_boot to an internal controller that touched every model. That ran the middleware, routing and Active Record paths, but it ran them in 63 workers at once — exactly the CPU spike we were trying to avoid. And a request with no real data renders no real views, so most of the app stayed cold.

Spreading YJIT compilation out. Delaying YJIT in each worker by a random interval spread the compiling out over a few minutes, but every worker still ran interpreted until its delay ended. The queue didn’t change.

Reforking from a warm worker. This is what Shopify’s Pitchfork does: let one worker serve traffic until it’s warm, then fork the others from it. Puma has an experimental version called fork_worker, and on beta it worked — the reforked workers were warm after three to five requests, where fresh ones took up to 30 seconds. But with fork_worker the template is worker 0, and it keeps serving requests. If it exits, the workers waiting to be forked never start (puma/puma#3596). If it gets no traffic, the refork never happens, which is what we saw on beta. Instacart have a mold_worker patch that promotes a warm worker to a template that stops serving, but it isn’t in a Puma release. We have a branch of it, and we may come back to it.

That last experiment did show us where the fix was, though. Everything a warm worker has that a cold one lacks is in its memory, and fork copies memory. The master already has the app loaded. It just never runs it.

Run the requests in the master

So now, before the master binds its socket and forks, it makes the app’s own requests, in-process, the way a signed-in user would.

Rack has a hook for exactly this. Rack::Builder#warmup takes a block that’s called once with the built app, before the server starts. rails server builds the app from config.ru, so the change to boot is one line:

require_relative "config/environment"

warmup { WarmUp.configured.run } if ENV["WARM_UP"]

run Rails.application

With preload_app! this runs in the master, and the workers inherit whatever it did. Puma binds its socket after the app is built, so until the warm-up finishes the health check’s connection is refused and kamal-proxy keeps retrying. No request reaches a worker that hasn’t been warmed.

The warm-up has three steps. After precompiling the views, it gives the page requests and schema loading a shared 20-second budget, checked before each page or model.

1. Precompile the views

actionview_precompiler reads every template for its render calls and compiles each one with the locals it’s passed. For us that’s 1,394 templates in about two seconds. A first request to a project page then compiles 2 templates instead of 44.

2. Request the pages, signed in

A small browser class makes the requests through Rack::MockRequest, with the two cookies a real sign-in sets, then goes back for each page’s lazy Turbo frames:

class WarmUp::Browser
  def initialize(signed_in_as:)
    @client = Rack::MockRequest.new(Rails.application)
    @headers = {
      "HTTP_USER_AGENT" => "Basecamp warm-up",
      "HTTP_COOKIE" => cookie_for(signed_in_as),
      "bc3.warm_up" => true
    }
  end

  def visit(path)
    page = get(path)
    frames_in(page).each { |id, src| get(src, "HTTP_TURBO_FRAME" => id) }
  end

  private
    def get(path, headers = {})
      @client.get("https://#{host}#{path}", @headers.merge(headers))
    end

    def frames_in(page)
      Nokogiri::HTML5(page.body).css("turbo-frame[src]").map { |frame| [ frame["id"], frame["src"] ] }
    end
end

The requests are signed in. The user is a monitoring account we already use for automated checks, and the pages are its own project, Campfire, to-dos, documents and messages. Public pages weren’t enough: after warming up with signed-out pages only, the first signed-in request to the projects page still took 131 ms, because authentication, the signed-in controllers and their views had never run. With signed-in pages it took 40 ms. cookie_for writes the same signed cookie the sign-in controller does, using the app’s own cookie jar, so there’s no API token and no secret to store.

The frames are followed. The busiest HTML requests in production aren’t pages at all but Turbo frames — the sidebar badge, the inbox, the navigation menus. The browser parses each page and requests its <turbo-frame src> URLs with the Turbo-Frame header, so those controllers and views get warmed too. Our first four pages turned into 60 requests.

The requests are excluded from rate limiting. They are internal, so they do not count against the rate limits that apply to real visitors.

3. Load the rest of the schema

The page requests load the schema for the models they touch. The last step loads the rest, from the read replica:

ApplicationRecord.reading do
  models.lazy.take_while { time_left? }.each { |model| model.load_schema if model.table_exists? }
end

The step checks 261 models and loads any schema information still missing. Those database round trips add up when the primary is far away: outside a request, Active Record uses the writing role, and from a host a long way from the primary each round trip is tens of milliseconds. Reading from the local replica brings the step down from about 20 seconds to 3.5. The pages go first because they load most of the schema anyway. If the time budget runs out, the step stops, logs how many models it got through, and the workers load the rest on first use like they always did.

Rails can also load the schema from a dumped cache file at boot (bin/rails db:schema:cache:dump), which would make this step unnecessary. We don’t ship one in our image yet, because the dump needs a database to read from at build time, and we have several databases to cover. It’s on the list.

What to close before the fork

Running requests in the master opens things the master never opened before, and every worker inherits them. Two processes writing to the same socket will corrupt each other’s traffic, so you need to know what’s open before you fork.

The way to find out is to list the master’s open file descriptors — ls -l /proc/<pid>/fd — before and after a warm-up, in an environment set up like production. Development wasn’t enough for us: it stores files on disk, so our S3 connections only showed up in production. Then, for each thing that’s open, check how its library handles a fork. We found three kinds:

Already handled. Plenty of libraries detect a fork on their own, either by recording the PID they connected from and reconnecting in the child, by opening per-process files, or by resetting their thread pools. Redis clients, metrics libraries and concurrency libraries tend to be in this group. Check, but you probably don’t need to do anything.
Already closed. Database connections are the classic one, and most Puma configs already clear them in before_fork. Anything else that’s opened per process — we have a SQLite cache the workers open on boot — needs closing when the warm-up finishes.
Needs a new step. HTTP clients with keep-alive connections are the ones to look for: cloud SDKs with connection pools, tracing exporters, error reporters. They usually have no fork handling at all. We empty the aws-sdk connection pools in before_fork, and we run the warm-up untraced so the OpenTelemetry exporter never opens its connection to Tempo in the first place.

Once that’s done, before_fork finishes with Process.warmup, which Ruby 3.3 added for this purpose: a major GC, a heap compaction, and every surviving object promoted to the old generation, so the memory pages the workers share change as little as possible afterwards.

Choosing the pages

The first list was the four pages that ran the busiest requests on beta. Once the warm-up was live, production showed us which endpoints were still cold. For one deploy, we compared each endpoint’s mean duration in the six minutes after kamal-proxy moved traffic to the new container with the same endpoint an hour later, then multiplied the difference by the number of requests in those six minutes. That gives the extra time each endpoint cost us because it was cold:

Endpoint	Cold	Warm	Requests in 6 min	Extra seconds
Campfire	246 ms	70 ms	6,490	1,140
Projects (JSON API)	84 ms	50 ms	22,077	771
Docs & Files	262 ms	177 ms	4,996	421
To-dos tool	205 ms	113 ms	4,018	371
To-dos (JSON API)	33 ms	16 ms	18,738	320

The pages already in the warm-up showed what to expect: the project page kept a 36 ms gap after a deploy, and the to-do page 10 ms. We’ve proposed adding these five requests, and expect them to add about five to seven seconds to the page step. The two JSON endpoints were a surprise. The warm-up’s page list had no API requests in it, so nothing on the API path had run before the first real request: not the API controllers, and not the Jbuilder templates rendering real records. Precompiling the views covers JSON templates too, but it isn’t a substitute for running the request.

Results

The warm-up is on for all 68 web hosts. With the first four pages it took 12 to 16 seconds per host: about 2 seconds to precompile the views, 7 to 9 for the 60 requests, and 3.5 for the schema. Deploys take that much longer per host, and we raised the deploy timeout from 30 to 60 seconds to cover it.

In Amsterdam, at a peak-hour deploy:

During deploy	Before	After
Peak Puma backlog per host	250–2,238 requests	19–223 requests
Peak kamal-proxy p99	about 10 s	2.4–4.8 s
Peak CPU time per request	201–214 ms	88–132 ms
Peak database time per request	56–69 ms	39–47 ms

The deploy in the middle, with four hosts warmed and four not, shows why every host needed the warm-up. Each warmed host recovered faster on its own: mean request duration peaked at 130 to 173 ms, against 203 to 311 ms on the hosts that weren’t warmed. But the backlog on all eight was about the same, because they were all waiting on the same database.

Memory came down too. The workers now share compiled templates, YJIT code and the schema with the master instead of each building their own copy. On beta, the view precompiler alone took a busy worker’s private memory from 174–202 MB to 119–135 MB. Thirty minutes after the deploy, the web containers used about 39 GB less memory than the previous day’s containers at the same age and traffic. Amsterdam served most of our traffic at the times we tested. In Amsterdam, each new container used about 2 GB less just after traffic moved to it, which lowers the peak while the old and new containers overlap.

Working with Claude

Claude Code helped throughout. It combed through the per-worker backlogs and per-endpoint timings in Prometheus and Loki after each deploy, worked out the cold-versus-warm cost of each endpoint, and prepared the changes and the pull request descriptions with the benchmarks in them. We decided what to try, deployed it and read the results.

If you do this

Warm the master before it forks. Compile common code and templates and load their schema in the master, so workers inherit that work. With preload_app!, Rack::Builder#warmup runs before the workers start accepting traffic.

Use the app’s real requests. Public pages, internal endpoints and synthetic queries warm the paths they run and nothing else. Signed-in requests to real records, frames included, run what production runs.

Measure the cold penalty per endpoint. The difference between an endpoint’s cold and warm duration, times its request count after a deploy, ranks the pages worth adding. Ours weren’t the ones we’d have guessed, and two of them were JSON.

Check what the warm-up leaves open. List the master’s file descriptors after a warm-up and account for every one before the fork. Two of ours needed changes.

Set a time budget. A warm-up that runs long on one slow host fails the deploy on that host. Ours gives the page requests and schema loading a shared 20-second budget, checked before each page or model, puts the most valuable pages first, and logs what it skipped.

Reforking from a warm worker, as Pitchfork does, solves the same problem continuously rather than once at boot, and it would warm paths no fixed list of pages covers. We may still get there: our branch brings Instacart’s mold_worker up to date with Puma’s main branch and fixes the bugs we found in it. But warming the master works with the Puma we already run, took a few days to implement, and substantially reduced the queues after deployment.

Lexxy 1.0 is here

Jorge Manrubia — Mon, 28 Sep 2026 15:21:52 +0000

Today we are releasing the version 1.0 of Lexxy. Lexxy is a rich text editor for Rails built on Lexical. It already powers Basecamp, Fizzy and many others, and it will become the default editor in Rails. I recently presented it in Rails World (slides, video coming soon). This is the article version of my talk.

Trix hit a wall

Trix has been our editor since 2015, and every Rails app’s editor since Action Text shipped in Rails 6. It’s small and reliable, and it has served millions of people for a decade. But in the last few years our customers kept asking for features like tables or code highlighting, and we kept struggling to deliver them. The reason is the Trix document model.

A Trix document is a flat list of blocks. A block is a line of text with some labels attached, like quote, bullet list, bullet. There is no tree, and a block can never contain another block. Nesting is an illusion: at render time, adjacent blocks with the same labels get wrapped together.

That design bought a lot of simplicity, but you can’t express something like a table with it. Two cells next to each other would carry exactly the same labels, so Trix would merge them into one. The model can say “this bullet is one level deeper”. It cannot say “this cell is different from the cell next to it”. A tables issue has been opened since 2015! The model just can’t do it.

The second problem was maintenance. An editor built on contenteditable behaves differently in every browser and even changes from time with operating system releases. In 2024, three iOS releases in a row broke typing, dictation or the caret in Trix, and each one cost us real effort to work around. Check this one as an example.

Why Lexical

This was a conversation we had at 37signals for years:

2022. We started a project to add tables to Trix. We gave up after a week. The document model can’t represent two-dimensional things.
2023. I built a proof of concept with Tiptap inside HEY. We liked it, but we never started a serious project with it.
2024. We built House, our own Markdown editor, for Writebook. Not WYSIWYG, but WYSIWYM: what you see is what you mean. A wonderful editor for long-form writing like books or technical documentation.
2025. We tried House in another product, and it didn’t fit. For most apps, WYSIWYG was just the right answer.
2025. We had the discussion again, and this time we looked at the whole field.

David ruled out Tiptap, CKEditor and the other commercial editors: an open source core with features kept proprietary, and a sales team behind them. We didn’t want our editor to depend on somebody else’s licensing decisions.

Then we found Lexical: MIT, from Meta, and very powerful. I spent two weeks evaluating it, and in May we made the call to go with it.

Lexical’s core has zero dependencies and weighs forty-two kilobytes. In a way, it validates the approach that Trix pioneered. The document is an immutable state you never mutate directly, you just get new snapshots when performing updates; contenteditable is an input device and a rendering surface, never the source of truth. It has a DOM reconciler to update the actual DOM very efficiently, and other primitives to deal with handling commands and node transformations. Everything else, from lists to tables to markdown, is a package in the orbit. Lexxy uses thirteen of them.

Lexical solved the maintenance problem too. Meta’s products like Facebook or Instagram use Lexical and their user count is in the hundreds of millions. This means that even small issues with new keyboards and devices are fixed fast by the dedicated Meta team that maintains it. Furthermore, Meta’s business is the products, not the editor. We much rather liked this structure of incentives for the long-term investment an editor represents.

The iceberg

The plan was simple. Pick Lexical, wire it up to Action Text, add a toolbar and ship it. Well, it didn’t go exactly like that.

Lexxy today is thirteen thousand lines of vanilla JavaScript on top of Lexical. A great editing experience is very hard to get right. An editor is a machine where the user can change the state in a thousand different ways, and the possible permutations are countless. For example. you have two images one after the other and want to put the cursor between them, but there is nothing there to put a cursor in. Or somebody pastes from Google Docs, and you have to turn a pile of inline styles and empty spans into clean markup. And then Safari, and Android keyboards, and the clipboard, and undo, and…

The first pull request landed in May 2025. Fizzy launched with Lexxy in December, and Basecamp 5 in May this year. Basecamp was the real test: twenty years of content written with Trix, and people who use the editor all day, every day. Zoltán Hosszú and Samuel Péchèr were the key people who made this happen. The took a very green version of Lexxy, added a ton of features (including Tables) and polish, and they fixed countless bugs. They also pulled off a remarkable milestone: seamlessly switching millions of Basecamp users from Trix to Lexxy.

What’s included?

We didn’t want a to build Trix with tables. We had Lexical and we had agents to help, so we wanted to be ambitious here. We went for the whole package.

Features

In terms of major features:

A color highlighter, built in instead of this being a custom Basecamp extension, as it was with Trix.
Tables, with an interface we worked hard to keep simple and accessible.
Markdown. You type it, you get rich text.
Code blocks with syntax highlighting as you type, in more than twenty languages.
Image galleries you can navigate and reorder with the keyboard.
Prompts. Type a character, get a menu: mentions, emoji, or whatever your app needs.
Links by pasting a URL over selected text.
Previews of attachments like videos and PDFs, rendered as your app renders them.

Action Text Native

Lexxy is also Action Text native. Action Text stores attachments in a canonical format that Trix doesn’t speak, so it translates on save and again on render. We taught Lexxy to emit exactly that markup. What you see in the editor is what gets saved, and what gets saved is what your app renders. Your existing content, attachments and views keep working.

That opened another door. Action Text now talks to an editor adapter, with an implementation for Trix and one for Lexxy, so switching is one line: config.action_text.editor = :lexxy. Here the credit goes to Sean Doyle, who started that pull request before Lexxy existed and took it to the finish line with our input. It ships with Rails 8.2, and we hope other editors will use it too.

Extensibility

And you can extend Lexxy. Extensions are built on Lexical’s own mechanism, and this is not a second-class API: Lexxy itself is thirteen extensions, tables included, and Basecamp has nine more.

class MyExtension extends Lexxy.Extension {
  get enabled() { … }

  get allowedElements() { … }

  get lexicalExtension() {
    return this.defineExtension({
      name: "my-extension",
      nodes: [ … ],
      register(editor) { … }
    })
  }

  initializeToolbar(toolbar) { … }

  dispose() { … }
}

Lexxy.configure({ global: { extensions: [ MyExtension ] } })

My favorite of how extensible is Lexxy are voice notes in Basecamp: you record, you see the waveform while you talk, and it becomes a player inside the document. About a thousand lines, without forking or patching anything.

Performance

Lexxy is fast, because Lexical is fast. In a ten thousand word document, Trix takes thirty-eight milliseconds to process a keystroke. Lexxy takes four. Above fifty milliseconds, the editor starts feeling sluggish. Compared to Trix, Lexxy brought a whole new performance regime.

Accessibility

Accessibility in Trix was not great. In general, building accessible experiences for rich text editors built on top of contenteditable is quite hard.

We had a dream team to help with Lexxy accessibility. Bruno Prieto worked with Michael Berger, our accessibility champion at 37signals, to bring the bar to where we wanted it to be. Bruno is an outstanding programmer who happens to be blind, so he knows one thing or two about accessibility, and he delivered.

As a result, in Lexxy everything is reachable with the keyboard. The editor announces itself properly to screen readers, and it gets a thousand details right so that the editing experience using a screen reader is fantastic. You can learn more about accessibility in our docs.

Security

The latest AI models have resulted in an unprecedented explosion of vulnerabilities found, and we took this thread quite seriously.

Lexxy counted with programmers of the caliber of Jeremy Daer and Mike Dalessio helping to make it more secure. We have put a lot of attention to sanitizing the editor contents, validating attachment URLs and making sure that the types of attachments and nodes the editor support are allow-listed. Lexxy also comes with preliminary Trusted Types support, to offer CSP-level control over certain DOM manipulation APIs. The trusted types policy is there, but we are not enforcing it everywhere yet.

Agents

We started Lexxy using Claude since day one. A main lesson was that an agent needs to drive the editor like a user does. The best decision we made in this project was moving the system tests from Capybara to Playwright: three browsers instead of one, a suite that runs in seconds, and a much more faithful clipboard, keyboard and focus. This represented a tremendous improvement in how agents could close the loop by themselves. Write a test, see it fail, fix it, see it pass. We have more than six hundred browser tests today.

With a solid testing foundation in place, we could start fixing bugs in large batches. As mentioned, getting a text editor right implies a ton of work, and the kind of backlog we got at some point would have have buried us in pre-agent times. We also used agents to validate the fixes: an agent reproduces the bug in the public Lexxy sandbox, checks that it’s gone with the branch applied, and labels the pull request.

Agents were essential to get Lexxy done with the people and the deadlines we had: we are a small company, and the same people were shipping two products in parallel.

The new Rails default

We believe Lexxy is the best rich text editor out there right now, and we are going to make it the default in Rails next.

If you’re starting a Rails application today, use Lexxy. If you’re using Action Text with a standard configuration, switch. It’s one line, and we’ve worked hard to make it seamless.

The ONCE app server

David Heinemeier Hansson, Kevin McConnell, Kimberly Rhodes, Fernando Olivares — Fri, 17 Apr 2026 17:00:00 +0000

In this episode of RECORDABLES, we dive into the thinking behind the ONCE open source app server and what it takes to turn self-hosted software into something anyone can run. David Heinemeier Hansson and Kevin McConnell walk through the shift from single-app installs to a console-like system that can run multiple applications on one machine, and the philosophy that guided that evolution.

Along the way, they explore the tradeoffs between simplicity and flexibility, the role of Docker and Kamal in making deployments seamless, and why ease of use matters more than raw capability. This conversation is a look at packaging complexity into something approachable and how ONCE aims to make running your own software feel fast, fun, and accessible.

Watch the full video episode on YouTube.

Timestamps

00:00:00 — Introduction
00:01:06 — What ONCE is and why it became open source
00:03:42 — Running multiple apps on a single machine
00:10:16 — Why single-machine deployments work today
00:17:47 — Balancing simplicity with flexibility
00:26:11 — Installing apps in seconds
00:36:47 — When to use ONCE vs. Kamal

Links & Resources

ONCE GitHub Repo

Transcript

Episode Highlights (00:00:00): The vast majority of people, they want something that’s convenient. They want something that’s quick. They want something where they can get the value without all the hassle. Making it a little bit more flexible is really useful, but not losing sight of that super targeted flow was really important. Making something that’s possible, accessible, easy, fun, aesthetically pleasing, that’s the mission we’ve been on with ONCE.

Kimberly (00:27): Welcome back to Recordables. This is where we’re sitting down with the programmers from 37signals to hear a little bit about what they’ve been working on behind the scenes so you can learn a few things and learn from some of our mistakes. I’m Kimberly, your host, joined by my trusty co-host, Fernando. Hello, Fernando.

Fernando (00:41): Hello, hello.

Kimberly (00:42): And then we have back with us, Kevin McConnell, who’s been with us on Recordables before, and then a special guest. You know him, David Heinemeier Hansson. Our team has recently released a new app server. We’re going to talk about it today. It is called ONCE. You’ve heard that name before. We’ve rebranded a couple of things, so we’re going to dive into all those details today. David, why don’t you kick us off on what this new open source tool is?

David (01:06): Sure. So this is a continuation of all the work we did originally to launch the ONCE brand and the applications that sat under it, like Campfire and Writebook. And those applications were intended for the ONCE business model where we would sell you a piece of web application software that you could install and run on your own machine. And it worked pretty well, but it wasn’t kind of the rocket ship that it needed to be for us to continue in that direction as a commercial branch of what we do here at 37signals. And instead we thought, you know what? We’ve recouped the investment that we did into Campfire. We had already released Writebook just as a free tool under this setup, but there are factors in how you run these services or how you did run them that weren’t the best if you wanted to run multiple.

(02:02): So we thought, you know what? Let’s just make all of this open source. Let’s just make all of it free. And then let’s really upgrade the platform itself that these applications run on so that you can run all of it at the same time on a single machine. Because when we first launched Campfire, it was really easy to install. You got one bash command that you’d paste into your terminal and it would set everything up, but then it would kind of dedicate that machine or that VM if you were hosting in the cloud to just that application. And getting anything running next to it was kind of cumbersome. So we actually already encountered this feedback when we launched Writebook because there were a bunch of people who were already running Campfire and who liked to run Writebook as well. And we kind of had to tell them, “You need a second machine.

(02:52): You need a new VM.” Which I don’t know, for us who are very familiar with this stuff, perhaps wasn’t the biggest deal in the world, but it was a real blocker for a ton of people. So we thought now that we’re releasing all this stuff as open source and that we ended up releasing Fizzy under the same paradigm, it had a dual license, the O’Saasy license where we allowed you to do basically whatever you wanted on your own installation as long as you didn’t compete with our SaaS setup, but it functioned the same way as Campfire and Writebook. And now you really have three applications that someone might want to run and you’re not going to ask someone to use three machines to dedicate to that. That’s wasteful, it’s cumbersome, all of that. So as we were moving into that direction of making all this free and open source, we thought, you know what?

(03:42): We could do better. And one of the reasons we could do better is because we’ve already built many of the underpinnings that we’re using for ONCE for our commercial products. We built something called Kamal, which is the main deployment tool that we used to get out of the cloud. Where in the cloud, we were running some Kubernetes and all this other stuff that was just way too complicated. So when we moved Basecamp and HEY and all the other services onto our own machines, we built something called Kamal. And Kamal has kept growing over the last few years. And one of the things that it has gained is something called Kamal Proxy. And Kamal Proxy allows us to do a lot of the really nice things you need for commercial services like deploying a new version of it without taking the application down, not losing any in-flight requests.

(04:35): And we thought, you know what? That piece, that proxy would fit really well if we could put it into the ONCE setup such that when you’re updating any of these applications, now there are multiple of them on your setup, you also get that nice no downtime deployment. But just as important, you can run multiple applications on a single machine. So that was really the impetus to this. But even broader than that, I had from the beginning with ONCE, once we kind of corralled around this vision of turning it into complete open source deployment setup, that it should work more like Nintendo, like an old Nintendo with cartridges where you could have multiple cartridges. The older version of Nintendo was you bought Donkey Kong on this handheld and that was the one game. And Donkey Kong is a phenomenal game, but at some point you’re probably going to get bored with that.

(05:32): And well, here’s a piece of hardware that only works with that. The Nintendo, Super Nintendo systems and even the original NES, they all have cartridges. You could change them out and you could plug in a new game. And what I really loved about that setup was just the simplicity of it. Cartridge out, cartridge in, it just works. There’s almost nothing to it, but it is interchangeable. And I thought, can we bring that level of simplicity to something like web applications? Now, we’d sort of already done that with ONCE, but that was kind of like Donkey Kong and we wanted to do the full console experience with multiple games here, multiple web apps. So that was the drive, right? Thinking about this as a console experience, which actually when Kevin is going to show how it really works, it was a little bit of the inspiration too about the aesthetics.

(06:25): Can we make this feel like a bootscreen? Like you’re starting up a new game and it has some interesting visual effects to it. Can we lean something like a web app server into the direction of something that’s a little fun, something that perhaps blurs the line a bit between developers and consumers where tech savvy, not even that savvy, just marginally savvy individuals could actually use this, could actually run their own applications on it, even if they weren’t programmers. I think this is something we’ve seen now a lot with the AI revolution, that there’s suddenly a bunch of people who were not capital P programmers who are programming because we’ve reduced the barriers of entry. We’ve made it so easy, or these agents have made it so easy, to create everything from an iOS app, to a web app, to a Go App, to whatever app. So I think we’re getting more people in that direction.

(07:23): And then nicely enough too, that whole flow has been through its terminal. So Claude Code and OpenCode and Gemini CLI, they’ve all been tracking people who might’ve been a little unwilling at first into the terminal. And that’s basically all we’re asking anyone to do when they set up this ONCE web app server. We’ve even made it interesting point that by the time you set up ONCE, you can install all our three basic applications. So you can install Campfire, you can install Writebook and you can install Fizzy. They’re just selectable from a multi-select menu. Or just as importantly, in my little Nintendo metaphor here, those three applications are our first party. That’s our Mario Kart, that’s our Super Mario, that’s our Pilot Wings. But then you can also bring your own software. You can bring your third party Docker containers to the setup and they can run the same way right next to the three applications we’re shipping with.

(08:22): Anyone can make open applications, open source applications that are distributed this way, comply to a few small rules. Basically, I think we have two rules. One is your application has to answer on slash up or you’d have to configure where that is. If you answer on slash up, we know when your application is ready to serve, that’s very useful when you’re both setting it up in the first place. And then also when you’re doing updates that we can do that gapless deploy. And then secondly, that in almost all cases that it’s based on SQLite because we give you one directory called slash storage where you can put anything into and then it’s very nicely contained. So in that setup, it’s really designed for single computer deployments, which is also this trade-off. If you look at something like Kamal like I was talking about that we’re using to deploy Basecamp and HEY with, that’s deploying on dozens of machines at the same time.

(09:19): And there’s a lot of coordination, there’s a lot of auxiliary services. We need to have MySQL running, we have Redis running, we have all these other things running, which is great. Kamal is ideal for that kind of bespoke internal SaaS operations setup. ONCE is much more sort of the console stage. There’s just one machine that you dedicate to run ONCE on, and you can run as many applications on that as you can fit on a single box, but it is constrained to one box and the applications are constrained to one directory that they can save into, which means that in almost all cases they’re going to use something like SQLite. Now, the great thing about SQLite is it’s ideal for this purpose that these applications are meant for single user if it’s a personal application or single company deployments. So it’s not like you’re going to deploy Campfire and then run it for, I don’t know, a million different companies at once.

(10:16): No, when you run ONCE, each company has their own ONCE console. They click in the cartridge for Campfire and then they use it within their organization. And it just so happens that technology has gotten really good. Servers have gotten incredibly fast. Even tiny little mini PCs, let alone droplets or other micro VMs in the cloud have gotten so good that almost any size organization can run one of these applications even with thousands of people on that single box. So it’s kind of trying to take advantage of all these advantages that we’ve seen over just even the last few years. Docker is phenomenal for packaging this stuff up. We have counted technology with something like Kamal Proxy. Server hardware has gotten really good just that you can dedicate a single machine to it and that’s everything you need for even quite large deployments. The missing bit has been to make this so easy that almost anyone can do it.

(11:18): And when we were talking about this and originally said, no, I wanted to make it so easy in all the instances. I wanted to make it really easy if you just want to run this on your laptop. If you just want to run it for yourself, Writebook is a wonderful way of writing a book even if you’re the sole consumer of it. You can use Campfire with just a study group. You can do Fizzy just with you and your agents to coordinate work. These applications scale down really well to single individuals or tiny groups. But then of course, you can also take that and start it on your laptop and move it into the closet. You might have an old PC or something like that. You stick into the closet and suddenly you have all the firepower in the world. They can stay up all the time even when you close the lid on your laptop.

(12:06): And then at some point you might want to move this onto the cloud or a bigger service. So that’s also kind of the magic of the constraints around this slash storage directory because we’ve also built in backup and restore. So you can start an application on the laptop, do a backup, get it running in the closet, restore that backup, get it and run it on a cloud server if you want or move in any other direction that you care to. And I think that model of web applications that they are for single organizations fulfill a enormous amount of checkboxes that people are increasingly starting to care about. And a key part of it is you own your data. Now in the EU, for example, that’s become a really hot topic that lots of organizations actually can’t host their stuff on American servers or are feeling queasy about it and would prefer to have it on their own setup.

(13:06): And with something like ONCE, it’s completely trivial to do. The second thing is this software is yours forever. So if Fizzy, for example, 15 years from now is no longer operational because we as a business no longer are around, that Docker container can run literally forever. This is the other parallel of why I like this console metaphor. People are playing the original Atari Pong from 19, whatever it is, 76 or 79 or whenever that was from, they’re vintage machines. Now, if you go into the 80s, there’s tons of them, right? There’s tons of operational Nintendos and Segas and whatever that are perfectly fine and they still work. Do you know how many sort of SaaS style games today that were launched even five or 10 years ago that are no longer running because the company behind them went out of business and then everything that depended on the operational servers that’s just gone and then all this stuff disappears.

(14:08): And I think, do you know what? Web applications is such a great form of making applications, such a great platform, but it has been tied intrinsically to the SaaS paradigm that there’s some company that needs to be around and you need to be, in almost all cases, a continuous customer of that company, paying them every month or every year for the privilege to use those servers. And at any point you’re at the mercy of them going out of business, yanking the application, handing your data over to someone who asks politely for it with a summary judgment. None of those things are true with ONCE. So there are all these different factors that I find this is really interesting. Even if we couldn’t make the original ONCE business model work, maybe we were just too early and I want to make sure that we keep this medium, this approach alive because I think back of Campfire, it’s funny we launched that….

(15:04): Campfire in its SaaS version was launched in 2005 and we realized quite quickly that, oh my gosh, chatting with your own team, especially if you distribute is a complete game changer. Now, the market wasn’t ready for that at all. It wasn’t until Slack took the business world by storm in, what was that? 12 or 11 or 13 or whenever they really took off. And you’re like, do you know what? You can just be too early. I don’t want to give up on this. I think there’s a ton of really valuable properties to being able to run your own applications on a super easy to operate console-like platform, both the ones we make, the ones you might make, the ones anyone might make. And this could become a really powerful platform. And especially now that the cost of producing these new applications have really dropped.

(15:57): I did a screencast for introducing ONCE and during the screencast, I made two applications using AI. You did a HTML application that was just a snake game. And then I did an application that kind of didn’t work on the first go, well, written also in Go, but we can make it so easy now that the imagination of all these new people who weren’t capital P programmers can suddenly create these apps and let’s see where it goes. But that bit, having the console, I think is really a prerequisite because if it’s too complicated for you to share your vibe coded app with someone else, it’s not going to go anywhere. But if we make it really easy, and if we build it into things like Omarchy is going to ship with ONCE just pre-baked because it’s such a tiny little CLI setup, we can simply just have it as part of it and now there’s an install platform to use this.

(16:57): So anyway, this is why we built it. This is why we dreamed it up. This is why it has these influences. This is why the TUI interface forward looks really playful and fun and throwback to a Commodore 64 game. It’s drawing on all those kind of highlights of programming and our industry and trying to put it forward to something that could only happen today because all these moving pieces are now coming together, the hardware, the software, the packaging, the vibe coding, all of it.

Kimberly (17:32): Okay. Because Kevin, I know you worked on the technology for self-installing Campfire, Writebook, our original ONCE products. You’ve worked on this. Tell us a little bit about the challenges that you faced or anything that we should know about the changes with this version.

Kevin (17:47): Sure. Yeah. So some of the components are the same. Some of the ideas are the same. We were able to look at what worked well with that first Campfire version and kind of roll that into the next one. So the first version, as David kind of mentioned, the first version was we focused so much on how simple can we make this that we deliberately restricted a lot of the things that you could do. So there was only one path to go down. You went through one install flow, you told it one piece of information, the host name you were deploying to, and it would run, but that was kind of all you could do. And I think what we learned from that was that making it a little bit more flexible is really useful, but not losing sight of that super targeted flow was really important.

(18:33): So we kind of hung onto that idea. We still use some of the same sort of underlying components, like the way manages Docker containers to run is based on the same idea, but it’s just expanded a bit of flexibility. So where we used to just run a single container that contained the app and everything, and that’s all you would run on that machine. Now the tool basically makes us as its own Docker network and installs a proxy along with one or more applications that it can run. It sort of manages that group of things together. So it’s kind of the same ideas from the first one, but it’s just like leveled up a little bit to become a bit more flexible.

Fernando (19:11): That’s so cool. I think what’s really interesting about this is that we went from like, oh, the cloud, there’s this massive sprawling of dependencies things. And we’re like, no, let’s simplify that. And then out of that comes Kamal, then the obvious necessities of running Kamal, it’s like, oh, we need a proxy to just do no downtime deploys. And then we’re still like, you know what, this is great, but what if we can make it simpler? And out of all the tiny pieces and all the open source programs that we ship, we’re like, you know what, at some point it’s almost like Lego, which I find fascinating about ONCE this new platform. It’s like, well, we have this and we have this and we have this. We just plug it in here, you just run a Rails app with up and the other requirements and that’s it, right? Am I getting it right?

Kevin (20:03): It is. And also a lot of it is continuing that same convention over configuration idea that Rails has always done is like, rather than you have to set something up and say, “Here’s where I’m storing the data, here’s what you would need to do if you make a backup or if you need to restart it.” Instead, we just say, here’s a small set of conventions. If the apps adhere to those, then you don’t have to tell anything. There’s virtually nothing to do when you set it up because we know all the decisions, we already made the decisions. So I think that sort of philosophy continues into the design of this.

David (20:37): This is something I was really keen to focus on since there is actually already a fair amount of open source software out there that you can run on your own server. And usually when you go to check it out and you go to the install page, it’s like freaking three pages of this dependency and this piece of configuration and all this stuff. And I always thought, this seems unnecessary. There’s got to be a way where we can boil this down to just putting in, as we do now, we ask for your host name, where do you want to put it? You can put something in as simple as chat.localhost if you want to install Campfire on your own laptop. And then in that case, we even already know the Docker container because this is one of our applications. But if it’s another application, either a third party or something you wrote yourself, you just put in the Docker image URL and that’s it.

(21:30): Literally everything else is just taken care of for you. And some of this infrastructure that is actually quite important, you can do things like compose up, right? You can do your own Docker Compose file and then you go like, well, what if I need to upgrade it? Well, sorry. Well, I guess you could put Caddy in front and you could do something. Yeah, you could do a lot of things. All of this stuff was always possible if you wanted to dig the trenches yourself. Now, as I’ve found literally for 20 years, most people don’t want to dig the trenches themselves. They would like trenches just to appear the place where they want to have some water flowing. And as with Ruby on Rails, the magic is not necessarily just in making something possible. Almost everything is possible. This is something we say at 37signals, or I at least reply whenever Jason asked me or I wanted to design, “Can we do this?”

(22:28): I always say, “We can do whatever we want. It’s just software.” Literally everything is changeable. It’s a factor of how long is it going to take? How much concrete do we have to dig up if the architecture is really sticky in this area and in a way, everything’s possible, but that’s not how you get people excited about things. I mean, I had this exact experience with Omarchy. So Omarchy takes a bunch of components that already exist and just glues it all together. Anyone could have done that if they had 300 hours to spend on Linux by themselves. And lots of people did, right? That’s what that subreddit that I got hooked into our Linux point was all about. It was people who’d spend, if not hundreds of hours, then tens and dozens of hours on really perfecting their own system. And it’s beautiful when there are folks who are that engaged with the super nitty gritty of it and want to go real deep.

(23:28): It’s just a tiny niche. The vast majority of people, they want something that’s convenient. They want something that’s quick. They want something where they can get the value without all the hassle. So ONCE is trying to do that for something I kind of believe just should exist, like all this technology, Docker is a huge part of it. Without Docker, without this way of being able to containerize applications and bundle up all the dependencies, none of this would work. This is why, for example, we couldn’t have had ONCE in 2005 when Rails was first released. That was not going to be a thing because we did not have the technology. And then even when Docker first appeared in, I don’t know, was that early 2010s, computers were just not fast enough actually to run something like Campfire for a few thousand people on a single machine if you were going to build in these kind of technologies on a tiny box.

(24:20): So sometimes you just got to wait until all the little Lego pieces are ready, but then you also got to build the castle. You can’t just dump it all on people. Well, actually Lego is a bad example because I find it a lot of fun to put Legos together. I don’t know, a million piece puzzle. I can appreciate a finished puzzle. You’re not going to get me to put a million piece puzzle together. I don’t have 300 hours for that. I have 300 hours for putting a Linux puzzle together, but not an actual puzzle. So a huge part of this is just fiddling away all the little annoyances, all the little gotchas, all the things you’d have to document. This is one of the things that both Rails and Omarchy and now ONCE live under is this document driven development. If I have to write something down, we’ve done something wrong.

(25:11): I should not have to explain anything. You know what? When you unwrap that cartridge of Super Mario 3, you’re not reading through the manual. You’re like, here’s the slot, bam, jam it in. And if it doesn’t work, pull it out, blow on it for a second and jam it back in. It’s going to work. It’s going to boot. You don’t need to actually read anything. And that should be a model for software of all kinds. In fact, it’s been a model for our kind of software for a very long time. We didn’t use the cartridge metaphor. The metaphor we used to use was Fisher Price. It should be Fisher Price easy. Literally a two-year-old should be able to make this stuff work. And if it’s colorful, all the better. So getting all that stuff together, making something that’s possible, accessible, easy, fun, aesthetically pleasing, that’s the mission we’ve been on with ONCE.

Kimberly (26:06): Okay. We’ve talked about how easy this is. Kevin, do you want to show us actually how easy it is?

Kevin (26:11): Sure. Yeah. Let me just show you what it looks like to install an app or two with it. Okay. So if you can see my terminal okay here. So installing the tool is basically, there’s a command in the ONCE repo and on the website, we share this command with you that you just need to copy and paste into a terminal to sort of bootstrap the install. We have direct versions of this command for specific apps. So like Fizzy can have its own Fizzy specific link, which will take you straight to installing Fizzy, but there’s also just a general one for installing the tool, and then you can choose apps from there. So if you run that, that will pop up like the first running experience of what the TUI looks like with some of this rather 80’s aesthetic that David mentioned.

(27:02): So as we said, it knows about the three ONCE apps already that are built in. So if you want to install one of those, you can just pick one from a menu and go onto the next stage. If you want to install something else, you can pick this other option here, which would let you put in the path to any Docker image that contains a web app that would work with ONCE and you can install there. And one of the nice things about that, which we could probably talk about a bit more later is that this is also a path for you to take one of the built-in apps. And if you want to make your own changes, you can make your own fork of that app and just make a few changes here and there and publish the image up to your own path.

(27:40): And then you could use it in this sort of custom Docker image path to do your customized version of Fizzy or whatever. But to start with, we’ll just pick one of the built-in apps like Campfire. And then the only thing required piece of information that we have, much like with the old ones, is telling it what the host name is you’re installing onto. So if you’re installing onto a server on the internet, you would’ve set up a DNS entry that points to that machine, and that’s where you would type in here. And by doing that, we can make sure that we’ve got SSL certificates set up and everything and make sure the app’s accessible on that path. If you want to just run it local in your machine, you can just do a localhost address. So if I want to install Campfire chat on localhost, I can just put that in there and that’s basically it. It’ll go and fetch the container. It will set up the proxy, set up the Docker network and everything and land you back into this dashboard. And this dashboard is where you get to see all of the apps that you’ve installed and whether they’re currently running or not, how much memory and CPU they’re using.

Fernando (28:47): Wait, that’s it.

Kevin (28:48): That kind of thing. And that’s it. That’s all you need to do to install it. So these bits over here that list the app names are links. So I could click on that, which would launch the app that I just installed in a browser and you could start using it. You can also from this dashboard, as well as see how things are running, this is also sort of the central place where you can then carry out other operations or make changes to your app. So for example, if I wanted to just stop this app from running, if I wasn’t going to use it for a while, then I can use the menus in here to do that. So there’s a little actions menu that lets you stop and start or delete apps. If I wanted to install a second app, so let’s say I wanted to add Writebook as well, it would be just a case of selecting that, giving it its own name. It will run through the same dance. I guess I didn’t have this one downloaded locally, so it’ll just take a second longer, but you’ll see when this returns. So now we’ve got two apps running here side by side in the same machine because they have different names, the traffic to these will route automatically to the correct app.

Kimberly (29:56): Kevin, you said that took a long time or it took longer. Like that was really fast.

Kevin (30:01): Yeah, it’s all relative, I guess. I must have recently installed Campfire on this machine, which is why that one was like a second. Sometimes it might be a few seconds, but the whole process should always be really pretty quick. It’s a little bit dependent on your internet connection obviously, but these Docker containers are not typically massive. I think our apps are sort of in the hundreds of megabytes size normally, I think. So it’s basically the time it takes to download that is the slowest part of setting up an app. The whole thing should definitely be up and running under a minute.

David (30:37): What’s really nice about using Docker as the underlying mechanism here is that we get to rely on the Docker layers when we update. So if someone like us puts out a new version of Writebook and we update this books.localhost, you’re only going to get whatever layer changes, which is typically just the Rails layer. And those updates are even tinier. We’re not talking anything close to hundreds of megabytes to get that up. Docker is really efficient at sort of separating the things that change from the things that don’t change and then making those updates really quick to do. Which, Kevin, maybe you can show that just the calling for an update and just the fact that we actually do poll for updates too. So these are not just like a one and done install and you can never get any updates to the applications. These are somewhere between the cartridge and the steam game and the fact that they can actually get updates.

Kevin (31:35): Yeah. So there’s a bunch of background tasks that work to take care of all that stuff. And if you look in the settings menu from this dashboard, you can see some of the things that happen. So the updates part there that we’re talking about is that by default, this feature is checked on. And just by the nature of the fact that you’ve installed it, it knows that it should check periodically for those updates. So once every 24 hours, it will go do that Docker pull. If there’s changes, it will fetch the new version and then redeploy new version. As we were saying earlier, that’s like a gapless process. So there’s no downtime where it does that. It’s just a case of get the change Docker layers, reboot onto a new container that runs the newest version. You can also check anytime you want just by clicking that button.

(32:19): It’ll see if there’s anything there to get. There’s a similar kind of thing with backups. So if you want to have backups happen automatically, you can just specify a path on your machine where you want those backups to go to and then click on this option. And then once a day, it will make a backup of all the things that are in storage and all the app settings and everything like that. And put that in a file in that directory. It’ll keep the last 30 days. So you can just sort of set this on and forget about it, and you’ll know that you’ll always have a month’s worth of daily backups without having to do any other kind of maintenance on that.

(33:00): Some of the other things you can do is customize settings. So it’s all designed so that you shouldn’t … In the common case, you don’t really have to customize settings most of the time, but you might want to for some other reasons. So some applications might have special environment variables you could set to control how features in the app work or something like that. So we have a screen where you could enter some of those. If you want to do that, you can do things like for apps that need to be able to send emails, Fizzy is one of these. Fizzy use these emails as its way of logging in through Magic Links. So when you set up Fizzy, you’ll actually want to tell it where it can use an SMTP server to send email. So we’ve got a screen that you can pop open where you can put in the details of whichever SMTP service you’re using, which could be something you run locally, or it could be like a third party email provider that you use.

(33:57): Oh yeah, sorry. It’s got these application settings, which is basically the things that you entered when you first installed it. You can see that by installing … In this case, this was Writebook. So by picking Writebook from the menu, it knew that this is where it can get the official Writebook image. And so what I was saying earlier about being able to fork the apps if you want to make your own changes is that since all of these built-in apps are on GitHub and they’re open source, you can use GitHub to just create your own private fork, make some changes, commit those. And when you do that, the same CI setup that we have in that repo for building images, that will apply to your fork. And so you’ll end up getting your own version of this. So if I forked it, then I would probably have a path like this or something.

(34:47): So you can use this as a way to, a really low friction way to start making changes to some of those apps, try your own little customizations and just point your installed version of the app at your fork. And then when you apply that, it will go grab that container boot onto that with all your data. And it gives you quite a low barrier way to start trying out customizations or getting an AI to do some customizations perhaps. One other thing I was going to show here is just you can also use this to watch the logs of the running apps. There’s no traffic going to this one just now, obviously, but if it was, this would be like a scrolling screen where you can see the logs that are happening in real time and you can filter down certain things that match certain bits of text. So you can use that kind of as a way to just any, I don’t know, inspect what’s running, I guess, diagnose any problems that you might run into.

David (35:45): Now what’s pretty cool with this setup too is, as Kevin showed, you can install your own stuff. And Rails applications out of the box ship with all these kind of setups for using slash storage for storing the database and then having slash up. But you can use any language, whether you want to create something in Go or you want to create something in Node or whatever backend language you want to use, if you are able to package it up as a Docker container, which you can do with anything, you can ship it and run it on this setup. Now, this is where the question of, should I use Kamal or should I use ONCE sometimes comes in. And as we talked about earlier, Kamal is really for deploying your own private applications in a more sort of involved manner. I mean, you can use it for small things too, but it’s meant to scale all the way up to a Basecamp or HEY or even bigger applications where with ONCE it’s meant to be a console.

(36:47): It’s meant for one-machine deployments where the applications are open and public and where they can be shared more so than just for your own setup. So in the spirit of this, as you’ll see, at least right now, we also don’t have any form of authentication. The Docker URL that you put in has to be a public one. It’s intended to be a console system where we share all the cartridges. So if you fork Writebook and you make a really cool special version of it, you should have that available for anyone else to try it. So in that sense, there’s a little bit of almost like GPL-like spirit, even if none of the software’s actually licensed under that. But I like this idea that here we can have a common platform where we can collaborate and make it really easy to fork and try different things on and see where it goes.

Kevin (37:42): Yeah. So one other thing I thought would be useful to mention here, which comes from this integration between the proxy and the ONCE tooling is the fact that on the dashboard, we’re able to show things like the traffic that’s reaching your service and the number of unique visitors that go to your service. So these are being done in a way that’s, it’s not specific to the application. Like any application you install through this, we can get this information, but we do it because we can hook into some of the stuff that comes through at the proxy level and kind of inspect the traffic and figure these things out. So the traffic, obviously these apps I’ve just installed have zero traffic because I just installed them on my laptop, but normally you would be able to see a ray of requests here and which percentage of them are errors.

(38:28): Those are actually coming out of Prometheus metrics that Kamal Proxy exports. So the tool is able to just kind of query those the way a normal Prometheus dashboard would be. It’s just like a very mini version of it and it uses it to render this little needle that would go up and down as the traffic varies. And then on the visitor counters, we actually have a process that looks through the proxy logs to see all the requests that are being processed and it can count the unique IP addresses or client IP addresses that sees come through there. So client IP is kind of a reasonable sort of approximate measure of users. Obviously we can’t tell exactly which users are which without, there could be an application specific thing, but I think counting client IP addresses is a fairly good approximation of that. So we have a process that processes the logs, grabs the client IP addresses. It uses a thing called HyperLogLog, which is a way, kind of an efficient way to count unique values over time ranges without having to use tons of memory to store them all. And so that lets us calculate these unique visitor accounts on a daily and weekly basis. And we can do that efficiently, even if there was millions of them, it wouldn’t take inordinate. It’s like kind of a fixed amount of memory regardless of how many there are.

Kimberly (39:45): Okay. So we’ve talked a little bit about what we’ve done already. Do you guys have any ideas of what’s coming next for this that we can share?

David (39:52): Well, we have some secret plans for some of the things that could be coming next that we’re not going to go into, but there are a bunch of ways you could take this. I mean, Kevin and I talked about a bunch of ideas for, for example, having offsite backups as something that’s built in. Could you do something with SCP or whatever? There’s a million ways we could take this, but I also just do want to see if, are we, do you have the right timing for this? Is this now that something like this could work? And what we have now is everything you need to start doing this cartridge sharing. But what I love about all of this stuff is, as with the earlier versions of ONCE, we keep rolling forward. So there’s some experiment that have some degree of success or doesn’t have some degree of success or as logical says, I’m like, we just pile it all in.

(40:39): It just all goes into the mixer and then we put out something that’s the sum of all of those experiments and attempts. And I think this is going to inform and already has informed and work in parallel with all this stuff that we’ve been doing for Kamal or the ways we can make Kamal simpler. So there’s this symbiotic relationship where we get to experiment with this kind of stuff, especially since it’s open source. And now there’s this base of applications that people can use. It’s not just one thing. There’s literally three applications right now off the gate that you can install of ours. That’s enough to get things rolling.

Kimberly (41:15): Okay. Well, this has been an episode of Recordables. To hear more from our technical team, you can read their blog at dev.37signals.com.

Other posts in the “RECORDABLES” series

Behind the Fizzy Infrastructure

Kevin McConnell, Kimberly Rhodes, Fernando Olivares — Thu, 26 Feb 2026 18:00:00 +0000

In this episode of RECORDABLES, we dive into the infrastructure journey behind Fizzy. Lead Programmer Kevin McConnell walks through the ambitious plan to give every customer their own SQLite database and the challenges the team ran into along the way. What started as a unique way to support both self-hosted and SaaS models evolved into a performance experiment, pushing multi-tenant design further than we had before.

But as launch day approached, the tradeoffs became harder to ignore. Kevin shares what worked, what got complicated, and the pivotal decision — days before release — to unwind months of work and revert to a more conventional setup. This conversation is a candid look at architectural bets, emotional attachment to big ideas, and knowing when to change course.

Watch the full video episode on YouTube.

Timestamps

00:00:00 — Introduction
00:02:18 — Exploring infrastructure options
00:09:08 — Making the app feel fast everywhere
00:14:17 — The per-customer SQLite experiment
00:31:05 — When the architecture started to feel heavy
00:42:22 — Choosing Plan B
00:46:00 — What we kept and lessons learned

Links & Resources

Kevin McConnell’s Rails World 2025 talk: SQLite Replication with Beamer

Transcript

Episode Highlights (00:00:00): A year from now when the app’s growing and we have lots of customers and we’re trying to add new features, are we going to be kicking ourselves for setting it up in this particular way? The only other option would be to kind of go ahead and do it while not feeling confident that it was the right choice, if you see what I mean, which just that option just isn’t really an option. We risked having an app that broke, which no one wants.

Kimberly (00:23): This is Recordables, a place where the 37signals team shares their behind the scenes work building products like Basecamp, HEY, Fizzy, and open source products. We are sharing the behind the scenes of what we’ve done, how we’ve done it, so you can learn from us and avoid some of the mistakes we’ve made. I’m Kimberly with my trusty co-host in tech, Fernando. Hello Fernando.

Fernando (00:45): Hello. Hello.

Kimberly (00:46): We talked recently with Mike Dalessio about Rails’ multi-tenant structure of working with databases. This week we’re diving a little bit deeper, specifically into Fizzy and the infrastructure that we investigated with that product and ended up using. So to do that, we have Kevin McConnell from our programming team. Kevin, welcome to Recordables. Well, before we dive into the Fizzy infrastructure, tell us a little bit about you, how long you’ve worked here, and then we’ll dive into the topic at hand.

Kevin (01:14): Sure. So yeah, I’m a programmer here at 37signals. I’ve worked here for I think coming up on four years, something like that.

Kimberly (01:23): Nice.

Kevin (01:25): For probably about the first year I worked on kind of across a few of the products, worked on HEY for a while, worked on Basecamp a little bit, actually a lot, a little bit. And then I kind of joined the team building the ONCE products, which was part of what, well we’ll learn more about this I think when we talk a bit more about the architecture, but some of the things we explored in Fizzy, they sort of started out life as part of the things we explored in ONCE and kind of grew from there. So yeah, so that was my first, I would say the first three years we’re I’m working on some of the products and working on the ONCE for a couple of years and then the last sort of year in a bit has been more Fizzy.

Kimberly (02:03): Okay, awesome. And I feel like we could do a whole episode just about the ONCE products and how, if that all came to be, but we’re going to talk a little bit more about Fizzy today. Let’s start maybe with why we wanted to even investigate a different infrastructure, what we were trying to accomplish, and we’ll go from there.

Kevin (02:18): Sure. So there were really two reasons for it. I think there was one initial reason, and that’s actually the part that ties back to the ONCE stuff like I was saying. So the ONCE products, the idea behind those was that where most of our products are normally SaaS based and people subscribe, we run all the software on our hardware and people use it. The ONCE project was sort an experiment to see maybe people would like to buy software and run it themselves, kind of the way people used to buy software back in the day.

(02:50): So the notion was instead of subscribing, you’d pay once, but there’s also this, that was part of it, kind of the business side I guess was pay once rather than subscribing, but the technical side is run it yourself rather than us running it for you. And when we did those products it was quite interesting, but I think we found that some people really gravitated towards the idea of running things themselves. And so when it came time to start working on Fizzy, having done that a little bit, we had this idea about should we make a product that you could do either. So rather than be like Basecamp is subscription, but Campfire is ONCE, it’s like what if Fizzy was a thing where you could choose, you could pay for it once upfront and you would own it your own copy and you could run it yourself or you could just use a normal SaaS subscription thing.

(03:38): And so that’s what led us into exploring architecture as a way to answer that because normally you would build things slightly differently for those two use cases, if you see what I mean. So Campfire, the first ONCE product that shipped is very self-contained as a Docker container, it runs SQLite, so it’s kind of like maintenance free on the database side. Everything that it needs to do is built into a single Docker container. It’s really easy to run it, but for a SaaS application, that’s not what you would typically do. You would normally use database servers that could hold data for lots of accounts at the same time. They would be separate from your application server, you’d spread things out a lot more. Your job servers would be different than your application servers and that kind of thing. And so this was one of the angles for looking at new architecture for Fizzy was if we want it to work in both places, what should we build to do that? Should it be the same thing that works in two? Should it sort of be switchable in some way where you could change it depending on how you could package it differently for sales as a one-off versus us running a SaaS. So those were the questions for one part.

Kimberly (04:43): I would imagine too, there’s the maintenance question of keeping two versions for multiple people in different ways consistent with each other.

Kevin (04:53): Yeah, that’s definitely part of it. Anywhere you’ve kind of made a branch where you say it in this situation for doing it this way and the other situation we’re doing the other way, then you’re kind of potentially making things difficult for the future because you’ve got to make sure why any change you make has to work in both sort of thing. But that was half of the reason. The other reason for exploring the new architecture was really more about speed and performance. And I think it wasn’t the first reason for us to look at new architecture, but once we had the other reason, the ones thing, and we started thinking about the ways we might solve it, it occurred to us that some of them would have an impact on performance. And then I think we sort of changed gears a little bit and start chasing this idea of how can we make this really fast?

(05:35): And that became just as much or if not more of a driver of exploring new architectures. A big part of that really is just about, it’s about where you put data, locality of data to people. So normally when we run SaaS applications, the database lives somewhere geographically and we can use things like read replicas to make copies of it near different locations. But essentially you have one main copy of your data somewhere. So that could be in something like Chicago. The further you are from that, the slower the app’s going to feel to you just because of the latency, the speed of light limits and so on. And this is something that comes up a lot I think, because the kinds of speeds that you have to reach for an app to feel fast are short enough that common distances are too long for that to feel fast, if you see what I mean.

(06:27): So for me in Edinburgh, if I have to use an application that’s running out of Chicago say then realistically there’s probably like 100 to 150 milliseconds of latency for me to send anything there and back before the server even does any work, just the time for my request to get there and come back. The theoretical speed of light latency would be a bit less than that. It’d be like 60 or something probably, but in practice it’s going to be like 150 milliseconds to do nothing. If you’re actually doing something like rendering a page that takes 50 milliseconds to render, then that’s 200 milliseconds for a request. And in practice, I think it usually feels like if you can get your pages to render, like all your requests to complete in a hundred milliseconds or less, then the app feels nice and fast. More than a hundred starts to feel slow. More than by 200

(07:22): You really feel like something slow is happening there. And so this is an issue that’s, it’s hard to avoid if you have all your data in one place just because there’s going to be a bunch of people where no matter how fast you make the bits of your app, it’s going to feel slow to them just because far away. But if you took, for my situation being in Edinburgh, one of the other places that we run servers out of at the moment is Amsterdam. So if we moved everything to Amsterdam, it would be fast for me because my round trip to Amsterdam and back is like 25 milliseconds or something. So that 50 millisecond page that we talked about plus 25 milliseconds of roundtrip is nicely under that a hundred milliseconds. Does that kind of make sense?

Fernando (08:03): Yeah, yeah. The threshold.

Kevin (08:04): So that was a little bit of a tangent, but thinking about that stuff was what got us to think about as we’re exploring different architectures for running self hosted applications and SaaS applications, maybe we could take something from the model of the self-hosted ones where people would run things on their own servers next to themselves. Their data was generally right next to them. If you’re going to self-host something, you might be running on a server that’s in your office or closet or if you rent something on a cloud provider, you can pick a region that’s near you or something and you get the fastest data. When it’s SaaS typically you plop it in one place and that’s where the day is for everybody. For some people it was fast. For some people it’s slow. So we started to think about maybe we could take something from the self-hosted model. So even when we run it as SaaS, can we set it up somehow so we put people’s data near where those people are and then everyone gets a faster application.

Fernando (09:03): And how did that go?

Kevin (09:05): Answering that is jumping right to the end of the whole story probably.

Fernando (09:08): Spoilers.

Kevin (09:10): Yeah, I don’t know if you want to go through there bit by bit or just answer it, but I will say that you can quite easily make reading data local to people because you can take databases and replicate all the data out. So whenever you make a change in the place where the data is normally kept, the copies can go to different data centers and then whenever someone has to read their data, they can just read from one of those read copies and it’s near them and it’s fast. This is basically what we’ve ended up doing. We do this in other apps as well and Fizzy where we landed in the end is what we do. So all the writes are still happening centrally in one place, but all your reads come from the closest reader that there is to you. It actually works out pretty good in practice because most web applications do far more reads than writes. A lot of it’s just because the way you use apps, you tend to click around and look a lot of things and every now and again you’ll change something, but the ratio of reading to writing is really heavily skewed to reading. I think Basecamp I once lookeded and it was something like 94% reads, like 6% writes.

Fernando (10:17): That makes sense.

Kevin (10:18): So even if all you make faster is the reads, you’ve already helped most of the things most of the time. There’s also I think a bit of a perception about it where you kind, like changes that you make, sometimes it’s okay for them to feel a bit slower because it feels like you did something if you see what I mean. You’re sending an email, if you click a button to send the email and it took half a second or something to happen, it wouldn’t feel that bad. But if you’re trying to read through a set of pages on a website and every time you click on a new page it takes half a second to come up, that does feel slow. So yeah, we ended up focusing on speeding up reads by making them local and not worrying too much about trying to divide up the data and move it around to make loca writes because that’s where a lot of the complexity is from. But I can say that’s jumping to the end of this story because the way what we set out to do at the start was exactly split up everything and pre-reads and writes closer to people.

Kimberly (11:17): So Kevin, let’s start there on what did we try or what were the things we thought about in terms of here are our options for this new infrastructure?

Kevin (11:26): There were three, I think kind of three things that we thought were worth trying initially. One of them wouldn’t have helped so much with the speed, but more just about the packaging of the two different ways of use the software. And that was just to basically squeeze all the things we need on the SaaS side into the self-hosted side. So when we made the ONCE applications, they ran on SQLite because it’s nice and easy and convenient. We don’t run SaaS applications on SQLite typically because it’s a lot easier to sort of scale things when you have separate database servers and we have a lot of experience with making MySQL run really fast. So we usually use that. So one of the options that we had would be just take the way we do in SaaS, take by MySQL Everything and squeeze those into this self-hosted version so we don’t have to have two different ways of running software, but people can still buy it and run it themselves.

(12:23): That would’ve helped with the packaging sort of side, but it doesn’t really change anything about the speed. So that wasn’t the one that we decided to go to first. Another one that we kind of considered but quickly I think decided not to pursue is that because we knew how these self-hosted apps worked when running as individual Docker containers, which is the way the ONCE stuff works, we could always just basically host everyone’s own copy of the app on the SaaS site in the same way we could run a whole bunch of Docker containers so that everyone who bought it would have their own little container. And that’s what we would run to access things, which is in theory quite nice conceptually, but in practice quite complicated because it’d be really inefficient, like to do it naively would be very inefficient because there’s a lot of overhead in every container and if you have a hundred thousand customers running a hundred thousand Docker containers would be a huge waste of resources because most of them wouldn’t be active all of the time and you’d be using up a bunch of memory and CPU for things that are really being used.

(13:31): If you see, even for customers that are active, there’s a lot of idle time between actions usually. So you do something and then you don’t do anything for a few seconds and then you do something else. And if you kept a container running all the time, it’s just going to be very inefficient to do. And so to make that work, you end up having to invent ways to do things like have containers that you can run but then put to sleep between request and then wake them up really fast when the next request comes in, which some things do you basically be reinventing the fly.io service, which is kind of what they do is like run containers everywhere. The thing that would’ve been nice about that would be that we could put them anywhere and they would have the data alongside all the app servers and job servers and everything they need to run.

(14:17): So we could spread them around and let people run their version just like they would self-host it, we would just be kind of self-hosting it for them, if you see what I mean. But as I say, that isn’t something we pursued because I think getting that to work in an efficient way, it’s quite a big project. So then the third way, which is the one that we did decide to pursue and looked at way seriously for a while was the idea of giving everyone their own database but running those databases inside the normal sort of setup of app servers that we would normally have. So we might have half a dozen servers running Rails that would normally talk to one MySQL database with everybody’s data, but instead we would have half a dozen web servers running Rails that would talk to lots of little MySQL databases. Each one would be one customer’s data and we could put those databases on those app servers wherever we want.

(15:09): They don’t have to be in the same central place because they’re separate files. And it felt like a nice thing to look into because then we get this sort of isolation of data. It’s like one customer’s data is all in a single file and wherever you want to put that only affects that customer, if you see what I mean. And so that’s what this is part of what we pursued. That’s what became Mike’s Active Record Tenanted, which I know you talked to him about that recently and he’ll have gone and tell the details of that. But that was kind of the idea behind it was it lets you put data wherever you want without having to also chop up the rest of your infrastructure in order to do so.

Kimberly (15:47): And that also gets you the speed that you’re looking for because the data can be closer to each customer.

Kevin (15:51): Exactly. The data can be closer to people so they get less latency. It’s also quite nice because it was one of the routes that could make it practical to run SQLite as the database server. Like SQLite can scale to very large amounts of data and in the right circumstances, but if you want to run SaaS on it, there’s some challenges about concurrency, how many actions you can do at the same time on the same database. Like SQLite limits you to one write at a time. For an individual customer it’s probably fine. You can do a lot of writes very quickly and one customer is only going to be able to change so many things in a certain space of time that you can keep up. But if you used one SQLite database for all your customers, you can only do one write in your whole application at one time, you’re going to start actually getting locked contention bumping up against that just because lots of customers will try and do things at the same time. They’ll end up waiting on locks. But if you split the databases up into individual databases, those types of locks and those types of kind of limits, they apply a database level. So now it’s back to each customer can do one thing at a time. Not the whole system could do one thing at a time.

Fernando (17:05): So I have a question about that. I know this is Fizzy infrastructure, but isn’t Campfire running with SQLite?

Kevin (17:12): But we don’t run Campfire SaaS, so Campfire, right?

Fernando (17:16): But even what I’m wondering about is that each, I assume each message needs to be a write.

Kevin (17:23): Yeah.

Fernando (17:24): So…

Kevin (17:26): You can scale quite far and that’s not a problem because they’re all so fast. With Campfire, we did run into it sometimes when we were first developing Campfire and we would notice when we had done something where a write operation was slow, if a right operation took too long or if you start a transaction, try and do too many things before you the transaction or something like that, then you will start to see that. You’ll start to see the requests get held up behind the one that you’re trying to process.

Fernando (17:57): How does that look like for someone as a user?

Kevin (18:01): How would you know if that was happening?

Fernando (18:03): Yeah, yeah, because what you’re telling me is a little bit of programming magic, right? Oh, I can see the transaction and the requests queued up behind it, but is that noticeable for the end user?

Kevin (18:16): It would become noticeable when it’s bad enough. You would feel the sluggishness of the app is probably what would happen. So if you start to have requests waiting on locks, they’re sort of queuing up. And so a thing that you would try and do posting a message should normally be really fast, but you might post it and be waiting for the response to come back because it’s waiting to try and get the lock on the database, if you see what I mean.

Fernando (18:41): Yeah, that makes sense.

Kevin (18:42): So generally when working with, this is one of the things we learned with Campfire I think was when working with SQLite, it’s good to be conscious of how long you’re likely to be taking locks for. So you would keep your write operations short and efficient, don’t hold transactions open while you sort, if you want a bunch of statements to work as an atomic unit with a database, you can start a transaction, you can do some work, then you could go off and do some other things in your Ruby code while you finish processing things and then you can do more and then commit the transaction.

(19:13): And that whole time in the SQLite world, that whole time you would have a write lock that you’d be blocking anybody else from writing to the database. So it just makes you be a bit more careful about that.

Fernando (19:24): And one more question about that is that I assume that Rails is prepared for that. It just queues up the requests. This isn’t a Rails limitation, it’s a SQLite limitation.

Kevin (19:37): Yeah, it’s a SQLite limitation. Exactly.

Fernando (19:39): So you could have the write log whatever and Rails is happy to wait until you release that log and then continue processing the requests.

Kevin (19:48): Exactly. Rails doesn’t even really know about it. I mean as long as you don’t hit some kind of time out or something that you’ve set up, Rails wouldn’t even really know. It just asks the database to do something and that takes some amount of time. It’s just that would slow down. But as I say, if you keep your transactions short, it doesn’t tend to come up a lot in SQLite, at least if you’re using SQLite with the write mode, you have to use it in wall mode and that kind of thing. But you can go pretty far with it. But once you get into the world of running SaaS with lots of customers, then you would start hitting those limits. And so that was part of what was, I think interesting about this idea of giving everyone their own SQLite database because that meant since we’re not having these logs, then it could be practical to run SQLite as our SaaS database rather than MySQL.

(20:40): And that has some really nice properties and one of them that’s nice for speed as well as the thing of getting the data close to people to cut it and latency. There’s another effect that’s happening, which is more about getting the data close to the app server. So normally if you use something like MySQL, you’ll have a Rail server somewhere and then MySQL server running somewhere else, and whenever it needs to do anything with the database, that’s a network call between the two, and they’ll be close together in the same data center usually it’s usually a very fast network call, but still it does add some time.

Fernando (21:14): But why is this, I don’t know, a CPU limitation where you want to run them separately?

Kevin (21:21): Usually you’ll want to run them separately partly for resources, partly for scaling because you want to be able to add more of one without necessarily the other, if you see what I mean. So for example, you might start with having two app servers in one database and that’s enough load, but then as you get more customers, you might find your app servers are quite busy, but the database server is fine. And so you can add more app servers and get more capacity there rather than, I mean if they were just on the same one box and it got busy, you’d just have to keep making that a bigger and bigger box.

Fernando (21:56): That makes sense.

Kevin (21:56): So yeah, so this thing where because SQLite works, it’s not on a separate server, it’s not even a separate process from your Rails process. So anytime you need to access the database, it doesn’t have to go across the network. It doesn’t even have to go from one process to another and make some kind of IPC call. It can just directly carry out the query, it just grabs the data from disk or if quite often a lot of the active data will be cached. So it’s basically pulling stuff out of memory. So it’s really, really fast and it’s enough of a difference to be noticeable I would say. I think often in a typical web application, a request might do a handful of database operations, now a handful of queries, things that have to be separate queries really because they’re unrelated things, if you see what I mean.

(22:47): Quite often you’ll be loading up the user record to make sure they’ve got permission to see what they’re looking at. You’ll be querying for whatever it is they’re trying to see, like the Fizzy card that they want to look at or something. And maybe there’s some sidebar content with a menu in your pulling out. So you probably do a handful of different database operations and with a client server database like MySQL, that’s a handful of back and forth, back and forth between machines. When you do it with MySQL and you’re nodding it back and forth, it is a noticeable difference. MySQL is super fast, sorry, SQLite sorry is super fast when you use it that way. So that was one of the other things that was nice was like, okay, with this architecture we’ll get all the speed of having our databases inside the process and we’ll be able to move them close to the people, so it’ll be fast and yeah, that was one of the things that kind of drove us to chase this for a while I think.

Kimberly (23:42): I mean it sounds like a big win. Tell us when we started going down that path, what happened or what did we find?

Kevin (23:49): So a lot of it turns out to go quite well, but I think it starts to get a little bit complicated just because although those are the nice advantages to that kind of architecture. Basically the nice parts of using SQLite means that the data is right there inside your Rails process. But the flip side to that is that now your data has to live on the same machine as your app server. You can’t just put a database server somewhere and then have however many app servers you need that all talk to it. And as you get more load and more customers, you can add more app servers. They have to be on the same machine. So if you want to increase capacity, then you can add servers, but then you also have to move the data. So you have to take some of your customers that were stored on server A and put those on server B, which starts to become a bit more complicated.

(24:46): You also have to have a way to not have each particular server be the only place that customer’s data lives. And you need to have some kind of redundant copies because that machine could fail suddenly and then you don’t have access to customer’s data. If you want to spread out the reload to multiple machines like we talked about before, then that also needs a way to replicate data from one server to another because otherwise it’s always in that one place. One thing I probably should have mentioned about this idea of making data close to people is it sort of assumes that people tend to be accessing the data from the same place. So you expect all your customers should… if all your customers work in Edinburgh and you put your data in Edinburgh, that’s great for them. But if they’re like 37signals and everyone’s spread all over the world, there’s not really a good spot that’s good for everybody.

(25:40): So you can’t avoid still trying to find some way of doing the thing of like, well, we can spread the read on the copies around and everyone gets fast reads. You still need to do those types of things if you want to work well in situations like that, if you see what I mean. When you want to do that with SQLite, there wasn’t really a good way that we found to just do that with existing software that kind of fit our use case well. So we had to build that part too. So there’s one of the places where the seemingly simple architecture starts to get a bigger project, a bigger project like, well, this bit is good, but we also need to build this other thing. We also need to build this other thing. And so it starts to grow arms and legs a little bit.

Fernando (26:23): And some of these concepts seem deceptively simple, right? From what we learned yesterday with Mike SQLite is just a file. So at first it seems like quite simple, you just copy the file everywhere. You just do it right and then copy the file somewhere else.

Kevin (26:42): Yeah

Fernando (26:44): That should suffice. And then I assume that when you start doing something simple like that, oh, there’s this edge case that you know when this happens.

Kevin (26:51): Yeah. Exactly. It is just a file, but it’s a file that is being changed in various different parts of it, potentially very quickly. And you need, so in the case of replication, you want another copy of it that you can use for reads. Then whatever changes in that first file has to change in that second file. But it has to do it really quickly, and it has to only do the parts that change. It can’t recopy the whole file because it’ll get too big. You would end up try to send far too much data across the network if every time you made a change, you copy the whole funnel.

Fernando (27:23): How big is the SQLite file, like in your experience while we were testing this?

Kevin (27:29): So it really depends. Since it’s per customer, I mean I think a lot of customers will be quite small. So megabytes probably. Big customers might be a gigabyte or something. They’re not massive, but they’re big enough that you couldn’t copy the whole thing each time someone changes. One thing I was getting to that you need to at least find out what’s the part that changed and then just apply that change. And you want to do it as close to instantly as possible because you want people to be able to use those read only copies to access all the reads without them getting behind and showing stale information, if you see what I mean. So we built a system for replication. We also did build in something about that notion of stale information because you can’t make the replication actually instantly. There’s always going to be some amount of delay and usually it should be short, but I think there’s always a potential that it gets held up for some reason because essentially you’re sending changes across networks between servers.

(28:35): If a lot of changes happen in one place, it’s possible that it might take a moment for all of those changes to get across the network, get applied in the other side. It shouldn’t normally be long. It should be well under a second most of the time, but there’s always a chance that a lot happens suddenly or something. Or just computers. It might just get slow for a little bit. So you usually need some mechanism to make sure that what people are looking at is not stale data. You wouldn’t know it’s stale data really if you weren’t the one making the change. If you’re looking at a page that someone’s updating and it takes two seconds for the change to show up, you’re probably just not going to know. You didn’t know it was there until you saw it, and that was in two seconds is nothing.

(29:22): But the case that quite often comes up is if you change something and then you go to another page where it’s you make a new post, add a new card in Fizzy or something, and then you go to the list of cards. If yours isn’t there because you read from a server that didn’t have it yet, it’s going to look broken. Normally, or I think commonly the way people handle that, which we do in some other apps is just have a short delay where you pin your activity to the place where you wrote it. So when you make the change, it has to go where the writer is. When you go to make a read, normally you would read it from the reader, but because the system knows that you just wrote something, it’ll make sure that you actually read from the writer for the next second or something, which works well.

(30:06): But it does mean that more of the requests go to the writer than have to, because it’s a pessimistic kind of approach, right? It’s like we don’t know if it’s in the other place yet, so we’ll just assume the worst and keep serving you from the writer for a couple of seconds until we’re sure it would be there. So in the architecture that we were building for Fizzy, we were quite conscious of trying to avoid doing any more work on the writer than we absolutely had to because we have these kind of tighter restrictions about how many writes we can push through in each app server and how many customers can go on each app server as a result. So we took a different approach there and instead we actually track what the idea of the last transaction is that you wrote. And then every time when your request comes in, we are able to assume that it’s probably safe to serve it from the reader and send you to the reader, but detect if we were wrong.

(31:05): And so in those rare cases where your next request goes to the reader and it doesn’t yet have the transaction you just wrote, at that point, it can quickly resubmit the request to the writer to get the fresh copy, which means it potentially might be a wee bit slower if that happens, but that very, very rarely happens. So it gives you the protection to never seem broken, but with the more optimistic behavior, that is fine to just read from the reader all the time. I feel like we just went down a really deep rabbit hole there. I forgot what we were talking about before we went down there.

Fernando (31:38): No, no, no. I think this is great. The reason is it seems like, wow, it seems so simple at first, and I’m sure you felt this and you and the team felt this, but I’m sure the desire to make something like SQLite work was really, really high.

Kevin (31:56): It was. I mean, I think it was at the start because it seemed cool basically.

Fernando (32:02): Yeah, it is cool.

Kevin (32:03): The advantages seemed good also. It just seemed like a fun thing to do and we all grew to really SQLite a lot. I think a few of us had used it and quite liked for other things, but the more we worked with it we’re like, this thing is cool. We just want to make this work and use this. And I think also there’s a thing where the more time you spend on a project, you also get a little bit emotionally attached to it and there’s a why get to the point where we’re like, I’m going to make it work because I said I was going to make it work. That like, it’s not going to beat me. I’m going to do this. Which actually, I think probably for me at least, it led me to chase it for a bit longer than… I think in hindsight, I think there was a point where I probably, it would’ve been better to stop, realize that some things weren’t going quite the way we wanted to and then change tech earlier. But instead, I was so determined to make it work because it mostly did as well. That’s the thing. Most of this did turn out to work really quite well.

Fernando (32:57): Right, and I was going to say, there doesn’t seem to be a huge wall in front of you, right? It was just tiny pebbles that made you like, oh, clean this up. I need to move this here. But they just kept coming.

Kevin (33:15): Yeah, well…

Fernando (33:18): Or was there something huge where you’re like, oh.

Kevin (33:21): I don’t think there was anything so much huge. I actually think it was a bit more of, it became a timing issue to get it done because it did turn out to be quiet quite big and quite hard to crack. So the way it played out in practice was that we built some of the parts of this first. We had the tentanted SQLite part. We had replication working. We set things up so we had one app server that was the writer, and then we had a couple of readers that replicated from that. We set up this geolocation routing stuff. We use. CloudFlare is a load balancer service that you can use to sort of direct traffic to one of your, like to the nearest data center from your data centers. So we put these servers in different places. We had traffic routeing to the right one, and that worked really well.

(34:15): And that’s what we used internally for a long time. We had this sort of small group, this sort of invite only early testers group that were using Fizzy for a while, we’re running on this. And it was great. It was super fast and worked really reliably, but as we started to add the multiple writer part, there’s a lot of things in there that are quite difficult to get right. So it took a while, I think to land on a design for that that we liked. We got there in the end, but we were already, we’re getting later and later in the project. And so we were working on the infrastructure part while it was basically for the longest time it was just Mike and myself working on this, and then Stanko joined. So there’s three of us towards the end. But while we’re doing that, there’s other people working on the product and so early on it’s great.

(35:05): You have all the time in the world, they haven’t built the product yet, so you can take your time figuring out the infrastructure, but we get to a point where the product’s good to go, ready to go out the door, and we are not quite ready with the infrastructure. We still have some things to figure out there, and for a little while you could kind of get away with that. We can be like, well, if we need a couple more weeks to finish something up, then they’ll add more things to the product for a couple more weeks. There’s always more things to add, but I think we were getting to the point where it started to feel like we’re going to end up slowing this down. We’re going to end up not being able to release a thing we’ve built just because we still have questions on our side.

(35:44): We figured out how we wanted to do the multi-writer thing. We built that kind of towards the last minute in project terms, the last minute. We got that stuff all working. But I think for me, there were two problems that we were staring at. One was that although we kind of got this stuff working, we had intended to do a lot more preparation in terms of how we were going to run this and make sure that we were prepared for whatever happened. So having run books for how to handle operational situations like if a machine breaks or if something goes wrong with a replication, then it gets, there’s a lot of replication lag. How do we deal with that? There were a lot of things that we ideally would have practiced and researched and written up. So we knew that we were prepared so that when we launched this for real, if anything went wrong, we could quickly recover.

(36:42): And we hadn’t really done enough of that by that point, mostly because we were so busy figuring out these other questions that we had to keep pushing that further down the line. Same with benchmarking. We’d done some basic benchmarking to know how fast this was, and we knew that it was pretty fast, those tests. But I think there’s always the potential that there’s kind of limits and ceilings that you haven’t uncovered yet. And to be confident in releasing the app, I think we needed to have spent a bit more time trying out different benchmarking scenarios, knowing that if loads and loads of people sign up on day one, it’s not going to catch fire because of some limit that we didn’t know about or something. So that was half of the concern was just not being quite ready enough to do it in a responsible way. We could have shipped it and launched it. I think it would’ve probably been fine, but if it wasn’t fine, I think we would’ve had a bad time. And we risked having an app that broke, which no one wants, right?

Fernando (37:40): It’s all fun and games until you get the customer data loss.

Kevin (37:45): Exactly because all this time we’ve been running internally, it was working well, but the stakes were low at that point because if it broke, we would just go, oh, I wonder what went wrong. We’d fix it and we’d put it back together. But I think the point where it’s out in the wild and people are using it and also especially the early point where you’re trying to tell the world you’ve made this new thing and you want everybody to come and look at it at the same time, you don’t want it to break in that moment. And as I say, it’s not that I thought it would break, but it was more that I didn’t feel confident enough that we had made sure it wasn’t, or that we had made sure that we could fix it really fast if it did. There was an amount of preparation that we hadn’t been able to do in time, I think.

(38:29): So that was half of it. The other reason though was that for all that we liked about that architecture, the longer we worked with it, I think we start to feel like there were some parts we didn’t like about it as well. There’s some parts that become harder and a lot of it just boils down to that same, it’s that same constraint that you have where if you’re using SQLite and your data is where your app server is, then even though you can have your read only copies and you can read from multiple machines, if you want to change the data, you have to change the data on a specific machine. If you’ve divided your data up per customer, then most of the time that’s fine because usually you know for any specific request, this is only for this customer. I know which machine has their data, I can route the request to the right place and they can do the write there.

(39:19): But sometimes you have requests that aren’t like that. There’s some things that do span customers. So for us, it came up with things like the way login works. When you want to log in, you need a way to enter your credentials and get authenticated before we could show you, here’s all your accounts. Those accounts are essentially the different tenants. And so we need to find out, we need to get you authenticated, show your list of different accounts, even though they’re all on different machines. So the information about you and how you log in can’t be just in each of your tenants. It’s a layer above that. It’s across all of those. And so as we were working on some of that side of it, we started to run into these situations where things that seemed like they should be easy were hard because you’d have a feature…

(40:12): I think one was that we had, I’m trying to remember the exact detail, it was something like we wanted to make it so that your profile picture wasn’t per account but was per person. You have the same profile picture regardless of accounts. And initially we had built this into the tenanted database because it was a per account thing. Changing that normally would be super easy. But in this architecture, we’re like, well, we don’t even know which database is holding the information about what your profile picture is, and if it’s not the machine where your account’s on now the request of this machine has to talk to this other machine. And we kind of worked our way through it, but we just started to feel like we’ve made things harder for ourselves here. There’s a lot we liked where, I dunno, I was going to say where we made things easier for ourselves.

(41:00): I dunno if it’s exactly fair to say we made things easier for ourself, but more like there was a lot where we could do things the way we normally did and get benefits. But then there’s these other cases where we’re like, this is kind of awkward. I don’t actually know if we’re going to be pleased that we did this a year from now when the app’s growing and we have lots of customers and we’re trying add new features or are we going to be kicking ourselves for setting it up in this particular way, if you see what I mean? And I don’t think it’s super clear right now whether it would’ve been, I think we were happy about or I think we weren’t happy about, but there was doubt. And so I think the combination of, we’re not quite ready to go with this, but the app is ready, the product is ready to go, that combined with, and we’re not even really that sure we still want it anymore kind of thing. That’s why we decided that the right thing to do for that situation was actually unwind some of that and go back to a more traditional for us architecture and ship on that.

Fernando (42:05): And I believe what’s interesting about this is that it’s in the git history, right?

Kevin (42:09): It is. Yeah. So actually I can show you, I have bookmarked the right PR for this in case, so that I could kind of show…

Fernando (42:20) The exact point in time. We were like, yeah, nope.

Kevin (42:22): Yeah. So it was a dramatic week in some ways. So the decision to go from what we were going to do to what this PR here is called Plan B, because we always had this idea of behind that if this didn’t work out, the plan B was that we would just convert the app to run on MySQL, use the same kind of set that we typically use for our other apps. And so the decision to do that, I think the day on this says November 18, I can’t remember the dates very well, but if this is November 18th, that probably means that I think November 19th probably was the planned day for shipping the app. And I think the day before this PR is probably when I kind of went, I don’t think this is going to work. I think it was on a Sunday evening, I pinged David to say, I think we should change. I think we should do Plan B. We should bail on the new architecture. It was literally two days before we were supposed to ship it.

Fernando (43:25): That must have been difficult.

Kevin (43:28): It was, like I said earlier, I think in hindsight it makes sense. It still feels like it was the appropriate thing to do in that situation for all the reasons that we just talked about at the time. But at the time of doing that, it was quite a hard thing to do because we’d invested a lot of time in it. I was really, like I say, I was determined to make it work. I was kind of attached a bit to the idea of shipping. It felt like failure to not do it. I knew it was the right decision to do, but at the same time, you get for one thing, you get kind of emotionally attached a bit to the project you’re working on for a long time, I think. But also, we had talked about this a lot. I gave a talk at Rails World about Beamer and how we were using it for Fizzy, and David and his keynote talked about how we were doing this whole new architecture and we’re going to change the world with this new architecture. So all of a sudden to be like, we’re not actually going to be doing that. It’s hard to not have that in your mind when you’re deciding to change the plan.

Fernando (44:29): For sure. Just knocking on David’s door, like hey. Have a second?

Kevin (44:32): Yeah like, sorry I told you we were going to do this thing, now we’re not going to do this thing. But it did really feel like the choice between, you either just put your hands up and say, you know what? This hasn’t worked out. We need to do something else. Or the only other option would be to go ahead and do it while not feeling confident that it was the right choice, if you see what I mean. Which that option just isn’t really an option, I think.

Fernando (44:58): It’s already been a few months and Fizzy’s out. People love it. They really, really love it. Do you see it as a failure or is this just like software development?

Kevin (45:10): Mostly the latter. So I think exploring this made a lot of sense. I think we learned a lot from this. There’s things that we took out of this like Fizzy, although we reverted to a more conventional, for us, architecture, it’s not exactly the same. We did get to keep some things from our explorations on the other architecture. So for example, the thing I described there about dealing with replication lag by tracking the transaction that you wrote rather than pinning all your reads to the writer for some period of time, that we figured out how to do that when working on this new architecture and when we switched to Plan B, we kept the idea and we ported it to the MySQL side. So now Fizzy still has that improvement because we had already put the work in while we were doing the other stuff. And then there’s a couple other things like that.

(46:00): Part of that I didn’t really talk about much earlier, there was that as well as replication and the database location stuff. The other thing that you have to do in all of this is routing requests to the right machines, you have to have a more, I dunno the right word to call it, but more dynamic I guess sort of routing. So when requests come in on a particular data center, it has to know which is the right server for this customer for this action. And so there’s a bit more behavior going on at that kind of level. And so we built a bunch of stuff into Kamal proxy, which is the proxy server that we have in Kamal our deployment tool. We built a bunch of load balancing features into it so that we could build this original architecture. But when we switched to the other, the MySQL version, it turned out to still be really good to be able to do this at that level.

(46:55): So we still use Kamal proxy as a load balancer. We have six load balancers that run Fizzy that are all Kamal proxy using the same new stuff that we had built originally for the first architecture, if you see what I mean. So we did take some stuff out of it. We didn’t throw it all away, but we did change a lot. So I was going to show you the PR just to give you a sense of what the work was like to do it because it was quite a sudden and big change in a sense. So one thing I didn’t mention is that there’s 14 participants it says in here. A couple of these are just people who I think commented and discussed things. So it’s not exactly 14, but there are probably eight or 10 people involved in doing this change. It has tons of commits.

(47:43): It’s one of those things that GitHub is not very good at displaying. It has too many things, but it’s basically a week of work by the whole team. So it was kind of a intense week where we all said, all right, let’s make this work. Let’s go with plan B. And we all made the changes. The changes themselves, most of them are not actually that difficult or complicated, it’s just that there’s a lot of ‘em. So I dunno if it’s interesting to the sort of thing that we had to do to change this. I could probably point a couple of things.

Kimberly (48:18): Yeah, let’s do that.

Kevin (48:19): The way this played out, you could see at the start I actually dunno why this first commit that wasn’t on the PR before. That’s just a weird git history thing. But we basically started by pulling stuff out. So Beamer’s the replication system that we built. So we weren’t going to need the replication anymore. We weren’t going to need this test bed, which was there to test how replication worked across machines. We added the trilogy adapter, which is the MySQL adapter. with two active record tenanting. And so you could see, we basically started this project where we’re like, take out all the new stuff, set it up on MySQL, and then there’s a bunch of, and then make it work things. So there’s a lot of update queries, fixing tests and stuff like that until it worked against MySQL. And a lot of it was, as you just described, there’s a lot of places where in the tenanted world, the database had the data for just that one customer or that one account.

(49:18): And that’s usually the level that you query things at. So if you want to look up your account, we didn’t have to say find the account for the customer, we could just say account.sol. It would just be like the account. And that would give us the account information. If we wanted to get a list of all the boards in your Fizzy account, we just query for all the boards because they’re all your account because of the database they’re in. Obviously once you stop tenanting and you go into this model where one big MySQL database has everybody’s data, then you have to go through, if we find the models down there, if you look at something like board, the model, yeah, there’s tons of this sort of stuff. So board had to have a belongs to account because before it didn’t matter. There only was one account. Now it has to have that relationship. And then all the places where we do queries, we have to make sure that we’re actually accessing things through current account or current user rather than just the only account. So there’s a lot of mechanical changes that are of that sort of form.

Kimberly (50:31): Kevin, is this made any more complicated because we had beta testers, like people with actual account data, not just our team in there. Did that make any of this change unraveling any more complex or would it have been the same regardless?

Kevin (50:47): It was a bit more complicated because yeah, we had people using it. We wanted to make sure that everyone’s data was preserved properly and we also didn’t want to interrupt their use of it too much, even though it was pre-release. So it was sort of okay to say we’re taking it down for maintenance, but we wanted to minimize the interruptions. So yeah, we couldn’t just start fresh with a new database. We had to write scripts that imported the data from all the individual tenanted SQLite databases and then copy them into the MySQL world.

Fernando (51:17): I saw in one of the commits that it says Remove Beamer. Beamer is not yet open source, is it?

Kevin (51:26): So it’s not yet. I want to open source it, but there’s a couple of things I wanted to tidy up to make sure it was ready before it goes out. And because we were quite busy getting Fizzy launched, at the point where we stopped, we decided we weren’t using it, haven’t really had a chance to go back and just make sure it’s ready, I think. It works. We were using it for a long time, but there are just a couple of rough edges that I don’t want there, and I think I want to share it because we built it and I think it works quite well. It’s super fast. It turned out to be pretty reliable, but I think as soon as you share something, you kind of… you only want to do that if you’re ready for other people to use it. If you see, I mean, I don’t want to share it insane, but it’s a bit wonky over here or not be around to help if someone has questions or something. Do you see what I mean? Kind of want to do it, share it responsibly if you see what I mean. So I will.

Fernando (52:23): I think that’s really interesting. How do you marry that with David’s very famous, the gift philosophy.

Kevin (52:31): I do think it’s a gift. I think you can give things to people and they don’t… I don’t think people have the right to demand that you start doing certain things like add features that they want if you don’t want them. I think that’s kind of where the gift thing come from, right? If you make a thing and you think other people can benefit from it and you want to give it to them, then you can and don’t really, they’re not really entitled to then demand more things. It’s a gift they could take or leave it. They can build their own if they don’t like the one that you made. I think that’s fine. But I do think also there’s an amount of just giving people something in a good form. You don’t want to waste people’s time by saying, I built this thing, it’s 95% done. It’s up to you to finish it or something. You know what I mean? It’s sort of a balance though I think.

Fernando (53:21): I get it. And this is still your baby in a way., right? I know this is a team effort and everything, but from what I’ve heard, Beamer is something you’ve poured a lot of time into.

Kevin (53:33): Yeah, because although all these things are, they’re team efforts, but we’re a small team, so in practice, in practice you might find that there’s two people or three people working the thing and there’s two or three things that you need to make for it to work. So you tend to end up each having a thing. A lot of the things we build end up being mostly built by one person, especially those kind of supporting tools and stuff. It’s usually one person that came up with an idea and built a thing.

Kimberly (53:59): This path that we went down with Fizzy infrastructure that we didn’t end up using. Do you imagine that in the future we’ll revisit it? We’ll try to go back to it given more time. It seems like there was a time element where the product was ready, the backend wasn’t. Given enough time, do you think it’s something you’d want to re-explore?

Kevin (54:18): Think there’s parts of it that we want to re-explore. I think we learned a lot from it that I think informed how we would do it again if we were to do it again. And I don’t know that we would go back and exactly continue that same journey and finish the exact same thing. But I think there were some things that we thought that we were hoping to get out of it that we ended up because we didn’t ship on that version, we don’t have these things that we might want to go back and say, well, how would they apply in this new architecture? So one of the things that I’d really like to look at, we will try to do that soon, think is some other form of doing the local writers to people. So even though we don’t have individual tenanted databases per customer that we can move around to different places,

(55:07): right now we have one big MySQL database with everybody, but we could have four or five or something. You could have the European database, the East U.S., the West U.S. and segment customers into a small number of large databases and use a lot of the existing, the load balancer routing that we already built. I think we could apply a lot of what we built for the MySQL version onto, sorry for the SQLite version. We could apply a lot of that onto a sort of MySQL sort of form of this and still get that benefit of, now my data could be in MySQL at Amsterdam, and so it’s faster here in Edinburgh. So I think that’s more like where we might go back and look at, it’s like what could we pick and choose from the things that we did and apply them in this place.

Kimberly (55:57): That makes sense.

Fernando (55:58): Listening to what Mike had to say about Active Record being multi-tenant and all of this, it seems like there’s the gem that he is working on. The Kamal proxy load balancing stuff, the Beamer, which in the future may come up and Fizzy overall, there was no actual real harm done by this exploration. It seems to be that it was like a net win even if we didn’t get a hundred percent of everything that we wanted, right?

Kevin (56:31): Yeah. Yeah, I think so. I think it was a really useful exploration. I think we learned a lot of stuff and some of the things you just mentioned, there are things that we have kept. It’s not like none of it shipped. It was more that we shipped some parts, we didn’t ship other parts, but we did still get quite a lot out of it. I’m quite excited with some of the load balancing stuff that we built into Kamal proxy and that I think we’ll be genuinely useful to a lot of people. We still have a little bit of work there to make it easier for people to set up those load bouncers and we’re going to add some things to Kamal to do it, but I think that’s actually going to be really nice for people and that came out of that work. So I think it was all good.

(57:14): I think, like I said earlier, I think the one part, in hindsight, I would’ve liked to notice maybe one month sooner than I did that we were going to change our mind. So instead of having that two days before launch going, let’s change everything and then we had to have that crazy week, I think it would probably would be good to notice slightly earlier. But other than that, I don’t regret looking into it and I do think that it was the right thing to explore and it was also the right choice in the end to go where we went. So I think it’s all good.

Kimberly (57:46): Yeah. Well Kevin, thanks for sharing all of that with us. This has been an episode of Recordables, which is a production of 37signals. To hear more from our technical team, check out their blog at dev.37signals.com.

Other posts in the “RECORDABLES” series

Introducing Upright: An Open Source Synthetic Monitoring System

Lewis Buckley — Mon, 16 Feb 2026 18:00:00 +0000

We’re open-sourcing Upright, the synthetic monitoring system we built to watch over Basecamp, HEY, Fizzy and our many other services.

Upright runs health checks from multiple geographic locations and tells us when something breaks. It’s a Rails engine that you deploy to cheap VPS nodes around the world using Kamal. Each node runs probes against your services and reports back via Prometheus metrics, so you can alert with AlertManager and visualize with Grafana.

Why we built it

We’ve used Pingdom for years and it’s generally worked fine, but has been somewhat of a black box. We couldn’t customize checks, couldn’t run authenticated browser flows without handing over more cash, and couldn’t control exactly where checks ran from. Occasionally, probes would fail and trigger a page, but the issue would resolve itself before we could investigate.

We wanted something we owned, something we could extend, and something that fit into the open source observability stack we already use.

Upright started as an internal tool to complement our existing monitoring. It now handles all of our synthetic monitoring across five global sites.

What it does

Upright supports four types of probes:

Playwright probes are the headline feature. These are real browser checks that can log into your app, navigate through flows, and verify that things work end-to-end. When a check fails, Upright captures a video recording and logs so you can see exactly what went wrong. You write them as Ruby classes using the Playwright API.

HTTP probes are simple health checks — hit a URL, check the status code. They run every 30 seconds from every site. Logs are in a familiar cURL format.

SMTP probes verify EHLO responsiveness, STARTTLS support, and SSL certificate validity.

Traceroute probes map the network path to your services with hop-by-hop latency tracking, thanks to MTR reports.

All probes run from every site you deploy to, optionally staggered by a few seconds. Upright uses this multi-site data to distinguish between real outages and regional blips. If one site in Amsterdam can’t reach your app but four others can, that’s a regional issue, not a full outage.

How it’s built

Upright is a Rails engine. You create a new Rails app, add the gem, run the install generator, and you have a working monitoring system. It uses:

SQLite for probe results storage
Solid Queue for background and recurring jobs
Kamal for multi-server deployment
Prometheus for metrics and uptime queries
AlertManager for notifications
OpenTelemetry for tracing and logging
Dark and light mode for the dashboard UI

The architecture is simple: deploy Upright to a few cheap VPS nodes (DigitalOcean, Hetzner, etc.), point DNS subdomains at each one, and configure your probes. Our setup runs across five sites — four DigitalOcean droplets (2 vCPU, 4GB RAM, ~$24/mo each) and one Hetzner server (3 vCPU, 4GB RAM) — for about $110/mo total. You could run a minimal two-site setup purely on Hetzner for under $20/mo.

Each node identifies itself via environment variables and runs probes independently. Metrics are scraped locally with an OpenTelemetry collector, and then fed back to the primary site which runs Prometheus and Alertmanager.

We actually send the metrics to three Prometheus instances, one in each of our datacenters as well as the primary Upright instance, providing full redundancy.

Getting started

rails new my-upright --database=sqlite3 --skip-test
cd my-upright
bundle add upright
bin/rails generate upright:install
bin/rails db:migrate
bin/dev

The install generator creates everything you need: Prometheus and AlertManager configs, OpenTelemetry collector config, probe directories, Kamal deploy templates, and a Docker Compose file for local development services.

Define an HTTP probe in probes/http_probes.yml:

- name: "37signals.com"
  url: "https://37signals.com/"

- name: "Help Ticket Creation"
  url: "https://help.37signals.com/billing/tickets/new"
  expected_status: 301

Or write a Playwright probe that logs into your app and checks a real user flow:

class Probes::Playwright::Bc4ProjectProbe < Upright::Probes::Playwright::Base
  authenticate_with_form :bc4

  def probe_name = "Basecamp 4: view project"

  def check
    page.goto("https://3.basecamp.com/5228151")

    page.get_by_text("37s Monitoring HQ").visible?
  end
end

Check out the README for the full setup guide, including multi-site Kamal deployment, alerting rules, and authentication configuration.

Try it out

Upright is available now on RubyGems and GitHub under the MIT license.

Rails Multi-Tenancy

Mike Dalessio, Kimberly Rhodes, Fernando Olivares — Fri, 13 Feb 2026 18:00:00 +0000

In this episode of RECORDABLES, we dig into our ambitious goal of moving from a commingled database to separate SQLite databases for every Fizzy customer. Lead Programmer Mike Dalessio walks through what multi-tenancy actually means in practice, why Rails doesn’t make it easy, and how the open sourced Active Record Tenanted gem makes it more seamless.

During the conversation, Mike does a live demo showing what it takes to convert an existing Rails app to multi-tenanted and the safeguards built in to prevent accidental data leaks. You’ll also hear about the edge cases of globally replicated SQLite, and why we ended up launching Fizzy without it.

Watch the full video episode on YouTube.

Timestamps

00:00:00 — Introduction
00:02:56 — The Fizzy bet: one SQLite database per customer
00:06:57 — The challenge with SQLite and global writes
00:14:35 — Switching to MySQL (and fixing the fallout)
00:18:55 — Why the Apartment gem wasn’t enough
00:22:55 — Live Demo: Making Writebook multi-tenanted in minutes
00:31:36 — Built-in safety checks to prevent data leaks
00:35:17 — Replication, failover & “emergent behavior”
00:43:28 — What’s Next: upstreaming to Rails and future plans

Links & Resources

Transcript

Episode Highlights (00:00:00): That’s the kind of thing that multi-tenancy can give you as well. It’s not necessarily between customers, but it’s just keeping your data separate and having that database connection handled really gracefully and very seamlessly by Rails. Rails does not make your life easy when you want to work in this way, when you have multiple tenants and all sitting in one database. I wanted to make it as hard as possible for a developer to accidentally shoot themselves in the foot.

Kimberly (00:28): Welcome back to another episode of Recordables. This is where we’re sitting down with the 37signals technical team to dive into some of the products that we make, including Basecamp, HEY, Fizzy and open source products. I’m Kimberly, your host, along with my co-host from the mobile team, Fernando. Hello, Fernando.

Fernando (00:46): Hello, hello.

Kimberly (00:47): And this week we are talking a little bit about multi-tenant Rails. We’re diving deep into a Rails topic, and to do that, we have one of our programmers, Mike Dalessio, with us to do that. Mike, welcome to Recordables. Thanks for being here.

Mike (00:59): Really good to be here. Thank you for inviting me.

Kimberly (01:01): Before we dive into this Rails specific topic, Mike, tell us a little bit about you, what you do here at 37signals. Oh, and how long you’ve been here.

Mike (01:08): Sure. I’ve been with 37signals for about a year and a half, about a year and a half, and I work on the SIP team that stands for Security Infrastructure and Performance. I do a little bit less of the product work and a little bit more of the backend infrastructure and keeping things running around here work, which I love. It’s what I did before I came to 37signals at Shopify where I was on the team that worked on Ruby Performance there as well.

Kimberly (01:35): Amazing. Also another huge Ruby on Rails shop, so that’s very cool. Okay, so I know you’ve talked about this multi-tenant Rails a bit at Rails Worlds last year, but we’re going to have time to dive into it deep. I know you’re going to do some screen shares with us, so why don’t you just tell us a little bit about this topic and obviously why you’re so passionate about it.

Mike (01:55): Oh, sure. So the concept of multi-tenancy really just means keeping your customer’s data separate and private from each other because ideally you don’t want one customer to be able to access another customer’s data, but when it’s all together co-mingled in one big database, that is a risk. A developer could introduce a bug that lets that data go. And so multi-tenancy is a little bit about just keeping the boundaries between customer data present. And so Rails doesn’t do a great job of making this easy. It forces you to think about it as an application level concern. In all of my models, I have to keep in mind as a developer that, oh, this model actually contains data for all of my customers and I need to remember to select data that only belongs to account ID 1234. And if I forget to do that, it’ll work in tests.

(02:56): And as soon as I push into production, people will see each other’s. So when we were building Fizzy here at 37signals, we wanted it to be multi-tenant and we wanted one application to work for all the customers similar to what Basecamp does. And David cooked up this hair brained scheme of what if we used SQLite, which is a file-based database. So instead of it being a server in the cloud that you connect to, it’s a file that sits on your local machine and that’s where the database is. And so you don’t have to go over the network so it’s faster. It’s also a little more self-contained where you can just delete the database or move it around or copy it or whatever you need to do. And he really wanted to push the boundaries of what we could do in Rails with SQLite. And so his idea was what if we made Fizzy use the multi-tenant with SQLite where there’s a separate database file on disk for each of our customers in Fizzy, if you think about it, that could be tens of thousands, hundreds of thousands of customers, we’re going to have hundreds of thousands of SQLite files sitting on disk. Wouldn’t that be really neat if we could pull this off and kind of pointed me at that problem and said go.

Kimberly (04:11): Okay, wait, can I ask a quick question?

Mike (04:12): Sure.

Kimberly (04:13): This commingled database, I would assume that’s common, like what most people are doing. Yeah?

Mike (04:19): Yes. That is the dominant mode of deploying web applications today in the world, and it works fine, but it also means that when you scale up, you have to scale your database machine way up. If you look at what we have to do to run Basecamp and HEY, we have large machines running the database. That database is replicated across multiple machines for redundancy and it takes a lot of operational money and effort to keep that running.

Kimberly (04:52): And we don’t currently have any of our products that are not commingled databases where each user has their own database. This would’ve been the first iteration of that. Is that a true statement?

Mike (05:03): Correct, yes. That’s what we were hoping for was to really kind of change the paradigm about how you deploy SaaS apps. Can you do it without having to have a big humongo database machine that everything reads and writes its data to? And we almost got there. I guess maybe the punchline to this story is we didn’t launch Fizzy with SQLite. Unfortunately, it was a really complicated problem and we ran out of time. It doesn’t mean it’s not solvable and it doesn’t mean we won’t solve it, it just means it wasn’t solved when we had to go live with Fizzy.

Fernando (05:36): So I sort of asked this question last time with Jeremy when we were talking about the S3 migration, and I was like, isn’t it just copying files? Well, with multi-tenancy, isn’t this just sort of like you have the file there, what’s the complicated aspect of it?

Mike (05:53): Yeah, that’s a really good question. I know there’s a plan to have Kevin on to talk about SQLite replication, and so he can probably talk about this a little more deeply than I can, but I’ll try. If you only have one machine, you only have one computer running your website, that machine takes all the connections, all the requests that come in, it can go to its local files for all of its data and return the response and life is good, and this is fine. If that was what we wanted to launch with Fizzy, we totally could have flipped the switch and we would be live on it today. But the reality is what you want to do to have a world-class product is you want to have it geographically distributed. You want to have a data center in Europe, you want to have a data center in the PacRim, you want to have a data center on the East coast and on the west coast, and you want customers to be able to go to the closest machine when they do a read. And you want them to be able to all write to the correct machine.

(06:57): So if you think about SQLite, if I had four machines, one in each of those regions, if I do a write, how does the network know where to send my request to? Because that database can only be written to on one of those machines. That’s the thing about SQLite is there’s only one master copy that can be written to at any point in time. So if I’m in Europe and I try to do a write, how does the network know that it needs to go to the US and do the write there because that’s where my database is? And that’s where the complications come in because we can replicate the database so it’s readable everywhere. But then when I do the request, all of a sudden this integration between the top and the bottom of your stack, the network stack at the top, which is what carries the request, and then you have Rails and everything down to the database and the database level needs to be coupled to the network level in a way that’s very unnatural. And so that’s what Kevin and I had been trying to build, mostly Kevin, is an integrated stack where the database routing and the request routing all worked the same way. I’m not doing it justice there, but Kevin can talk about it in way more detail than I can.

Fernando (08:11): No, no, no, that makes sense.

Mike (08:12): And that was actually the bit that caused us to say this isn’t quite ready yet. And we launched Fizzy with MySQL instead of with SQLite.

Fernando (08:23): That makes perfect sense. When you have a stack of things, you usually don’t want to jump between more. One, you don’t want to jump between more than one level, so you want to go straight down or straight up. You don’t want go like, oh, the database needs the network and then that’s where things get fizzy. No, I’m kidding. My question is, if I’m understanding this correctly, there is a single source of truth. Let’s talk about this for database example. There’s a single source of truth. So is that by design due to how SQ Lite? Oh my god, I’ve only read that. I’ve never heard it spoken.

Mike (09:03): I think most people say SQLite. I think the maintainers say S Q L Lite.

Fernando (09:09): That’s how I would say it in Spanish. I’ll say SQLite. Is that a SQLite design decision or is that like a 37signals decision when you’re trying to explore?

Mike (09:20): That’s how the database works. So SQLite is intended to be, it’s just a file that’s on disk, which makes it great for things like mobile applications, which you can talk about right now, right? SQLite is the most widely deployed database in the universe because it’s on every Android and iOS device in the world. And it’s great because on a phone you only have one writer, which is the phone. Using SQLite for a web app is a little bit different. It’s not necessarily what SQLite was designed for, but again, so because it’s a flat file, you can have multiple processes on the local machine open up that file and write to it or read to it. So on a single machine I might have a dozen web server processes running, I can all read and write to the file and it’s fine, but as soon as I go to another machine that doesn’t have access to that hard drive, that machine can no longer write to the file. So that’s really the boundary we’re talking about, about the file system boundary.

Fernando (10:18): And if you say that these are database level design decisions, I assume that there’s some sort of conflict resolution or something that can be done at the file level so that oh, okay, there’s synchronization between two SQLite databases?

Mike (10:39): This is a great question. So if I have multiple processes that are reading and writing to the database, they can all read at the same time, but only one can write at a time. And for this, I think the default is to use POSIX file locking. So it’s just using the file system lock system call to prevent multiple processes from writing to it at the same time. So this also means that SQLite has an interesting bottleneck that RDBMs like MySQL or Postgres don’t have, which is you can only have one writer at a time, but on balance, because the writes are so fast, right? They’re like, I don’t know, nanoseconds, right? It’s you’re writing to the local disk and not having to go over the network, which is maybe milliseconds Most of the time it’s not a big deal that you can only have one writer at a time.

Fernando (11:29): That is really interesting.

Mike (11:31): So we’ve drifted a little bit, so I want to just tie what we’re talking about back to the multi-tenant conversation real quick. So we wanted to use SQLite and we wanted it to be multi-tenant, but those two things don’t necessarily need to be coupled together. So the Active Record Tenanted gem, which we open sourced a few months ago, we have had a couple of contributors who’ve started working on MySQL and Postgres support for this gem. So the same kind of thing applies where I might have for, I’m going to imagine a scenario here, regulatory reasons I need to have a separate database for Europeans than I do for North Americans, GDPR, what have you, contractual regions, I don’t know. But if you have that problem, Rails does not make your life very easy and a lot of people end up deploying two versions of their app and one for Europe, one for North America, and you don’t necessarily need to. If you think about tenant, not necessarily as a customer-specific thing, but maybe it’s a region-specific thing. You could have all of your US customers route their MySQL database connection to your US East to Amazon region, and you can have all of your European customers routed to a Hetzner box that you have sitting in Germany. That’s the kind of thing that multi-tenancy can give you as well. It’s not necessarily between customers, but it’s just keeping your data separate and having that database connection handled really gracefully and very seamlessly by Rails where today it’s very difficult to do that.

Fernando (13:12): Exactly. Yeah. I was going to eventually get to that point. When you explained it like, oh, the database layer is down here and the network layer up there, and you were trying to make it work, I’m sure both you and Kevin plenty of times where were like, we’re trying to make a square peg fin into, right? So my follow-up question was like, well, this sounds like sort of an application concern. And then you mentioned the decoupling, like, oh, okay, we were trying to make SQLite work with multi-tenancy, but they’re not necessarily tightly coupled. You could just split them apart and then use MySQL or Postgres or whatever. That is really interesting. My follow up question then is what is actually implemented in Fizzy? Yeah, what did we go live with? Yeah, did we just cut it all up? Open source the gem and Fizzy works like regular and there’s some open source work that’s happening?

Mike (14:16): Yeah, so right now, Fizzy doesn’t rely on the multi-tenant thing at all. The version, that version that we open source, but Fizzy itself is an open source project. And so you can actually go back through history and you can see the point in time that we had the multi-tenant libraries all working and we ripped it out and replaced it with MySQL.

Fernando (14:35): What is that like on a technical level?

Mike (14:37): Yeah, sure. Okay. Do you remember I mentioned that Rails does not make your life easy if you want to do multi-tenancy. So there was a long tail of bugs that were introduced by switching from SQLite to MySQL or from multi-tenant SQLite to using normal Rails and MySQL. Because if you think about if you are in multi-tenant mode, you have your database, your Rails processes connected to the database, and so you can do something like, I want to find the user named Fernando, so select star from users where name equals Fernando and it’ll bring you back. But now if we are in this MySQL database where all of the customer data is co-mingled, right? I can no longer say select star where name is Fernando, I need to say, and where account ID equals 1234. And so there were a bunch of places where we were relying on this implicit assumption that we’re connected to a database where all of the data belongs to the same customer.

(15:44): And now we switched over to MySQL and now all of a sudden we had this assumption sprinkled throughout our code where we weren’t specifying the account ID the way that we should. And so we had to go in and find that. So you can actually go through the Fizzy source code history and see us fixing these bugs one by one as we went through it. I think it was all one big bang, one big pull request, but there were a lot of very small changes we had to make in order to do that. And that’s the sign I’m talking about here, which is that Rails does not make your life easy when you want to work in this way when you have multiple tenants and all sitting in one database.

Fernando (16:19): I’m not a Rails expert, but the example that you gave sounds very low level when it comes to SQL instructions. Like Rails wouldn’t… do something like, no, no, no. My question is specifically what part makes it difficult from Rails to actually build multi-tenant?

Mike (16:42): It, it’s just the usability. There’s nothing technical preventing. So there’s a really good mature, well-tested gem called Apartment because tenants and Apartments and everything.

Fernando (16:55): That’s a great name.

Mike (16:56): So a word play and Apartment actually does make this easy. Where Apartment will you have to decorate your Rails models with like, oh, I’m tenanted by account id, and you have to put that in all of your models.

(17:10): But once you do that, then Apartment will do some of the heavy lifting and say that you have to have an account ID specified or else it will raise an exception to make it a little safer. And the drawback, I talked about this a little bit in my Rails World talk, Apartment was built a long time ago. I think it’s like 12 years old and it’s still, it’s showing signs of its age and it’s not totally thread safe in all circumstances. So it’s not taking advantage of a lot of modern Rails. That connection handling got completely rewritten a few years ago. So Apartment is still, it’s a little inefficient. So Apartment I think does this thing where it closes the database connection and it opens a new one, and then if you want to switch back, it has to close this one and open the new one as opposed to just keeping a pool of connections open and maybe doing something where you like, oh, there’s a max on a number of connections you can have open at a time. It will reap them if they’re unused. And I built all of that into the Active Record Tenanted gem. It’s all there. So Apartment does handle the usability issues that you’re talking about though. Why is it hard? It’s simply like this implicit where account ID equals 1234 has to be in all of your database queries and Apartment will do a little bit of the heavy lifting there. It does a good job,.

Kimberly (18:32): But it didn’t do what we needed it to do for Fizzy.

Mike (18:34): It did not at all. No, no. It was a little slow. It wasn’t thread safe, like I said. And also, oh, this is the other thing. It’s Apartment only deals with the Active Record bit of multi-tenancy and it doesn’t make the rest of Rails work in multi-tenant mode.

Fernando (18:55): What is the rest of Rails?

Mike (18:57): Let me give you an example. So Rails is a really big framework. So I wrote a list, okay, if you think about, there’s a lot of things that Rails does that need to be aware that we’re working in this multi-tenant space. So one is fragment caching. If we generate a view in Rails. Rails is a really great job of caching that view so that we don’t have to regenerate it the next time you ask for it. But the cache is based on the record ID. And so you need to make sure that if you have two records with the same ID that belong to different accounts, so account A, record one and account B, record one by default will both try to write to the same cache record. So this is another way that you can accidentally get data from one customer being shown to a different customer because you’re not hitting the database, but you’re hitting the view fragment cache.

Fernando (19:58): Oh.

Mike (19:59): If that makes sense.

Fernando (20:01): Yeah.

Mike (20:02): So you’ve got to be careful about that. You want to maybe include your tenant ID when you are uploading blobs, attachments, pictures, whatever. If you have a customer who deletes their account, you want to be able to easily delete all of those attachments or maybe they want an export. There’s no good way in Rails by default to say, oh, these are all the files that belong to customer A because they’re all dumped into one big directory or one big S3 bucket with no differentiation. So having the tenant ID be in there would be really great so that you can say, oh, this is the first tenant account ID slash the blob ID is just a good way to keep all of files organized on disk. That’s something else that the Active Record Tenanted gem does.

Fernando (20:53): It sounds like a lot of work.

Mike (20:55): It was a long tail. It was a long tail and it took about six months to get it all working well and Fizzy.

Fernando (21:00): Wow.

Mike (21:01): But as a result though, it’s super easy to make an existing application multi-tenant. And I can walk you through that if you want. I can share my screen, actually walk you through what that looks like.

Kimberly (21:14): Yeah, let’s do it. Yeah. Mike, it kind of sounds like some of this solution is to eliminate the human factor. As you’re describing this, you’re like there’s all these ways that you can mess this up, and it sounds like it’s trying to prevent some of that human error potential.

Mike (21:33): Yes. I think that’s a big part of it is making sure that it is… well, I mean you can turn that on its head a little bit and be like, oh, well, I’m trying to make it usable because people will invariably, if you design something that’s not very usable, then invariably people are going to hold it the wrong way. If you don’t make the hammer have a nice long handle, they’re going to hold it by the head and try to pound something with it. So usability and also safety or for me, at least the same dimension, right? It’s kind of the same topic.

Kimberly (22:05): Okay, what are we looking at here?

Mike (22:06): So here’s what I’m going to show you. We have an application called Writebook that is open source and that you can get through the ONCE program and you can run it yourself, and it’s just for handling documentation. So we got a bunch of books here that are our internal documentation at 37signals. For example, if I go into the programmer’s handbook, then I can get a nice list of topics and each of these is page in our documentation. It’s just Writebook. This is a SQLite-backed application. The idea is you can run Writebook anywhere you want Digital Ocean or on your local machine. It’ll be writing to a local file that’s just a SQLite database. And so I thought this would be a really great way to show how easy it is to make something multi-tenant. Why don’t I convert Writebook into a multi-tenant application.

(22:55): So instead of having one global, everybody sees the same documents, it’ll actually be multiple instances. You can actually have multiple instances of Writebook. So it’ll be like fu.writebook.com or bar.writebook.com for individual customers. So this is what I did and it turned out to be super, super easy. So I’m going to show you the diff. And I’m not lying. This is the entire set of changes that I had to make to run in code to make Writebook multi-tenant, that is right to multiple SQLite databases. So I’ll walk through the changes real quick. You have to say that your main database connection is tenanted. It’s just… if I wanted to connect to a specific database, I could add some arguments here, but by default I’m just saying, hey, listen, this is the class that I want tenanted and all of our models inherit from this class, so they are all also tenanted.

Fernando (23:55): Can I ask tenanted based on what?

Mike (23:58): Oh, so the tenant, the tenant is just a string. It’s just a name.

Fernando (24:05): Oh my god, of course.

Mike (24:06): And the name is used by default. The name is used, I’ll tie it together in a minute, the name is used by default for the subdomain that you’re going to go to. So fu.writebook.com, the tenant is going to be fu.

Fernando (24:19): Yeah.

Mike (24:20): And then the database name on disk is also going to be fu.

Fernando (24:24): Of course.

Mike (24:24): So that’s how, again, the network layer and the database layer get tied together because we’ve named them the same thing here. And so at the database level, this is our database config. What we’ve had to do is change the name of the database, the path where it used to be storage/db/development. It’s now storage/… oh, I’ve got this little percent tenant in here. So, if you’re familiar with printf() this is just a format specifier that at runtime is going to be replaced with the name of the tenant. And I have to do that for all of our environments, so development, test and production is the same change. And then I have to make sure that when I actually hit in development, I hit a host that it has a wild card where I can hit any host I want. So lemme show you how this actually works.

(25:20): It might be a little bit easier. So I’m going to start off from scratch. I’ve deleted all the databases. I’m going to start up the server and it’s going to say, if I try to go to writebook.localhost, it’s going to give me an error and it’s going to say, oh, well, I can’t connect to a tenanted database while you’re untenanted. So it’s telling me that I’m untenanted because I haven’t provided that subdomain. So if I go and I do mike.writebook, it’s not going to say, ah, tenant not found Mike. Very cool.

(25:58): So if I go to the console, the Rails console, and I say application record, and that was the class that had tenanted added to it. It’s our base class. Create tenant, fubar. So what it just did was it just created the database and it applied all of the schema migrations. And now that database is live on disk. If I go and I look in find storage, here’s now a database called fubar db.development.sqlite. If I go here instead of Mike I now go to fubar, it’s going to kick me into the first run and prompt me to enter everything. So I could be like, oh, my name is Mike, my fake password, right? What it’s going to do now is it’s booting up the database, loading up the initial, oh, it didn’t do it. Hang on, because my password wasn’t long enough. There we go.

(27:04): Now it’s going to work. Ah. So now I’ve got the Writebook manual, which is by default this gets added. And if I look at what’s in storage now, you’ll see this is not great. Tree…. right? You can see that now I’ve also got all my active storage files are being stored in a directory called fubar as well, like automatically, because I mentioned I think that all of the blob keys get the tenant slash added to the front of them, which turns into on disk, I’ve now got a directory that has all of the fubar tenants images from this document.

Fernando (27:44): This is super cool.

Mike (27:46): Thank you. That’s great to hear.

Fernando (27:47): Is this a new default for, should this be the new default for Rails? It feels like a straight up improvement.

Mike (27:53): I would love for this to land in Rails. I feel like it’s got to actually be running somewhere in production first.

Fernando (28:00): No, no, no, of course, of course, of course. But the concept…

Mike (28:02): My plan, my dream is this would get upstream into Rails because it’s additional, if you don’t want it…

Fernando (28:08): Who cares?

Mike (28:09): You can continue to use Rails the way you always did. But if you want to work in this way, then yeah, turn on a bunch of these configs and you’ll get it for free. Yeah, I would love to see this in Rails at some point.

Fernando (28:20): Now, one question I have is, in this hypothetical scenario, if you were to have multi-tenant as a default, what would change in the configuration so that you didn’t need to have subdomains tied to tenants? I’m sure it’s possible, right?

Mike (28:45): Yes. So the gem by default ships with what I will call a, it’s actually it’s Rack Middleware, which maybe isn’t a helpful phrase, but the web server framework is called Rack, and that handles the request. And what we can do, and the way it works is the request gets handed to each stage before it finally arrives at the app. And you can insert a stage in there, and that’s what the gem does or inserts the stage that looks at the subdomain. It says, ah, there’s a subdomain in here. I will now look on disc to see if that is a valid tenant. And if it is, I will connect to it. And if it is not, I will raise an error. So by default, that looks at subdomain, but you can override that and you can use whatever logic you want it’s provided. I will see if I can get the gem…

Fernando (29:39): You could just consider… sorry.

Mike (29:39): No, go ahead.

Fernando (29:42): Oh, I was going to say, you could basically consider the base, like the default domain as a tenant, right?

Mike (29:53): Oh, you could. Yes, you could.

Fernando (29:55): And then that way you could get both the subdomains, each as individually named tenants and the top level domain as a tenant itself.

Mike (30:02): Yes, you could.

Fernando (30:03): Wow.

Mike (30:04): You can do whatever you want.

Fernando (30:05): That is super cool.

Mike (30:06): It’s literally the way this case is configured is you actually just pass in a lambda, an anonymous function…

Fernando (30:11): Right, right.

Mike (30:13): That takes the request and hands back the tenant name. So by default it’s just going to use the subdomain. And so if you look in Fizzy, in the source code, what we’re actually doing there is we look at the path and the first section of the path is a big number, and that’s the account ID. And we do that just like Basecamp does. So it’s that you can take it out of the path, you can take it out of the domain, you can take it out of the host name if you want to. Some of these are operationally easier than others. Having a separate domain for every customer means you have to worry about regenerating SSL keys, and how is your web server going to do that? And can you dynamic, can you automatically regenerate those SSL keys every 90 days? So some are harder or easier, but you have the flexibility to do whatever you want.

Kimberly (31:03): Mike, tell me this, I know in your Rails World talk, you mentioned something about safety checks. Is there something that you can show us on how that exactly works or what you’re checking for?

Fernando (31:13): Safety clearly.

Mike (31:15): Yes. Yes, absolutely. So one of the things that I think is, we talked about usability a little bit earlier and usability and safety kind of being different points on the same spectrum. I wanted to make it as hard as possible for a developer to accidentally shoot themselves in the foot. And so if I….

Kimberly (31:36): Eliminating that human error of possibilities again.

Mike (31:39): Yes, exactly. So if I do user.first, this is on the Rails console, so I’m running Ruby code, I get an exception saying, oh, well there’s there, there’s a default tenant here, which is like development tenant when I’m in development mode, and that tenant doesn’t exist. So if I do our application record with tenants and then I can pass a block and whatever I do in this block is going to be in the context of this tenant. So I can just say fubar, which is the tenant we just created, I can do user.first and it’ll return me a user from that database. And I know it’s from that database because I can see in here it has tenant fubar as an attribute on that model, which is great. Tells me that that object belongs to that tenant. So I can pull this out.

(32:33): I could say user equals blah. Now what happens if I try to update that user or write to that user while I’m in the context of a different tenant? It should raise an error for that too. So I have to create a second tenant, which I will just call second. And again, it migrates the whole database. So now what if I am in the context of the second database and I tried to do user.update, name is… I’m going to try to change the name on that user. It’s going to give me a safety exception saying with the user model belongs to tenant fubar, but you’re currently connected to tenant second to prevent you from doing this at runtime. So if there’s any kind of a bug that’s introduced where you might be saving one record to a different tenant’s database, you can’t cross the streams anymore because it does the safety checks where it compares the objects tenant string to the connection string, if that makes sense.

Kimberly (33:47): So those safety checks are already built into the gem that you created.

Mike (33:51): That’s correct. They’re built into the gem to make it as hard as possible for people to cross the streams and mix commingle customer data.

Kimberly (33:59): You’re making this sound very straightforward and very simple, but we also didn’t put this in Fizzy. I’m curious if we just needed more time because we’re working against a deadline or there was something else that made it not appropriate for what we were doing. Why did we not? How come it didn’t happen?

Mike (34:17): Yeah, the main reason is that we ran out of time. We had everything working and there were some edge cases. If you think about going live with a global product, you want it to have automatic failover where if a machine goes down, you want everything to immediately cut over to the backup. And with the SQLite database, that becomes really interesting to do because you need to make sure that you’ve got the database file being replicated in real time to a second machine where it’s available in read only mode. And then as soon as this machine goes down, you need to have something outside cut requests over to the second machine and it go into writeable mode. So this all of a sudden becomes the primary, where it used to be a secondary or a backup machine, and that was where things, there’s, there’s a long list of edge cases there.

(35:17): There’s a phrase people use, which is emergent behavior. Complex systems have interesting emergent behavior, that is behavior that you only see in when certain weird edge cases happen or when things happen in a certain order. And what we were going through was about once a week finding a new edge case in this complicated system we had built where there’s SQLite and there’s tenanting in Rails, and that was pretty solid. That was working well, but then we needed to replicate it globally and we needed to have network routing route requests to the right place. And then we also needed failover to work properly. And making all of that work in time for the release date just didn’t happen. We just didn’t have enough time to work out all of the bugs.

Fernando (36:05): I’d be remiss if I didn’t ask this, but this work was like last year, right? Middle of last year more or less?

Mike (36:13): It was. So the multi-tenant gem was worked on mostly the first half of 2025, and then the replication and failover work was like the middle of 2025, like the summer and fall of 2025.

Fernando (36:32): So given that this is 2026, the obvious question I have to ask is do you think the amount of time would’ve been reduced had you been using AI?

Mike (36:47): Oh the AI question. I don’t, I don’t know. I don’t know. If we could get an AI, an agent that was able to do things like deploy to our staging environment, actively monitor all of the systems like simulate, it might’ve been helpful to have an agent help us with a lot of those testing scenarios.

(37:16): But we actually, we did use AI for quite a bit of the replication stuff to if you go look at it. So I’m hopeful that Kevin will, open source Beamer. Beamer was the code name of the replication stack that he wrote. And a lot of that is Go code, which LLMs are great at writing Go code because it’s very simple language, syntactically anyway, it’s simple. So we did have a lot of AI help on some of this. The multi-tenant gem was I think a little bit tougher task for agents because it was something that was novel and it was trying to shoehorn something new into the existing framework. And so there were a lot of design decisions that had to be done. So the version of the gem that is open source now I think is version four if you believe that. I think it’s version four, where the first version was just spike threw it away, but then I wrote two more versions and I was unhappy with the API and threw them away before I landed on an API that I was really happy with.

(38:28): That felt Railsy and that felt like it wasn’t getting in the way. And also that was extensible in ways that would lead to creative problem solving. Let me give you an example of that. So I always think that you can tell that something is designed well, if people can use it for purposes that it was not originally designed for. And I came across one of these late in the Fizzy development work where we realized that we were already tenanting by customer. So every customer has their own SQLite database, but we wanted to replicate by region. So we wanted to have, for example, all of the North American customers in our Chicago data center and all the European customers and our Amsterdam data center. And then that meant that when we were running background jobs, we would have to have one database for North America and one database for Amsterdam.

(39:34): Because the job workers are local, they have to write to the local SQLite disks, and they have to keep their state in their own database too. So it’s like you’d have a job worker in the US that connects to any of the US customer databases, but then had its own database as well to keep state in. And they had to have the same thing in Amsterdam. And if we had multiple data centers, we’d do that in multiple places. And so what I ended up doing was we had the customer databases tended by account ID, and then we had the Solid Queue databases tenanted by region. So within the same application, we had two different dimensions of tenanting going on. And the Apartment gem would not be able to handle this because the Apartment gem acts as a, it’s a global, it’s a singleton Apartment, and you’d be like apartment.createtenant.

(40:31): And you’ll notice that when I was just showing you in the Rails console that I was actually using the application record. Application record was annotated with tenanted and I do applicationrecord.createtenant. And so all of the tenanting methods live on your application models. They don’t live on some other class that the gem is bringing. They live on your application models. And because I did that, I mostly did because it just felt a little bit more natural to me. But because I did that than late in the Fizzy development process, I was like, aha, I can use tenanting for Solid Queue as well. And then I would do Solid Queue record base.create tenant, and I’d pass in the region. And that was how you create the Solid Cache database because that API just ended up working much better for that use case. Anyway, where was I going with this? I was going to the point where it took me three generations of API design before I landed on one that I thought was Railsy and flexible enough to do some interesting things. And I don’t think an AI could have necessarily helped me get there better, get there better or faster.

Fernando (41:45): I agree. Sorry, I’m trying to wrap my mind around this. Why are you tenanting the Solid Queue by region?

Mike (41:54): The current version of Fizzy does not do this, but when we were still using SQLite, again, the idea was that our Solid Queue database is still SQLite and needs to live on a machine and a process needs to know that if it’s a running in the US, it should connect to the US database. And if it’s running in Amsterdam, it should connect to the Amsterdam database, and we needed to replicate that also. So we were also replicating the Solid Queue databases so that if the US went down, it wouldn’t be a complete outage. We would reroute everything over to Europe and North American customers would just get routed to Europe until we were able to get the US back up and running. So for failover and replication purposes, we’re still replicating those databases. It’s just that then in that case in Europe, we would have two Solid Queue clusters running. One would connect to the European database and one would connect to the US database. They’re both still local though. So that’s why we were using, we could have done something more complicated, but the fact that tenanting was right there fits and it just automatically multiplexed to the databases based on whatever region they were running for. You look at the customer database and you say that customer’s in the US region, so I should connect to the US Solid Queue database. It just ended up working really well.

Fernando (43:20): That’s awesome. That’s really interesting.

Mike (43:21): Well, if it had gone live in production, it would’ve been even more interesting, but maybe someday we’ll get there.

Fernando (43:25): Do you see a path forward?

Mike (43:28): I do. So I showed you, I just made Writebook with, honestly, that was five minutes of work. I was able to make it multi-tenant. Now there’s a little bit more you need to do around like, oh, I need to document how this works for people and I need to give them a script so they can create a new account when they want to, and they may need to create a wild card SSL cert if they want to run this themselves on their own machine. So there’s a long tail of little things, but the bulk of the code change was very quick, five minutes. So one possible path forward that I’ve been talking to David about is can we take all of our ONCE products that are all running on SQLite and make them all multi-tenant, demonstrate that the multi-tenant gem, Active Record Tenanted does work, get some miles on it. I feel like I really want to run it in production somewhere so that then it’s a more compelling argument about upstreaming it to Rails. I would love to try to get a future product that we’re working on on this as well. But again, I feel like the Active Record Multi-tenant gem is pretty solid. It had a year almost where we were running it internally on Fizzy. We were using Fizzy most of last year, and it was solid, it was work, it worked well.

(44:50): We didn’t really have any problems with it for the last six months or so. The complication was around replication and failover, and really we need to solve that. And so I want to keep working with Kevin to hopefully finish up Beamer and really get that working. Because there’s another missing concern in Rails that I would love to fix at some point, which is failover. There’s no concept in Rails of what host am I running on and should I fail over? Solid Queue has started to scratch at this with how it manages its connection handling. And I actually built, I have a fork of Solid, Solid Queue that knows whether it is in active or passive mode based on something that’s in the database. You can have a flag in the database that says, you have your primary in North America and your backup is in Europe.

(45:41): You have the Solid Queue job cluster running in both places, but nobody’s doing anything in Europe because the US is the primary. So this is all code that we already have in HEY, that it basically lifted, I jammed into Solid Queue. It totally worked in Solid Queue. And I was like, all right, we’re using this in one app, we’re using it in a library. Why isn’t this primitive in Rails? So why shouldn’t there be a class in Rails that tells you, are you in fail… are you in primary mode or are you in passive backup mode? And if you have that, then a lot of the Beamer replication code becomes much simpler as well. So there’s a whole path here, but I really want Rails to become a little bit more capable around failover and replication than it is today.

Kimberly (46:28): Mike, question for you, for anyone who’s listening and is like, okay, I totally get this and I can use this gem and this doesn’t seem hard, do you have any pieces of advice or things you should make sure to look out for that you can share?

Mike (46:42): Yeah, that’s a really good question. So the one thing I think that I would love some more feedback on is how the gem approaches Action Cable. So brief summary of what Action Cable is, is it’s the ability to do, to push data to a web client. Under the hood it’s using web sockets and maintaining an open connection. And so the most common use for this these days in Rails is Turbo, Hotwired and Turbo, where you can essentially broadcast data. So we use this everywhere in our apps to push notifications, to push chats, to push card updates if someone changes the status on it. So you’re pushing data to the web client. That connection also has to be tenanted because you don’t want to push data for one customer to a different customer. And a lot of that is reusing the same middleware I referenced earlier where request comes in. How do you find out what tenant that request is for? I’ve only solved one problem with that, which was Fizzy’s problem. And so I would ask people if they’re going to kick the tires on the gem to make sure that Action cable is wired up correctly for your use case. And let me know if it’s not.

Fernando (48:01): So Mike, I’m on the mobile side. So when it comes to rich text, where’s that stored? How do you handle that with the multi-tenant?

Mike (48:11): Thanks for asking. So Rails has a relatively complicated system for how it stores its own data. And by that I mean there’s metadata around Active Storage uploads. There’s some, the actual action, the rich text content is stored in a separate record. So Rails needs its own tables, basically. So if you’re going to use any of the rich text stuff, you need Action Text and Active Storage. And that means that those need to be in separate tables. And so the gem actually jumps through some hoops to make sure that the Rails models are using your tenanted database too. So the idea is if in my tenanted database I have my card class, and the card has some rich text and the rich text has some attachments. You want to be able to do a join across all of those tables. So they all need to be in the same database.

(49:08): And so there’s actually a concept in the gem of being a subtenant of your database. So your application models are the official tenants, they are the tenants, but then you can have these Rails records that normally would be, I don’t know, in some other, in your primary database. If your primary database is tenanted, then all of a sudden these Rails records are subtenants and we can’t do it through class inheritance. In your Rails application, all of your classes inherit from application records. So you make the change there and everything else gets it because the subclasses, but the Rails records are in a completely separate hierarchy of classes. And so we actually, we have to actually inject some behavior into those classes. But it all just works at the end of the day where then you can do that join across all of those classes. And it also means that those Rails data records also don’t get commingled. And we have that multi-tenant boundary between all that data.

Kimberly (50:11): Mike, so is there a place where people can find this gem? And if so, we’ll add it to our show notes.

Mike (50:16): Yes, there is a GitHub repository for the gem. It is open source. It’s been open source for a few months now. And also you can go to the Fizzy source code if you want, go back through history and actually see how it was being used.

Kimberly (50:26): That’s perfect. Well, thank you for joining us. This has been an episode of Recordables a production by 37signals. To hear more from our technical team, like Mike and Fernando, check out our developer’s blog. That is at blog. No, that’s at dev.37signals.com. I like don’t even know what the address is.

Other posts in the “RECORDABLES” series

Moving Mountains of Data off S3

Jeremy Daer, Kimberly Rhodes, Fernando Olivares — Thu, 8 Jan 2026 18:00:00 +0000

In this episode of RECORDABLES, we talk through the final and most nerve-racking part of our cloud exit — moving massive amounts of data out of Amazon S3. Principal Programmer Jeremy Daer shares how we moved billions of files with no downtime. He covers everything from dealing with bandwidth limits and AWS constraints to building custom tooling when off-the-shelf options won’t work.

The conversation gets into the human side of a project like this, including verification, anxiety, and the moment you finally hit delete. You’ll also hear how long it actually takes to move that much data and the tools we used to make it happen seamlessly.

Watch the full video episode on YouTube.

Timestamps

00:00:00 — Introduction
00:02:05 — Why S3 was the last (and scariest) piece
00:08:34 — The volume of data to move
00:11:11 — Bandwidth limits and AWS constraints
00:13:12 — The custom-built Rails tool for copying and reconciliation
00:21:25 — The logistics of hard drives, write speeds, and network connections
00:28:05 — The intentional order of moving data
00:49:55 — Anxiety, verification, and the fear with deleting data you can’t get back
00:54:13 — Was there any downtime?
00:58:56 — Essential tools that made the migration possible
01:07:03 — What happens next

Links & Resources

Transcript

Episode Highlights (00:00:00): Well yeah, there’s no real difficulty. It’s just copying some stuff, right? Among those five petabytes of data was spread across hundreds of buckets and on the order of about five billion objects. The concept is straightforward, but you got to be careful and do it right. And there are easy ways to do it right, but there are a lot of easy ways to go wrong too.

Kimberly (00:00:22): This is for Recordables, a place where the team at 37signals shares their behind the scenes work, building Basecamp, HEY and open source projects. We’re diving deep into what we’ve done and how we’ve done it so you can learn a thing or two and learn from some of our mistakes. I’m your host, Kimberly. I’m joined to help with the technical side of discussion by Fernando from our engineering team. Hello, Fernando.

Fernando (00:00:45): Hello, hello.

Kimberly (00:00:46): Well, we are talking this week a little bit more about our move out of the cloud. If he’s been following 37signals at all, you know that we started a move out of the cloud back in 2022. We have wrapped up the big portion of moving data out of S3. To join us, we have Jeremy Daer, principal programmer here at 37signals who did much of that final work to talk about it. Jeremy, thanks for being here. Before we dive into our topic today, tell us a little bit about you and how long you’ve been at 37signals.

Jeremy (00:01:17): Yeah, I’m a programmer here. I’ve been around for some ages and this storage migration is the most recent of probably half a dozen over 15, 20 years. So coming to it feeling like it’s time. We need to move all the bits across the continent again.

Kimberly (00:01:40): Okay. Well, I’m sure there’s a lot to dive into. Fernando, you’re going to help me with all the technical aspects of it, but let’s just start kind of where we ended, if you will. We know we’ve started this move back in 2022. S3, moving that piece of data was the last part of the project as I understand it. Kind of tell me why that was the last piece and then we’ll dive into the technical side.

Jeremy (00:02:05): Yeah, there are two things. It’s expensive and it’s scary. We’ve got a lot of things to move and we had built a lot of trust in S3. It’s super durable, super reliable. You can’t go wrong with S3. It’s like the old school IBM thing. Nobody’s going to get fired for using IBM. Whereas storing your own stuff, it’s all on you. And so if you haven’t been shouldering that risk for a long time, you kind of forget what it feels like. And so a bunch of planning, a bunch of mitigation work, risk assessment can help prepare for that. But kind of getting over that hump and just choosing to do it, we’re going to do it solves that problem. The other is cost. And what are we doing here? It’s easy to open up our wallet and pay AWS. And it causes us to cry a little bit or a lotta bit.

(00:03:05): But when we want to replace it, we’re looking at spending hundreds of thousands, millions of dollars on something. And we’ve got to be pretty sure it’s going to work. Well, we’ve got to be certain it’s going to work, but also not spin our wheels for weeks, months, trying to figure out and prove it for sure. So we need to come up with some kind of framework for understanding what we need and being able to prove it before spending a bunch of money. Third thing really, S3 also has this tricky bit which motivated some of the cloud exit of you got to pay. You got to pay for everything. And one of the things you got to pay for is a bandwidth coming out of S3. So if you want to move your data, you’ve got a little bit of, again, a lot bit of a handcuff situation.

(00:03:49): You got to pay to get your data out. So the EU came up with some kind of regulations around this and all the major cloud providers got out ahead of the regulations saying that you got to be able to exit and you can’t kind of keep people’s data for ransom. And it was like this before, people didn’t necessarily want to leave S3 because the alternatives were not great. And you’re on the upswing of cloud adoption of look at all the things we’re not doing. You just get to send it out to this abstracted thing and pay some monthly fee rather than doing it all yourself and paying hundreds of thousands upfront for storage hardware that you’ve got to run yourself and maintain yourself. So anyway, AWS came up with this waiver program wherein you could get your data out of S3 under certain conditions. And the conditions run like this.

(00:04:51): You’ve got 60 days, 90 days, whatever. You’ve got a ticking clock and you got to get all your data out. You can’t half ass it. You got to get it all out. And if by the end of that time you can prove that it’s all out, then you’ll get AWS credits for the bandwidth cost. So you got to estimate how much stuff you’re storing and then how much bandwidth it should take to get out. If you do it all perfectly, which of course everybody would do, then that would be your credit. So the public messaging is like, “We’re chill with this. We’ll let you get all your data out.” The reality is you’ve got this kind of tight binding contract and you got to do it perfectly and then we’ll give you a refund.

Kimberly (00:05:42): Oh, wow. Do we pick the timeframe?

Jeremy (00:05:45): We do not pick the timeframe.

Kimberly (00:05:46): Do we pick the 30, 60 or 90 days?

Jeremy (00:05:49): It’s negotiable. So depending on … Well, the internal messaging toward customers is a limited timeframe, like 60 days or something. The external messaging is, well, we’ll work with you. We’ll be reasonable. So the reality is kind of in between. We do have great account reps, so I can’t fault that end of things. It’s been wonderful interacting with AWS, but you see both sides of things of we’re going to look to the world like we’re ready for you to leave, but in fact, we’re going to make you jump through a bunch of very tight hoops.

Fernando (00:06:26): Well, I was going to joke like, “Oh, it doesn’t sound that difficult. You just, I don’t know, copy a bunch of files over to your hardest drive somewhere.” What is the real difficulty there?

Jeremy (00:06:38): Well, yeah, there’s no real difficulty. It’s just copying some stuff, right? It’s just that if you want to copy things, well, there’s a lot of things you got to do, especially if you’ve got a lot of files stored. For most people, this copy is not technically hard. The basic process would be clone your S3 bucket. A bucket of objects is like a folder full of files. You list the bucket to see what’s in there. You copy it over to your new place and typically be doing this with a live system. So you need to have some kind of awareness of your system would need to know that you’re moving to a new system and there’s a live old system. So if you want to do it without downtime, you need to be able to store files to both places. You need to mirror your files to both storage systems.

(00:07:23): And then you need to do a copy. And after that first copy, maybe you’ve got some stuff that’s in one storage system, but not the other and vice versa. So you need to bring them into sync. You need to reconcile them.

Fernando (00:07:34): And none of this is offered by AWS. It’s all on you.

Jeremy (00:07:37): AWS does offer something that does something like this, but you got to pay.

Kimberly (00:07:44): I feel like that’s a theme.

Jeremy (00:07:46): Yeah. So if there’s a data transfer service that can do something like this, and in fact, it can do a lot more. It can do incremental syncing between disparate systems, but it is very expensive. You’re looking at tens of thousands of dollars to move large scale buckets of stuff.

Fernando (00:08:06): Was it ever an option for us?

Jeremy (00:08:07): We did evaluate it. It would be very nice not to have to do this job ourselves and to have somebody else do it. And there are vendors that do stuff like this, but again, you’re looking on the order of like tens to hundreds of thousands of dollars, depending on the size of your buckets. And most of them will scale with your storage. So they’re looking to take a percentage cut because the more you’ve got, the more you can pay probably.

Kimberly (00:08:34): Jeremy, when you’re saying the size of our buckets, how much data are we talking about that was being moved?

Jeremy (00:08:41): So in aggregate, we had about 10 petabytes of data across a bunch of buckets. Some of our applications were responsible for a lot more than others. Our average object size was about 1.1 megabytes. Of all that data, we’re also geographically distributed. So deduplicated, we had probably about five petabytes of unique objects. And among those five petabytes of data was spread across hundreds of buckets and on the order of about five billion objects. It’s a lot of stuff. You get into the realm where you can’t … If this is a folder on your computer and you tried to open the folder, your computer would crash. And you can’t list a bucket of that size without taking literally days.

(00:09:36): So there are a bunch of interesting constraints that come into play when you try to do this conceptual process of you copy, then you stop your application or whatever, storing things for a little while. Then you do a catch-up copy to reconcile, to make sure your destination has all the stuff that was in the source. And then you cut over, you start using the new system and you got to be sure that you actually got everything, that everything was copied correctly and there were no mistakes and you didn’t miss anything and nothing showed up while you were copying, et cetera, et cetera. So even at the small scale, the concept is straightforward, but you got to be careful and do it right. And there are easy ways to do it right, but there are a lot of easy ways to go wrong too. And once you get to a larger scale, there are a lot of easy ways to go wrong and a lot more easy ways to go wrong and a lot of ways to go slow.

Kimberly (00:10:32): Well, I feel like we should talk about all of those things. Not only the things that are easy to do well, but what are all the things people should avoid if they’re trying to do this?

Jeremy (00:10:43): Well, what would be wonderful is if egress were free and we weren’t stuck in the situation of needing to move quickly because when we look at the kind of bigger picture of what the job to be done is we got to move five petabytes of stuff within … We negotiated 90 days and that was based on our back of the napkin math of what size of network connection we have available and how quickly we can move stuff.

Fernando (00:11:11): Is there a limit on the AWS side on the connection?

Jeremy (00:11:15): There are plenty of limits, yes. And there are some hidden limits and some stated limits. So that comes into play when we try to figure out what our limiting factors are.

Fernando (00:11:25): Wait, sorry.

Jeremy (00:11:25): So if we do a bounds analysis of…

Fernando (00:11:27): Yeah, let me backtrack just a little bit. You said like, okay, we negotiated 90 days. What was that like? Oh, we have five petabytes if we have access to, I don’t know, 10 gigabytes per second and we do this twice.

Jeremy (00:11:40): I was chatting with our rep saying, “Here’s how fast we can possibly do it. “ So that’s what we need. And they said, “Okay.”

Fernando (00:11:48): Okay.

Jeremy (00:11:50): We built in plenty of buffer because part of this process is not just the copy. And something that anybody doing a process like this will quickly discover is that copying is, it seems like the central point and the purpose of the job, but it’s really reconciliation and verification, making sure that you did what you thought you did. And the doing what you thought you did is as costly as the copying. And it has different kinds of limitations because you need to go look and see what you’ve got. And that means listing all your files, listing all the objects in a bucket. And when this process can take days to perform, you need faster ways to do it and you need cheap ways to do it because again, you got to pay. If you want to list a bucket, you got to pay for API requests.

(00:12:43): And so you’ve got to be careful that you don’t get in a situation where you’re repeating work or doing unexpectedly expensive work. And in many cases, these like API requests are cheap. So it turns out we’re totally fine, but you got to do the due diligence first, so you don’t end up discovering that in order to save whatever tens of thousands of dollars on a paid service, we don’t end up spending tens of thousands of dollars on errant API requests ourselves.

Fernando (00:13:12): Yeah, that makes sense. How do you even … I’m trying to wrap my mind around this, where … this process must be automated somehow, right? Is it in the app? Where does this reconciliation process happen?

Jeremy (00:13:25): So that gets in a little bit deeper into the approach that we arrived at. So there’s more than one way to do reconciliation and something I’d recommend to people who are moving just a single application, which ideally it’s a situation you’d be in. So a little digression. Looking back, the ideal way to set up AWS accounts in the first place is around these kind of scopes of kind of responsibility radius, blast radius of if you ever wanted to leave, you got to take your whole account out. So don’t go putting a bunch of buckets from different places, don’t share an account. Like make accounts early and often so that accounts are aligned with your application. So that gives you the flexibility to move one application at a time. We were in that situation to some extent with some of our newer stuff, but our older applications, we had a shared account, so we needed to do everything all at once.

(00:14:30): And when you have multiple storage systems in a single account and you need to move them all, you can’t tailor the solution for the system that’s being moved. Instead, you need something that’s going to work for all of them. So that led us to, it’s got to be something that’s at a lower level and is mostly transparent to the applications we were running. So if I were doing this with a modern rails app, this is something I’d build into Active Storage at the application level. I’d build in some kind of modeling for where things are stored. So you get metadata within the application itself, knowing that I’ve got this object stored in AWS and I’ve also have it stored in my, whatever my new destination storage is. And so the process would be that I’ve got some rule that says I need to copy storage and they need to be in each of these locations.

(00:15:23): And so the application could do that with just a bunch of active jobs, copying things on its own, could do it lazily in the background and just let it trickle through, if you didn’t have these other constraints of you got to get it all done, you got to blast it at maximum speed and you’ve got to make it work with, and you got to make it work with older apps. So not all of our stuff is on latest Rails and we extracted active storage from our own apps. And so most of our older apps are using older abstractions that were kind of the source of the extraction. So we needed to be compatible with all these possible systems. So we needed something lower level.

Fernando (00:16:04): So we built…?

Jeremy (00:16:05): So we built something. We didn’t want to build something, but we also didn’t want to pay a lot of money. So kind of rock and hard place thing. And we are in the position of having expertise. We are a technical organization. We have programming and operations teams. We do all this stuff. So it’s natural for us to do our jobs. Maybe this isn’t the thing that we’d outsource. If there was a natural fit, we would. And in fact, we did to the greatest extent we could by picking purpose-built tools to do each of the jobs. And in fact, we went through multiple iterations. As we got closer and closer to the kickoff of we want to begin the copy, we reinvented the system that we built to do copying several times as we discovered kind of quicker, simpler, straighter paths to do that job. So we started out with something that would be dead simple, of distributing jobs over a bucket itself, would store objects and each object would be a file manifest.

(00:17:21): And then that file manifest would be available to a bunch of workers. We’d spin up as many workers as we needed to saturate whatever bottleneck we had. So the bottleneck could be our network connection, how fast we could pull things from S3. It could be our write rate to our destination storage. How quickly can we write objects and are we bandwith limited? Is there a metadata limit on the IO operations per second, or is it going to be the S3 read rate? So we can discover all these things.

Fernando (00:17:54): Read rate. Why does it have a read rate? They’re just throttling you.

Jeremy (00:17:59): Yeah, because S3 doesn’t technically have a cap on read rate, but they do have a cap on what they call bucket partition read rate. And so a normal bucket will only have a single partition and that’s kind of … Well, the way S3 is laid out, it looks like it’s kind of hierarchical folders, but it’s not. It’s actually key names that are separated typically with slashes. So it looks like a file path. And the natural way to partition a bucket would be, let’s take all the first file paths up to the first slash and we’re going to turn that into an internal S3 partition, which is like their own way of kind of sharding the bucket so that they can scale out. So that if you want to do things like write a ton of things to S3, it can be fanned out to multiple partitions.

(00:18:49): And as you write more, it’s smart about noticing that write rates are hot on certain partitions and it’ll split them automatically for you all behind the scenes. So you can’t even tell what your partitions are unless you give some kind of cues or use like a traditional slash character, which can help S3 figure out, but also you don’t need to. It’ll just do it for you, but it’ll do it based on your usage. So if you have a moderate usage app, the first time you’re going to have high usage is when you try to copy things out of it. So you’re going to hit the rate limits pretty quickly, especially if you have a lot of small objects. If you’ve got a lot of big objects, you’re going to be bandwidth constrained, probably on your own network connection. If you’ve got a lot of small objects, you can fill that pipe with tons of connections and you’re going to hit the rate limit quickly.

(00:19:46): The rate limit is, for the record, 5,500 GET requests per second. So that’s fetching…

Fernando (00:19:56): And we hit that.

Jeremy (00:19:57): Oh yeah. Yeah.

Fernando (00:19:58): Oh my God. That’s insane.

Jeremy (00:20:01): So out of the box tools can hit that pretty quickly, especially if you have small objects. We got a little bit lucky because when we did our migration into S3, we worked with S3 to pre-partition our buckets knowing that we’d be writing at high rates. So on the way in, we contacted S3 ahead of time saying, “Hey, here’s about how much stuff we’ve got. Here’s what the key layout looks like for the objects.” And we generally use random hash keys, so it’s like completely uniform. So on S3’s end, the job is easy because when you have a uniform key distribution, they can just say, “Let’s take whatever the first two or three characters of the key in, we’ll use that as partition keys.” So our big buckets were already partitioned, so who knows what the limit could be because when we did a load test, we didn’t hit a rate limit on those larger buckets. On smaller ones, we quickly hit a rate limit in our load testing.

Fernando (00:21:06): I know this is mostly about S3, but I’m also curious, did you really hit a write limit? Because David has spoken at length about this, about the hardware, about the amount of money that we spent on brand new hardware that’s blazing fast. Even that couldn’t keep up. We just went full throttle.

Jeremy (00:21:25): We certainly hit its limits. So it became a question of where are the limits and which one is … So we’ve got a system with a bunch of components and each component has a kind of maximum, has a cap, and which one is going to be the weakest link, which one’s going to slow us down. And it turns out that we ended up pretty similarly bound by our network connection. We got a hundred gigabit network connection dedicated to just this copying process that thanks to our data center pros at Summit, they set this up just for this job and we set up a separate VLAN for the machines that would be doing the work. So it was essentially their dedicated little network universe of, you can saturate this pipe. Turns out to not have been completely true. We were actually sharing it for a couple other things, which we discovered because as we tuned our system to eke out the maximum possible performance, we actually overshot a little bit and started interfering with other traffic, but that was what we thought would end up being our bottleneck.

(00:22:33): And to our wonder, it was not our destination storage. We had initially considered using MinIO and using hard drives and our read and write rate for normal application usage can easily be satisfied by spinning disk hard drives. We didn’t exactly relish the idea of maintaining a bunch of spinning hard drives because the failure rate can be notoriously poor depending on which batch you get and whether you have kind of a hot rack in your data center. And it’s just, we were not looking forward to it. And Eron, our head of ops team, had a line on a new storage system from Pure Storage Flashblade. They’ve got this fancy, super duper proprietary flashy, bashy setup where they’ve got, rather than using off the shelf flash stuff, they mounted their own flash on their own boards and did something a lot cheaper, kind of bringing the kind of flash of two years in the future back a couple of years, which made it kind of cost competitive with hard drives.

(00:23:46): So that was a surprise and ended up being a huge blessing, not because we need the performance for steady state usage, but because we needed the performance for the copy. So if we had been on hard drives, we definitely would have been limited by our write rate into that storage cluster.

Fernando (00:24:06): By physics, just how fast you can spin those.

Jeremy (00:24:09): Yeah. Yeah. And so, I mean, it would depend on the number of drives. And I mean, actually we probably … Yeah, thinking back, I think we probably would be able to satisfy 100 gigabit traffic on hard drives because we’d have so many of them. In any case, it was a blessing, not quite in disguise, but we’re all happy to take the other path. I’m digressing a little bit, but if you’re choosing storage systems, if you’re choosing storage systems, it can seem like make the choice based on what you can afford now, but it’s also what you’re paying over the course of five years or 10 years. And our total cost of ownership analysis was based on five years, seven year, 10 year. What it would be like to keep the system around for a long time. And the power savings alone from flash are significant. It’s a lot cheaper when your power is expensive in a costly data center that uses up less rack space.

Fernando (00:25:13): I hadn’t even thought of that. When you go from an actual … I’m just picturing a bunch of machines in a place, but if you’re going to clone five petabytes, it has to be a lot, right? A lot of m achines, a lot of power, a lot of network, a lot of everything.

Jeremy (00:25:29): Yeah. Yeah. I mean, it’s a crazy time in the storage world. There’s a bunch of new form factors for solid stage drives coming out right now. So it gave me a little bit of FOMO because I see these things coming out just as we purchase this giant system and we still made the right decision for the time, but in about a year or two, there’s going to be kind of a new generation of solid state drives coming out at 256 terabyte size per module. So you can fit … Shoot, what was it? 40 petabytes in two units of rack space.

Fernando (00:26:07): Wow.

Jeremy (00:26:07): So you could fit all of our storage into just a tiny little bit of a rack. And in the hard drive era, we’d be looking at like two full racks just for that storage. And so the shrinkage and power savings are dramatic and it’s all happening now.

(00:26:23): And a lot of this is driven by AI stuff of… people need a ton of data stored and super high bandwidth to it. And so new vendors are cropping up daily trying to do this kind of job. So they’re also driving the flash hardware side. So hopefully this will become just kind of a commodity storage problem. And you’ll be able to go to Super Micro or Dell or whatever and order up some servers that are packed full of these drives and you won’t need a special setup. You’ll just go to Newegg and…

Fernando (00:26:57): And buy one.

Jeremy (00:26:59): Yeah.

Fernando (00:26:59): Yeah.

Jeremy (00:27:00): The limiting factor by far is network bandwidth. So in almost any copy, that’s going to be the cap you’re going to hit. So having a good data center partner is essential there. We were able to get 100 gigabit connections set up within just a span of days. And we had plenty of lead time, et cetera, but still it’s just wonderful to be able to bring in a big pipe like that. And of all things, we have AWS direct connects already, but not allowed. You cannot use your special direct, fast connection to AWS to do the data egress. You’ve got to use the public internet for-

Kimberly (00:27:45): Even if you pay money, pay more?

Fernando (00:27:47): Oh, wow.

Jeremy (00:27:48): Yeah. So you pay money, but to get this other bandwidth covered, you got to use the egress cost. And I guess maybe it’s just tied up in some kind of red tape in the direct … Would they cover the cost of the direct connect? Maybe they can’t account for it. Who knows?

Kimberly (00:28:05): Jeremy, I do have a question because obviously we were moving multiple applications or working with multiple applications to make this move. Was there a specific order that you were moving them or was it just you just kind of picked an application and did that? What was the reasoning behind the order for the move?

Jeremy (00:28:24): There were two phases. So we chose some smaller applications with less storage, but that would be representative of our applications with larger storage needs that we would migrate before the egress window opened. So essentially we wanted to test the process. We wanted to be prepared that when the window opened, we’d be able to blast. Didn’t turn out to be that way. We had some delays as we optimized and restarted things and whatever, but it was crucial to do that, identify some systems that we could do real life copies and not just kind of test runs. So we chose a couple of representative systems and did those first to prove kind of a blueprint for how we do it for our other applications. And each application ended up being not quite cookie cutter, but once you’ve kind of improvised a recipe a few times, you kind of know what you need to do.

(00:29:26): So you come into it with a plan of attack of you need to do dual rights to multiple places, you need to do a reconciliation step, you need to have a well-defined cutover process. And so you come up with a checklist, validate the checklist by doing a live migration and iterate, fine tune, and then you’re ready to go. Then when it comes to the actual copy, we go criticality first. The things that we want to be absolutely sure and the business critical stuff, Basecamp, HEY, all our primary revenue generating apps with the big data. We want to get those started as soon as possible because there’s the most a copy and we want the greatest assurance that we did it all properly. So give us the most headroom for unknown unknowns. We’ve got our known unknowns of things that might crop up, but we also have room for just we don’t know. Who knows what would happen?

(00:30:27): And we came up with a bunch of those. So we’re grateful for having started them early. But yeah, definitely dive into the trickiest, biggest thing first.

Fernando (00:30:42): That makes sense. I’m still trying to wrap my mind around this. You go to a small app and you’re like, okay, you know what? We need to migrate this. You start building the program that you mentioned, the reconciliation program. Is that program a Rails server on its own? Is it like the modifications that you mentioned to Active Storage?

Jeremy (00:31:08): No. So we built a new thing that kind of scales up the basic idea I started with of I need to list things to know what’s in the source. I need to copy everything in that list over to the destination. And at a small scale, I can use a single program. We used one called Rclone. There’s one called RSync. And most folks, technical folks have used one of these before. And you just fire it up, you give it a source and a destination and it churns and it does the job. At most scales, that’ll work fine. And in fact, for AWS, their free bandwidth egress limit is pretty generous, so most people would fit within it. And you could just Rsync and do it yourself or Rclone and call it done. And it does all the bookkeeping for you. At our scale, we needed to fan out to a bunch of workers doing this job.

(00:32:00): So we needed to do it in parallel. That means splitting up, batching it up. And so it becomes kind of a classic like map reduce problem. You’ve got a big input, you need to spread out to a bunch of jobs, and then they’ve got all their individual outputs. In this case, it’s take a big list of files, split it up into batches, send those to workers that are going to do the copies. Have those workers have some kind of supervision that’s tracking what they’re doing, their progress, whether there’s an error, retries, all that kind of stuff. That’s where we used a Rails app for command and control, for wrapping up the jobs and the work. We arrived at a Rails app after trying what I thought might be some kind of simpler lo-fi ways of doing things. And of all things, I kind of backed my way into doing, to using Rails because I was missing some of the conveniences of home.

(00:32:59): One of them was secrets management, credentials. Here we’re doing something that’s copying between a bunch of AWS accounts to a bunch of destination buckets. You’ve got a ton of sensitive credentials in one application. And so as I was building this kind of script-based simple system, I realized I’m rebuilding a credentials manager. This is not the life I want to be leading right now. And how about I do something that’s already built for me? And so going essentially to Vanilla Rails because of this mildly auxiliary concern. I had a bunch of other kind of pressures swirling in my mind that were resolved by this, so who knows whether my unconscious was also kind of like, uhhhh… I can kind of feel that change is going to need to be made and here’s the thing that just triggered it. And the trigger then led to a bunch of nice outcomes like being able to use ActiveJob and Solid Queue. And a lot of things we’re familiar with for basically how do you distribute this work?

Fernando (00:34:03): So in the end, you have a single Rails app that you’re constantly looking doing this window, you’re constantly monitoring like, okay, did it complete everything or are there any retries errors? And then you go and fix them like you say, right? Oh, we hit this limit. Let’s try and work around it.

Jeremy (00:34:21): So that’s part of the approach based on the initial migrations as our kind of test runs. As we discovered in the test runs, you’re going to have new errors, kind of surprises crop up. And so we prioritized failing fast and not trying to be resilient and automated way. So not building in things like exponential back off too early because sometimes things were not things we wanted to retry. There were actual errors. So being able to identify something that was truly like a transient failure and then automate it late rather than early. So we treated it kind of like an Andon cord of like, here we’ve got a production line, we notice something’s failing, we pull it, we stop everything, we fix it, and then we proceed from there. So that drove a lot of other decisions of when you break things up into chunks, they need to be observable, they need to be retryable, they need to be supervised.

(00:35:19): And especially for a diagnostics and troubleshooting, you need to be able to see what’s going on. So in these cases, you’ve got a active job process that invokes another tool and you’ve got hundreds of these running. And so you’ve got this standard output and standard error from a bunch of tools. You’ve got the exit status. How do you see them? And so this, for me, this was kind of a crucial stumbling block of like, if I can’t see exactly what’s going on, I don’t know what’s going on. And so I don’t want to spend a bunch of time guessing at and troubleshooting. I want to just look at the output and I want to be able to figure it out as if I were running it on my own console. So that was a critical step early on too, of making something that was easy to supervise and just witness.

(00:36:12): If something’s failing, I could try it myself, I could invoke it myself, or I could pull up a transcript of what that process had done. So this was the job of the Rails app of coordinate, pulling an inventory or a catalog from the source, split that catalog or inventory up into a bunch of pieces, which was its own whole thing. If you’ve got something that’s huge and you need … Anyway, you can get into tooling in a little bit here, and then make a bunch of jobs for all that stuff. And each job, well, it has its responsibility is its chunk of files and the output for that job is a bunch of things like status and transcripts and whatnot. And those themselves are actually stored in a storage bucket as well. So every job, there’s a unique idea associated with it and you can go inspect the whole process.

(00:37:11): And in fact, we did have a live tail. So since these things can take a while, you can visually see what’s going on. You could just kind of snoop on any transfer.

Fernando (00:37:25): So you were basically Neo for 90 days?

Jeremy (00:37:28): Not 90 days, thank goodness. We ended up getting it down thanks to Pure Storage and the very fat pipe of bandwidth to less than 10 days of transfer.

Kimberly (00:37:42): What?

Fernando (00:37:42): Wow.

Jeremy (00:37:44): Which turned out to be pretty critical because that gave us lots of time to do reconciliation, verification. Yeah, we didn’t know that. And so we had built in a lot more buffer than we needed, but I’m sure I’m glad we had the buffer because I was also going to … I was headed on vacation on sabbatical right after this was going to wrap up, which was not the wisest of career choices, but …

Kimberly (00:38:11): Jeremy, tell us a little bit about those 10 days. Is it 10 days just nonstop? Are you breaking … Kind of walk us through…

Jeremy (00:38:18): Oh, it’s nonstop. Yes.

Kimberly (00:38:19): Yeah. Okay. Yeah.

Jeremy (00:38:20): So the setup is I didn’t want to trickle things into a pipe and have to be carefully tending things. I wanted to feed a pipe and have a backlog. And so I can go through some of the technology stuff just briefly. If you’re copying a bucket with billions of objects from S3, use S3 inventory reports. It’s something you can turn on S3 console. It’s easy to do. You got to pay. You got to pay.

Kimberly (00:38:49): It’s the theme. It’s the theme.

Jeremy (00:38:51): But it is the most efficient, effective way to get a large scale bucket listing without doing the work yourself. It is delayed. The most frequent you can do it is daily, and so they drop on a schedule. And so for this kind of process where you want to do a big bulk transfer, daily’s fine, and particularly if the system that you’re migrating is doing dual writes, you’re writing to both the old source and your new destination, you know that you’re already in sync, so there’s not going to be missed writes. So the thing you do here is you turn on dual writes to both places, and then you take the inventory from the day before you turned on writes. So you know all new objects are being written to both places. So the old inventory is sufficient for knowing that you’re going to get a bulk copy of, and it’s going to bring you into accord.

(00:39:50): Everything’s going to be the same. So you start from that snapshot of the bucket. So we used S3 inventory reports. We didn’t have that turned on everywhere. We developed … We looked for another tool that could do something like this. And there are a bunch that take a similar approach. If you try to list an S3 bucket, it’ll take literally days because you need to sequentially list files and it lists whatever, 1,000 at a time or some 10,000 at a time, something like that. Anyway, order magnitude-wise, it’s ages, but there’s some tricky, very clever ways of doing this where you can do it in parallel by estimating what the prefixes of a bucket are. And you can ask S3 for bucket listing starting from a certain prefix. So if you know your key distribution, then rather than doing a single sequential listing, you can instead do thousands of parallel listings for every prefix you’ve got.

(00:40:50): So you can turn a multi-day bucket listing into something that takes like 30 minutes. So I had that in my back pocket in case we needed live listings. If we discovered that we were going to be in a situation where we needed to do a downtime or we weren’t able to do dual writes in a system, so you’d need a downtime to be able to stop writes from the old system, but also not write to the new system so you wouldn’t get out of sync. And also to list the objects in the destination buckets because Pure Storage and the Flash Blade product does not have an equivalent to inventory reports. So you’ve got to do the listing yourself. So on that other side, you want to take the inventory report from S3 and then you want to develop your own report of the destination and compare them and any discrepancies need to be accounted for.

(00:41:45): So you need a fast way to do that, so there’s a tool called S3 Fast List that’s on our GitHub. We forked it from AWS samples and we adapted it so that it would support non-S3 storage systems so that we can use it to list flashblade buckets. Works great. Very clever approach. Really pleased to find that. We didn’t end up needing it that much, but it was a wonderful diversion and felt like kind of an insurance policy of a special built tool in the toolbox. The next thing was, how do we split these things up? And we started with, gosh, how are we going to take this S3 inventory report comes in either CSV format or Parquet format and parquet being like flooring, it’s like kind of split up and actually I don’t know how far the metaphor goes, but it’s a very efficient format for doing columnner data storage.

(00:42:51): So it’s great for analytical processing and whatnot where you know which columns you want to work with and you need to do some transformations on them. And it’s particularly nice for something like this because there’s tooling that can ingest it, that can stream it from a remote destination and operate on it and then emit it again. So took us a while to discover this because I was looking initially for just something like polars. There are different tools that can ingest parquet and operate on it, split it up, whatever, like using a windowing function to … We wanted to do something like split this not just into number of objects, but in total batch size, so that we would evenly distribute batches across machines so that we wouldn’t end up with uneven bandwidth demands. So you wouldn’t have one machine that is working on a batch of a bunch of small objects and it can’t fill the pipe.

(00:43:45): What you want is to have an even distribution size wise so that you’re maxing out the pipe on each of the worker machines. So to do that, you need a windowing function that goes through the inventory report and does a cumulative sum on the bite size of the objects. And each time it reaches 10 gigabytes, it says, “Oh, I’m going to do a split right there and I’m going to turn that into a chunk.” And this turned out to be hard, use a lot of memory and kind of work, but ended up maxing out the memory on a machine I was using. And it’s like, “Well, okay, this is probably, might be feasible, might not be feasible.” And then I discovered DuckDB, frigging awesome. DuckDB is amazing. I cannot sing its praises enough. It’s like somebody discovering SQLite for the first time, although DuckDB is like SQLite on whatever next generation steroids, because it can even do the stuff SQLite does maybe better because I’m just glowing with its capabilities, but it can do SQL, it can like connect to remote databases, it can work with a local database end process just like SQLight.

(00:45:03): It can work with CSV files and parquet files on the local file system. So most kind of big data, data science stuff, DuckDB can do locally on a single machine and super efficiently. It’s really smart about spreading out IOs to do things as smartly as possible to try to avoid doing things like bringing everything into memory. So I’ll sing this praises a little bit more.

(00:45:34): Not only can connect to everything like a Swiss Army knife of data analysis, it can also connect to remote URLs and to S3. So I had this whole system built of ingesting data from S3, downloading it, and then staging it on local file system, and then doing the splitting myself using my own tooling, and then storing those split files in another bucket as a staging area for jobs that would then be dispatched to work each of those chunks. Turns out I could skip all of that. With one DuckDB invocation, I can point it at a glob that’s referencing multiple S3 files. So the inventory report is split up into hundreds of files. It can reference all those files, stream them all in, partition them the way I like, and then write them to a remote S3 compatible file store, all streaming. So there’s no local file system, there’s no other code I need to write.

(00:46:36): It’s just you got to configure it properly, you got to know what you’re doing. But when you get it working, it’s like, “Oh God, yes, this is sweet.” You’re working with the remote thing, streaming it all through, not using a bunch of memory, and then writing it out to remote storage. And then I was able to take the files that had been written and I kind of wrapped Active Storage records around them. I said, I’m going to make Active Storage records that point to where those batch files had been stored. And then I distribute those active storage records out to the jobs to work.

Fernando (00:47:13): That’s a benefit that you were in the Rails app, right?

Jeremy (00:47:15): That is a benefit of the Rails app, yes. Yeah. And I had that abstraction to work with. And thankfully, Active Storage was able to accommodate this where I was kind of going behind its back because I was using DuckDB to write the files rather than using Active Storage. And I said, haha, Active Storage, I’ve got these files. Can you make use of them? And it’s like, “Yeah, of course I can do that. “ So you just feed it the key of where it’s stored and Active Storage will be happy to work with it as you bring it.

Fernando (00:47:39): That is so cool.

Jeremy (00:47:39): So any case, DuckDB, amazing, able to partition the problem and kind of eliminate a whole step of what would otherwise need to be custom code.

Fernando (00:47:51): It just comes on to show you that nothing is new. How did the DuckDB guys know like, “Oh, you know what? What process would be really nice if you take this, partition it and then put it this way?” That’s so, so cool. I love that.

Jeremy (00:48:07): Yeah. The thing in common, people’s resource constraints, everybody’s constrained in similar ways and they all have different problems, but they’re all stuck in similar ways. And here somebody comes along and solves it elegantly and does it with open source.

Fernando (00:48:18): Very cool. And this process, this had to happen once a day because of the limitations of the inventory. Am I right?

Jeremy (00:48:27): So I did end up automating it.

Fernando (00:48:31): Of course.

Jeremy (00:48:32): But it can happen at most once a day. So I made a scheduled scan for new inventory reports and I automatically process them so they’d be ready if I did choose to use them. And I only did this because it turned out that partitioning was so easy and cheap now. I had anticipated it being like a pretty slow process. And so I wouldn’t want to just be firing it off all the time. I’d want to choose which specific inventory report I used and it would take hours or who knows how long to do the partitioning, but now it’s a matter of minutes. So I’m like, well, I’m just going to do it, make it easy to choose which inventory report I want to use as a sync, as a copy source. So from the app dashboard, I could have a list for every app and every AWS account in every bucket, which things were transferred using which source manifest. I’ve got a bunch of inventory reports. For each one, I can see that I’ve kicked off a copy. I have state tracking for every part of the process of partitioning to copying to errors to reconciliation.

Fernando (00:49:44): Wow.

Kimberly (00:49:44): As a non-technical person here, I do have a question because all of this sounds very hard and scary. I’m curious, Jeremy, what was the most nerve-wracking part of this process?

Jeremy (00:49:55): Deletion. Yeah, the final deletion. I mean, you know, I mean, it’s like anything that’s high stakes, your brain’s got a lot of things going on. My brain’s got a lot of things going on. I’ve got feelings, sensations, I’ve got some kind of cognitive whatnot that’s blinking on and off sometimes. And some parts are telling me like, “I know that things are fine, but anxiety’s telling me,

(00:50:17): Maybe you should discover why it’s not. “ And those things need to work together. And I can use my anxiety as a guide that maybe I haven’t figured everything out, but I could then use my cognitive process of here’s the things I’ve worked out, I’ve ruled all these things out and I have some standards of proof. I can demonstrate conclusively in a way that is not dependent on my anxiety, that can be externally verified, that it worked and that I’m done. Nonetheless, deletion is still dicey, but when you do press delete, then it’s, I mean, oh… Somebody else take the wheel. It’s just all happening now. Yeah, right. There’s no going back. We’re now doing deletions. Yeah. Coming up with the biggest unlock feeling was adding kind of a belt and suspenders step, you know, keeping your pants up you want more than one way. When you’re verifying and reconciling, you want to be a little bit more than sure.

(00:51:23): And so thanks to Pure Flashblade’s extraordinary metadata read/write rates, we can do hundreds of thousands of metadata operations per second just without breaking a sweat. It made it easy to do rather than doing reconciliation against inventory reports, to do reconciliation by doing a live sync. And so with an inventory report, you’ve got lag time between the listing and kind of the live state of things. Whereas with a live synchronization, you can see exactly how many objects were needed to be copied and the size of them. And so you could essentially do repeat copies fairly cheaply until you can see that everything is done. And it really helped just psychologically and as a matter of certifiable proof that you get the final copy that says nothing needed to be copied. Everything was up to date and that typically came after, there’s another key step of when you’re doing dual writes to a previous source and the new destination, you turn off dual writes and you go to single write just to your new storage destination.

(00:52:49): And then at that point you know only your new one is new. And if you did need to roll back, you’re kind of like, ah, if I did need to roll back, now you’re out of sync. Now you need to copy the new stuff back to the old place. So that happens after you get your green, everything was cool, nothing new needed to be copied. Turn off the dual writes and a final sync and double verify. Nothing is changing. Nothing is accidentally writing. So I did a bunch of other, more than just belt and suspenders also, whatever else, contraptions you couldn’t imagine keeping your pants up. One of them was just changing the permissions on the S3 buckets. Some of these, we had old systems that had multiple things writing to them and you just, you kind of know, but do you really know? And one of the ways to be sure is to just turn off writes.

(00:53:42): So if something was writing to it, it would error. So that’s kind of the final straw of assurance. When I go to delete this or I go to turn things off, I don’t have some straggler that’s going to surprise me. And it was only writing some hours of the day or it was on a cron job or something, so I wouldn’t have caught it in the initial sync.

Fernando (00:54:04): My hands are sweating and I had nothing to do with this. The amount of … Okay, the million dollar question, was there any downtime?

Jeremy (00:54:13): There was no downtime. Everything worked and it was really quite wonderful. There were some things we broke, so I suppose-

Fernando (00:54:24): We don’t need to talk about that. No, I’m kidding.

Jeremy (00:54:25): So the 100 gigabit link was actually on a shared link with some other things that we’re using a portion of the bandwidth. And so when we pushed over about 80 gigabits, we started kind of impinging on some other stuff that needed to not be impinged on. And so we did cause some errors elsewhere, but the copy was fine.

Fernando (00:54:53): Suspenders worked, right?

Jeremy (00:54:54): Yes.

Kimberly (00:54:58): Jeremy, question for you. Now that this is done, like it’s tied up with a bow, looking back, are there things that you’re like, “Oh, I wish I’d done this differently?” You can say no.

Jeremy (00:55:08): No.

Kimberly (00:55:09): Okay.

Jeremy (00:55:09): No. Yeah, I’m pretty happy with how things worked out. I appreciated the incremental approach to starting simple and focusing on the epicenter of, what I thought was the epicenter of the problem because I didn’t build too much as I discovered that the true epicenter was in verification, reconciliation, inventory management. It’s about how do I track what’s going on and the copy itself was fairly simple. I did rabbit hole a couple of times. I built out a live view system that I didn’t actually end up using much, but in key times I did. So it’s a kind of like hindsight bias of like I didn’t need to do all that, but I kind of did to discover that I didn’t need it. It’s a little circular, but it allowed me to diagnose and troubleshoot things that were blockers that would’ve been really hard to work out otherwise because it’s things that turn into like little Heisenbugs of, if I run this myself outside of supervision, it works.

(00:56:11): But then in my supervision framework, something breaks. And it’s stuff like I’m opening a pipe and I’m feeding things to standard input and I got another pipe that’s reading and like a pipe could get wedged because somebody hasn’t read from it frequently enough for something like this and it can manifest as some other kind of error. Anyway, having that kind of visibility was really helpful, but looking back, I could probably delete it from the app we built.

Fernando (00:56:43): My question would be, I mean, you are an eminence within the Ruby on Rails community. Let’s assume I’m just an average Rails developer. What is the complexity of the project I can take on from moving out of history to our own hardware?

Jeremy (00:57:04): You could take on this whole project. And this is one of the … And the magic of this is that a lot of the feeling of criticality is the business criticality. It’s not technical difficulty. It’s a modeling problem. And there’s some tricky things with process supervision that Ruby doesn’t make super easy, but it’s not bad. And there’s plenty of other kind of worked examples you could start with. And otherwise, the degree to which Vanilla Rails just works is quite gratifying. And in particular, well, Kamal also, in fact, leading into this as part of figuring out our upper bounds was doing S3 load testing against our flashblade. So we got this new S3 service. What can it actually do? Well, there are load testing tools out here that can do that. And I used Kamal to deploy one of these tools out to a bunch of nodes and hammer it as much as I could. Worked great.

(00:58:06): And I used Kamal to deploy our copying application called Nostos and worked fantastically too. And I was able to use accessories, Kamal Accessories to stand up the database that did all the state tracking, do open telemetry observability, do logging. It was all just single system, like a single developer pushing to some VMs somewhere. So it’s all kind of bog standard stuff, but it’s being used in the employment of a kind of critical operation. But again, the criticality is all in our heads. The actual app is fairly simple.

Fernando (00:58:50): You mentioned DuckDB as being like, wow, an incredible tool in your arsenal.

Jeremy (00:58:56): I cannot sing as praises enough.

Fernando (00:58:59): Are there any other tools that were completely necessary in this process?

Jeremy (00:59:03): Yeah, I sung Rails praises a little bit. Vanilla Rails turns out to be the way to go. And I don’t mind tooting that horn a little bit. This other tool, S3 Fastlist, was like a little pleasant discovery, diamond in the rough. In our clone itself, what we did to do the heavy lifting copies was incredible. It did all the stuff we needed. It’s open source. It was easy to contribute to. In fact, as part of this, we added a flashblade, an official flashblade destination to our clone. So when you go do an R-clone of your own, and it’s got this nice kind of interactive thing where it asks about where you’re coming from, where you’re going to, and now Pure Storage Flashblade is one of the places you’re going to. And what that is, it’s essentially like a list of characteristics of the system so that Rclone knows how to best do its transfers.

(00:59:57): There are certain quirks and asynchronicities with different S3 compatible file stores and Pure Storage has some of them, and now it’s just all set up out of the box where you don’t need to go figure out the command line flags yourself.

Fernando (01:00:14): That change was upstreamed.

Jeremy (01:00:15): Yes, that’s part of Rclone now.

(01:00:19): The other key thing is that Rclone is bandwidth and metadata operation efficient and it’s oriented around resilience first. So you can operate it in a bunch of different ways. You can kind of back off on the kinds of checks it does, but it does nice things like pull the checksum from the source and check it against the destination. And it can even do a kind of extreme check where normally you would write the file to the destination, you get a checksum back and you compare that the checksum that it says was written is what you had. And you say, “Okay, cool. It looks good.” But you can also do it in kind of really be careful mode, which is write to the destination, get the checksum, then download it from the destination and actual checksum of the bits. So it’s got you covered for every degree of risk mitigation you want to have at play.

(01:01:16): So if you don’t trust your destination yet, because funny things can happen where like a bit gets flipped on a hard drive or a gamma ray hits something and messes something up where it says that you got the checksum you expect, but it turns out on disk, there was a modification. So depending on the level criticality of your data, how many infinite nines you’re going for after that decimal point, Rclone’’s got your back. I’m just going to sing his praises a little more. Sorry.

Fernando (01:01:42): Yes, go for it.

Jeremy (01:01:43): So there are certain things you can be very efficient with Rclone where you can skip operations that don’t matter. And if I don’t want to do a bunch of metadata operations against S3, like checking last modified time, whatever, things that are not present in the S3 file listing or the inventory report that would normally need to make a head request against S3, which you got to pay.

(01:02:08): I don’t want to go do that and I don’t want to bottleneck on making those calls to S3, because again, there’s just adds to the rate limit bucket. And on the flip side, if you don’t want to do excess metadata operations on the destination, you can tune Rclone to your heart’s content.

Fernando (01:02:26): I was going to ask, did we at any point consider doing random statistical analysis with like a full, give me the bytes back, do the checksum here.

Jeremy (01:02:36): We did.

Fernando (01:02:37): Okay.

Jeremy (01:02:37): We did. So we did some of the copies with the full pull and we didn’t run into any issues. We just did kind of a grab bag sample. I just manually ran some of the batches just to satisfy myself.

Fernando (01:02:51): Yeah, because I mean, you could do it for everything and we completely quote unquote completely sure, but it would be insanely expensive, insanely time consuming, but that makes sense.

Jeremy (01:03:00): Yeah. And it would suck a lot of our bandwidth. So all the bandwidth we want to be using exclusively for writes, we don’t want to be eating it up on reads, verification reads. Whereas metadata operations, they don’t use much bandwidth. They just eat up CPU time on the flashblades. So we were able to calculate all that out too, depending on the number of, kind of, compute cores in the flashblade cluster, how many metadata operations you can possibly do concurrently. So essentially we wouldn’t flood it. We would edge it just a little bit over what it ought to be possible, what ought to be able to do to keep it full utilized.

Fernando (01:03:43): That makes sense. I want to go back to the open sourcing. I feel it’s such a great part of this, of 37signals that we’re both contributing to open source in the patch that went out. But I’m curious if we are going to open source our tool, like this tool that you built.

Jeremy (01:04:05): I did build it with a mind toward open sourcing, and part of that was to … It’s almost like a kind of a design discipline that I’m not going to make something like too bespoke. And there were some decision points along the way. At one point, it’s easy for us to do an NFS mount, for example, and to use an NFS mount to share files, but it’s not terribly different to use an object store and to use Active Storage. So in a case like that, I erred on the side of using Vanilla Rails of using object storage rather than developing a different kind of file storage backend. And if I was just doing batch scripts, I would probably do an NFS mount, but as soon as I moved into Rails, it’s like, let’s just do it all the Rails way. And once you’re doing things the Rails way, it becomes almost hard to do it in a 37signals specific way.

(01:04:58): Of the things you hard code in, it’s easy not to. And things like where your credentials go, well, goes into a separate credentials area and things like AWS accounts and credentials are modeled in the database. I used Active Record encryption to store credentials in the database so they’re not part of the repo. It’s part of your onboarding process. You start with a blank app, you add AWS accounts, it takes your credentials and goes and scans for all the buckets you have and imports them into the app. And on a regular basis, it can go pull them and then you can start copies from that stuff. And similarly, you set up destinations and just the same way. So you can support different kinds of sources, different kinds of destinations, perfect setup for being a general purpose tool. As we got closer to the end, I did start specializing some of my design decisions based on the phase of the copying that we were in.

(01:06:00): So I started tuning the dashboard to reflect what I needed to know. And that sacrificed a little bit the things that are important earlier on. So as it got into like, “Okay, I just need this from me, I’m going to totally revamp things right here.” Now the UI is, it’s a little bit narrower and purpose built for my needs and not everybody else could necessarily understand it.

(01:06:26): So in any case, yeah, it’s very open sourceable. We’d like to do it. I would like to do it. The investment in doing it is like, okay, got to tease apart some things. And so one possibility is sharing it kind of like an artifact of here’s what this look like frozen in time at the end. And if you wish to take it and adapt it and turn it into something, because this is certainly not our line of business. We’re not going to make a product out of doing this.

Fernando (01:06:52): That’s right.

Jeremy (01:06:52): So I’m not going to spend a six-week cycle of work on polishing up something that then we end up being open source maintainers for. It’s like, no, I’m not going to … So I’ll share it, but I’m not going to maintain it.

Fernando (01:07:03): That’s awesome. And what do you feel are the next steps now that the full transfer is done, you deleted everything? Where do we go from here? What about backups? What about …

Jeremy (01:07:18): Yeah. So it’s all the other stuff. This is all the programming side of … I drove the copies and the transfers from the application and the software looking out. Our awesome ops team and Matt and in particular did all the operations side of how do we stand up these storage systems on the back end and feed them and keep them well fed. And now that we’ve got all the bits on a disk somewhere, how do we make sure we don’t lose them? It’s all the standard kind of data reliability stuff. And so we run through the cases of like, what can happen, what are the risks we need to mitigate? And things like data loss on a drive or losing a system, losing a power supply. Each of these things has its own kind of redundancy and that’s all on the system side. And there are other kinds of redundancy concerns like what happens when a truck backs into the power transformers at a data center and takes the whole thing down.

(01:08:20): Well, we’re not going to lose our data, but it’s going to be unavailable. So then we got the availability problem. Well, we’ve got a second site, which is our backup of the first. It’s kept in sync live and it lags in a sense that we don’t write to it directly. So we’re insulated from things like software bugs in our applications. If we accidentally delete something like crap, we wrote a bug that deleted some stuff. It’s like you need a backup. It’s not sufficient just to have like high durability. You also need to be insulated from other kinds of mistakes. And our second site is essentially that. We’ve got another storage system that similar size, similar class, similar write rate, and we replicate from the first to the second. So if there is an issue availability wise or durability wise, we’ve got a place to go and we’ll just flip the applications over to use the second one.

(01:09:17): Now, that’s not the whole story. For backups, there’s the kind of age old, three, two, one rule where you’ve got three copies of your data on two different kinds of media and one offsite. It’s a little bit rusty around the corners and the age of cloud stuff and particularly with modern flash because the old school thing was typically hard drives and hard drive failures and your other media would often be tape.

(01:09:47): And these days, that picture looks a little bit different. What is different media really? Is it other kinds of flash? Is all flash one kind of media. You need sufficiently different characteristics that if something catastrophic were to happen to one mode of file storage, your other mode would not be affected by the same risk factor. So we’re looking at doing a third site with hard drive storage or different kind of flash and acting as our insurance policy. And the other factor that we’re pulling in here is that our two sites are using the same vendor. So we’ve been very happy with Pure’s Flashblade product, and we had similar resource constraints, both in power and cooling and rack space in both data centers. So using Pure for both made sense and we’re able to smush them together into one contract, just sensible step, but that’s a single vendor and we run the same operating system.

(01:10:47): We do stagger upgrades, but if there’s a bug in the system and we deploy to both systems, well, we essentially have a single point of failure. We don’t have two systems anymore and we’ve got one operating system and the same bug affecting both. So what we’d like is a third system with a different vendor. So we’re looking at using MinIO, the kind of open core storage S3 combatible storage system. They’ve got a dual license where you can use MinIO free or you can get a support plan with them. And it’s what we had been considering using before we discovered Pure. So we’ll be going back a little bit and having not just a second vendor, but also going to have an open source fallback. That’s our insurance policy against things like storage vendors being acquired. So that is often the final destination that’s pushed us off of other vendors in the past of you got bought by some big company and now what do we do?

Fernando (01:11:46): Wow, that is a lot of things to consider.

Jeremy (01:11:48): I will say the other storage medium that was kind of a romantic diversion for quite some time was using tape for real. Back in the day, I operated tape libraries and there’s something that’s just kind of a retrofuture tech satisfying about seeing a literal robot going and grabbing a tape from a library and putting in a little drive and like, “Oh, I’m going to copy my stuff onto there.” And it’s super durable. You don’t need power for tapes just sitting there and I mean, their lifetime is like 30 years and you can go put them in whatever salt mine somewhere and you’re good. And you could pack up your tapes and you could carry them in luggage if you wanted to. There’s just a lot of aesthetically pleasing characteristics about it. So we really tried to make this work because you could get a whole multi-petabyte storage system going for like less than a hundred grand, which is as far as like capital outlay, it’s like fairly cheap per terabyte, but devil’s in the detail.

(01:12:57): So like these tape systems are built for older school systems that have a directory, like a file system that you can scroll through to see what needs to be backed up. Object storage is a little bit different. To figure out what needs to be backed up with object storage you need some kind of gateway, which pulls new objects into a scratch space and then backs those up to tape and keeps a catalog. And there’s just a whole finicky kind of additional system you need. And then you’d have the troubles of like, how do actually restore from tape? Well, the only case it would be in is like a truly end of the company kind of scenario where some kind of nuke has hit and you want to recover things and it’s going to take weeks to recover from tape. So yeah, it’s truly an insurance policy at that point.

(01:13:49): And like, well, it’d be an aesthetically pleasing insurance policy and one that’s fun to consider, but it’s not actually going to work. Dang it.

Kimberly (01:13:57): Jeremy, thank you for joining us. This has been Recordables, a production of 37signals. To learn more from our technical team, check out the developer’s blog at dev.37signals.com.

Other posts in the “RECORDABLES” series

The Rails Delegated Type Pattern

Jeffrey Hardy, Kimberly Rhodes, Fernando Olivares — Fri, 19 Dec 2025 18:00:00 +0000

Here on the 37signals developer blog, we go deep into technical topics, but some ideas are easier to explain through conversation, a little back-and-forth, and an occasional screen share. That’s why we’re launching RECORDABLES, a new video series hosted by Kimberly Rhodes, who you may know from The REWORK podcast, alongside Fernando Olivares, Lead Programmer on our Mobile Team.

We couldn’t think of a better way to kickoff RECORDABLES than exploring recordables! We’ve had many requests over the years to explain how and why we use the delegated types pattern from Rails in our products and we’re finally explaining it in-depth.

Kimberly and Fernando sit down with Principal Programmer Jeffrey Hardy to unpack the recordables architecture that powers Basecamp and HEY. Jeffrey explains how delegated types made it possible to scale Basecamp for over a decade without constant rewrites. The discussion explores why this approach makes copying, version history, timelines, and mobile support dramatically easier, along with the tradeoffs and learning curve. We’re sharing a behind-the-scenes look at how 37signals builds and evolves complex products with a small team and the Rails pattern that helps make that possible.

Watch the full video episode on YouTube. And stay tuned for another episode of Recordables with Jeremy Daer in the new year!

Timestamps

00:00 — Introduction
01:20 — What “recordables” are and why they matter
02:15 — The delegated type hierarchy
04:15 — The challenge with single-table inheritance
09:05 — Organizing recordables and recordings
11:45 — Tracking change history with recordables and events
22:40 — Copying and moving content efficiently
27:54 — Pagination and querying across content types
34:00 — The learning curve with delegated types
39:54 — Building new features faster with reusable behavior
43:03 — Long-term scalability and performance

Links & Resources

Transcript

Episode Highlights (00:00:00): You got to think of the recordables as being pretty dumb. They’re just like, in the case of a message, it’s literally just a title and content. That’s it. They have no connection to the outside world. You can build entirely new features that should take months in like a week, two weeks. We’ve been running this architecture for 10 years.

Kimberly (00:00:19): This is Recordables, a place where the 37signals team share some of the technical work behind Basecamp, HEY, and some of our other open source projects. I’m Kimberly from the product team joined by Fernando who’s helping out with some of the technical aspects of today’s discussion. And this one, we’re excited to kick off this series with an episode about recordables. To do that, we have principal programmer Jeffrey Hardy with us today. Jeffrey, thanks for being here.

Jeff (00:00:46): Hey, thank you. This is great.

Kimberly (00:00:47): Before we get started, I know this is a topic people really want to hear about, but tell us a little bit about you and what you do here at 37signals and how long you’ve been here. You’ve been here for a while.

Jeff (00:00:56): Yeah, I’ve been at 37signals for 18 years, so I’ve worked on all our products, all versions of Basecamp, including the most recent version, and HEY, and everything we’ve done. It’s been a really great place to spend 18 years and have the advantage of seeing the evolution of Basecamp through all these different versions. So yeah, I work on the product team.

Kimberly (00:01:20): Amazing. Well, we know people want to hear about recordables. We’re excited to kick off this new series where we’re sharing some of the behind the scenes stuff with this episode. And before we dive in, I think I have to ask kind of the obvious question as the non-technical person here. I always hear people talking about recordings and recordables. Will you just kind of break that down? What does that even mean before we deep dive into the topic?

Jeff (00:01:43): Right, it’s a little bit abstract. Recordings and recordables and buckets and… what is this? So in Basecamp, pretty much all content, things like documents, messages, comments, uploads, these are modeled as a pattern we use called a delegated type. And so in the delegated type pattern, which comes from Rails, you have a primary table. In our case it’s recordings. And recordings reference recordables, and the recordables are like the concrete types. A recordable would be a message, a comment, a document, or an upload. And so with this pattern, we’re able to model all the various content in Basecamp, which is essentially all the same. There’s a bunch of advantages to treating all that content in the same way. And Basecamp’s architecture hinges on this concept, and it’s our third iteration of Basecamp. We call the version of Basecamp, the internal version is three. We call that the chassis, like the architecture on which it’s built.

(00:02:51): It’s our third attempt. And so when building the third version, we came up with this pattern that we use and it’s called a delegated type, which I know it doesn’t make a lot of sense. What does that mean? But I can describe the sort of hierarchy. So you have recordings, and this is a table of records that stores all the metadata and common information between recordable types like messages and documents and uploads, but it doesn’t have any specific information. It doesn’t know the title of a message or the content of a message or the location of a file or other things. It delegates that knowledge to other records. Those records have their own tables in the database, messages, documents, uploads comments, all separate tables. And the advantage here is that you have your single recordings table that delegates its types can be lean and tight.

(00:03:53): It only has metadata. It has references to recordables. So it has a recordable ID and a recordable type. It has timestamps. It has a creator id. It has tiny little columns, no text columns, nothing big and heavy. And so all of the specific information is contained in the recordable tables, which can vary. So to back up a little bit, in Rails and in software architecture patterns in general, there are a few ways to handle this idea of things are mostly the same, but they’re kind of different. And how do you account for the differences without repeating yourself? And one common way is a pattern is called single table inheritance. You have a database table and you have subtypes that also use that same table. That’s where you would have your shared metadata, the things you have in common. Then comments might have a body and messages also have a body.

(00:04:50): And so you’re like, great, I have a body column. I can use the same table to store these. I just have to keep the type in the table. Is this a message row? Is this a comment row that works good for things that are super similar, like maybe messages and comments. But what about things like events? They have a starts at and an ends at column. Now you’ve got to add those to the main single table. And so the table keeps getting bigger widthwise. It has to have all the different kinds of columns that you might have for any type. And adding a new type means you have to modify that single table and that table gets really, really big. It’s just sort of intrinsic to having a single table that does that. The bigger the table is, the harder and slower it is to migrate.

(00:05:38): So that’s one way to do it. Another way is to have just different tables for everything. And then you take the things that are the same and instead of belonging to a specific kind of record, lots of things, for example can be commented on in a system like Basecamp. So you might have a documents table and you might have an uploads table and a messages table and you want comments and all these things. So the comments table could instead of belonging to, having a reference to a specific thing, like having a document ID or an upload ID or a message ID, it can belong to a polymorphic type. It can store a commentable ID and a commentable type. And so now you get this is called polymorphic relationships. That’s another way to model this sort of, everything is the same but slightly different. We need to differentiate. And the delegated type pattern is neither of these, but it’s similar and it’s something that I think we pioneered it at Basecamp. Maybe some other really smart people have come up with this, but it’s possible they haven’t. But that’s what I can talk about today is how this system works. It’s a really neat system and it’s enabled us to scale Basecamp and to use the same version of the chassis of Basecamp version 3 for multiple subsequent versions without having to do a rewrite.

Kimberly (00:07:12): So just to clarify, we’re currently working on Basecamp 5, we said that, but we’re building it on the same platform as 3 was built and 4 was built. And that is because of this recordable situation?

Jeff (00:07:27): Yeah, I mean the main reason we made other versions of Basecamp was because they become too difficult to work with. They’re kind of rigid and we also want to introduce new features or have a different take on how we do something. And that’s hard to change when existing customers are using it. But it’s also been good to sort have a blank slate and start over because you don’t have all this baggage, this big ship that’s hard to turn around. While using this delegated type recording recordables pattern has made the ship less big.

(00:08:03): I was talking about single table inheritance and how you’ve constantly got to modify that single table and migrate it every time you want to add a new kind of thing. You don’t have that with the delegated type recordable pattern. Your recordings table basically never changes. You just add new types, which are their own fresh tables. And so there’s no penalty to adding a new type, deciding that you want to add comments or messages or documents or any other thing that we can imagine. You just make a new table and you’ll have new recording rows that reference it, but you don’t have to modify recordings. So it’s been much easier to scale than other versions of Basecamp. And consequently, we don’t start hating the effort of doing a small thing. You want to be able to iterate on your product easily. And I think that in the past, making a new version of Basecamp was the best way to do that brand new, and we just haven’t felt that pain with the version 3 architecture.

(00:09:05): And it goes entirely to this recordings recordable pattern. Now, there are a few things about the, it’s not just the delegated type. So for those that don’t know delegated type, it’s available in Rails. The documentation in the Rails docs is pretty good, but I think a lot of people don’t quite understand when they would use it or why to use it. Hopefully we can clarify that today. But there are a few other patterns that we use in addition to the delegated type with recordings that make this content system possible. One is that the recordables themselves, we treat them as immutable, so we don’t modify them in place. We create new ones. And another thing that we do is the recordings are organized in a tree, like a parent child relationship. So I’ll go back to the messages example because it’s the most intuitive. We have a message board, and a message board’s children are messages. And a messages children are comments. And a comments children are like attachments. If you’ve uploaded a file to your comment.

Kimberly (00:10:17): What about a boost? Would a boost be a child?

Jeff (00:10:20): Ah. No, boosts are different. Boost actually used that other pattern I was describing called polymorphic relationships where boosts belongs to a boostable thing. And that’s because we want to boost things that aren’t necessarily recordings. You might want to boost a status notification or an event. Events are another key part of the pattern. So that helps me tie into events. So we have the delegated type recording recordable pattern. We have the arrangement of recordings as a tree, we have immutable recordables, and then we have an event system. And the event system can tie a recording to a recordable at a particular moment in time. And so this is why said the recordables themselves are immutable. So we never change them, but we can move the pointers to them around. And at any given time, a recording only points to one recordable. But if you want to have a history of what are all the recordables that it ever pointed to, we track that in an event model. The event has a reference to the recording and it has a reference to a recordable. This means we can look at the history of a recording and see all of its changes and look at that recordable that is immutable at any moment in time to see how it looked.

Kimberly (00:11:43): Yeah, here’s a good example here.

Jeff (00:11:44): Yes, yes, I know what you’re going to show here. This is great.

Kimberly (00:11:49): Do you want to walk us through this change log?

Jeff (00:11:53): Yes, this is a document in Basecamp and what enables this, what we have the change log here, and what enables this is this combination of immutable recordables and events. So what you’re seeing here in this history of changes is really a listing of all the events for this particular recording. And so where you see save to change to this document and you can say, see what changed, what we’re able to do is find the document recordable instance at that moment in time. This is how it looked, this is what its content was, and compare it to the previous version. And we can only do that because we didn’t update the document in place. We created a new one. Now we can compare version A to version B, and you’ll see there’s a button there. Make this the current version. When you click that button, all we’re going to do is update the recording record to point to this version of the document instead of the current version.

Fernando (00:12:49): So recordings are not immutable,

Jeff (00:12:51): No recordings are fully mutable. We change them. It’s the recordables, the type that we delegate to,

Fernando (00:12:57): But what are we changing in a recording? We’re only changing the point of reference to the recordable, right? Yes.

Jeff (00:13:05): There are a few other things that can change. One is its timestamp, like it’s updated at timestamp.

Fernando (00:13:09): Oh, of course.

Jeff (00:13:11): Or if… most other things won’t change. We store a color value on the recording so you can change its color, presentational, things like that, but…

Fernando (00:13:24): Oh, that makes sense.

Jeff (00:13:26): Yeah, you’re right. The recording is mutable, but many things about it don’t change. All of the meat, like all of the action is in the recordables themselves.

Fernando (00:13:36): And you mentioned, for example, a document can have comments. And so the logic I have in my mind here is I want to get to this document. I have the recording. The recording has a reference to the recordable document. When I fetch that recordable, that document, which is a recordable, are the reference to its children… are the references to its children recordings.

Jeff (00:14:08): Yeah, so it’s only the recordings that have children. So you look up a recording by primary key, that would be the ID that you see in the URL. And then you ask for its recordable. And the thing is you don’t care often. You don’t care what kind of recordable it is, if it’s a document or a message. When you do care, the pattern that Rails exposes lets you ask for that type of recordable by name. So if you know want a document, you’re expecting a document to be the recordable, you wouldn’t just ask for the recordable, the generic recordable. You would say recording.document or recording.comment. Another convenience that Rails is delegated, type provides is the ability to query and filter for the type of recordable, the type of delegated type. So you would say recordings.messages would only return recordings that have a type of message. Or recordings.comments would only return recordings that have a type of comment.

Fernando (00:15:09): And if I do like recordings.messages.children, I would get back an array of recordings.

Jeff (00:15:16): Yes. That were messages, right?

Fernando (00:15:20): No, no, no. The children of messages. Let’s say comments.

Jeff (00:15:24): Yes.

Fernando (00:15:25): Okay.

Jeff (00:15:25): Yeah, you’re always going to get recordings. So the children and the parent, they’re always recordings.

Fernando (00:15:28): Always?

Jeff (00:15:30): Yeah, this is a good point to clarify.

Kimberly (00:15:33): Is this something we can show? Or no?

Jeff (00:15:34): I kind of can. Yeah, I can show this. It might help to see how the code is organized. Let me share my screen. So here I’m on my Mac. I have been using Omarchy on a Framework 13 laptop, but my camera is not working great right now. Now it’s probably configuration, I don’t want to say anything bad about Framework. It’s an excellent device, but I didn’t want to risk it for this presentation yet. But ok, so this is the recording class in Basecamp. So what you’re seeing is a whole bunch of concerns. Basically, you see there’s very little, the main model in Basecamp is 42 lines long. It’s kind of incredible. But you can see a few things immediately that are important about a recording. It belongs to a bucket. And I’ll talk about, I can just introduce the bucket briefly. The bucket is just a container for recordings. Buckets are how we control access in Basecamp. So buckets have accesses. You add people to a bucket. If you can have access to the bucket, you can see all the recordings that are in the bucket. Buckets are also a delegated type. A kind of bucket is a project, a template, a ping. Anything that can contain a distinct set of recordings is a bucket.

Fernando (00:16:55): So is a bucket a recordable?

Jeff (00:16:58): No. A bucket is a delegated type, which creates a bucketable. So have the recordable is just the name we’ve given to the target of the delegated type. So in HEY we use the same pattern except in HEY we’ve called them entries. Entry has a delegated type, which we’ve named entryable and in HEY those are things like messages, replies, notes on a message, and these represent emails. So the naming can be whatever you want, but the pattern is the same. So buckets have bucketables. Bucket is the thing that delegates to a bucketable.

Fernando (00:17:36): Oh, okay, perfect, yes.

Jeff (00:17:37): A recording is a thing that delegates to a recordable. And in HEY, an entry delegates to an entryable.

Fernando (00:17:44): And so a bucketable can be a recordable depending on the context, right?

Jeff (00:17:48): I mean technically, but no, you wouldn’t do that. Bucketable would be a different thing. But there’s no reason, they’re just classes. They can reference any kind of thing they want, but we want all of our recordables to behave the same or have a similar interface. And you would want that of all the types you intended to delegate to. So you want all bucket things to quack like buckets.

Fernando (00:18:14): To bucket buckets.

Jeff (00:18:16): Right? As containers of recordings in this case. Or in HEY, as things that are like email messages. Or in Basecamp, recordables, things that are like our chunks of content, documents, messages, uploads. So mixing the types while it would be technically possible would be kind of confusing. So the thing I want to point out here in the recording class is right here, this little inclusion of a Ruby module called recordables, and I have that up here. This is what the recordables concern looks like. And right here on line 5, this is what does all this work. So we say, imagine that this is mixed into recording. So we say that recording has a delegated type called recordable. All the types that it can have are named in this constant, which is in another file, which I’ll show you later. And this does all of the work. And so these are by mixing this into recording, these are all the things that a recording can do.

(00:19:24): But you see this is also a really small file. It’s like a hundred lines long. It doesn’t do a whole lot. It doesn’t even include a lot of other behavior. So where does all this come from? And so this is sort of the key. So this is the recordable class. This is what every individual recordable class, like a message, a comment or whatever includes, and this is where all the types are defined. So all of these, an attachment, an auto link image, a bulletin message, a bulletin link, all the chat things, like a chat, everything from a chat line to a chat transcript, a cloud file, a comment, a doc, a document, a door. All of these things are recordables. So every one of these content types, which is tons are all modeled in the same way as far as recordings and recordables go. And then all of the things that recordables have in common are listed here.

(00:20:23): So I’ll scroll down a bit. These are things that they can do. Are they auto subscribable? Are they subscribable in general? So this is where the type that we delegate to defines its capabilities. So by default, no recordables are subscribable, but if we go to a document and we look like, oh, it enables subscribable. Here it includes the recordable class, the recordable mix in, and now it overrides the subscribable method to say, yes, I can be subscribed to. I can be exported. I can be commented on. I’m auto positioned when I’m placed inside a list. So all of these things you can kind of like, respondable, backlinking, like when one recording references another, we can create a link to that so you can navigate it easy in Basecamp. If something’s copyable or movable, if it’s recurrable or recurring, if it repeats. These are things that you can ask of any kind of recordable.

Kimberly (00:21:25): Jeff, is there something in here we have public links that we can make, is that part of this?

Jeff (00:21:30): Yeah, it probably would be as publishable.

Kimberly (00:21:34): Publishable.

Jeff (00:21:36): Yeah.

Kimberly (00:21:37): Publicly linkable.

Jeff (00:21:37): I’m looking for it. Right? Is it memorizable? But I think… do we have it? I think all recordings we’ve defined as are publishable, so we tend to list only the things that might need to vary. So if some things are commentable, but some things aren’t, then we define a commentable capability and let other recordable types define it, override it. You can see some of our comments here too. There’s lifecycle events that happen. A recordable is recorded, meaning we’ve created a recording that references this recordable. And often you have specific behavior that isn’t going to apply unless you… it only applies to specific types. You might want to do something with a comment being recorded that you wouldn’t want to do when a document is recorded. And so this is one of the challenges of the delegated type and type pattern in general is that where do you stick that specific behavior?

(00:22:41): You’re mostly working with your type that does the delegating, the recording, but you’ve got specific behavior that should only apply to certain recordable types. And in that case, we pass in the context of the recording because recordables themselves, they don’t know they don’t belong to any particular recording. They can be referenced by many recordings. And in fact, that’s how copying works, and that’s how we make copying really efficient. Instead of actually copying the content of a message, we just create a new recording row and point to the same message recordable that already exists. We don’t need to copy it at all, and it’s super fast and storage efficient for that reason. But it means that a message doesn’t know, it doesn’t have any particular recording that owns it, so it can be owned by many or none. And so sometimes when you need to work with a recording, we pass it in. So here we can say the comment reads overwrite to perform recordable specific actions after making a recording. So comment uses this to compose its title based on the parent of its recording. So…

Fernando (00:23:56): That’s the whole purpose of buckets then. Recordables have no understanding of access control. So if you’re in a private project, you have your own document there and you want to copy it over to a public, it’s just referencing the same recordable. The recordable, there’s no change in access to that recordable because access goes through the bucket.

Jeff (00:24:18): Yes, right. They’re sort of orthogonal in that sense though. It’s more that the recordable is, it doesn’t even need to, we don’t even keep things like timestamps on our recordable records because they’re just chunks of content.

Fernando (00:24:34): Yes.

Jeff (00:24:36): Know what I mean? Some recordables don’t even have any row, any columns. They’re just a table that has an ID, their rows just have an ID. They sort of act as a placeholder, if they don’t have any distinct content to store, or they might just have a title, you know?

Fernando (00:24:56): Right. That makes sense. So they can be as small or as large as they need to be.

Jeff (00:25:02): They can be, yeah, they could be as smaller as large as they need to be, and you get to modify them without having to modify all of the recordings, which becomes the big table. The table with billions and billions of rows. I know it’s a little abstract even when I’m showing the code, it’s kind of like…

Fernando (00:25:20): No, no, no, I think it makes perfect sense. You have the recordings, which themselves are just fixed point. They’re the latest pointer to a recordable.

Jeff (00:25:30): Yes, they reference usually the latest version.

Fernando (00:25:34): Yeah, usually. And then, however, if you needed to go through the history of how that recording got to a certain recordable, you go to the events table, right?

Jeff (00:25:44): Yes. You ask the recording for its events. And you could go over each event and say, what was your recordable when this event was created?

Fernando (00:25:53): Now for example, if I’m in a project, what is the algorithm? What are the steps? If I’m opening a tool and I want to see the messages.

Jeff (00:26:06): Right. So the bucket has recordings. The bucket is the project. A bucket delegates its type. Buckets are just things that have recordings. And so if you were in a project, we would be looking for buckets that have a type of project. Then from the bucket record, you can ask a bucket for all of its recordings. You would want to filter those recordings to just those that are messages. Your call would look like bucket.recordings.messages. Now you’re going to get a list of recordings that all point to message recordables, right? So in Basecamp, the messages themselves because of the tree structure, are owned by a message board. So every bucket has one or more message… every project bucket has one or more message boards. You would find a message board — bucket.recordings.messageboards.first. This would give you a recording of type message board, and you would ask for its children.

(00:27:04): Now in Basecamp, the only children of a message board are going to be messages, but you could have different types. So you would want to filter that down and say, just show me the messages. So you would say message board, your message board recording. You would say children.messages. Now you’ll only get its children that have a type of message, still recordings, it’s recordings all the way down. When you finally get to the message recording that you want, you would ask for the recordable. That’s where the title is and where the content is and where other message specific things are.

Fernando (00:27:40): So how do you enforce that? I am coming from a very compiler part of the programming world, and all I’m hearing is let’s make a message board hold pings and chats.

Jeff (00:27:54): Right? You don’t enforce it, right? There’s no method to enforce that. It’s just how you do it. I guess you could say it’s convention, and that’s why the conveniences the delegated type provides like the ability to filter on type are useful. You probably wouldn’t want to just ask for any child because it could contain mixed types. Let’s think about in Basecamp, you have documents, but you can have Google documents and uploads, and all of these go into a container, a parent that we call a vault. A vault is a thing that holds document like things. So you wouldn’t want to just, if you ask a vault for its documents, you’re only going to get documents. So there’s an instance where you do just want to ask for children. All of the things that this might contain, you want the mixed result. So this actually brings me to another cool thing.

(00:28:50): This is one of the reasons we have this pattern is this mixed use case. Let’s say you have a timeline. We have a timeline in Basecamp. We want to see everything that’s happened at a global level. How do you do that and paginate it if all of your records are in different tables? You have to select from the comments table and the documents table and the uploads table and the messages table. Like everything, it’s super hard. But with recordings, you’re just querying for recordings. You can say recordings where recordable type is message, document, comment. You’re going to get all the ones that are that type in one query that you can paginate with a limit and an offset. Super handy. And that query itself is cheap because you’ve only asked for recordings rows that are not big. They don’t have any text columns.

(00:29:50): It’s sort of n plus one by design. But as you iterate over those, then you go and fetch the recordable records. And to improve performance, you can preload them, get them all in one batch for each type. But the key thing that you’re able to do is work with just recordings as if they’re all the same. So I’ll give you another example, unless this is too much, but another example where this uniformity matters. Copying, right? Or any, actually just think not just copying, but any generic process that you’d like to do with a recording. We can export recordings, right? We have an HTML export feature in Basecamp. It’s nice to be able to write the exporter service that doesn’t care about its individual types. All of these, the stuff that would apply to exporting a recording is common to any recordable type, you know? It knows how to write itself… it knows that it has to write something, some output, and then it can delegate the what to do to its recordable.

(00:30:53): So we basically say recordable, write out your export format. Thank you. It means that you don’t need to change the exporter when you add new recordable types, right? Each recordable type just has to define the format that it exports as. Same thing, the copying and moving. You write a copier that works with recordings. That’s it. It doesn’t need to care about what kinds of recordings it’s copying as it iterates over each recording that it’s copying, that recording sends a message to its recordable that says, I would like to, you’ve been requested. A copy has been requested of you, and it just does its thing. So again, adding new types, there’s no penalty. You just add them and the system can work with them. So any of those, what we have in Basecamp, like a bunch of controllers, these are what field requests that come in over HTTP.

(00:31:50): So instead of having to have, like if you want to trash a document or archive a document, we don’t need a separate documents controller for this. We have a recordings controller that knows how to trash things and archive things and restore things. We just need one. It works with any kind of recording. Same thing, that’s how copying and moving works too. We don’t need a document copier and a message copier and an upload copier. We have a recording copier, one controller. It takes the recording that you want to copy and a destination and the copier itself and the controller that orchestrates it doesn’t care what kinds of recordables it’s working with. It just works with recordings. So I think that’s another one of the reasons why Basecamp 3, that architecture has been so easy to maintain and scale. When you decide like, oh, we’re going to add Hill Charts. Oh, we’re going to add, I don’t know, templates. You don’t have to change or migrate the existing system. It adapts well to change. You just insert new types and the system knows how to deal with them.

Fernando (00:32:52): Are you at any point worried that the recording stable is massive? Billions of entries?

Jeff (00:33:00): I mean, it is billions of entries. Not really because its size on disc is small, so it doesn’t take much to index it. But I mean, and if it did, let’s say it did get too big and we had to shard it or break it into pieces, it would be much easier to do because it’s an inherently lightweight table. It really just has foreign key references in it. That’s it. By contrast, the messages table can have many megabytes of text, and these are big on disc. And so the way copying or making an index on disc works is that you’ve got to copy the table. It’s still an entity on disc that you need to copy, and the bigger it is, the slower. So yeah, there is a danger that recordings gets too big, but if it does, it’s much easier to deal with than if it was storing content.

Kimberly (00:33:50): Jeffrey, you’ve made it sound like this recordables pattern that we’re using is much simpler, makes things a lot easier. Are there any downsides?

Jeff (00:34:02): Yeah, I think probably the main downside is familiarity and with the way you would quote unquote normally do this in Rails. Normally you would have a message class, a document class, and everything it knows about being a message or a document would be encoded in that class. I would call that a rich class. It has all the functionality in it. This is what the active record pattern is about. It’s backed by a database table, but you never interact with the database table directly. You interact with your message model. And so if there’s specific things that a message would do that are not generic, you would write those methods right inside the message class or the comment class. And with the delegated type pattern, you kind of want all of your types to be generic and you’re not working with them directly, ever. We never look up just a message because the message on its own is it’s not linked to anything.

(00:35:01): It’s not useful. So in order for a message to implement its specific work, it often needs a reference to a particular recording. If a comment wants to know, you saw in that one comment when I was showing the code where it said comments can set their title based on the parent that created them. Well, a comment doesn’t have a parent. A recording can have a parent. So in order for the comment to be able to do this, it needs to be given a recording to work with. Now it can ask for, it’s that recording’s parent and be like, oh, okay, great. Now I can change myself based on some context. You got to think of the recordables as being pretty dumb. They’re just like in the case of a message, it’s literally just a title and content. That’s it. They have no connection to the outside world.

(00:35:53): They don’t have any associations and nothing points to them directly except for recordings and events. So I think some of that richness gets a little bit lost. You don’t do things quite in the Rails standard way, where you tend to think more abstractly, like how is this a generic concern? How if I want something to be commentable, do I make this work with any kind of recording? So it means that you would have to add some facility to the recording class that helps it stay generic. That’s where more delegation comes in. So you would never just add a comment to a recording. You would ask, is this recording commentable? The recording can’t answer this itself, but it can ask it’s recordable and pass the message along. So it can say like, document, what do you say when I ask are you commentable? And document says yes. But something like, I don’t know, message board. The message board container itself says, no, I’m not commentable. You can’t put comments on me directly. So in the generic comments controller, one of the preconditions is is that the recording has to be commentable. Now you can have one generic controller that handles comments and you can pass it any kind of recording, and it just asks that recording, are you commentable? Can I work with you? And if the recording says, yes, you’re good. And if it says no, it’s an error.

Fernando (00:37:21): Speaking of drawbacks, is that, I mean, I think I know the answer to this, but is that slower? Are you trading speed for clarity?

Jeff (00:37:31): I mean like development speed?

Fernando (00:37:33): Yes.

Jeff (00:37:34): Maybe a little upfront. There’s a higher learning curve cost, right? So when we onboard new developers, it can be non-intuitive. It’s like, ah, I want to add some behavior to the messages. I just opened the message file and there’s nothing in it. Where’s all this behavior? And we’re like, oh, yes, it wouldn’t go here because…

Fernando (00:37:57): Rails magic.

Jeff (00:37:59): But it’s like, no, you wouldn’t put it here. You sort of have to define it in a more abstract, generic way. And then I think that then it becomes a huge net benefit. Once you’ve learned the pattern, now you start to get, because then you’re like, well, I want my message to be copyable. And it’s like, oh, that already works. I want it to be commentable. Oh, just add a def commentable method and return true, done. All the commenting works. I want it to be exportable. Done. Just add an exportable method. So you start to realize, oh, what I’ve sacrificed in being able to make changes to message directly and add some richness to that… I’ll give you an example. Messages can be categorized in Basecamp. You can make categories. It’s useful, but it’s not really useful for other types, maybe documents, but we don’t have that feature enabled.

(00:38:55): So I think in a standard Rails app, you would say that a message belongs to a category and categories have many messages, but in the delegated type recordable pattern, you’ve got to make it so all recordings have the potential to be categorizable and only message opts in. So that cost is there upfront, but then a month later when someone’s like, hey, can we have comments or categories on documents? It’s like, yes, that’s easy, no problem. And so we’ve gotten a ton of leverage out of that. It allows you to build a lot of things that should be hard, just are not hard. They come down to a matter of configuration. You have the system behaves in a uniform way. And where I think this works best, ‘cause this won’t work in all application domains, and we don’t use a delegated type for every kind of model.

(00:39:54): But in Basecamp where you have basically the same chunks of content, like you would have in a content management system or a Wiki or something Notion, what these chunks of content have in common is the operations you can perform on them and with them. And so their ability to just participate in that system just by being defined is incredible. We added the Card Table, our Kanban board system, we call the Card Table, and it was super easy to do because we’re just like, all right, we have a board. It’s got children, you can watch a column. Well, that’s just a subscription. So we’ll make those columns are a kind of recording. Their children are the cards themselves. Each of the cards can have comments. The column itself can be watched, the board itself can also be watched. You’re sort of mixing and matching behavior that already exists.

(00:40:58): And then when it’s like, well, how would I move a column to another column? It’s just a move operation. All recordings are movable. They can move to different containers. So that just works. How do I get a reminder about a due date I’ve given myself on the card? Oh, that just works because all recordings are remindable. And so you’re like, you can build entirely new features that should take months in a week, two weeks, I mean excluding the design, but the software modeling part. And it’s all uniform. So once you’ve figured out how this works, you understand the whole system. It doesn’t scale up with the number of types. It behaves the same whether you have one kind of recordable or 200.

Kimberly (00:41:46): It kind of sounds like, Jeff, I’m imagining as you’re describing this, a whole bunch of Legos or puzzle pieces that are just being redone in a different way. You’re taking elements that you guys already have and reconfiguring them to make a new feature.

Jeff (00:42:02): And if you think about Legos in particular….

Kimberly (00:42:03): I’m probably grossly, oversimplifying that.

Jeff (00:42:06): No, no, but Legos are a good example. If the Legos didn’t fit together, if they didn’t have the same hole and bump pattern, it wouldn’t work. What you get is being able to say, okay, here’s all the Legos, and they’re all interchangeable. Every Lego works with every other Lego and that’s why it works. So there were all these things that motivated the pattern, and what I don’t think we could have anticipated was how well it worked out in the long term, right? From being able to query efficiently to query across types with pagination, to be able to copy and move things really efficiently because we are not having to recreate the message content every time. If a message gets copied 100 times, there’s still only one message recordable. That’s it. Whereas in earlier versions of Basecamp, we would have 100 different rows that each had a copy of that content.

(00:43:03): So things like this adds up. Whenever you start a new Basecamp account, you get a demo project. So with thousands and thousands of Basecamp accounts that all have the same starting content, and we would need thousands and thousands, tens of thousands of rows to represent this. And so you get a really, really big database. Basecamp is the majestic monolith. We don’t use microservices. It’s one app with one big database, and there’s a lot of efficiency in terms of workflow that you get from just having one database. And this is one of the patterns. This delegated immutable types in particular make that possible. Stay in the majestic monolith as long as you can. We’ve been running this architecture for 10 years. Other versions of Basecamp didn’t make it quite that long before we created a new version, and then therefore they’re still growing, but it really reduced the rate that the content is growing in those systems.

(00:44:05): But Basecamp has just been going up and up and up for over 10 years, version 3. So this architecture scales really well. We’re totally confident to build Basecamp 5 on it. It’s just a little bit different maybe than what you’re used to. So once you, and I don’t know how widely used it is because of that. So David is going to do a talk on and a code demonstration of the mechanics, but that’s useful. But we sort of already have that too in documentation and still people miss the big picture. So I’m hoping that that’s what I am able to convey here is why would you do this? Why not just have messages and documents and separate records like the universe intended? And that’s why. I mean, it is fine for a lot of purposes. There is a cost to this level of abstraction, but it’s not as great as you might think and it pays dividends once you’ve made the investment.

Kimberly (00:45:05): I mean, I’m fascinated by this.

Fernando (00:45:08): Oh, I agree. I’ve been trying to think…

Jeff (00:45:10): Well, Fernando, you work on the mobile team and think about the API. We’re able to expose a single recordings, JSON API. Pretty great.

(00:45:19): You can just ask for recordings and you can provide filters between this time and that time. You’re going to get all different types, but this is really easy to implement. And you can do things like your native code can do things and not… It means that things just work. So we add a new type and we don’t need to issue a new, create a new build of the mobile apps. It’s going to get a new kind of recording and it can deal with it in a generic way. It knows that all recordings have titles, all recordings might have a position. All recordings can be archived or active or trashed. It’s not like, oh, everything breaks because we introduced a new type. And that’s one of the other ways because we also are super small team. I mean super small at times. There’s only a handful of people working on any of this stuff, and you would need way more people if you had to coordinate these sorts of changes.

(00:46:15): It means that we just deploy Basecamp whenever and we’re confident that nothing is going to break even as we had new things. You can’t do that for everything. But as far as recordings go, which is the bread and butter of Basecamp, these different kinds of content, these content blocks that are fundamentally content oriented, all behave in the same way. And it means that we ship new stuff and we don’t have to wake Fernando up at night or anybody on his team to say, you need to do a release because we busted something. That’s the kind of efficiency I think you get, and that’s the kinds of things we’re looking for so that we can stay a small team.

Kimberly (00:46:54): If someone is like, yes, this is a better way to work or this is a better way to do this programming with the delegated types, all of this, is it something that you can incorporate into something you’re already doing or do you have to start from scratch?

Jeff (00:47:09): You can incorporate it into what you’re doing. It is a little hard though. It’s definitely better to start from scratch. The migration path would look like it would be a little bit complicated. You would introduce your main table, we’ll call it recordings that delegates its types, and you would either create new types, so if you already had comments, you would need to name it something else, like recordable comments and migrate your content into that. Create recordings as you go. Because the way you create a recording is you create a recording record and you pass it a new recordable, and then when we save the recording, that’s when we update the reference and persist the recordable. So you would sort of do that process and move your content into it. So we’ve done this before when we worked on the HEY calendar. The HEY calendar also uses the delegated type pattern for all of its calendary things. Think of all the things that can appear on a calendar day. You’ve got the event itself. You might want to circle the event. You can have a countdown.

Kimberly (00:48:11): You might want to name it.

Jeff (00:48:12): Name it, it has a title, right.

Kimberly (00:48:14): You can add a picture.

Jeff (00:48:15): Yes, all of these things are implemented as… a calendar is the bucket, and the bucket has recordings. And the recordings are things that are eventlike. Now I talked about how the recordings table, the main table, stores like the metadata, the stuff that all of the types have in common. So in an application like HEY calendar, they have… everything that can appear on a calendar has to have a starts at and an ends at timestamp. You can’t put it on a calendar unless it has knowledge of when it exists.

(00:48:49): So in that case starts at and ends at aren’t delegated to an event type, they’re on the recordings table. That means we can query for any kind of thing that might be on the calendar and any moment in time from one place. That’s what allows us to in one query, fetch all the things that we need to display on a given day or in a given week or month. So part of it is, it’s that you find the stuff that never changes, the things that all of the types would need, and that becomes your main class, your main table. And then you’ve got it when you can just query that one table to get everything you need. Our first iteration of HEY had us going through the different tables, so we had to query for events and we had to query when they’re like, oh, we’re going to add other things to the calendar. I forget, even just being able to circle a certain day or something. And they were all slightly different.

Kimberly (00:49:51): And it’s like tasks. Sometimes this week tasks.

Jeff (00:49:54): That’s another great example. And so we did this and then we had to migrate into that. This was still while we were developing, but we didn’t do it by deleting everything. We were able to transition to it. But that’s a great question. It is the kind of pattern you want to start with. The single table inheritance pattern and the polymorphic pattern are similar. Yes, you can migrate into them, but it’s a pain. It’s better to identify upfront that this is something you might want to use. And similarly, you could go the other way. You could start a delegated type and turn these all into concrete types that have their own tables. You would iterate over the recordings. For each recording you would create a new record, you know. Take its recordable content, take the recording metadata, stick them together and insert that in a new table. So there’s migration paths to and from. But you’re right, you want to start with something like this.

Fernando (00:50:53): Jeff. I’m skeptical. I have to say, I work at 37signals. I work with Basecamp all the time. It is very fast. Why is this not more popular? Is it like obscurity? Is it like, what? Because like, come on, we’ve been talking about this for however long the episode has been going on and it’s like, I keep trying to, I’ve been thinking about poking holes at it. There’s like very few…

Jeff (00:51:25): I don’t know. I think it’s just familiarity, which is why I’m glad we’re doing this and sort of educating people. I think it would be a lot more used if people understood its strengths. The Rails documentation on it is good, but it’s kind of terse. It’s kind of like describes a shallow version of the benefits. But you talked about fast, speed. That’s another great example. Caching. We don’t cache at the recordable level.

Fernando (00:51:50): Oh my. That’s right.

Jeff (00:51:52): We cache at the recording level. It means that the way caching works for any kind of content type is identical across content types, across recordable types. We cache on the recording. When you want to expire the cache, you touch the recording. In the tree pattern, if the tree needs to touch its children, it can, but it’s more common for a child to touch up its tree. But then we have like, we call it the Russian doll caching pattern, where it doesn’t matter, you can invalidate a child way down the tree and not need to invalidate the parent. The parent can still render itself, and then when it gets to this child, it’s like, oh, this part I need to change. But yeah, you don’t get to use the same caching scheme for everything when you don’t have a uniform thing like a recording. So what you’ll see in our views is like “cache (recording) do…end”. Everything is wrapped in a cache block keyed only on the recording. So you get to share a ton of stuff in terms of, it’s not just the, I’ve been mostly talking about the domain model, but this also this cascades to the controllers and the views. It’s all the same. It’s recordings all the way down.

Fernando (00:53:05): You mentioned something about Rails. Is this pattern in Rails somehow? Or part of Rails?

Jeff (00:53:14): Yeah, it’s part of Rails. So it started life in Basecamp as, you know, just in our application code. And then once we realized how good it was, David packaged it up and wrote documentation for it and tests and extracted it to Rails. That’s another reason we know it works. It sounds very abstract, but it’s not ivory tower sort of theory thing. This is what we’re actually using and we’ve proven it out over a decade. So it really, really works. And it’s replaced our use of single table inheritance and even polymorphic relationships. We almost don’t… when you’re going to reach for one of those, you should give delegated type a look and be like, could I do the same thing with the delegated type? Because you get the same sort of sharing, but I think that it’s sort of inverted, right? It’s particularly an inversion of the polymorphic type. In a polymorphic pattern, you have a record that can belong to any kind of record right?

(00:54:16): And in the delegated type pattern, it’s the other way. It’s like, so it’s not your child record that can belong to any kind of thing. It’s the parent record that can have any kind of thing, but it means you get to work with those parents. You’re starting at the top because if you think about something like a comment can have any kind of commentable, when are you ever working from the comment? When do you want to start with the comment table and find all the things? No, it’s at the wrong end of the relationship. You want to start at the top or that’s why the tree pattern works really well too. So I think that it just needs more people once they try it, would realize this is really cool and this can change a lot.

Kimberly (00:55:02): Okay. Jeff, I have a question for you. So you’re clearly an expert at all this recordings, recordables, all of the things. What would you say, if you had to give someone advice, like someone who is using this, what do you think are some of the biggest mistakes people have? I know I didn’t prep you for that question.

Jeff (00:55:22): No, it’s okay. I don’t know if there’s any mistakes so much as… maybe the mistake is not taking it far enough. You know what I mean? You can model all the things that you have in common in this way and you can go pretty far with it. We did Hill Charts in Basecamp, like how do you model those as recordings and recordables? But there’s a way, and you need other patterns. Or how do you take, lilke you want document history or event history. The event stream is another thing. Not only do we want to show you events, but we want the events that we’re listing to be accurate. So you have a timeline and it’s like, Kimberly saved a new document called this. Well, then you went and renamed it, but when you created it, it had the original name. And so when I’m scrolling back, I’m like, Kimberly didn’t name this…

(00:56:16): It says new features March, but I’m reading this, I’ve scrolled back to February. That doesn’t make sense. Does she have a time machine? No. You want accurate history and that’s another way to get it is with this. So I think it’s combining the delegate pattern, the delegate type pattern with other patterns and they really can build off of each other. So once you have this concept of the type doesn’t matter, you can do interesting things with the type. Like, don’t change them, keep a log of them, right, so that you can easily move between them. Move pointers around. It gives you ideas for how you could copy and move things efficiently. So I think that people don’t realize where, it’s not that they use it wrong, it’s that they don’t realize in how many places it’s actually applicable. Not nearly as rigid as STI or polymorphism. It’s super, super flexible.

Fernando (00:57:11): Were you a part of the…. or was this David’s invention?

Jeff (00:57:17): It was a big part of David… David came up with the original idea and it was out of the frustration of how expensive and slow it had gotten to copy things in Basecamp 2, and how every time you introduced a new thing, you also had to introduce changes to the copier. And so yeah, David had the first idea for this. Typical David thing, like how can I invent a brand new pattern that nobody is using? That’s why I think it originated here. And it’s sort of like it has a lot in common with STI, but it’s flipped on its head. And I think also a thing that only David can do is that it was unproven when he introduced it. It was like, eh, this might work. It’s harder to get away with the speculative, let’s give it a shot, unless you’re David, but it’s David and he can just do that.

(00:58:11): And then it was like, let’s see how this works. Because at the time, we hadn’t been designing Basecamp like that. It was very Basecamp 2 like, and then we just, from there, it became a shared thing we could iterate on. And we didn’t extract it for several years after it was in use, maybe a year or two years after it was in use in Basecamp before it proved itself to be a viable pattern. It went through a bunch of iterations. But yeah, the kernel of that idea that you have one row of one table that holds most of the information about the records, but none of the specifics came from David motivated by copying efficiency and the need to paginate, the need to have a timeline of different types that we could paginate with one query. Because the other way to do something like that is to sort of create a table of copies that you insert different records in the table that you’re going to paginate, but now you have a copying system. And we do have a version of that in Basecamp, but it’s a performance optimization. It’s not the main way that we query for recordings. Think of it like caching. We can relay events into another table so that we can very efficiently be like, here’s all the events that apply to just this person. Everybody can see different things in Basecamp. My timeline doesn’t look exactly like your timeline because we’re on different projects.

Fernando (00:59:42): No, I could see how that would be a nightmare to implement if you had different types. I can imagine it in my head like a PM going like, okay, I want you to give me a timeline of things that are happening and the programmers going, oof! That’s going to be rough.

Jeff (01:00:01): And you pay that cost every time you add a new thing. And even when we’re doing work in beta, that’s the one time… so we do work in beta. We’ll modify the database, we’ll add new types and production needs to know how to deal with these new types. In most cases, it just can, because we’ve programmed it in a way that if they don’t know how to deal with a particular recordable, it just deals with it in a generic way. It displays the generic icon. It asks for things that it knows that all recordables have — a title, possibly content, subscribers. And so nothing breaks.

Fernando (01:00:41): Nothing breaks

Jeff (01:00:42): Yeah.

(01:00:43): And so that was another key element of the design. How can we have something that’s flexible? What we found with Basecamp 2 is that, yeah, it works, but it’s slow. And every time we’re like, hey, we want to add a new thing, we’re like, eh, that’s expensive. That’s going to be a pain in the ass. How do you make it not a pain in the ass? And I think good patterns emerge when that’s your criteria. We come up with something to make it easier. What would make this easier? And then so you find the specific problems we had and then correct for those. And David is basically an expert at that.

Fernando (01:01:17): And he hates when things break.

Jeff (01:01:19): And he hates friction…

Fernando (01:01:21): He does, yeah.

Jeff (01:01:22): When the common thing is hard, the common thing should be easy. The common thing we do is introduce new features, new kinds of content. The common thing we do is, and with those content, that content, we need it to work on our mobile apps with no extra effort without having to hire new mobile developers. This is why we have Hotwire and why we have Hotwire native, why we have Rails. Make these super common things easy. The uncommon things just need to be possible. You don’t do them that often. It’s okay to have friction there. Look at deploying apps, right? Everyone was deploying apps with Docker before using Docker and production before we were, but David looks at it and is like, this is a pain. Why is it like this? And so he even invents Kamal. It’s that response. He has a low tolerance for friction on the critical path, and I think people just, they become oblivious to that friction. They sort of ramp up slowly with it and they’re like, this is just the way it is. David never accepts that this is just the way it is. He’s got a real eye for like, this could be different. And he’s not just going to complain about it. He’ll do something. He’ll disappear for a week or two and then be like, here. I fixed it. It’s great.

Kimberly (01:02:47): Jeff, you mentioned some documentation, some Rails documentation over delegated types. We’ll link to that in the notes for this episode. Are there any other places people should go for information about all of this?

Jeff (01:02:58): I mean, right now there’s not a ton of places. There are a few people who have written articles that are pretty good, and they’re sort of the exploratory kind of article. They’re like, hey, you know, I haven’t found a lot of documentation about this cool feature of Rails. And so they’ve written about it, so that’s really good. The Rails documentation, while terse, is also very good, and I think that what we’re doing right now is adding to the growing body of documentation on this pattern. And what David’s going to do is also going to help that. But yeah, if I can find any, I’ll send you the links and we can add that to the show notes because yeah, I think that’s the main thing we’re trying to correct by doing an episode like this is to share the benefits of this underrepresented pattern.

Kimberly (01:03:45): Awesome. Well, with that, we’re going to wrap it up. This has been a production of Recordables by the 37signals team. To hear more from our technical team about their behind the scenes work, visit our developer’s blog at dev.37signals.com.

Other posts in the “RECORDABLES” series

Lexxy: A new rich text editor for Rails

Jorge Manrubia — Thu, 4 Sep 2025 17:00:00 +0000

Today, we are introducing Lexxy, a new rich text editor for Action Text. It’s based on Lexical — Meta’s text editing framework — and it brings a much better text editing experience to Rails:

Good HTML semantics. Paragraphs are real <p> tags, as they should be.
Markdown support: shortcuts, auto-formatting on paste.
Real-time code syntax highlighting.
Create links by pasting URLs on selected text.
Configurable prompts. Support for mentions and other interactive prompts with multiple loading and filtering strategies.
Preview attachments like PDFs and Videos in the editor.
Works seamlessly with Action Text and Active Storage.

We created Lexxy because Trix was falling short of expectations in certain areas, and we encountered technical barriers when attempting to offer the experience we wanted. Lexxy comes with a bunch of juicy improvements, but more than that, we now have a fantastic foundation to build on top of.

Lexxy will also bring a great improvement to Action Text: we will let you configure the editor in Action Text just like you configure the database in Active Record. This will open the door to integrating other editors in Rails.

Text editing is central to our products. We believe Lexxy will let us deliver the editing experience we want. We are launching an early beta today, give it a try!

Running our Docker registry on-prem with Harbor

Farah Schüller — Sun, 24 Aug 2025 17:00:00 +0000

As of early 2025, we’re deploying all of our applications with Kamal using Docker as our containerization platform. The container registry that holds our app images is one of the most integral pieces of our deployment pipeline.

Like many organizations, we’d been using external container registries for years. Our ecosystem was tightly coupled to both Dockerhub and Amazon’s Elastic Container Registry.

However, as part of our cloud exit and kamalization journey, several issues started emerging:

Cost: Not only does the paid license for Dockerhub produce a considerable invoice — pulling and pushing our images over the internet dozens of times a day caused us to hit the contracted bandwidth limit with our datacenter provider Deft repeatedly. We tried working around this by running pull-through caches, but this still locked us to Dockerhub.
Performance: Migrating HEY to Kamal and expanding the deployment to another continent caused deploy time penalties — up to 45 seconds on uncached pulls per host. This was exacerbated once our largest application Basecamp 4 was moved to Kamal — suddenly deployments took minutes longer simply because of push/pull speeds out of our control.
Security and Governance: We all hope to never leak credentials in our images, and yet it still happens — the scale ranging from easily mitigated to catastrophic. We wanted to eliminate that threat surface once and for all by keeping our artifacts where they belong — with us.
Independence: Despite being on a paid account, we fell into the crunch of API limitations for arbitrary reasons a couple of times. In addition, we’d been keeping all of the images used in our Chef CI/CD infrastructure still on AWS.

Our criteria for the solution to pick were fairly simple: reliable, performant, easy to set up, open-source.

We evaluated running the default distribution implementation as our registry, but quickly set our eyes on Harbor. Harbor provided us with a more expandable and rich feature set right out of the box, and required minimal extra tooling to make it robust and scalable.

Setting up Harbor

Harbor’s deployment is optimized for using it within Kubernetes environments, but the single-server setup using the pre-packaged docker-compose configuration proved to be exactly what we were looking for.

We had three key points to cover in our plan for the v1 of our on-premise registry:

Use our own S3 storage.
Make sure we have at least two replicating sites that can be easily failed over.
Keep the storage footprint as small as possible by enabling retention policies.

Configuring S3 storage

At 37signals, we’re running our own Pure FlashBlade storage cluster providing us with S3 object storage right out of the box, but for Harbor, any S3-compatible backend will do.

The configuration in Harbor was easy, but it was crucial to get the permissions set right on the Pure backend. You can obviously run an s3:* policy, but let’s be real, we want to do better! After some trial and error with broken image pushes, these are the minimal permissions needed on the bucket to operate Harbor with a custom S3 backend:

s3:AbortMultipartUpload
s3:DeleteObject
s3:GetBucketLocation
s3:GetObject
s3:ListBucket
s3:ListBucketMultipartUploads
s3:ListMultipartUploadParts
s3:PutObject

Configuring multiple instances

For the v1 of the Harbor deployment, we opted to run two stand-alone instances at first: one in our Ashburn and one in our Chicago location.

Harbor comes with several components, such as PostgreSQL and Redis services, handling manifest/user management and job scheduling. We explored an elaborate HA per datacenter with colocated instances of those services, but decided to wait for the first results of the all-in-one stand-alone deployment before making it more complicated than it has to be.

This is an excerpt of our harbor.yml in use, which gives you a functional instance, including the s3 configuration and enabled monitoring:

hostname: "#{node['fqdn']}"
http:
  port: 80
data_volume: /data
harbor_admin_password: "#{admin_password}"
storage_service:
  s3:
    bucket: docker-registry-bucket
    accesskey: "#{bucket_credentials['access_key']}"
    secretkey: "#{bucket_credentials['secret_key']}"
    regionendpoint: "https://purestorage.#{node["domain"]}"
    region: us-east-1
    encrypt: false
    secure: true
    v4auth: true
    chunksize: 5242880
    loglevel: debug
metric:
  enabled: true
  port: 9090
  path: /metrics
database:
  password: "#{db_password}"
  max_idle_conns: 50
  max_open_conns: 100
clair:
  updaters_interval: 12
jobservice:
  max_job_workers: 20
  job_loggers:
    - FILE
  logger_sweeper_duration: 3600
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
notification:
  webhook_job_max_retry: 3
  webhook_job_http_client_timeout: 10
_version: 2.10.0

As you can see, it is a fairly default config. Encapsulated in a Chef recipe, this will be executed on the respective nodes in each DC, setting the correct FQDN and pointing to the correct storage endpoint. These nodes are then fronted by our F5 loadbalancers for SSL termination and region-specific domains.

Each Harbor node is currently a virtual machine equipped with 64GB of RAM, 32 vCPU and 320GB of storage.

Configuring replication

The initial Chef setup only needs to run once for bootstrapping. For further configuration we decided to rely on the terraform provider for Harbor. In addition to the initial user management setup, here we could also configure replication between the endpoints easily. We decided on a two-way replication scheme to keep it all in sync, inspired by this setup.

Images are pushed to a registry endpoint.
The endpoint pulls data from the opposite registry every 10 minutes.

resource "harbor_replication" "replication_push_sc_chi" {
  provider               = harbor.sc-chi
  name                   = "Replicate images on push to df-iad"
  action                 = "push"
  registry_id            = harbor_registry.df_iad.registry_id
  schedule               = "event_based"
  dest_namespace_replace = -1
  filters {
    name = "**"
  }
  filters {
    tag = "**"
  }
}

resource "harbor_replication" "replication_pull_sc_chi" {
  provider               = harbor.sc-chi
  name                   = "Replicate missing images/artifacts from df-iad"
  action                 = "pull"
  registry_id            = harbor_registry.df_iad.registry_id
  schedule               = "0 0/10 * * * *"
  enabled                = false
  dest_namespace_replace = -1
  filters {
    name = "**"
  }
  filters {
    tag = "**"
  }
}

In addition, we’re replicating the underlying S3 buckets directly on our Pure cluster for an extra failsafe backup mechanism. It’s important to note however that this is not enough to make Harbor aware of the data in the other location — explicit replication on the Harbor level like the setup above must be configured.

Syncing the catalogue

You could of course start with an empty registry and fill it as you go, but this isn’t very feasible if you want a drop-in replacement for your current registry. In our case, we had to make sure that the entire image catalog from Dockerhub gets copied into Harbor — the challenge being that this meant dealing with 80+ individual image repositories. Thankfully, Harbor offers replication directly from Dockerhub, so we opted for that.

Sounds straightforward? Here’s a funny caveat — depending on the amount of repositories you want to fetch and replicate, Dockerhub is likely going to throttle you on the API level if you want to do this all at once. You could totally write a functional replication rule that just targets **/**, only to be showered with 429 — even on a paid account.

Thus, the replication has to happen in batches. For this, we chose to create individual replication rules per repository with a “manual” (also scripted) trigger to avoid overloading the API.

The definition for the replication rules in Terraform:

variable "repositories" {
  type = map(string)
}

resource "harbor_registry" "dockerhub" {
  provider      = harbor.sc-chi
  provider_name = "docker-hub"
  name          = "DockerHub"
  endpoint_url  = "https://registry-1.docker.io"
  description   = "Endpoint for replicating the existing catalogue"
  access_id     = var.dockerhub_username
  access_secret = var.dockerhub_password
}

resource "harbor_replication" "dockerhub_mirror" {
  for_each = var.repositories

  provider               = harbor.sc-chi
  name                   = "mirror-dockerhub-${each.key}"
  description            = "Replicate and mirror images from DockerHub"
  registry_id            = harbor_registry.dockerhub.registry_id
  dest_namespace         = "yourorg"
  override               = true
  dest_namespace_replace = 1
  copy_by_chunk          = true

  filters {
    name = "yourorg/${each.key}"
  }

  filters {
    tag = "**"
  }

  action = "pull"
}

The script to pull out a list of repositories from Dockerhub, make it accessible to Terraform and create the individual replication rules per repository:

dockerhub-to-harbor.sh

#!/usr/bin/env bash

set -euo pipefail

# -----------------------------
# Config
# -----------------------------
DOCKERHUB_USER="${DOCKERHUB_USER:-your-dockerhub-username}"
DOCKERHUB_PASSWORD="${DOCKERHUB_PASSWORD:-your-dockerhub-password}"
DOCKERHUB_ORG="yourorg"
TFVARS_DIR="./generated_tfvars"
TFVARS_FILE="$TFVARS_DIR/all_repos.tfvars.json"

mkdir -p "$TFVARS_DIR"

# -----------------------------
# Authentication
# -----------------------------
echo "🔐 Getting Docker Hub token..."
TOKEN=$(curl -s -X POST https://hub.docker.com/v2/users/login/ \
  -H "Content-Type: application/json" \
  -d '{"username": "'"$DOCKERHUB_USER"'", "password": "'"$DOCKERHUB_PASSWORD"'"}' |
  jq -r .token)

if [[ "$TOKEN" == "null" || -z "$TOKEN" ]]; then
  echo "❌ Failed to authenticate. Check Docker Hub credentials."
  exit 1
fi

export AUTH_HEADER="Authorization: Bearer $TOKEN"

# -----------------------------
# Helper Functions
# -----------------------------
fetch_repos_starting_with() {
  local letter="$1"
  local page=1
  local repos=()

  while :; do
    local url="https://hub.docker.com/v2/repositories/${DOCKERHUB_ORG}/?page=$page&page_size=100"
    local response=$(curl -s -H "$AUTH_HEADER" "$url")

    local matched=$(echo "$response" |
      jq -r ".results[] | select(.name | startswith(\"$letter\")) | .name")

    repos+=($matched)

    local next=$(echo "$response" | jq -r ".next")
    [[ "$next" == "null" ]] && break
    ((page++))
  done

  echo "${repos[@]}"
}

generate_tfvars_file() {
  local repos=("$@")
  local tfvars_file="$TFVARS_FILE"

  echo "{ \"repositories\": {" > "$tfvars_file"
  for repo in "${repos[@]}"; do
    echo "  \"$repo\": \"$repo\"," >> "$tfvars_file"
  done
  sed -i '' '$ s/,$//' "$tfvars_file"
  echo "} }" >> "$tfvars_file"

  >&2 echo "💾 Created tfvars file: $tfvars_file"
  >&2 ls -l "$tfvars_file"

  # Only echo the filename to stdout
  echo "$tfvars_file"
}

run_terraform_once() {
  local tfvars="$1"

  local abs_tfvars
  abs_tfvars=$(realpath "$tfvars")

  echo "🔁 Applying Terraform with $abs_tfvars"

  (
    cd harbor-production || exit 1
    local rel_tfvars
    rel_tfvars=$(python3 -c "import os.path; print(os.path.relpath('$abs_tfvars', '.'))")

    terraform apply -var-file="$rel_tfvars" -auto-approve
  )
}

# -----------------------------
# Execution
# -----------------------------
main() {
  echo "🚀 Starting repository sync..."
  letters=(a b c d e f g h i j k l m n o p q r s t u v w x y z)
  all_repos=()

  for letter in "${letters[@]}"; do
    echo "📦 Fetching repos for prefix: $letter"
    repos=($(fetch_repos_starting_with "$letter"))
    all_repos+=("${repos[@]}")
  done

  if [[ "${#all_repos[@]}" -eq 0 ]]; then
    echo "⚠️ No repositories found."
    exit 0
  fi

  tfvars_file=$(generate_tfvars_file "${all_repos[@]}" 2>/dev/null)
  run_terraform_once "$tfvars_file"

  echo "✅ All repositories synced."
}

main "$@"

And the script to enable those rules alphabetically in batches:

trigger-replication.sh

#!/usr/bin/env bash
set -euo pipefail

# -----------------------------
# Config
# -----------------------------
HARBOR_URL="${HARBOR_URL:-https://registry.yourdomain.com}"
HARBOR_USER="${HARBOR_USER:-your-harbor-user}"
HARBOR_PASSWORD="${HARBOR_PASSWORD:-your-harbor-password}"
BATCH_DELAY=10  # seconds between batches
PREFIX="mirror-dockerhub-"

# -----------------------------
# CLI Args
# -----------------------------
DRY_RUN=false
RANGE_START="a"
RANGE_END="z"

usage() {
  cat <<EOF
Usage: $0 [--dry-run] [--range <start-end>]

Options:
  --dry-run         Only print the rules that would be triggered, no API calls.
  --range a-d       Trigger only rules whose names start with '${PREFIX}' plus letter in <start-end>.
                    Example: --range a-d
EOF
  exit 1
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run)
      DRY_RUN=true
      shift
      ;;
    --range)
      if [[ "$2" =~ ^[a-zA-Z]-[a-zA-Z]$ ]]; then
        RANGE_START=$(echo "$2" | cut -d- -f1 | tr '[:upper:]' '[:lower:]')
        RANGE_END=$(echo "$2" | cut -d- -f2 | tr '[:upper:]' '[:lower:]')
        shift 2
      else
        echo "Invalid range format. Expected like 'a-d'."
        usage
      fi
      ;;
    *)
      echo "Unknown argument: $1"
      usage
      ;;
  esac
done

if [[ "$RANGE_START" > "$RANGE_END" ]]; then
  echo "Invalid range: start ($RANGE_START) > end ($RANGE_END)"
  exit 1
fi

# -----------------------------
# Auth & Token
# -----------------------------
echo "🔐 Authenticating with Harbor..."
AUTH_HEADER="Authorization: Basic $(echo -n "$HARBOR_USER:$HARBOR_PASSWORD" | base64)"

curl -s -H "$AUTH_HEADER" "$HARBOR_URL/api/v2.0/users/current" | jq -e .username > /dev/null || {
  echo "❌ Harbor auth failed"
  exit 1
}

echo "📋 Fetching all replication rules..."
rules=$(curl -s -H "$AUTH_HEADER" "$HARBOR_URL/api/v2.0/replication/policies?page_size=100")

declare -A letter_to_ids

while IFS= read -r rule; do
  id=$(echo "$rule" | jq -r '.id')
  name=$(echo "$rule" | jq -r '.name')

  # Filter by prefix first
  if [[ "$name" == "$PREFIX"* ]]; then
    # Get letter after the prefix
    suffix_letter=$(echo "${name#$PREFIX}" | cut -c1 | tr '[:upper:]' '[:lower:]')
    if [[ "$suffix_letter" < "$RANGE_START" || "$suffix_letter" > "$RANGE_END" ]]; then
      continue
    fi
    letter_to_ids["$suffix_letter"]+="$id "
  fi
done < <(echo "$rules" | jq -c '.[]')

echo "✅ Loaded replication rules matching prefix '$PREFIX'."

increment_letter() {
  local c=$1
  printf "\\$(printf '%03o' "$(( $(printf '%d' "'$c") + 1 ))")"
}

current="$RANGE_START"
while [[ $current < $RANGE_END || $current == $RANGE_END ]]; do
  ids=${letter_to_ids[$current]:-}
  if [[ -n "$ids" ]]; then
    echo "🚀 Processing prefix '$PREFIX$current' with rule IDs: $ids"
    for id in $ids; do
      if $DRY_RUN; then
        echo "   (dry-run) Would trigger rule ID: $id"
      else
        echo "   🔁 Triggering rule ID: $id"
        curl -s -X POST -H "$AUTH_HEADER" \
          -H "Content-Type: application/json" \
          -d "{\"policy_id\": $id}" \
          "$HARBOR_URL/api/v2.0/replication/executions" > /dev/null
      fi
    done
    if ! $DRY_RUN; then
      echo "⏳ Waiting $BATCH_DELAY seconds before next batch..."
      sleep "$BATCH_DELAY"
    fi
  fi
  current=$(increment_letter "$current")
done

echo "✅ Done."

The progress of all replication tasks at the same time is quite hard to monitor within Harbor’s UI (despite excellent logging). Thus, another small script helped summarize this:

harbor-replication-monitor.sh

#!/usr/bin/env bash

HARBOR_URL="${HARBOR_URL:-https://registry.yourdomain.com}"
HARBOR_USER="${HARBOR_USER:-your-harbor-user}"
HARBOR_PASSWORD="${HARBOR_PASSWORD:-your-harbor-password}"

# Check required env vars
if [[ -z "$HARBOR_USERNAME" || -z "$HARBOR_PASSWORD" || -z "$HARBOR_URL" ]]; then
  echo "❌ Missing HARBOR_USERNAME, HARBOR_PASSWORD or HARBOR_URL. Set them as env vars."
  exit 1
fi

# -----------------------------
# Auth
# -----------------------------
echo "🔐 Authenticating with Harbor..."
pong=$(curl -s -u "$HARBOR_USERNAME:$HARBOR_PASSWORD" "$HARBOR_URL/ping")
if [ "$pong" != "Pong" ]; then
  echo "❌ Authentication failed. Harbor did not return expected 'Pong'."
  echo "Response: $pong"
  exit 1
fi
echo "✅ Auth successful."

# -----------------------------
# Fetch executions
# -----------------------------
echo "📋 Fetching replication executions..."
executions=$(curl -s -u "$HARBOR_USERNAME:$HARBOR_PASSWORD" "$HARBOR_URL/replication/executions?page_size=100")

# Cache for policy ID to name mapping
declare -A POLICY_NAMES
get_policy_name() {
  local policy_id="$1"
  if [[ -n "${POLICY_NAMES[$policy_id]}" ]]; then
    echo "${POLICY_NAMES[$policy_id]}"
  else
    local name=$(curl -s -u "$HARBOR_USERNAME:$HARBOR_PASSWORD" "$HARBOR_URL/replication/policies/$policy_id" | jq -r '.name // "unknown"')
    POLICY_NAMES[$policy_id]="$name"
    echo "$name"
  fi
}

# -----------------------------
# Build table
# -----------------------------
rows=()
while IFS= read -r exec; do
  id=$(echo "$exec" | jq -r '.id')
  policy_id=$(echo "$exec" | jq -r '.policy_id')
  status=$(echo "$exec" | jq -r '.status')
  start_time=$(echo "$exec" | jq -r '.start_time' | sed 's/\.[0-9]*Z$/Z/')

  start_epoch=$(date -u -d "$start_time" +%s 2>/dev/null)
  now_epoch=$(date +%s)
  runtime_min=$(( (now_epoch - start_epoch) / 60 ))

  policy_name=$(get_policy_name "$policy_id")

  row=$(printf "%-8s %-10s %-25s %-12s %-14s %s" "$id" "$policy_id" "$policy_name" "$status" "$runtime_min" "$start_time")
  rows+=("$runtime_min $row")
done < <(echo "$executions" | jq -c '.[] | select(.status == "InProgress")')

echo
echo "🟡 In-progress replication tasks (sorted by runtime):"
printf "%-8s %-10s %-25s %-12s %-14s %s\n" "ID" "Policy_ID" "Rule Name" "Status" "Runtime(min)" "Start Time"

# -----------------------------
# Print
# -----------------------------
for line in "${rows[@]}"; do
  echo "$line"
done | sort -rn | cut -d' ' -f2-

After enabling those rules in batches, it’s also crucial to make sure enough job worker resources are available to sufficiently speed up this process.

Analyzing performance

After migrating all our Kamal-ized apps to push and pull from our new on-premise registry, it was finally time to actually get some numbers on performance in. We grabbed this data directly from the deployment logs printed by Kamal.

You can use these quick one-liners for extracting the pull times from the Kamal log on your terminal:

# Mac
pbpaste | sort | sed 's/  INFO //' | grep -E "Running docker pull" -B1 | grep Finished | awk '{print $4}' | sort | tr '\n' ' '
# Linux
xclip -o | sort | sed 's/  INFO //' | grep -E "Running docker pull" -B1 | grep Finished | awk '{print $4}' | sort | tr '\n' ' '

Or from a logfile:

awk '
/Running .*docker pull/ {
  if (match($0, /\[([a-f0-9]+)\]/, m)) {
    id = m[1]
    running[id] = $0
  }
}
/Finished/ {
  if (match($0, /\[([a-f0-9]+)\]/, m)) {
    id = m[1]
    if (id in running) {
      print running[id]
      print $0
      print ""
      delete running[id]
    }
  }
}
' /path/to/kamal.log

After analyzing the numbers, we were quite happy to see that:

The overall image pull timings on our fleet decreased by up to 25 seconds for HEY, Basecamp 4 and Basecamp 2 (our three largest apps), with the lion’s share of improvement on our HEY nodes on the Amsterdam outposts.
Deploy times decreased by 15 seconds for HEY.

In addition, it allowed us to:

Retire the Dockerhub cache setup, further detangling our infrastructure.
Implement proper retention policies and garbage collection to decrease the overall storage quota from almost 9 TiB to 1.5 TiB.
Save roughly $5k/year on subscription fees going forward.

Remember: this is basically a single-node infrastructure, with the primary endpoint being in Chicago, and the Ashburn site providing the backup. We found that this small setup has been reliable for roughly two months now. During this time, Harbor has served more than 32,000 pulls under company-wide use in day-to-day business.

Conclusion

This project proved to us that it’s — again — worth considering a departure from large SaaS offerings and public cloud providers. We’ve been dependent on external registries keeping our app images for years, but the simplicity and benefits of our current setup give little reason to doubt that cutting the cord was the right decision: better performance at less cost with minimal infrastructure.

Introducing Action Push Native

Jacopo Beschi — Mon, 18 Aug 2025 17:00:00 +0000

Note: Shortly after releasing this gem, we renamed it from Action Native Push to Action Push Native, in case you arrived here looking for the gem of the former name. More details here.

We’ve open-sourced Action Push Native, a Rails gem for sending push notifications to mobile platforms. It supports both Apple and Google push notification services.

Why did we build it?

We created it to migrate off Amazon SNS and Pinpoint, as part of our broader cloud exit. We’re using it in Basecamp and HEY to send more than 10 million push notifications per day without a hitch.

Action Push Native relies on HTTP/2 persistent connections to the Apple Push Notification service, which significantly reduced job duration compared to our previous HTTP/1 setup with AWS Pinpoint:

How does it work?

The gem connects directly to the Apple (APNs) and Google (FCM) push notification services. It handles retries, rate-limiting, and deleting dead devices automatically. Configure each platform with your credentials, and you can start sending notifications like this:

device = ApplicationPushDevice.create! \
  name: "iPhone 16",
  token: "6c267f26b173cd9595ae2f6702b1ab560371a60e7c8a9e27419bd0fa4a42e58f",
  platform: "apple"

notification = ApplicationPushNotification.new \
  title: "Hello world!",
  body:  "Welcome to Action Push Native"

notification.deliver_later_to(device)

Version 0.1.0 is available now. You can read more on GitHub.

We hope you find it useful!

Announcing Hotwire Native 1.2

Jay Ohms — Wed, 23 Apr 2025 17:00:00 +0000

We’ve just launched Hotwire Native v1.2 and it’s the biggest update since the initial launch last year. The update has several key improvements, bug fixes, and more API consistency between platforms. And we’ve created all new iOS and Android demo apps to show it off!

Improvements

There are a few significant changes in v1.2 that are worth specifically highlighting.

Route decision handlers

Hotwire Native apps route internal urls to screens in your app, and route external urls to the device’s browser. Historically, though, it wasn’t straightforward to customize the default behavior for unique app needs.

In v1.2, we’ve introduced the RouteDecisionHandler concept to iOS (formerly only on Android). Route decisions handlers offer a flexible way to decide how to route urls in your app. Out-of-the-box, Hotwire Native registers these route decision handlers to control how urls are routed:

AppNavigationRouteDecisionHandler: Routes all internal urls on your app’s domain through your app.
SafariViewControllerRouteDecisionHandler: (iOS Only) Routes all external http/https urls to a SFSafariViewController in your app.
BrowserTabRouteDecisionHandler: (Android Only) Routes all external http/https urls to a Custom Tab in your app.
SystemNavigationRouteDecisionHandler: Routes all remaining external urls (such as sms: or mailto:) through device’s system navigation.

If you’d like to customize this behavior you can register your own RouteDecisionHandler implementations in your app. See the documentation for details.

Server-driven historical location urls

If you’re using Ruby on Rails, the turbo-rails gem provides the following historical location routes. You can use these to manipulate the navigation stack in Hotwire Native apps.

recede_or_redirect_to(url, **options) — Pops the visible screen off of the navigation stack.
refresh_or_redirect_to(url, **options) — Refreshes the visible screen on the navigation stack.
resume_or_redirect_to(url, **options) — Resumes the visible screen on the navigation stack with no further action.

In v1.2 there is now built-in support to handle these “command” urls with no additional path configuration setup necessary. We’ve also made improvements so they handle dismissing modal screens automatically. See the documentation for details.

Bottom tabs

When starting with Hotwire Native, one of the most common questions developers ask is how to support native bottom tab navigation in their apps. We finally have an official answer! We’ve introduced a HotwireTabBarController for iOS and a HotwireBottomNavigationController for Android. And we’ve updated the demo apps for both platforms to show you exactly how to set them up.

New demo apps

To better show off all the features in Hotwire Native, we’ve created new demo apps for iOS and Android. And there’s a brand new Rails web app for the native apps to leverage.

Clone the GitHub repos to build and run the demo apps to try them out:

Huge thanks to Joe Masilotti for all the demo app improvements. If you’re looking for more resources, Joe even wrote a Hotwire Native for Rails Developers book!

Release notes

v1.2 contains dozens of other improvements and bug fixes across both platforms. See the full release notes to learn about all the additional changes:

Take a look

If you’ve been curious about using Hotwire Native for your mobile apps, now is a great time to take a look. We have documentation and guides available on native.hotwired.dev and we’ve created really great demo apps for iOS and Android to help you get started.

Monitoring 10 Petabytes of data in Pure Storage

Victor Bogo — Thu, 2 Jan 2025 18:00:00 +0000

As the final part of our move out of the cloud, we are working on moving 10 petabytes of data out of AWS Simple Storage Service (S3). After exploring different alternatives, we decided to go with Pure Storage FlashBlade solution.

We store different kinds of information on S3, from the attachments customers upload to Basecamp to the Prometheus long-term metrics. On top of that, Pure’s system also provides filesystem-based capabilities, enabling other relevant usages, such as database backup storage. This makes the system a top priority for observability.

Although the system has great reliability, out-of-the-box internal alerting, and autonomous ticket creation, it would also be good to have our metrics and alerts to facilitate problem-solving and ensure any disruptions are prioritized and handled.

For more context on our current Prometheus setup, see how we use Prometheus at 37signals.

Pure OpenMetrics exporter

Pure maintains two OpenMetrics exporters, pure-fb-openmetrics-exporter and pure-fa-openmetrics-exporter. Since we use Pure FlashBlade (fb), this post covers pure-fb-openmetrics-exporter, although overall usage should be similar. The setup is straightforward and requires only binary and basic authentication installation.

Here is a snippet of our Chef recipe that installs it:

pure_api_token = "token" # If you use Chef, your token should come from an encrypted databag. Changed to hardcoded here to simplify
PURE_EXPORTER_VERSION = "1.0.13".freeze # Generally, we use Chef node metadata for version management. Changed to hardcoded to simplify

directory "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}" do
 recursive true
 owner 'pure_exporter'
 group 'pure_exporter'
end

# Avoid recreating under /tmp after reboot if target_binary is already there
target_binary = "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter"

remote_file "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz" do
 source "https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/releases/download/v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz"
 not_if { ::File.exist?(target_binary) }
end

archive_file "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz" do
 destination "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}"
 action :extract
 not_if { ::File.exist?(target_binary) }
end

execute "copy binary" do
 command "sudo cp /tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter /opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter"
 creates "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter"
 not_if { ::File.exist?(target_binary) }
end

tokens = <<EOF
main:
 address: purestorage-mgmt.mydomain.com
 api_token: #{pure_api_token['token']}
EOF

file "/opt/pure_exporter/tokens.yml" do
 content tokens
 owner 'pure_exporter'
 group 'pure_exporter'
 sensitive true
end

systemd_unit 'pure-exporter.service' do
 content <<-EOU
 # Caution: Chef managed content. This is a file resource from #{cookbook_name}::#{recipe_name}
 #

 [Unit]
 Description=Pure Exporter
 After=network.target

 [Service]
 Restart=on-failure
 PIDFile=/var/run/pure-exporter.pid
 User=pure_exporter
 Group=pure_exporter
 ExecStart=/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter \
 --tokens=/opt/pure_exporter/tokens.yml

 ExecReload=/bin/kill -HUP $MAINPID
 SyslogIdentifier=pure-exporter

 [Install]
 WantedBy=multi-user.target
 EOU

 action [ :create, :enable, :start ]
 notifies :reload, "service[pure-exporter]"
end

service 'pure-exporter'

Prometheus Job Configuration

The simplest way of ingesting the metrics is to configure a basic Job without any customization:

- job_name: pure_exporter
  metrics_path: /metrics
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main] # From the tokens configuration above

For a production-ready setup, we are using a slightly different approach. The exporter supports the usage of specific metric paths to allow for split Prometheus jobs configuration that reduces the overhead of pulling the metrics all at once:

- job_name: pure_exporter_array
  metrics_path: /metrics/array
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  metric_relabel_configs:
  - source_labels: [name]
    target_label: ch
    regex: "([^.]+).*"
    replacement: "$1"
    action: replace
  - source_labels: [name]
    target_label: fb
    regex: "[^.]+\\.([^.]+).*"
    replacement: "$1"
    action: replace
  - source_labels: [name]
    target_label: bay
    regex: "[^.]+\\.[^.]+\\.([^.]+)"
    replacement: "$1"
    action: replace
  params:
    endpoint: [main] # From the tokens configuration above
- job_name: pure_exporter_clients
  metrics_path: /metrics/clients
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main] # From the tokens configuration above
- job_name: pure_exporter_usage
  metrics_path: /metrics/usage
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main]
- job_name: pure_exporter_policies
  metrics_path: /metrics/policies
  static_configs:
  - targets: ['<%= @hostname %>:9491']
    labels:
      environment: 'production'
      job: pure_exporter
  params:
    endpoint: [main] # From the tokens configuration above

We also configure some metric_relabel_configs to extract labels from name using regex. Those labels help reduce the complexity of queries that aggregate metrics by different components.

Detailed documentation on the available metrics can be found here.

Alerts

Auto Generated Alerts

As I shared earlier, the system has an internal Alerting module that automatically triggers alerts for critical situations and creates tickets. To cover those alerts on the Prometheus side, we added an alerting configuration of our own that relies on the incoming severities:

- alert: PureAlert
  annotations:
    summary: '{{ $labels.summary }}'
    description: '{{ $labels.component_type }} - {{ $labels.component_name }} - {{ $labels.action }} - {{ $labels.kburl }}'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_alerts_open{environment="production"} == 1
  for: 1m

We still need to evaluate how the pure-generated alerts will interact with the custom alerts I will cover below, and we might decide to stick to one or the other depending on what we find out.

Hardware

Before I continue, the image below helps visualize how some of the Pure FlashBlade components are physically organized:

Because of Pure’s reliability, most isolated hardware failures do not require the immediate attention of an Ops team member. To cover the most basic hardware failures, we configure an alert that sends a message to the Ops Basecamp 4 project chat:

- alert: PureHardwareFailed
  annotations:
    summary: Hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed
    description: 'The Pure Storage hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_hardware_health == 0
  for: 1m
  labels:
    severity: chat-notification

We also configure alerts that check for multiple hardware failures of the same type. This doesn’t mean two simultaneous failures will result in a critical state, but it is a fair guardrail for unexpected scenarios. We also expect those situations to be rare, keeping the risk of causing unnecessary noise low.

- alert: PureMultipleHardwareFailed
  annotations:
    summary: Pure chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}
    description: 'The Pure Storage chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'
    dashboard: 'https://grafana/your-dashboard'
  expr: count(purefb_hardware_health{type!~"eth|mgmt_port|bay"} == 0) by (ch,type,environment) > 1
  for: 1m
  labels:
    severity: page
 # We are looking for multiple failed bays in the same blade
- alert: PureMultipleBaysFailed
  annotations:
    summary: Pure chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays
    description: 'The Pure Storage chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'
    dashboard: 'https://grafana/your-dashboard'
  expr: count(purefb_hardware_health{type="bay"} == 0) by (ch,type,fb,environment) > 1
  for: 1m
  labels:
    severity: page

Finally, we configure high-level alerts for chassis and XFM failures:

- alert: PureChassisFailed
  annotations:
    summary: Chassis {{ $labels.name }} is failed
    description: 'The Pure Storage hardware chassis {{ $labels.name }} is failed'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_hardware_health{type="ch"} == 0
  for: 1m
  labels:
    severity: page
- alert: PureXFMFailed
  annotations:
    summary: Xternal Fabric Module {{ $labels.name }} is failed
    description: 'The Pure Storage hardware Xternal fabric module {{ $labels.name }} is failed'
    dashboard: 'https://grafana/your-dashboard'
  expr: purefb_hardware_health{type="xfm"} == 0
  for: 1m
  labels:
  severity: page

Latency

Using the metric purefb_array_performance_latency_usec we can set a threshold for all the different protocols and dimensions (read, write, etc), so we are alerted if any problem causes the latency to go above an expected level.

- alert: PureLatencyHigh
  annotations:
    summary: Pure {{ $labels.dimension }} - {{ $labels.protocol }} latency high
    description: 'Pure {{ $labels.protocol }} latency for dimension {{ $labels.dimension }} is above 100ms'
    dashboard: 'https://grafana/your-dashboard'
  expr: (avg_over_time(purefb_array_performance_latency_usec{protocol="all"}[30m]) * 0.001)
  for: 1m
  labels:
    severity: chat-notification

Saturation

For saturation, we are primarily worried about something unexpected causing excessive use of array space, increasing the risk of hitting the cluster capacity. With that in mind, it’s good to have a simple alert in place, even if we don’t expect it to fire anytime soon:

- alert: PureArraySpace
  annotations:
    summary: Pure Cluster {{ $labels.instance }} available space is expected to be below 10%
    description: 'The array space for pure cluster {{ $labels.instance }} is expected to be below 10% in a month, please investigate and ensure there is no risk of running out of capacity'
    dashboard: 'https://grafana/your-dashboard'
  expr: (predict_linear(purefb_array_space_bytes{space="empty",type="array"}[30d], 730 * 3600)) < (purefb_array_space_bytes{space="capacity",type="array"} * 0.10)
  for: 1m
  labels:
    severity: chat-notification

HTTP

We use BigIp load balancers to front-end the cluster, which means that all the alerts we already had in place for the BigIp HTTP profiles, virtual servers, and pools also cover access to Pure. The solution for each organization on this topic will be different, but it is a good practice to keep an eye on HTTP status codes and throughput.

Grafana Dashboards

The project’s GitHub repository includes JSON files for Grafana dashboards that are based on the metrics generated by the exporter. With simple adjustments to fit each setup, it’s possible to import them quickly.

Wrapping up

On top of the system’s built-in capabilities, Pure also provides options to integrate their system into well-known tools like Prometheus and Grafana, facilitating the process of managing the cluster the same way we manage everything else. I hope this post helps any other team interested in working with them better understand the effort involved.

Thanks for reading!

Other posts in the “On-prem observability” series

Announcing Hotwire Spark: live reloading for Rails applications

Jorge Manrubia — Wed, 18 Dec 2024 18:00:00 +0000

Today, we are releasing Hotwire Spark, a live-reloading system for Rails Applications.

Reloading the browser automatically on source changes is a problem that has been well-solved for a long time. Here, we wanted to put an accent on smoothness. If the reload operation is very noticeable, the feedback loop is similar to just reloading the page yourself. But if it’s smooth enough — if you only perceive the intended change — the feedback loop becomes terrific.

To use, just install the gem in development:

group :development do
  gem "hotwire-spark"
end

It will update the current page on three types of change: HTML content, CSS, and Stimulus controllers. How do we achieve that desired smoothness with each?

For HTML content, it morphs the <body> of the page into the new <body>. Also, it disconnects and reconnects all the Stimulus controllers on the page.
For CSS, it reloads the changed stylesheet.
For Stimulus controllers, it fetches the changed controller, replaces its module in Stimulus, and reconnects all the controllers.

We designed Hotwire Spark to shine with the nobuild approach we use and recommend. Serving CSS and JS assets as standalone files is ideal when you want to fetch and update only what has changed. There is no need to use bundling or any tooling. Hot Module Replacement for Stimulus controllers without any frontend building tool is pretty cool!

2024 has been a very special year for Rails. We’re thrilled to share Hotwire Spark before the year wraps up.

Wishing you all a joyful holiday season and a fantastic start to 2025.

A vanilla Rails stack is plenty

Jorge Manrubia — Thu, 12 Dec 2024 18:00:00 +0000

Minimal dependencies, maximum productivity. Staying vanilla pays long term dividends for your Rails apps.

If you have the luxury of starting a new Rails app today, here’s our recommendation: go vanilla.

Fight hard before adding Ruby dependencies. Keep that Gemfile that Rails generates as close to the original one as possible.
Fight even harder before adding Javascript dependencies. You don’t need React or any other front-end frameworks, nor a JSON API to feed those.
Hotwire is a fantastic, pragmatic, and ridiculously productive technology for the front end. Use it.
The same goes for mobile apps: use Hotwire Native. With a hybrid approach you can combine the very same web app you have built with a wonderful native experience right where you want it. The productivity compared to a purely native approach is night and day.
Embrace and celebrate rendering things on the server. It has become cool again.
ERB templates and view helpers will take you as long as you need, and they are a fantastic common ground for designers to collaborate hands-on with the code.
#nobuild is the simplest way to go; don’t close this door with your choices. Instead of bundling Javascript, use import maps. Don’t bundle CSS, just use modern standard CSS goodies and serve them all with Propshaft. If you have 100 Javascript files and 100 stylesheets, serve 200 standalone requests multiplexed over HTTP2. You will be delighted.
Don’t add Redis to the mix. Use solid_cache for caching, solid_queue for jobs, and solid_cable for Action Cable. They will all work on your beloved relational database and are battle-tested.
Test your apps with Minitest. Use fixtures and build a realistic set of those as you cook your app.
Make your app a PWA, which is fully supported by Rails 8. This may be more than enough before caring about mobile apps at all.
Deploy your app with Kamal.

If you want heuristics, your importmap.rb should import Turbo, Stimulus, your app controllers, and little else. Your Gemfile should be almost identical to the one that Rails generates. I know it sounds radical, but going vanilla is a radical stance in this convoluted world of endless choices.

This is the Rails 8 stack we have chosen for our new apps at 37signals. We are a tiny crew, so we care a lot about productivity. And we sell products, not stacks, so we care a lot about delighting our users. This is our Omakase stack because it offers the optimal balance for achieving both.

Vanilla means your app stays nimble. Fewer dependencies mean fewer future headaches. You get a tight integration out of the box, so you can focus on building things. It also maximizes the odds of having smoother future upgrades. Vanilla requires determination, though, because new dependencies always look shiny and shinier. It’s always clear what you get when you add them, but never what you lose in the long term.

It is certainly up to you. Rails is a wonderful big tent. These are our opinions.

If it resonates, choose vanilla!

Guess what our advice is for architecting your app internals?

Mission Control — Jobs 1.0 released

Rosa Gutiérrez — Wed, 4 Dec 2024 18:00:00 +0000

We’ve just released Mission Control — Jobs v1.0.0, the dashboard and set of extensions to operate background jobs that we introduced earlier this year. This new version is the result of 92 pull requests, 67 issues and the help of 35 different contributors. It includes many bugfixes and improvements, such as:

Support for Solid Queue’s recurring tasks, including running them on-demand.
Support for API-only apps.
Allowing immediate dispatching of scheduled and blocked jobs.
Backtrace cleaning for failed jobs’ backtraces.
A safer default for authentication, with Basic HTTP authentication enabled and initially closed unless configured or explicitly disabled.

We use Mission Control — Jobs daily to manage jobs HEY and Basecamp 4, with both Solid Queue and Resque, and it’s the dashboard we recommend if you’re using Solid Queue for your jobs. Our plan is to upstream some of the extensions we’ve made to Active Job and continue improving it until it’s ready to be included by default in Rails together with Solid Queue.

If you want to help us with that, are interested in learning more or have any issues or questions, head over to the repo in GitHub. We hope you like it!

37signals Dev

Parsing JSON-structured Docker logs with the OpenTelemetry Collector

No built-in Docker log receiver

The problem: JSON inside JSON

The operator chain

1. container: unwrap the Docker envelope

2. filter: drop non-JSON lines

3. recombine: reassemble split JSON

4. json_parser: parse the app JSON

Two things we got wrong

What comes next

Hot Cell v1.0: Securing Active Storage in the age of AI

Where we are

Part 1: You are not worried enough (probably)

Part 2: Reducing the blast radius

Step 1: Define the requirements

Step 2: An obscure, too-clever name 🤣

Step 3: Jam it into Rails

Step 4: Inputs and outputs

Step 5: Doing the work

Part 3: Rolling it out

Observability

The cost

Coloring outside the box

What production taught us

What’s next

Part 4: The moment we are in

Warming up the Puma master before it forks

Why 63 single-threaded workers?

What happens when we deploy

What didn’t work

Run the requests in the master

1. Precompile the views

2. Request the pages, signed in

3. Load the rest of the schema

What to close before the fork

Choosing the pages

Results

Working with Claude

If you do this

Lexxy 1.0 is here

Trix hit a wall

Why Lexical

The iceberg

What’s included?

Features

Action Text Native

Extensibility

Performance

Accessibility

Security

Agents

The new Rails default

The ONCE app server

Timestamps

Links & Resources

Transcript

Behind the Fizzy Infrastructure

Timestamps

Links & Resources

Transcript

Introducing Upright: An Open Source Synthetic Monitoring System

Why we built it

What it does

How it’s built

Getting started

Try it out

Rails Multi-Tenancy

Timestamps

Links & Resources

Transcript

Moving Mountains of Data off S3

Timestamps

Links & Resources

Transcript

The Rails Delegated Type Pattern

Timestamps

Links & Resources

Transcript

Lexxy: A new rich text editor for Rails

1. `container`: unwrap the Docker envelope

2. `filter`: drop non-JSON lines

3. `recombine`: reassemble split JSON

4. `json_parser`: parse the app JSON