Julia Evans

Replacing the old battery on rechargeable bike lights

Sun, 27 Sep 2026 00:00:00 +0000

Hello! Recently I needed bike lights for my bike. And I remembered that I already had rechargeable bike lights that I bought ten years ago, that I hadn’t tried in a long time. I tried to recharge them, but after fully charging them, they only worked for maybe 5 minutes before they turned off again.

I don’t know much about electronics, but I’ve been curious about whether it’s possible to fix old electronics for a long time, and this seemed like the perfect repair project because I might just need to replace the battery.

So I went to the local queer makerspace where I’m a member to use the soldering iron and try to do it! I don’t know much about electronics and this post does not contain any safety advice because I don’t know much about safety. I think it’s nice to do projects in a community space where you can get help.

step 1: cut it open

The bike light felt like it was made of silicone, so I cut open the silicone in a haphazard way along something that vaguely looked like a seam.

I definitely ripped some silicone in the process and it was pretty messy but I got it open and found the circuit board.

I don’t know the model number of the bike lights but there’s a photo of them at the end of the post.

step 2: remove the screws

There were some screws attaching things together so I removed them so I could get the circuit board out.

Mostly I tried to remove as few screws as possible because I was worried about losing them or not being able to put them back after. I probably put the screws in a bag or something.

step 3: get the circuit board out

I took out the circuit board. Here’s what it looked like:

You can see where the battery is attached, I think it’s left of RI3 and above Q2.

Here’s what the battery looked like:

step 4: desolder the battery

I’d never desoldered anything before, so I found the iFixit guide to desoldering and read it. Also I asked my friends Lee and Lauria for advice.

Here were the steps I ended up following based on the guide & the advice I got:

Use a desoldering pump to remove most of the solder
Once most of it is gone, kind of pull them apart to try to separate them
Also try to avoid getting the battery too hot in the process by taking breaks to let it cool down. I’m not very good with a soldering iron so it took a while.
The battery has an attachment that is welded to the top. For a while I thought I needed to remove this and it seemed impossible, but it turned out the replacement battery comes with that part so actually I was supposed to leave it alone.

step 5: identify the battery

In the picture of the battery in Step 3, you can see it says something like “3” and “LI???77”. There’s a piece of metal that I think is welded or something to the top of the battery. It seemed impossible and also maybe not smart to try to remove so I wasn’t sure how to find out what an “LI????77” was or how to order another one.

I’ve been trying to avoid using LLMs (though I will not get into that because I am exhausted by LLM discourse and I’m sure you are too), but I really had no idea how to figure out what the battery was so I asked an LLM. It gave the response “LIR2477”, which (when I looked it up) looked exactly the same as my battery so I figured that was plausible.

I would be interested to learn non-LLM ways to figure this out though. There must be a way. Lauria showed me how to use DigiKey’s search which was very cool though DigiKey didn’t have that part.

step 6: buy the battery

I went to AliExpress and ordered:

2 batteries (I had 2 bike lights and I wanted to fix them both)
some silicone glue to glue things back together

I think the batteries were $3 each and the glue was $8.

step 7: solder the new batteries in and glue it back together

The parts took maybe 2 weeks to arive, and once they arrived, I went back to the makerspace and:

soldered in the new batteries
put the screws back in. The screws were very small and hard to hold, so at this point I dropped some screws on the ground and couldn’t find them because they were too small. So I just used fewer screws and hoped for the best.
used the glue to try to put everything back together.
Make a somewhat halfhearted attempt to clamp the parts I was gluing together

Then after waiting some amount of time for the glue to dry I took it home and waited 24 hours for the glue to cure.

Also I took the old batteries to somewhere nearby that accepts old batteries.

it works!

The lights work! I have used them to bike at night! I still haven’t needed to recharge them (and tragically I had to order a new Mini USB cable because I got rid of all my Mini USB cables, so I’m still waiting for that), so I still don’t know for sure how long the lifetime of the new battery will be.

Here’s what the light looks like after re-gluing. You can see that I didn’t glue very carefully. It didn’t really go back together that well but I’m hoping it’ll be good enough.

I thought it was really cool that I was able to do this with extremely minimal electronics skills! It cost about $20 CAD to buy the parts, and (whether or not the repair holds up, I’ll try to update this post in the future!), it was fun to try to repair something and learn something new.

Some more things about Django I've been enjoying

Tue, 21 Jul 2026 00:00:00 +0000

Hello! I’m on a funny journey right now where I’m trying to learn how to make websites in a sort of 2010 style, where I have an SQL database and render some HTML on the backend.

It’s kind of an interesting journey because it doesn’t necessarily feel “easy” to me to make websites in this way: I never learned how to do it in the 2000s or 2010s, and there’s a lot I need to learn.

So here are some Django features that make building this kind of site feel more achievable than when I was trying and failing to use Go’s standard library or Flask. And I’ll talk about a couple of issues with Django I’ve run into.

why learn to make websites like it’s 2010?

Previously the toolkit I felt confident with for making websites was:

static site generators (like for this blog)
static sites that do some fun stuff with Javascript (like this sql playground)
simple Vue.js single page apps with either a Lambda as a backend or a Go backend (like mess with dns)

I really liked this frontend-heavy approach for these super simple applications but when I started thinking about making something with a lot of different pages (instead of literally just one page), I didn’t feel so excited about the options I saw that involved a lot of frontend code. So I figured I’d try the backend.

Writing a backend-focused site that uses as little JS as possible feels the same to me in a way as writing a single-page JS website that does as little on the backend as possible, even though they might seem like opposites. In both cases I’m just trying to keep as much of the logic as possible in one place.

Now for some thoughts about Django!

I’m enjoying query builders

I learned that I can define a “query set” class in Django with a bunch of methods with different WHERE statements I might want to use while constructing a query:

Here’s how I use it in my view code once I’ve defined what all the methods mean:

Events.objects.approved()
  .for_tab(tab)
  .with_festivals(tab_params.festival_slugs)
  .is_free(tab_params.free)
  .is_outdoors(tab_params.outdoors)

and here’s how I define the methods:

class EventQuerySet(SearchableQuerySetMixin, models.QuerySet):
    def approved(self):
        return self.filter(approved_at__isnull=False)

    def future(self):
        today = timezone.localdate()
        return self.filter(end__gt=self._midnight(today))

    def with_tags(self, tags):
        if tags:
            return self.filter(tags__name__in=tags).distinct()
        return self

The syntax for defining the filters isn’t my favourite, but I spend most of my time just using the methods, and it feels super readable and nice to use, and it makes me want to look into other query builder libraries in the future. In the past I thought “I know SQL, who needs a query builder?”, but this kind of structure does make it really nice to read.

I found an example of someone who wrote their own small query builder in Python that I want to read later to think about whether I would enjoy using a more minimal version of this.

the template filters are awesome

There are a bunch of little quality of life filters available in Django templates that are super useful for generating HTML. The ones I’ve used so far are:

translating plain text URLs into links, or line breaks into <br> ({{ event.description|urlize|linebreaksbr }})
formatting dates ({{ row.date|date:"M j" }})
json_script, which takes a Python dictionary and automatically converts it to JSON and inserts it into the HTML as a <script> tag in a safe way

These are all small things individually but I feel like it makes a big difference somehow to just have them available.

`querystring` is cool

I think my favourite template filter is querystring: in this site sometimes we use filters like ?date=2026-06-01 to decide what’s displayed. querystring that will make a link to the same query string with one change, like this to link to the previous date:

<a href="{% querystring date=nav.prev_date%}">

Or to remove the outdoors parameter:

<a href="{% querystring outdoors=None %}">

automatic database migrations are still great

I still really love Django’s automatic database system. It’s amazing to be able to just edit a model to add a new field or whatever, and then Django automatically generates the migration.

So far we have done 19 database migrations and I think there will probably be more! It makes a huge difference for me to be able to just easily change the database as my understanding of the problem changes.

I do not want to organize my code with inheritance

Django’s documentation sometimes offers the option of using class-based views and inheritance to organize the code in your views. For example I have four views that share a lot of code, and I could use inheritance to manage that by defining some kind of parent class and then having my other views inherit from it.

I tried it out and I did not enjoy the experience of using inheritance to share code between views. I switched to using functions instead, sort of how this post advocates, and that was a lot more straightforward. I’ve never had a good experience using inheritance in Python and I don’t think I’ll try to use it again.

But I don’t mind using inheritance to use the interfaces Django itself provides: for example if I want to define a query set I need to write something like class EventQuerySet(SearchableQuerySetMixin, models.QuerySet). I don’t think too hard about it and it seems to work.

(as a meta comment: I’ve been working on talking about my programming opinions by just saying “THING does not feel good to me, I prefer OTHER THING instead”. That post I linked to says that function-based views are the “right way”. I’m not very invested in whether it’s “right”, but it’s validating to know that other people feel similarly to me about inheritance)

I don’t know how to think about Django performance

At some point the LLM scrapers discovered our site, and started sending us maybe 10 requests per second. I blocked them which is working for now, but it made me think about what the site’s capacity is. I’m used to writing Go backends where the performance situation is pretty straightforward (usually everything is just fast enough), and a Django site is very different.

Some light load testing (with (ab -n 1000 -c 1) shows that right now we can serve about 2-3 requests per second (on a ~$10/month VM).

It’s tempting for me to go down a rabbit hole where I do a bunch of profiling to figure out what’s slow and try to make it faster (there’s py-spy for that, and py-spy is great and super easy to use, and profiling is fun!) But I really don’t understand what I should expect in terms of performance from a Django site and how I should be thinking about at a higher level.

Some things I haven’t figured out yet:

If I have a site that’s going to be getting occasional bursts of traffic, do I want to be able to scale up?
Do I want to design the site so that more things can be cached? (and do I really have to? caches are so annoying to get right!)
The django performance docs say that Jinja is faster for templating, do I want to think about switching templating systems?
Those docs also say “{% block %} is faster than using {% include %}”, I wonder if it’s a big difference and if so why

template caching might be important

I think one thing I’m learning about Django is that because it’s a Framework (tm), it’s easy to accidentally misconfigure it. For example, when I was thinking about why my site was slow just now, I read the django performance docs and I noticed a comment saying:

Enabling the cached template loader often improves performance drastically, as it avoids compiling each template every time it needs to be rendered.

When I’d done CPU profiling I’d noticed that it was spending a lot of time rendering templates! Maybe this could help me!

Clicking through the link, I saw that the cached template loader was supposed to be on by default, but I’d turned it off by accident while trying to do something else. I think this “I turned off the cached template loader by default” things is an example of how I still find the django settings file to be pretty confusing and difficult. I guess I should just be careful when I go in there.

After turning on template caching, it seems like the site can now pretty easily handle 12 requests per second or so without using all of the CPU. I have not carefully benchmarked the before and after but it seems like it’s made a pretty big difference.

One thing that’s been surprising to me about Django performance is that I’ve always heard the advice “if you have a performance problem, check your database queries! Maybe add an index!”. But I’ve been running into a variety of performance issues (like this template caching thing) that are not because of slow queries, so instead it’s been more useful for me so far to start by running a CPU profile. And since I’m using SQLite, any slow database query problem will show up on the CPU profile anyway.

Anyway I don’t want to get too far into site performance. Like I said it’s easy for me to get interested in profiling, but actually I know a lot about profiling and it’s not the most important thing for me to learn about.

that’s all for now!

I might say more about what I’m enjoying (or having a hard time with!) about Django later. Trying to write some shorter blog posts recently.

Learning a few things about running SQLite

Fri, 17 Jul 2026 00:00:00 +0000

Hello! I’ve been working on a Django site recently, and I decided to use SQLite as the database. When I was getting started with using SQLite as database for a website I read a bunch of blog posts about how it is totally fine to use SQLite in production for a small site and I think it is totally fine, but what I did not fully appreciate is that SQLite is still a database, databases are complicated, and I do not know a lot about operating databases.

So here are a couple of small things I’ve been learning about running SQLite. This is the 4th website I’ve used SQLite for, and I think this one is harder because with the power of the Django ORM I’ve been making the database do more work than I was previously without Django.

I started by turning on WAL mode like all the blog posts said to do and hoping for the best.

`ANALYZE` is apparently important

Today I was running a query (using SQLite’s FTS5 for full-text search) on a table with 4000 rows and it took 5 seconds. That seemed wrong to me: computers are fast!

It turned out that what I needed to do was to run ANALYZE! Immediately the problem query went from taking 5 seconds to like 0.05 seconds (or some other number small enough that I didn’t care to investigate further). I still don’t know exactly what went wrong in the query plan, but my best guess is that it was some sort of accidentally quadratic thing.

ANALYZE generates “statistics” (I guess about the number of rows in each table? and presumably other things?) so that the query planner can make better choices.

Maybe one day I’ll learn to read a query plan.

cleaning up the database is tricky

Occasionally I’ve run into situations where I accidentally put a bunch of rows in my database that I don’t want to be there (for example completed tasks from django-tasks-db), and I want to clean them up.

What’s happened to me a few times in this case is:

I run some kind of command to clean up the rows
The command takes more than 5 seconds, since there are a lot of rows (though I still have some questions about why these DELETE statements are so slow honestly, maybe there’s a bunch of Python code running inside a transaction, I’m not sure)
One of the other workers tries to write the database while this is happening, and times out after 5 seconds (I have a timeout of 5 seconds set)
The worker crashes because it couldn’t write to the database and the VM shuts down

My approach so far has been to just do these cleanup operations in small batches so that I don’t need to do database queries that take more than 5 seconds to run. This whole experience has given me more of an appreciation for why someone might want to use a “real” database like Postgres which can have more than one writer at the same time though.

Maybe in the future I’ll just take the site down for scheduled maintenance instead when I need to do this kind of thing, but I haven’t figured out a workflow for that yet.

no notes on performance of ORM queries yet

So far I’ve been using Django’s ORM to make any query I want without paying any attention at all to query performance and it’s mostly been going okay other than the ANALYZE thing. The database is pretty small (maybe 10000 rows?) and I expect it to stay pretty small forever, so I’m hoping that that plan will keep working.

backing up sqlite

I’ve done SQLite backups a couple of ways. I don’t think I’ve actually tested restoring from my backups but I do usually try to monitor them with a dead man’s switch.

way 1: restic

sqlite3 /data/calendar.db "VACUUM INTO '/tmp/calendar.sqlite'"
gzip /tmp/calendar.sqlite

# Upload backup to S3
# Sometimes the backup gets OOM killed and so it stays locked, do an unlock
restic -r s3://s3.amazonaws.com/some_bucket/ unlock
# Do the backup & prune old backups
restic -r s3://s3.amazonaws.com/some_bucket/ backup /tmp/calendar.sqlite.gz
restic -r s3://s3.amazonaws.com/some_bucket/ snapshots
restic -r s3://s3.amazonaws.com/some_bucket/ forget -l 1 -H 6 -d 2 -w 2 -m 2 -y 2
restic -r s3://s3.amazonaws.com/some_bucket/ prune

way 2: litestream

I started trying out Litestream recently because I felt like doing incremental backups might be more efficient: my restic backups were sometimes getting OOM killed, and I was a bit tired of it. Basically I just write a config file and run:

litestream replicate -config litestream.yml

I set retention: 400h in my config file in an attempt to retain some amount of history of the database but I have no idea if it works.

I’ve been backing up to AWS, which is always a pain because it’s annoying to navigate the AWS console to generate credentials. Maybe one day I’ll move away to some other S3-compatible alternative.

you can use multiple databases

My current project only has one database, but one trick I used with Mess with DNS was to split the tables into three separate database files because I didn’t actually need my tables to be in the same db. I think it was helpful.

Mess with DNS has been running on SQLite for 4 years now (since 2022) and it’s been great, I think the move from Postgres was a great choice for that project.

that’s all!

It’s always kind of fun to see how long it takes me to learn sort of basic things about the technologies I’m using. I think I used SQLite for a web project for the first time in 2022 and I only learned that ANALYZE existed today! I imagine in a year or two I’ll be learning about some other very basic feature.

some references

Some blog posts I’ve looked at, other than the official docs:

Moving away from Tailwind, and learning to structure my CSS

Fri, 15 May 2026 00:00:00 +0000

Hello! 8 years ago, I wrote excitedly about discovering Tailwind.

At that time I really had no idea how to structure my CSS code and given the choice between a pile of complete chaos and Tailwind, I was really happy to choose Tailwind. It helped me make a lot of tiny sites!

I spent the last week or so migrating a couple of sites away from Tailwind and towards more semantic HTML + vanilla CSS, and it was SO fun and SO interesting, so here are some things I learned!

As usual I’m not a full-time frontend developer and so all of my CSS learning has happened in fits and starts over many years.

it turns out Tailwind taught me a lot

When I started thinking about structuring CSS, I was intimidated at first: I’m not very good at structuring my CSS! But then I started reading blog posts talking about how to structure CSS (like A whole cascade of layers or How I write CSS in 2024) and I realized a couple of things:

Every CSS code base has a bunch of different things going on (layouts! fonts! colours! common components!)
It’s extremely useful to have systems or guidelines to manage each of those things, otherwise things descend into chaos
Tailwind has systems for some of these, and I already know those systems! Maybe I can imitate the systems I like!

For example, Tailwind has:

a reset stylesheet
a colour palette
a font scale

the systems I’m going to talk about

I’m going to talk about a few aspects of my CSS codebase and my thoughts so far what kind of rules I want to impose on the codebase for each one. Some of them are copied from Tailwind and some aren’t.

reset
components
colours
font sizes
utility classes
the base
spacing
responsive design
the build system

1. reset

I just copied Tailwind’s “preflight styles” by going into tailwind.css and copying the first 200 lines or so.

I noticed that I’ve developed a relationship with Tailwind’s CSS reset over time, for example Tailwind sets box-sizing: border-box on every element (which means that an element’s width includes its padding and border):

* { box-sizing: border-box; }

I think it would be a real adjustment for me to switch to writing CSS without these, and I’m sure there are lots of other things in the Tailwind reset (like html {line-height: 1.5;}) that I’m subconsciously used to and don’t even realize are there.

2. components

This next part is the bulk of the CSS!

The idea here is to organize CSS by “components”, in a way that’s spiritually related to Vue or React components. (though there might not actually be any Javascript at all in the site)

Basically the idea is that:

Each “component” has a unique class
The CSS for one component never overrides the CSS for any other component
Each component has its own CSS file

So editing the CSS for one component won’t mysteriously break something in another component. And probably like 80% of the CSS that I would actually want to change is in various component files, so if I’m editing a 100-line component, I just have to think about those 100 lines. It’s way easier for me to think about.

For example, this HTML might be the .zine “component”.

<figure class="zine horizontal">
    <img src="whatever.jpg">
</figure>

And the CSS looks something like this, using nested selectors:

.zine {
  ...
  &.horizontal {
    ...
  }
  &.vertical {
    ...
  }
  &:hover {
    ...
  }
}

I haven’t done anything programmatic (like web components or @scope) that ensures that components won’t interfere with each other, but just having a convention and trying my best already feels like a big improvement.

Next: conventions to maintain some consistency across the site and keep these components in line with each other!

3. colours

colours.css has a bunch of variables like this which I can use as necessary. Colour is really hard and I didn’t want to revisit my use of colour in this refactor, so I left this alone.

The only guideline I’m trying to enforce here is that all colours used in the site are listed in this file.

:root {
  --pink: #fea0c2;
  --pink-light: #F9B9B9;
  --red: #f91a55;
  --orange: rgb(222, 117, 31);
  ...
}

4. font sizes

One thing I appreciated about Tailwind was that if I wanted to set a font size, I could just think “hm, I want the text to be big”, write text-lg, and be done with it! And maybe if it’s not big enough I’d use xl or 2xl instead. No trying to remember whether I’m using em or px or rem.

So I defined a bunch of variables, taken from Tailwind, like this:

  --size-xs: 0.75rem;
  --line-height-xs: 1rem;

  --size-sm: 0.875rem;
  --line-height-sm: 1.25rem;

Then if I want to set a font size, I can do it like this. It’s a little more verbose than Tailwind but I’m happy with it for now.

h3 {
  font-size: var(--size-lg);
  line-height: var(--line-height-lg);
}

5. utilities

There are some things like buttons that appear in many different components. I’m calling these “utilities”.

I copied some utility classes from Tailwind (like .sr-only for things that should only appear for screenreader users).

This section is pretty small and I try to be careful about making changes here.

6. the base

“base” styles are styles that apply across the whole site that I chose myself. I have to keep this section really small because I’m not confident enough to enforce a lot of styles across the whole site. These are the only two I feel okay about right now, and I might change the <section> one:

/* put a 950px column in the middle of each <section> */
section {
  --inner-width: 950px;
  padding: 3rem max(1rem, (100% - var(--inner-width))/2);
}

a {
  color: var(--orange);
}

I think for the base styles it’s going to be easiest for me to work kind of bottom up – first start with almost nothing in the base styles, and then move some styles from the components into base styles as I identify common things I want.

7. spacing

I haven’t completely worked out an approach to managing padding and margins yet. I’m definitely trying to be more principled than how I was doing it in Tailwind though, where I would just haphazardly put padding and margins everywhere until it looked the way I wanted.

Right now I’m working towards making the outer layout components in charge of spacing as much as possible. For example if I have a <section> with a bunch of children that I want to have space between them, I might use this to space the children evenly:

section > *+* {
  margin-top: 1rem;
}

Some inspiration blog posts:

8. responsive design: use more grid!

The way I was doing responsive design in Tailwind was to use a lot of media queries. Tailwind has this md:text-xl syntax that means “apply the text-xl style at sizes md or larger”.

I’m trying something pretty different now, which is to make more flexible CSS grid layouts that don’t need as many breakpoints. This is hard but it’s really interesting to learn about what’s possible with grid, and it’s a good example of something that I don’t think is possible with Tailwind.

For example, I’ve been learning about how to use auto-fit to automatically use 2 columns on a big screen and 1 column on a small screen like this:

  display: grid;
  grid-template-columns: repeat(auto-fit, minmax(min(100%, 400px), max-content));
  justify-content: center;

I also used grid-template-areas a lot which is an amazing feature that I don’t think you can use with Tailwind.

Some inspiration:

A responsive grid layout with no media queries from CSS Tricks

9. the build system: esbuild

In development, I don’t need a build system: CSS now has both built in import statements, like this:

@import "reset.css";
@import "typography.css";
@import "colors.css";

and built in nested selectors, like this:

.page {
  h2 { ...}
}

If I want, I can use esbuild to bundle the CSS file for production. That looks something like this.

esbuild style.css --bundle --loader:.svg=dataurl  --loader:.woff2=file --outfile=/tmp/out.css

Even though I usually avoid using CSS and JS build systems, I don’t mind using esbuild (which I wrote about in 2021 here) because it’s based on web standards and because it’s a static Go binary.

why migrate away from Tailwind?

A few people asked why I was migrating away from Tailwind. A few factors that contributed are:

Tailwind has become much more reliant on a build system since 2018, I think it’s impossible (?) to use newer versions of Tailwind without using a build system. So I’ve been using Tailwind v2 for years. (there’s also litewind apparently)
It’s always been true that you’re supposed to use Tailwind with a build system, but I’ve never really done that, so I have 2.8MB tailwind.min.css files (270K gzipped) in a lot of my projects and it feels a little silly.
I’m a lot better at CSS than I was when I started using Tailwind
Ultimately Tailwind is limiting: if you want to do Weird Stuff in your CSS, it’s not always possible with Tailwind. Those limits can be extremely useful (a lot of this post is about me reimplementing some of Tailwind’s limits!) but at this point I’d like to be able to pick and choose.
I ended up with sites that mixed both vanilla CSS and Tailwind in the same project and that was not fun to maintain
I got curious about what writing more semantic HTML would feel like.

CSS features I’m curious about

While doing this I learned about a lot of CSS features that I didn’t use but am curious about learning about one day:

@layer (from A Whole Cascade of Layers)
@scope) (especially this example of how to use @scope in a “component” CSS design from the specification!)
container queries
subgrid

one last reason I moved away from Tailwind

I’ve been talking a lot in this post about what I learned from using Tailwind, and that’s all true.

But I read this post 3 years ago called Tailwind and the Femininity of CSS that really stuck with me. I honestly probably started out with an attitude towards CSS a little like that post describes:

They’ve heard it’s simple, so they assume it’s easy. But then when they try to use it, it doesn’t work. It must be the fault of the language, because they know that they are smart, and this is supposed to be easy.

But in the last 10 years I’ve learned to really love and respect CSS as a technology.

So I decided years ago that I wanted to react to “CSS is hard” by getting better at CSS and taking it seriously as a technology, instead of devaluing it. Doing that changed everything for me: I learned that so many of my frustrations (“centering is impossible”) had been addressed in CSS a long time ago, and that also what “centering” means is not always straightforward and it makes sense that there are many ways to do it. CSS is hard because it’s solving a hard problem!

I’ve been so impressed by the new CSS features that have been built in the last 10-15 years (some of which I’ve talked about in this post!) and how they make it easier to use CSS, and spending the time to improve my CSS skills has been a really cool experience.

And that post made me feel like Tailwind contributes to the devaluing of CSS expertise, and like that’s not something I want to be a part of, even if Tailwind has been a useful tool for me personally. Especially in this time of LLMs where it feels more important than ever to value humans’ expertise.

Another blog post criticizing Tailwind that influenced me:

Classic rock, Mario Kart, and why we can’t agree on Tailwind

that’s all for now!

Thanks to Melody Starling who originally designed and wrote the CSS for wizardzines.com, everything cool and fun about the site is thanks to Melody.

Also I read so many incredible blog posts about CSS while working on this (from CSS Tricks, Smashing Magazine, and more), I’ve tried to link some of them throughout this post and I really appreciate how much folks in the CSS community share their practices.

Links to CSS colour palettes

Mon, 4 May 2026 00:00:00 +0000

A while back I decided to stop using Tailwind for new projects and to just write vanilla CSS instead.

But one thing I missed about Tailwind was the colour palette (here as CSS). If I wanted a light blue I could just use blue-100 and if I didn’t like it maybe try blue-200 or blue-50. I’m not very good with colours so it makes a big difference to me to have a reasonable colour palette that somebody who is better at colour than me has thought about.

But I’m also a little tired of those Tailwind colours, so I asked on Mastodon today what other colour palettes were out there. And then a friend said they wanted links to those colour palettes, so here’s a blog post so my friend can see them, and all the rest of you too :)

my favourites

The ones I liked the most were:

uchū (css file, FAQ)
flexoki (css file)
reasonable colours, which seems to have a focus on accessibility (css file)

more colour palettes

colourscheme generators

Folks also linked to a bunch of colour palette generators

I’ve always found these types of generators too hard to use but maybe one day I will get better enough at colour that I’m able to use a colour palette generator successfully so I’ll leave those links there anyway.

and more colour tools:

colorhexa has some info about colorblindness

`oklch`

Generative colors with CSS gives an example of how to use the oklch CSS function to dynamically generate colors.

Testing Vue components in the browser

Sat, 2 May 2026 00:00:00 +0000

Hello! One of my long term projects on here is figuring out how to write frontend Javascript without using Node or any other server JS runtime.

One issue I run into a lot in my frontend JS projects is that I don’t know how to write tests for them. I’ve tried to use Playwright in the past, but it felt slow and unwieldy to be starting these new browser processes all the time, and it involved some Node code to orchestrate the tests.

The result is that I just don’t test my frontend code which doesn’t feel great. Usually I don’t update my projects much either so it doesn’t come up that much, but it would be nice to be able to make changes with more confidence! So a way to do frontend testing that I like has been on my wishlist for a long time.

idea: just run the tests in the browser tab

Alex Chan wrote a great post a while back called Testing JavaScript without a (third-party) framework in response to one of my previous posts in this series that explained how to write a tiny unit-testing framework that runs in a page in browser.

I loved this post at the time, but it only talked about unit testing and I wanted to write end-to-end integration tests for my Vue components, and I didn’t know how to do that.

So when I was talking to Marco the other day and he said something like “you know, you can just run tests for your Vue components in the browser”, I thought “hey, I should try that again!!!”

I just did all of this yesterday so certainly there’s a lot to improve but I wanted to write down a few things I noticed about the process before I forget.

This was a bit tricky for me because the Vue site usually assumes that you’re using Node as part of your build process in some way (there’s a lot of “step 1: npm install THING), and I didn’t want to use Node/Deno/etc. But it turned out to not be too complicated.

The project I’m going to talk about testing is this zine feedback site I wrote in 2023.

the test framework: QUnit

I used QUnit. It worked great but I don’t have anything interesting to say about how it works so I’ll leave it at that. I think that Alex’s “write your own test framework” approach would have worked too. I followed these directions.

I did appreciate that QUnit has a “rerun test” button that will only rerun 1 test. Because there are so many network requests in my tests, having a way to run just 1 test makes it a lot less confusing to debug the test.

step 1: set up the component for testing

The first thing I needed to do was get my Vue components set up in the test environment.

I changed my main app to put all my components in window._components, kind of like this:

const components = {
  'Feedback': FeedbackComponent,
  ...
}
window._components = components;

Then I was able to write a mountComponent function which does basically exactly the same thing my normal main app does (render a tiny template with the component I want to use). The only differences are:

I can optionally pass some some extra data to use as its props.
It mounts the component to a temporary invisible div which will get removed from the DOM after the test is done. The div is positioned off the page (position: absolute; top: -10000, ...) so you can’t see it.

Here’s what using the mountComponent function looks like:

const {div} = mountComponent(
  '<Page :feedbacks="feedbacks" id=2 />',
  {feedbacks: [testFeedback]},
);

and here’s the code for it:

function mountComponent(template, data) {
  const app = Vue.createApp({
    template: template,
    data: () => data,
  })
  for (const [c, v] of Object.entries(window._components)) {
    app.component(c, v);
  }
  const div = document.getElementById('qunit-fixture')
             .appendChild(document.createElement('div'));
  return div;
}

The result is a div where I can programmatically click, fill in form data, check that the right content appears, etc.

step 2: add some fixture data

Because I was writing end-to-end integration tests to make sure my client JS worked properly with my server, I needed to have some test data in my database. So I wrote ~25 lines of SQL to set up some test data in my database, and added an endpoint to my dev server to run the SQL to reset the test data to a known state.

async function reset() {
    return fetch('/api/reset_test_data', {method: "POST"})
}

Then I just run await reset() at the beginning of any test that needs the test data.

My reset() function actually doesn’t always totally reset everything which is kind of bad, but it was workable to start with and can always be improved.

step 3: a basic test

Here’s what a basic test looks like! Basically we’re rendering the div and make sure it contains some approximately correct data.

QUnit.test('renders feedback content', async function (assert) {
  const {div} = mountComponent(
    '<Page :feedbacks="feedbacks" id=2 image=2 page_hash=2 />',
    {feedbacks: [testFeedback]},
  );
  assert.ok(div.textContent.includes('loved this section'));
})

Those are all the basic pieces! Now here are a few issues I ran into along the way

waiting for parts of the page to render

I have a lot of network requests in my tests, and it takes time for them to finish and for the Vue code to do what it has to do with the results and update the DOM.

I think we all learned a long time ago that putting random sleep() calls in your tests and hoping that the timings are right is slow and flaky and extremely frustrating, so I needed a different way.

As far as I can tell the normal way to deal with this is to figure out a way to tell from the DOM whether it’s okay to proceed or not. Like “if this button is visible, we can “.

So I wrote a little waitFor() function that polls every 20ms to see if a condition has finished yet. It times out after 2 seconds.

Here’s what using it looks like:

QUnit.test("click item", async function (assert) {
  const {div} = mountComponent(
    '<Feedback zine_id="test123" image_width="800px" />',
    {});
  const item = await waitFor(() => div.querySelector('.feedback-item'));
  item.click();
  // rest of test goes here... 
})

It looks like there are a lot of implementations of this concept out there and they’re all better thought-through than mine. (from a quick Google: qunit-wait-for, playwright expect.poll)

figuring out the right thing to wait for is not straightforward

In some cases I thought I’d identified the right thing to wait for in the DOM (“just wait for this textarea to appear!’) but it turned out that because of some internal details of how my program works, actually I needed to wait for something else later on which was hard to pin down.

I ended up changing one of my components to add some random value to the DOM when it was finished an important action (like data-this-thing-is-ready=true) which didn’t feel great.

My best guess is that the right way to fix this kind of test issue is a refactor that also makes the app more reliable for the users: if there’s an element in the DOM that isn’t actually ready for the user to interact with, maybe I shouldn’t be displaying it yet!

adding some CSS classes to identify things (but is that right?)

I ended up adding a few classes to HTML elements that I needed to find in the tests, either because I needed to click on them or wait for them to appear in the DOM.

I might want to change this approach later - frontend testing frameworks seem to suggest avoiding using CSS classes and instead using something like getByRole or as a last resort something like a data-testid. Feels like there’s a way to make the app more accessible and easier to test at the same time.

filling out forms is tricky

To fill out a form, I can’t just set the value, I also need to dispatch an event to tell Vue that the element has changed. For example, checkbox and textarea need different kinds of events.

textarea.value = 'banana banana banana';
textarea.dispatchEvent(new Event('input'));

checkbox.checked = true;
checkbox.dispatchEvent(new Event('change'));

This is kind of annoying and it made me realize why I might want to use some kind of UI testing library, for example:

Testing Library’s example of filling out a form looks extremely different from what I’m doing
Vue Test Utils: their section on form handling looks like it simplifies this a lot.

test coverage

I want to have an idea of what my test coverage was, and it turns out that Chrome actually has a built-in code coverage feature for JS and CSS!

My JS is bundled into a file called bundle.js with esbuild, so I could just look at bundle.js and see which lines weren’t covered.

The process was a little finicky: I had to turn off sourcemaps in the Chrome devtools to get this to work, and there’s a specific not super obvious series of actions I have to do in order to see the coverage data.

this was so fun!

As usual with these posts I’ve never really worked as a frontend or backend developer (other than for myself!) and I feel like I’m constantly learning how to do super basic tasks.

I really had a blast doing this. My frontend projects always feel so fragile because they’re untested, and maybe one day I’ll have a test suite I’m confident in!

Some things I’m still thinking about:

While writing this post I found this frontend testing library called Testing Library that has a lot of guidelines for how to write tests that are very different from my initial ideas. I experimented with rewriting everything to use Testing Library and it felt pretty good, so we’ll see how that goes. They distribute a .umd.js file that works without Node.
I’m not sure how I feel about not having a way to run these tests on the command line at all. Maybe there’s a simple way to work primarily in the browser but have an way to run them in CI too if I want?

Examples for the tcpdump and dig man pages

Tue, 10 Mar 2026 00:00:00 +0000

Hello! My big takeaway from last month’s musings about man pages was that examples in man pages are really great, so I worked on adding (or improving) examples to two of my favourite tools’ man pages.

Here they are:

the dig man page (now with examples)
the tcpdump man page examples (this one is an update to the previous examples)

the goal: include the most basic examples

The goal here was really just to give the absolute most basic examples of how to use the tool, for people who use tcpdump or dig infrequently (or have never used it before!) and don’t remember how it works.

So far saying “hey, I want to write an examples section for beginners and infrequent users of this tools” has been working really well. It’s easy to explain, I think it makes sense from everything I’ve heard from users about what they want from a man page, and maintainers seem to find it compelling.

Thanks to Denis Ovsienko, Guy Harris, Ondřej Surý, and everyone else who reviewed the docs changes, it was a good experience and left me motivated to do a little more work on man pages.

why improve the man pages?

I’m interested in working on tools’ official documentation right now because:

Man pages can actually have close to 100% accurate information! Going through a review process to make sure that the information is actually true has a lot of value.
Even with basic questions “what are the most commonly used tcpdump flags”, often maintainers are aware of useful features that I’m not! For example I learned by working on these tcpdump examples that if you’re saving packets to a file with tcpdump -w out.pcap, it’s useful to pass -v to print a live summary of how many packets have been captured so far. That’s really useful, I didn’t know it, and I don’t think I ever would have noticed it on my own.

It’s kind of a weird place for me to be because honestly I always kind of assume documentation is going to be hard to read, and I usually just skip it and read a blog post or Stack Overflow comment or ask a friend instead. But right now I’m feeling optimistic, like maybe the documentation doesn’t have to be bad? Maybe it could be just as good as reading a really great blog post, but with the benefit of also being actually correct? I’ve been using the Django documentation recently, and it’s really good! We’ll see.

on avoiding writing the man page language

The tcpdump project tool’s man page is written in the roff language, which is kind of hard to use and that I really did not feel like learning it.

I handled this by writing a very basic markdown-to-roff script to convert Markdown to roff, using similar conventions to what the man page was already using. I could maybe have just used pandoc, but the output pandoc produced seemed pretty different, so I thought it might be better to write my own script instead. Who knows.

I did think it was cool to be able to just use an existing Markdown library’s ability to parse the Markdown AST and then implement my own code-emitting methods to format things in a way that seemed to make sense in this context.

man pages are complicated

I went on a whole rabbit hole learning about the history of roff, how it’s evolved since the 70s, and who’s working on it today, inspired by learning about the mandoc project that BSD systems (and some Linux systems, and I think Mac OS) use for formatting man pages. I won’t say more about that today though, maybe another time.

In general it seems like there’s a technical and cultural divide in how documentation works on BSD and on Linux that I still haven’t really understood, but I have been feeling curious about what’s going on in the BSD world.

The comments section is here.

Notes on clarifying man pages

Wed, 18 Feb 2026 00:00:00 +0000

Hello! After spending some time working on the Git man pages last year, I’ve been thinking a little more about what makes a good man page.

I’ve spent a lot of time writing cheat sheets for tools (tcpdump, git, dig, etc) which have a man page as their primary documentation. This is because I often find the man pages hard to navigate to get the information I want.

Lately I’ve wondering – could the man page itself have an amazing cheat sheet in it? What might make a man page easier to use? I’m still very early in thinking about this but I wanted to write down some quick notes.

I asked some people on Mastodon for their favourite man pages, and here are some examples of interesting things I saw on those man pages.

an OPTIONS SUMMARY

If you’ve read a lot of man pages you’ve probably seen something like this in the SYNOPSIS: once you’re listing almost the entire alphabet, it’s hard

ls [-@ABCFGHILOPRSTUWabcdefghiklmnopqrstuvwxy1%,]

grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz]

The rsync man page has a solution I’ve never seen before: it keeps its SYNOPSIS very terse, like this:

 Local:
     rsync [OPTION...] SRC... [DEST]

and then has an “OPTIONS SUMMARY” section with a 1-line summary of each option, like this:

--verbose, -v            increase verbosity
--info=FLAGS             fine-grained informational verbosity
--debug=FLAGS            fine-grained debug verbosity
--stderr=e|a|c           change stderr output mode (default: errors)
--quiet, -q              suppress non-error messages
--no-motd                suppress daemon-mode MOTD

Then later there’s the usual OPTIONS section with a full description of each option.

an OPTIONS section organized by category

The strace man page organizes its options by category (like “General”, “Startup”, “Tracing”, and “Filtering”, “Output Format”) instead of alphabetically.

As an experiment I tried to take the grep man page and make an “OPTIONS SUMMARY” section grouped by category, you can see the results here. I’m not sure what I think of the results but it was a fun exercise. When I was writing that I was thinking about how I can never remember the name of the -l grep option. It always takes me what feels like forever to find it in the man page and I was trying to think of what structure would make it easier for me to find. Maybe categories?

a cheat sheet

A couple of people pointed me to the suite of Perl man pages (perlfunc, perlre, etc), and one thing I noticed was man perlcheat, which has cheat sheet sections like this:

 SYNTAX
 foreach (LIST) { }     for (a;b;c) { }
 while   (e) { }        until (e)   { }
 if      (e) { } elsif (e) { } else { }
 unless  (e) { } elsif (e) { } else { }
 given   (e) { when (e) {} default {} }

I think this is so cool and it makes me wonder if there are other ways to write condensed ASCII 80-character-wide cheat sheets for use in man pages.

examples are very popular

A common comment was something to the effect of “I like any man page that has examples”. Someone mentioned the OpenBSD man pages, and the openbsd tail man page has examples of the exact 2 ways I use tail at the end.

I think I’ve most often seen the EXAMPLES section at the end of the man page, but some man pages (like the rsync man page from earlier) start with the examples. When I was working on the git-add and git rebase man pages I put a short example at the beginning.

a table of contents, and links between sections

This isn’t a property of the man page itself, but one issue with man pages in the terminal is it’s hard to know what sections the man page has.

When working on the Git man pages, one thing Marie and I did was to add a table of contents to the sidebar of the HTML versions of the man pages hosted on the Git site.

I’d also like to add more hyperlinks to the HTML versions of the Git man pages at some point, so that you can click on “INCOMPATIBLE OPTIONS” to get to that section. It’s very easy to add links like this in the Git project since Git’s man pages are generated with AsciiDoc.

I think adding a table of contents and adding internal hyperlinks is kind of a nice middle ground where we can make some improvements to the man page format (in the HTML version of the man page at least) without maintaining a totally different form of documentation. Though for this to work you do need to set up a toolchain like Git’s AsciiDoc system.

It would be amazing if there were some kind of universal system to make it easy to look up a specific option in a man page (“what does -a do?”). The best trick I know is use the man pager to search for something like ^ *-a but I never remember to do it and instead just end up going through every instance of -a in the man page until I find what I’m looking for.

examples for every option

The curl man page has examples for every option, and there’s also a table of contents on the HTML version so you can more easily jump to the option you’re interested in.

For instance the example for --cert makes it easy to see that you likely also want to pass the --key option, like this:

  curl --cert certfile --key keyfile https://example.com

The way they implement this is that there’s [one file for each option](https://github.com/curl/curl/blob/dc08922a61efe546b318daf964514ffbf41583 25/docs/cmdline-opts/append.md) and there’s an “Example” field in that file.

formatting data in a table

Quite a few people said that man ascii was their favourite man page, which looks like this:

 Oct   Dec   Hex   Char                     
 ───────────────────────────────────────────
 000   0     00    NUL '\0' (null character)
 001   1     01    SOH (start of heading)   
 002   2     02    STX (start of text)      
 003   3     03    ETX (end of text)        
 004   4     04    EOT (end of transmission)
 005   5     05    ENQ (enquiry)            
 006   6     06    ACK (acknowledge)        
 007   7     07    BEL '\a' (bell)          
 010   8     08    BS  '\b' (backspace)     
 011   9     09    HT  '\t' (horizontal tab)
 012   10    0A    LF  '\n' (new line)

Obviously man ascii is an unusual man page but I think what’s cool about this man page (other than the fact that it’s always useful to have an ASCII reference) is it’s very easy to scan to find the information you need because of the table format. It makes me wonder if there are more opportunities to display information in a “table” in a man page to make it easier to scan.

the GNU approach

When I talk about man pages it often comes up that the GNU coreutils man pages (for example man tail) don’t have examples, unlike the OpenBSD man pages, which do have examples.

I’m not going to get into this too much because it seems like a fairly political topic and I definitely can’t do it justice here, but here are some things I believe to be true:

The GNU project prefers to maintain documentation in “info” manuals instead of man pages. This page says “the man pages are no longer being maintained”.
There are 3 ways to read “info” manuals: their HTML version, in Emacs, or with a standalone info tool. I’ve heard from some Emacs users that they like the Emacs info browser. I don’t think I’ve ever talked to anyone who uses the standalone info tool.
The info manual entry for tail is linked at the bottom of the man page, and it does have examples
The FSF used to sell print books of the GNU software manuals (and maybe they still do sometimes?)

After a certain level of complexity a man page gets really hard to navigate: while I’ve never used the coreutils info manual and probably won’t, I would almost certainly prefer to use the GNU Bash reference manual or the The GNU C Library Reference Manual via their HTML documentation rather than through a man page.

a few more man-page-adjacent things

Here are some tools I think are interesting:

The fish shell comes with a Python script to automatically generate tab completions from man pages
tldr.sh is a community maintained database of examples, for example you can run it as tldr grep. Lots of people have told me they find it useful.
the Dash Mac docs browser has a nice man page viewer in it. I still use the terminal man page viewer but I like that it includes a table of contents, it looks like this:

it’s interesting to think about a constrained format

Man pages are such a constrained format and it’s fun to think about what you can do with such limited formatting options.

Even though I’m very into writing I’ve always had a bad habit of never reading documentation and so it’s a little bit hard for me to think about what I actually find useful in man pages, I’m not sure whether I think most of the things in this post would improve my experience or not. (Except for examples, I LOVE examples)

So I’d be interested to hear about other man pages that you think are well designed and what you like about them, the comments section is here.

Some notes on starting to use Django

Tue, 27 Jan 2026 00:00:00 +0000

Hello! One of my favourite things is starting to learn an Old Boring Technology that I’ve never tried before but that has been around for 20+ years. It feels really good when every problem I’m ever going to have has been solved already 1000 times and I can just get stuff done easily.

I’ve thought it would be cool to learn a popular web framework like Rails or Django or Laravel for a long time, but I’d never really managed to make it happen. But I started learning Django to make a website a few months back, I’ve been liking it so far, and here are a few quick notes!

less magic than Rails

I spent some time trying to learn Rails in 2020, and while it was cool and I really wanted to like Rails (the Ruby community is great!), I found that if I left my Rails project alone for months, when I came back to it it was hard for me to remember how to get anything done because (for example) if it says resources :topics in your routes.rb, on its own that doesn’t tell you where the topics routes are configured, you need to remember or look up the convention.

Being able to abandon a project for months or years and then come back to it is really important to me (that’s how all my projects work!), and Django feels easier to me because things are more explicit.

In my small Django project it feels like I just have 5 main files (other than the settings files): urls.py, models.py, views.py, admin.py, and tests.py, and if I want to know where something else is (like an HTML template) is then it’s usually explicitly referenced from one of those files.

a built-in admin

For this project I wanted to have an admin interface to manually edit or view some of the data in the database. Django has a really nice built-in admin interface, and I can customize it with just a little bit of code.

For example, here’s part of one of my admin classes, which sets up which fields to display in the “list” view, which field to search on, and how to order them by default.

@admin.register(Zine)
class ZineAdmin(admin.ModelAdmin):
    list_display = ["name", "publication_date", "free", "slug", "image_preview"]
    search_fields = ["name", "slug"]
    readonly_fields = ["image_preview"]
    ordering = ["-publication_date"]

it’s fun to have an ORM

In the past my attitude has been “ORMs? Who needs them? I can just write my own SQL queries!”. I’ve been enjoying Django’s ORM so far though, and I think it’s cool how Django uses __ to represent a JOIN, like this:

Zine.objects
    .exclude(product__order__email_hash=email_hash)

This query involves 5 tables: zines, zine_products, products, order_products, and orders. To make this work I just had to tell Django that there’s a ManyToManyField relating “orders” and “products”, and another ManyToManyField relating “zines”, and “products”, so that it knows how to connect zines, orders, products.

I definitely could write that query, but writing product__order__email_hash is a lot less typing, it feels a lot easier to read, and honestly I think it would take me a little while to figure out how to construct the query (which needs to do a few other things than just those joins).

I have zero concern about the performance of my ORM-generated queries so I’m pretty excited about ORMs for now, though I’m sure I’ll find things to be frustrated with eventually.

automatic migrations!

The other great thing about the ORM is migrations!

If I add, delete, or change a field in models.py, Django will automatically generate a migration script like migrations/0006_delete_imageblob.py.

I assume that I could edit those scripts if I wanted, but so far I’ve just been running the generated scripts with no change and it’s been going great. It really feels like magic.

I’m realizing that being able to do migrations easily is important for me right now because I’m changing my data model fairly often as I figure out how I want it to work.

I like the docs

I had a bad habit of never reading the documentation but I’ve been really enjoying the parts of Django’s docs that I’ve read so far. This isn’t by accident: Jacob Kaplan-Moss has a talk from PyCon 2011 on Django’s documentation culture.

For example the intro to models lists the most important common fields you might want to set when using the ORM.

using sqlite

After having a bad experience trying to operate Postgres and not being able to understand what was going on, I decided to run all of my small websites with SQLite instead. It’s been going way better, and I love being able to backup by just doing a VACUUM INTO and then copying the resulting single file.

I’ve been following these instructions for using SQLite with Django in production.

I think it should be fine because I’m expecting the site to have a few hundred writes per day at most, much less than Mess with DNS which has a lot more of writes and has been working well (though the writes are split across 3 different SQLite databases).

built in email (and more)

Django seems to be very “batteries-included”, which I love – if I want CSRF protection, or a Content-Security-Policy, or I want to send email, it’s all in there!

For example, I wanted to save the emails Django sends to a file in dev mode (so that it didn’t send real email to real people), which was just a little bit of configuration.

I just put this settings/dev.py:

EMAIL_BACKEND = "django.core.mail.backends.filebased.EmailBackend"
EMAIL_FILE_PATH = BASE_DIR / "emails"

and then set up the production email like this in settings/production.py

EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp.whatever.com"
EMAIL_PORT = 587
EMAIL_USE_TLS = True
EMAIL_HOST_USER = "xxxx"
EMAIL_HOST_PASSWORD = os.getenv('EMAIL_API_KEY')

That made me feel like if I want some other basic website feature, there’s likely to be an easy way to do it built into Django already.

the settings file still feels like a lot

I’m still a bit intimidated by the settings.py file: Django’s settings system works by setting a bunch of global variables in a file, and I feel a bit stressed about… what if I make a typo in the name of one of those variables? How will I know? What if I type WSGI_APPLICATOIN = "config.wsgi.application" instead of WSGI_APPLICATION?

I guess I’ve gotten used to having a Python language server tell me when I’ve made a typo and so now it feels a bit disorienting when I can’t rely on the language server support.

that’s all for now!

I haven’t really successfully used an actual web framework for a project before (right now almost all of my websites are either a single Go binary or static sites), so I’m interested in seeing how it goes!

There’s still lots for me to learn about, I still haven’t really gotten into Django’s form validation tooling or authentication systems.

Thanks to Marco Rogers for convincing me to give ORMs a chance.

(we’re still experimenting with the comments-on-Mastodon system! Here are the comments on Mastodon! tell me your favourite Django feature!)

A data model for Git (and other docs updates)

Thu, 8 Jan 2026 00:00:00 +0000

Hello! This past fall, I decided to take some time to work on Git’s documentation. I’ve been thinking about working on open source docs for a long time – usually if I think the documentation for something could be improved, I’ll write a blog post or a zine or something. But this time I wondered: could I instead make a few improvements to the official documentation?

So Marie and I made a few changes to the Git documentation!

a data model for Git

After a while working on the documentation, we noticed that Git uses the terms “object”, “reference”, or “index” in its documentation a lot, but that it didn’t have a great explanation of what those terms mean or how they relate to other core concepts like “commit” and “branch”. So we wrote a new “data model” document!

You can read the data model here for now. I assume at some point (after the next release?) it’ll also be on the Git website.

I’m excited about this because understanding how Git organizes its commit and branch data has really helped me reason about how Git works over the years, and I think it’s important to have a short (1600 words!) version of the data model that’s accurate.

The “accurate” part turned out to not be that easy: I knew the basics of how Git’s data model worked, but during the review process I learned some new details and had to make quite a few changes (for example how merge conflicts are stored in the staging area).

updates to `git push`, `git pull`, and more

I also worked on updating the introduction to some of Git’s core man pages. I quickly realized that “just try to improve it according to my best judgement” was not going to work: why should the maintainers believe me that my version is better?

I’ve seen a problem a lot when discussing open source documentation changes where 2 expert users of the software argue about whether an explanation is clear or not (“I think X would be a good way to explain it! Well, I think Y would be better!”)

I don’t think this is very productive (expert users of a piece of software are notoriously bad at being able to tell if an explanation will be clear to non-experts), so I needed to find a way to identify problems with the man pages that was a little more evidence-based.

getting test readers to identify problems

I asked for test readers on Mastodon to read the current version of documentation and tell me what they find confusing or what questions they have. About 80 test readers left comments, and I learned so much!

People left a huge amount of great feedback, for example:

terminology they didn’t understand (what’s a pathspec? what does “reference” mean? does “upstream” have a specific meaning in Git?)
specific confusing sentences
suggestions of things things to add (“I do X all the time, I think it should be included here”)
inconsistencies (“here it implies X is the default, but elsewhere it implies Y is the default”)

Most of the test readers had been using Git for at least 5-10 years, which I think worked well – if a group of test readers who have been using Git regularly for 5+ years find a sentence or term impossible to understand, it makes it easy to argue that the documentation should be updated to make it clearer.

I thought this “get users of the software to comment on the existing documentation and then fix the problems they find” pattern worked really well and I’m excited about potentially trying it again in the future.

the man page changes

We ended updating these 4 man pages:

git add (before, after)
git checkout (before, after)
git push (before, after)
git pull (before, after)

The git push and git pull changes were the most interesting to me: in addition to updating the intro to those pages, we also ended up writing:

a section describing what the term “upstream branch” means (which previously wasn’t really explained)
a cleaned-up description of what a “push refspec” is

Making those changes really gave me an appreciation for how much work it is to maintain open source documentation: it’s not easy to write things that are both clear and true, and sometimes we had to make compromises, for example the sentence “git push may fail if you haven’t set an upstream for the current branch, depending on what push.default is set to.” is a little vague, but the exact details of what “depending” means are really complicated and untangling that is a big project.

on the process for contributing to Git

It took me a while to understand Git’s development process. I’m not going to try to describe it here (that could be a whole other post!), but a few quick notes:

Git has a Discord server with a “my first contribution” channel for help with getting started contributing. I found people to be very welcoming on the Discord.
I used GitGitGadget to make all of my contributions. This meant that I could make a GitHub pull request (a workflow I’m comfortable with) and GitGitGadget would convert my PRs into the system the Git developers use (emails with patches attached). GitGitGadget worked great and I was very grateful to not have to learn how to send patches by email with Git.
Otherwise I used my normal email client (Fastmail’s web interface) to reply to emails, wrapping my text to 80 character lines since that’s the mailing list norm.

I also found the mailing list archives on lore.kernel.org hard to navigate, so I hacked together my own git list viewer to make it easier to read the long mailing list threads.

Many people helped me navigate the contribution process and review the changes: thanks to Emily Shaffer, Johannes Schindelin (the author of GitGitGadget), Patrick Steinhardt, Ben Knoble, Junio Hamano, and more.

(I’m experimenting with comments on Mastodon, you can see the comments here)

Notes on switching to Helix from vim

Fri, 10 Oct 2025 00:00:00 +0000

Hello! Earlier this summer I was talking to a friend about how much I love using fish, and how I love that I don’t have to configure it. They said that they feel the same way about the helix text editor, and so I decided to give it a try.

I’ve been using it for 3 months now and here are a few notes.

why helix: language servers

I think what motivated me to try Helix is that I’ve been trying to get a working language server setup (so I can do things like “go to definition”) and getting a setup that feels good in Vim or Neovim just felt like too much work.

After using Vim/Neovim for 20 years, I’ve tried both “build my own custom configuration from scratch” and “use someone else’s pre-buld configuration system” and even though I love Vim I was excited about having things just work without having to work on my configuration at all.

Helix comes with built in language server support, and it feels nice to be able to do things like “rename this symbol” in any language.

the search is great

One of my favourite things about Helix is the search! If I’m searching all the files in my repository for a string, it lets me scroll through the potential matching files and see the full context of the match, like this:

For comparison, here’s what the vim ripgrep plugin I’ve been using looks like:

There’s no context for what else is around that line.

the quick reference is nice

One thing I like about Helix is that when I press g, I get a little help popup telling me places I can go. I really appreciate this because I don’t often use the “go to definition” or “go to reference” feature and I often forget the keyboard shortcut.

some vim -> helix translations

Helix doesn’t have marks like ma, 'a, instead I’ve been using Ctrl+O and Ctrl+I to go back (or forward) to the last cursor location
I think Helix does have macros, but I’ve been using multiple cursors in every case that I would have previously used a macro. I like multiple cursors a lot more than writing macros all the time. If I want to batch change something in the document, my workflow is to press % (to highlight everything), then s to select (with a regex) the things I want to change, then I can just edit all of them as needed.
Helix doesn’t have neovim-style tabs, instead it has a nice buffer switcher (<space>b) I can use to switch to the buffer I want. There’s a pull request here to implement neovim-style tabs. There’s also a setting bufferline="multiple" which can act a bit like tabs with gp, gn for prev/next “tab” and :bc to close a “tab”.

some helix annoyances

Here’s everything that’s annoyed me about Helix so far.

I like the way Helix’s :reflow works much less than how vim reflows text with gq. It doesn’t work as well with lists. (github issue)
If I’m making a Markdown list, pressing “enter” at the end of a list item won’t continue the list. There’s a partial workaround for bulleted lists but I don’t know one for numbered lists.
No persistent undo yet: in vim I could use an undofile so that I could undo changes even after quitting. Helix doesn’t have that feature yet. (github PR)
Helix doesn’t autoreload files after they change on disk, I have to run :reload-all (:ra<tab>) to manually reload them. Not a big deal.
Sometimes it panics, maybe every week or so. I think it might be this issue.

The crashes look something like this:

thread 'main' panicked at helix-core/src/transaction.rs:499:9:
Positions [(2959, AfterSticky), (2959, AfterSticky)] are out of range for changeset len 2945!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The “markdown list” and reflowing issues come up a lot for me because I spend a lot of time editing Markdown lists, but I keep using Helix anyway so I guess they can’t be making me that mad.

switching was easier than I thought

I was worried that relearning 20 years of Vim muscle memory would be really hard.

It turned out to be easier than I expected, I started using Helix on a vacation for a little low-stakes coding project I was doing on the side and after a week or two it didn’t feel so disorienting anymore. I think it might be hard to switch back and forth between Vim and Helix, but I haven’t needed to use Vim recently so I don’t know if that’ll ever become an issue for me.

The first time I tried Helix I tried to force it to use keybindings that were more similar to Vim and that did not work for me. Just learning the “Helix way” was a lot easier.

There are still some things that throw me off: for example w in vim and w in Helix don’t have the same idea of what a “word” is (the Helix one includes the space after the word, the Vim one doesn’t).

using a terminal-based text editor

For many years I’d mostly been using a GUI version of vim/neovim, so switching to actually using an editor in the terminal was a bit of an adjustment.

I ended up deciding on:

Every project gets its own terminal window, and all of the tabs in that window (mostly) have the same working directory
I make my Helix tab the first tab in the terminal window

It works pretty well, I might actually like it better than my previous workflow.

my configuration

I appreciate that my configuration is really simple, compared to my neovim configuration which is hundreds of lines. It’s mostly just 4 keyboard shortcuts.

theme = "solarized_light"
[editor]
# Sync clipboard with system clipboard
default-yank-register = "+"

[keys.normal]
# I didn't like that Ctrl+C was the default "toggle comments" shortcut
"#" = "toggle_comments"

# I didn't feel like learning a different way
# to go to the beginning/end of a line so
# I remapped ^ and $
"^" = "goto_first_nonwhitespace"
"$" = "goto_line_end"

[keys.select]
"^" = "goto_first_nonwhitespace"
"$" = "goto_line_end"

[keys.normal.space]
# I write a lot of text so I need to constantly reflow,
# and missed vim's `gq` shortcut
l = ":reflow"

There’s a separate languages.toml configuration where I set some language preferences, like turning off autoformatting. For example, here’s my Python configuration:

[[language]]
name = "python"
formatter = { command = "black", args = ["--stdin-filename", "%{buffer_name}", "-"] }
language-servers = ["pyright"]
auto-format = false

we’ll see how it goes

Three months is not that long, and it’s possible that I’ll decide to go back to Vim at some point. For example, I wrote a post about switching to nix a while back but after maybe 8 months I switched back to Homebrew (though I’m still using NixOS to manage one little server, and I’m still satisfied with that).

New zine: The Secret Rules of the Terminal

Thu, 26 Jun 2025 00:00:00 +0000

Hello! After many months of writing deep dive blog posts about the terminal, on Tuesday I released a new zine called “The Secret Rules of the Terminal”!

You can get it for $12 here: https://wizardzines.com/zines/terminal, or get an 15-pack of all my zines here.

Here’s the cover:

the table of contents

Here’s the table of contents:

why the terminal?

I’ve been using the terminal every day for 20 years but even though I’m very confident in the terminal, I’ve always had a bit of an uneasy feeling about it. Usually things work fine, but sometimes something goes wrong and it just feels like investigating it is impossible, or at least like it would open up a huge can of worms.

So I started trying to write down a list of weird problems I’ve run into in terminal and I realized that the terminal has a lot of tiny inconsistencies like:

sometimes you can use the arrow keys to move around, but sometimes pressing the arrow keys just prints ^[[D
sometimes you can use the mouse to select text, but sometimes you can’t
sometimes your commands get saved to a history when you run them, and sometimes they don’t
some shells let you use the up arrow to see the previous command, and some don’t

If you use the terminal daily for 10 or 20 years, even if you don’t understand exactly why these things happen, you’ll probably build an intuition for them.

But having an intuition for them isn’t the same as understanding why they happen. When writing this zine I actually had to do a lot of work to figure out exactly what was happening in the terminal to be able to talk about how to reason about it.

the rules aren’t written down anywhere

It turns out that the “rules” for how the terminal works (how do you edit a command you type in? how do you quit a program? how do you fix your colours?) are extremely hard to fully understand, because “the terminal” is actually made of many different pieces of software (your terminal emulator, your operating system, your shell, the core utilities like grep, and every other random terminal program you’ve installed) which are written by different people with different ideas about how things should work.

So I wanted to write something that would explain:

how the 4 pieces of the terminal (your shell, terminal emulator, programs, and TTY driver) fit together to make everything work
some of the core conventions for how you can expect things in your terminal to work
lots of tips and tricks for how to use terminal programs

this zine explains the most useful parts of terminal internals

Terminal internals are a mess. A lot of it is just the way it is because someone made a decision in the 80s and now it’s impossible to change, and honestly I don’t think learning everything about terminal internals is worth it.

But some parts are not that hard to understand and can really make your experience in the terminal better, like:

if you understand what your shell is responsible for, you can configure your shell (or use a different one!) to access your history more easily, get great tab completion, and so much more
if you understand escape codes, it’s much less scary when cating a binary to stdout messes up your terminal, you can just type reset and move on
if you understand how colour works, you can get rid of bad colour contrast in your terminal so you can actually read the text

I learned a surprising amount writing this zine

When I wrote How Git Works, I thought I knew how Git worked, and I was right. But the terminal is different. Even though I feel totally confident in the terminal and even though I’ve used it every day for 20 years, I had a lot of misunderstandings about how the terminal works and (unless you’re the author of tmux or something) I think there’s a good chance you do too.

A few things I learned that are actually useful to me:

I understand the structure of the terminal better and so I feel more confident debugging weird terminal stuff that happens to me (I was even able to suggest a small improvement to fish!). Identifying exactly which piece of software is causing a weird thing to happen in my terminal still isn’t easy but I’m a lot better at it now.
you can write a shell script to copy to your clipboard over SSH
how reset works under the hood (it does the equivalent of stty sane; sleep 1; tput reset) – basically I learned that I don’t ever need to worry about remembering stty sane or tput reset and I can just run reset instead
how to look at the invisible escape codes that a program is printing out (run unbuffer program > out; less out)
why the builtin REPLs on my Mac like sqlite3 are so annoying to use (they use libedit instead of readline)

blog posts I wrote along the way

As usual these days I wrote a bunch of blog posts about various side quests:

How to add a directory to your PATH
“rules” that terminal problems follow
why pipes sometimes get “stuck”: buffering
some terminal frustrations
ASCII control characters in my terminal on “what’s the deal with Ctrl+A, Ctrl+B, Ctrl+C, etc?”
entering text in the terminal is complicated
what’s involved in getting a “modern” terminal setup?
reasons to use your shell’s job control
standards for ANSI escape codes, which is really me trying to figure out if I think the terminfo database is serving us well today

people who helped with this zine

A long time ago I used to write zines mostly by myself but with every project I get more and more help. I met with Marie Claire LeBlanc Flanagan every weekday from September to June to work on this one.

The cover is by Vladimir Kašiković, Lesley Trites did copy editing, Simon Tatham (who wrote PuTTY) did technical review, our Operations Manager Lee did the transcription as well as a million other things, and Jesse Luehrs (who is one of the very few people I know who actually understands the terminal’s cursed inner workings) had so many incredibly helpful conversations with me about what is going on in the terminal.

get the zine

Here are some links to get the zine again:

get The Secret Rules of the Terminal
get a 15-pack of all my zines here.

As always, you can get either a PDF version to print at home or a print version shipped to your house. The only caveat is print orders will ship in August – I need to wait for orders to come in to get an idea of how many I should print before sending it to the printer.

Using `make` to compile C programs (for non-C-programmers)

Tue, 10 Jun 2025 00:00:00 +0000

I have never been a C programmer but every so often I need to compile a C/C++ program from source. This has been kind of a struggle for me: for a long time, my approach was basically “install the dependencies, run make, if it doesn’t work, either try to find a binary someone has compiled or give up”.

“Hope someone else has compiled it” worked pretty well when I was running Linux but since I’ve been using a Mac for the last couple of years I’ve been running into more situations where I have to actually compile programs myself.

So let’s talk about what you might have to do to compile a C program! I’ll use a couple of examples of specific C programs I’ve compiled and talk about a few things that can go wrong. Here are three programs we’ll be talking about compiling:

paperjam
sqlite
qf (a pager you can run to quickly open files from a search with rg -n THING | qf)

step 1: install a C compiler

This is pretty simple: on an Ubuntu system if I don’t already have a C compiler I’ll install one with:

sudo apt-get install build-essential

This installs gcc, g++, and make. The situation on a Mac is more confusing but it’s something like “install xcode command line tools”.

step 2: install the program’s dependencies

Unlike some newer programming languages, C doesn’t have a dependency manager. So if a program has any dependencies, you need to hunt them down yourself. Thankfully because of this, C programmers usually keep their dependencies very minimal and often the dependencies will be available in whatever package manager you’re using.

There’s almost always a section explaining how to get the dependencies in the README, for example in paperjam’s README, it says:

To compile PaperJam, you need the headers for the libqpdf and libpaper libraries (usually available as libqpdf-dev and libpaper-dev packages).

You may need a2x (found in AsciiDoc) for building manual pages.

So on a Debian-based system you can install the dependencies like this.

sudo apt install -y libqpdf-dev libpaper-dev

If a README gives a name for a package (like libqpdf-dev), I’d basically always assume that they mean “in a Debian-based Linux distro”: if you’re on a Mac brew install libqpdf-dev will not work. I still have not 100% gotten the hang of developing on a Mac yet so I don’t have many tips there yet. I guess in this case it would be brew install qpdf if you’re using Homebrew.

step 3: run `./configure` (if needed)

Some C programs come with a Makefile and some instead come with a script called ./configure. For example, if you download sqlite’s source code, it has a ./configure script in it instead of a Makefile.

My understanding of this ./configure script is:

You run it, it prints out a lot of somewhat inscrutable output, and then it either generates a Makefile or fails because you’re missing some dependency
The ./configure script is part of a system called autotools that I have never needed to learn anything about beyond “run it to generate a Makefile”.

I think there might be some options you can pass to get the ./configure script to produce a different Makefile but I have never done that.

step 4: run `make`

The next step is to run make to try to build a program. Some notes about make:

Sometimes you can run make -j8 to parallelize the build and make it go faster
It usually prints out a million compiler warnings when compiling the program. I always just ignore them. I didn’t write the software! The compiler warnings are not my problem.

compiler errors are often dependency problems

Here’s an error I got while compiling paperjam on my Mac:

/opt/homebrew/Cellar/qpdf/12.0.0/include/qpdf/InputSource.hh:85:19: error: function definition does not declare parameters
   85 |     qpdf_offset_t last_offset{0};
      |                   ^

Over the years I’ve learned it’s usually best not to overthink problems like this: if it’s talking about qpdf, there’s a good change it just means that I’ve done something wrong with how I’m including the qpdf dependency.

Now let’s talk about some ways to get the qpdf dependency included in the right way.

the world’s shortest introduction to the compiler and linker

Before we talk about how to fix dependency problems: building C programs is split into 2 steps:

Compiling the code into object files (with gcc or clang)
Linking those object files into a final binary (with ld)

It’s important to know this when building a C program because sometimes you need to pass the right flags to the compiler and linker to tell them where to find the dependencies for the program you’re compiling.

`make` uses environment variables to configure the compiler and linker

If I run make on my Mac to install paperjam, I get this error:

c++ -o paperjam paperjam.o pdf-tools.o parse.o cmds.o pdf.o -lqpdf -lpaper
ld: library 'qpdf' not found

This is not because qpdf is not installed on my system (it actually is!). But the compiler and linker don’t know how to find the qpdf library. To fix this, we need to:

pass "-I/opt/homebrew/include" to the compiler (to tell it where to find the header files)
pass "-L/opt/homebrew/lib -liconv" to the linker (to tell it where to find library files and to link in iconv)

And we can get make to pass those extra parameters to the compiler and linker using environment variables! To see how this works: inside paperjam’s Makefile you can see a bunch of environment variables, like LDLIBS here:

paperjam: $(OBJS)
	$(LD) -o $@ $^ $(LDLIBS)

Everything you put into the LDLIBS environment variable gets passed to the linker (ld) as a command line argument.

secret environment variable: `CPPFLAGS`

Makefiles sometimes define their own environment variables that they pass to the compiler/linker, but make also has a bunch of “implicit” environment variables which it will automatically pass to the C compiler and linker. There’s a full list of implicit environment variables here, but one of them is CPPFLAGS, which gets automatically passed to the C compiler.

(technically it would be more normal to use CXXFLAGS for this, but this particular Makefile hardcodes CXXFLAGS so setting CPPFLAGS was the only way I could find to set the compiler flags without editing the Makefile)

As an aside: it took me a long time to realize how closely tied to C/C++ `make` is -- I used to think that `make` was just a general build system (and of course you can use it for anything!) but it has a lot of affordances for building C/C++ programs that it doesn't have for building any other kind of program.

two ways to pass environment variables to `make`

I learned thanks to @zwol that there are actually two ways to pass environment variables to make:

CXXFLAGS=xyz make (the usual way)
make CXXFLAGS=xyz

The difference between them is that make CXXFLAGS=xyz will override the value of CXXFLAGS set in the Makefile but CXXFLAGS=xyz make won’t.

I’m not sure which way is the norm but I’m going to use the first way in this post.

how to use `CPPFLAGS` and `LDLIBS` to fix this compiler error

Now that we’ve talked about how CPPFLAGS and LDLIBS get passed to the compiler and linker, here’s the final incantation that I used to get the program to build successfully!

CPPFLAGS="-I/opt/homebrew/include" LDLIBS="-L/opt/homebrew/lib -liconv" make paperjam

This passes -I/opt/homebrew/include to the compiler and -L/opt/homebrew/lib -liconv to the linker.

Also I don’t want to pretend that I “magically” knew that those were the right arguments to pass, figuring them out involved a bunch of confused Googling that I skipped over in this post. I will say that:

the -I compiler flag tells the compiler which directory to find header files in, like /opt/homebrew/include/qpdf/QPDF.hh
the -L linker flag tells the linker which directory to find libraries in, like /opt/homebrew/lib/libqpdf.a
the -l linker flag tells the linker which libraries to link in, like -liconv means “link in the iconv library”, or -lm means “link math”

tip: how to just build 1 specific file: `make $FILENAME`

Yesterday I discovered this cool tool called qf which you can use to quickly open files from the output of ripgrep.

qf is in a big directory of various tools, but I only wanted to compile qf. So I just compiled qf, like this:

make qf

Basically if you know (or can guess) the output filename of the file you’re trying to build, you can tell make to just build that file by running make $FILENAME

tip: you don’t need a Makefile

I sometimes write 5-line C programs with no dependencies, and I just learned that if I have a file called blah.c, I can just compile it like this without creating a Makefile:

make blah

It gets automaticaly expanded to cc -o blah blah.c, which saves a bit of typing. I have no idea if I’m going to remember this (I might just keep typing gcc -o blah blah.c anyway) but it seems like a fun trick.

tip: look at how other packaging systems built the same C program

If you’re having trouble building a C program, maybe other people had problems building it too! Every Linux distribution has build files for every package that they build, so even if you can’t install packages from that distribution directly, maybe you can get tips from that Linux distro for how to build the package. Realizing this (thanks to my friend Dave) was a huge ah-ha moment for me.

For example, this line from the nix package for paperjam says:

  env.NIX_LDFLAGS = lib.optionalString stdenv.hostPlatform.isDarwin "-liconv";

This is basically saying “pass the linker flag -liconv to build this on a Mac”, so that’s a clue we could use to build it.

That same file also says env.NIX_CFLAGS_COMPILE = "-DPOINTERHOLDER_TRANSITION=1";. I’m not sure what this means, but when I try to build the paperjam package I do get an error about something called a PointerHolder, so I guess that’s somehow related to the “PointerHolder transition”.

step 5: installing the binary

Once you’ve managed to compile the program, probably you want to install it somewhere! Some Makefiles have an install target that let you install the tool on your system with make install. I’m always a bit scared of this (where is it going to put the files? what if I want to uninstall them later?), so if I’m compiling a pretty simple program I’ll often just manually copy the binary to install it instead, like this:

cp qf ~/bin

step 6: maybe make your own package!

Once I figured out how to do all of this, I realized that I could use my new make knowledge to contribute a paperjam package to Homebrew! Then I could just brew install paperjam on future systems.

The good thing is that even if the details of how all of the different packaging systems, they fundamentally all use C compilers and linkers.

it can be useful to understand a little about C even if you’re not a C programmer

I think all of this is an interesting example of how it can useful to understand some basics of how C programs work (like “they have header files”) even if you’re never planning to write a nontrivial C program if your life.

It feels good to have some ability to compile C/C++ programs myself, even though I’m still not totally confident about all of the compiler and linker flags and I still plan to never learn anything about how autotools works other than “you run ./configure to generate the Makefile”.

Two things I left out of this post:

LD_LIBRARY_PATH / DYLD_LIBRARY_PATH (which you use to tell the dynamic linker at runtime where to find dynamically linked files) because I can’t remember the last time I ran into an LD_LIBRARY_PATH issue and couldn’t find an example.
pkg-config, which I think is important but I don’t understand yet

Standards for ANSI escape codes

Fri, 7 Mar 2025 00:00:00 +0000

Hello! Today I want to talk about ANSI escape codes.

For a long time I was vaguely aware of ANSI escape codes (“that’s how you make text red in the terminal and stuff”) but I had no real understanding of where they were supposed to be defined or whether or not there were standards for them. I just had a kind of vague “there be dragons” feeling around them. While learning about the terminal this year, I’ve learned that:

ANSI escape codes are responsible for a lot of usability improvements in the terminal (did you know there’s a way to copy to your system clipboard when SSHed into a remote machine?? It’s an escape code called OSC 52!)
They aren’t completely standardized, and because of that they don’t always work reliably. And because they’re also invisible, it’s extremely frustrating to troubleshoot escape code issues.

So I wanted to put together a list for myself of some standards that exist around escape codes, because I want to know if they have to feel unreliable and frustrating, or if there’s a future where we could all rely on them with more confidence.

what’s an escape code?

Have you ever pressed the left arrow key in your terminal and seen ^[[D? That’s an escape code! It’s called an “escape code” because the first character is the “escape” character, which is usually written as ESC, \x1b, \E, \033, or ^[.

Escape codes are how your terminal emulator communicates various kinds of information (colours, mouse movement, etc) with programs running in the terminal. There are two kind of escape codes:

input codes which your terminal emulator sends for keypresses or mouse movements that don’t fit into Unicode. For example “left arrow key” is ESC[D, “Ctrl+left arrow” might be ESC[1;5D, and clicking the mouse might be something like ESC[M :3.
output codes which programs can print out to colour text, move the cursor around, clear the screen, hide the cursor, copy text to the clipboard, enable mouse reporting, set the window title, etc.

Now let’s talk about standards!

ECMA-48

The first standard I found relating to escape codes was ECMA-48, which was originally published in 1976.

ECMA-48 does two things:

Define some general formats for escape codes (like “CSI” codes, which are ESC[ + something and “OSC” codes, which are ESC] + something)
Define some specific escape codes, like how “move the cursor to the left” is ESC[D, or “turn text red” is ESC[31m. In the spec, the “cursor left” one is called CURSOR LEFT and the one for changing colours is called SELECT GRAPHIC RENDITION.

The formats are extensible, so there’s room for others to define more escape codes in the future. Lots of escape codes that are popular today aren’t defined in ECMA-48: for example it’s pretty common for terminal applications (like vim, htop, or tmux) to support using the mouse, but ECMA-48 doesn’t define escape codes for the mouse.

xterm control sequences

There are a bunch of escape codes that aren’t defined in ECMA-48, for example:

enabling mouse reporting (where did you click in your terminal?)
bracketed paste (did you paste that text or type it in?)
OSC 52 (which terminal applications can use to copy text to your system clipboard)

I believe (correct me if I’m wrong!) that these and some others came from xterm, are documented in XTerm Control Sequences, and have been widely implemented by other terminal emulators.

This list of “what xterm supports” is not a standard exactly, but xterm is extremely influential and so it seems like an important document.

terminfo

In the 80s (and to some extent today, but my understanding is that it was MUCH more dramatic in the 80s) there was a huge amount of variation in what escape codes terminals actually supported.

To deal with this, there’s a database of escape codes for various terminals called “terminfo”.

It looks like the standard for terminfo is called X/Open Curses, though you need to create an account to view that standard for some reason. It defines the database format as well as a C library interface (“curses”) for accessing the database.

For example you can run this bash snippet to see every possible escape code for “clear screen” for all of the different terminals your system knows about:

for term in $(toe -a | awk '{print $1}')
do
  echo $term
  infocmp -1 -T "$term" 2>/dev/null | grep 'clear=' | sed 's/clear=//g;s/,//g'
done

On my system (and probably every system I’ve ever used?), the terminfo database is managed by ncurses.

should programs use terminfo?

I think it’s interesting that there are two main approaches that applications take to handling ANSI escape codes:

Use the terminfo database to figure out which escape codes to use, depending on what’s in the TERM environment variable. Fish does this, for example.
Identify a “single common set” of escape codes which works in “enough” terminal emulators and just hardcode those.

Some examples of programs/libraries that take approach #2 (“don’t use terminfo”) include:

I got curious about why folks might be moving away from terminfo and I found this very interesting and extremely detailed rant about terminfo from one of the fish maintainers, which argues that:

[the terminfo authors] have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is.

I’m not going to do it justice so I’m not going to summarize it, I think it’s worth reading.

is there a “single common set” of escape codes?

I was just talking about the idea that you can use a “common set” of escape codes that will work for most people. But what is that set? Is there any agreement?

I really do not know the answer to this at all, but from doing some reading it seems like it’s some combination of:

The codes that the VT100 supported (though some aren’t relevant on modern terminals)
what’s in ECMA-48 (which I think also has some things that are no longer relevant)
What xterm supports (though I’d guess that not everything in there is actually widely supported enough)

and maybe ultimately “identify the terminal emulators you think your users are going to use most frequently and test in those”, the same way web developers do when deciding which CSS features are okay to use

I don’t think there are any resources like Can I use…? or Baseline for the terminal though. (in theory terminfo is supposed to be the “caniuse” for the terminal but it seems like it often takes 10+ years to add new terminal features when people invent them which makes it very limited)

some reasons to use terminfo

I also asked on Mastodon why people found terminfo valuable in 2025 and got a few reasons that made sense to me:

some people expect to be able to use the TERM environment variable to control how programs behave (for example with TERM=dumb), and there’s no standard for how that should work in a post-terminfo world
even though there’s less variation between terminal emulators than there was in the 80s, there’s far from zero variation: there are graphical terminals, the Linux framebuffer console, the situation you’re in when connecting to a server via its serial console, Emacs shell mode, and probably more that I’m missing
there is no one standard for what the “single common set” of escape codes is, and sometimes programs use escape codes which aren’t actually widely supported enough

terminfo & user agent detection

The way that ncurses uses the TERM environment variable to decide which escape codes to use reminds me of how webservers used to sometimes use the browser user agent to decide which version of a website to serve.

It also seems like it’s had some of the same results – the way iTerm2 reports itself as being “xterm-256color” feels similar to how Safari’s user agent is “Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15”. In both cases the terminal emulator / browser ends up changing its user agent to get around user agent detection that isn’t working well.

On the web we ended up deciding that user agent detection was not a good practice and to instead focus on standardization so we can serve the same HTML/CSS to all browsers. I don’t know if the same approach is the future in the terminal though – I think the terminal landscape today is much more fragmented than the web ever was as well as being much less well funded.

some more documents/standards

A few more documents and standards related to escape codes, in no particular order:

the Linux console_codes man page documents escape codes that Linux supports
how the VT 100 handles escape codes & control sequences
the kitty keyboard protocol
OSC 8 for links in the terminal (and notes on adoption)
A summary of ANSI standards from tmux
this terminal features reporting specification from iTerm
sixel graphics

why I think this is interesting

I sometimes see people saying that the unix terminal is “outdated”, and since I love the terminal so much I’m always curious about what incremental changes might make it feel less “outdated”.

Maybe if we had a clearer standards landscape (like we do on the web!) it would be easier for terminal emulator developers to build new features and for authors of terminal applications to more confidently adopt those features so that we can all benefit from them and have a richer experience in the terminal.

Obviously standardizing ANSI escape codes is not easy (ECMA-48 was first published almost 50 years ago and we’re still not there!). I don’t even know what all of the challenges are. But the situation with HTML/CSS/JS used to be extremely bad too and now it’s MUCH better, so maybe there’s hope.

How to add a directory to your PATH

Thu, 13 Feb 2025 12:27:56 +0000

I was talking to a friend about how to add a directory to your PATH today. It’s something that feels “obvious” to me since I’ve been using the terminal for a long time, but when I searched for instructions for how to do it, I actually couldn’t find something that explained all of the steps – a lot of them just said “add this to ~/.bashrc”, but what if you’re not using bash? What if your bash config is actually in a different file? And how are you supposed to figure out which directory to add anyway?

So I wanted to try to write down some more complete directions and mention some of the gotchas I’ve run into over the years.

Here’s a table of contents:

step 1: what shell are you using?

If you’re not sure what shell you’re using, here’s a way to find out. Run this:

ps -p $$ -o pid,comm=

if you’re using bash, it’ll print out 97295 bash
if you’re using zsh, it’ll print out 97295 zsh
if you’re using fish, it’ll print out an error like “In fish, please use fishₚid” (`$` isn’t valid syntax in fish, but in any case the error message tells you that you’re using fish, which you probably already knew)

Also bash is the default on Linux and zsh is the default on Mac OS (as of 2024). I’ll only cover bash, zsh, and fish in these directions.

step 2: find your shell’s config file

in zsh, it’s probably ~/.zshrc
in bash, it might be ~/.bashrc, but it’s complicated, see the note in the next section
in fish, it’s probably ~/.config/fish/config.fish (you can run echo $__fish_config_dir if you want to be 100% sure)

a note on bash’s config file

Bash has three possible config files: ~/.bashrc, ~/.bash_profile, and ~/.profile.

If you’re not sure which one your system is set up to use, I’d recommend testing this way:

add echo hi there to your ~/.bashrc
Restart your terminal
If you see “hi there”, that means ~/.bashrc is being used! Hooray!
Otherwise remove it and try the same thing with ~/.bash_profile
You can also try ~/.profile if the first two options don’t work.

(there are a lot of elaborate flow charts out there that explain how bash decides which config file to use but IMO it’s not worth it to internalize them and just testing is the fastest way to be sure)

step 3: figure out which directory to add

Let’s say that you’re trying to install and run a program called http-server and it doesn’t work, like this:

$ npm install -g http-server
$ http-server
bash: http-server: command not found

How do you find what directory http-server is in? Honestly in general this is not that easy – often the answer is something like “it depends on how npm is configured”. A few ideas:

Often when setting up a new installer (like cargo, npm, homebrew, etc), when you first set it up it’ll print out some directions about how to update your PATH. So if you’re paying attention you can get the directions then.
Sometimes installers will automatically update your shell’s config file to update your PATH for you
Sometimes just Googling “where does npm install things?” will turn up the answer
Some tools have a subcommand that tells you where they’re configured to install things, like:
- Node/npm: npm config get prefix (then append /bin/)
- Go: go env GOPATH (then append /bin/)
- asdf: asdf info | grep ASDF_DIR (then append /bin/ and /shims/)

step 3.1: double check it’s the right directory

Once you’ve found a directory you think might be the right one, make sure it’s actually correct! For example, I found out that on my machine, http-server is in ~/.npm-global/bin. I can make sure that it’s the right directory by trying to run the program http-server in that directory like this:

$ ~/.npm-global/bin/http-server
Starting up http-server, serving ./public

It worked! Now that you know what directory you need to add to your PATH, let’s move to the next step!

step 4: edit your shell config

Now we have the 2 critical pieces of information we need:

Which directory you’re trying to add to your PATH (like ~/.npm-global/bin/)
Where your shell’s config is (like ~/.bashrc, ~/.zshrc, or ~/.config/fish/config.fish)

Now what you need to add depends on your shell:

bash instructions:

Open your shell’s config file, and add a line like this:

export PATH=$PATH:~/.npm-global/bin/

(obviously replace ~/.npm-global/bin with the actual directory you’re trying to add)

zsh instructions:

You can do the same thing as in bash, but zsh also has some slightly fancier syntax you can use if you prefer:

path=(
  $path
  ~/.npm-global/bin
)

fish instructions:

In fish, the syntax is different:

set PATH $PATH ~/.npm-global/bin

(in fish you can also use fish_add_path, some notes on that further down)

step 5: restart your shell

Now, an extremely important step: updating your shell’s config won’t take effect if you don’t restart it!

Two ways to do this:

open a new terminal (or terminal tab), and maybe close the old one so you don’t get confused
Run bash to start a new shell (or zsh if you’re using zsh, or fish if you’re using fish)

I’ve found that both of these usually work fine.

And you should be done! Try running the program you were trying to run and hopefully it works now.

If not, here are a couple of problems that you might run into:

problem 1: it ran the wrong program

If the wrong version of a program is running, you might need to add the directory to the beginning of your PATH instead of the end.

For example, on my system I have two versions of python3 installed, which I can see by running which -a:

$ which -a python3
/usr/bin/python3
/opt/homebrew/bin/python3

The one your shell will use is the first one listed.

If you want to use the Homebrew version, you need to add that directory (/opt/homebrew/bin) to the beginning of your PATH instead, by putting this in your shell’s config file (it’s /opt/homebrew/bin/:$PATH instead of the usual $PATH:/opt/homebrew/bin/)

export PATH=/opt/homebrew/bin/:$PATH

or in fish:

set PATH ~/.cargo/bin $PATH

problem 2: the program isn’t being run from your shell

All of these directions only work if you’re running the program from your shell. If you’re running the program from an IDE, from a GUI, in a cron job, or some other way, you’ll need to add the directory to your PATH in a different way, and the exact details might depend on the situation.

in a cron job

Some options:

use the full path to the program you’re running, like /home/bork/bin/my-program
put the full PATH you want as the first line of your crontab (something like PATH=/bin:/usr/bin:/usr/local/bin:….). You can get the full PATH you’re using in your shell by running echo "PATH=$PATH".

I’m honestly not sure how to handle it in an IDE/GUI because I haven’t run into that in a long time, will add directions here if someone points me in the right direction.

problem 3: duplicate `PATH` entries making it harder to debug

If you edit your path and start a new shell by running bash (or zsh, or fish), you’ll often end up with duplicate PATH entries, because the shell keeps adding new things to your PATH every time you start your shell.

Personally I don’t think I’ve run into a situation where this kind of duplication breaks anything, but the duplicates can make it harder to debug what’s going on with your PATH if you’re trying to understand its contents.

Some ways you could deal with this:

If you’re debugging your PATH, open a new terminal to do it in so you get a “fresh” state. This should avoid the duplication.
Deduplicate your PATH at the end of your shell’s config (for example in zsh apparently you can do this with typeset -U path)
Check that the directory isn’t already in your PATH when adding it (for example in fish I believe you can do this with fish_add_path --path /some/directory)

How to deduplicate your PATH is shell-specific and there isn’t always a built in way to do it so you’ll need to look up how to accomplish it in your shell.

problem 4: losing your history after updating your `PATH`

Here’s a situation that’s easy to get into in bash or zsh:

Run a command (it fails)
Update your PATH
Run bash to reload your config
Press the up arrow a couple of times to rerun the failed command (or open a new terminal)
The failed command isn’t in your history! Why not?

This happens because in bash, by default, history is not saved until you exit the shell.

Some options for fixing this:

Instead of running bash to reload your config, run source ~/.bashrc (or source ~/.zshrc in zsh). This will reload the config inside your current session.
Configure your shell to continuously save your history instead of only saving the history when the shell exits. (How to do this depends on whether you’re using bash or zsh, the history options in zsh are a bit complicated and I’m not exactly sure what the best way is)

a note on `source`

When you install cargo (Rust’s installer) for the first time, it gives you these instructions for how to set up your PATH, which don’t mention a specific directory at all.

This is usually done by running one of the following (note the leading DOT):

. "$HOME/.cargo/env"        	# For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish

The idea is that you add that line to your shell’s config, and their script automatically sets up your PATH (and potentially other things) for you.

This is pretty common (for example Homebrew suggests you eval brew shellenv), and there are two ways to approach this:

Just do what the tool suggests (like adding . "$HOME/.cargo/env" to your shell’s config)
Figure out which directories the script they’re telling you to run would add to your PATH, and then add those manually. Here’s how I’d do that:
- Run . "$HOME/.cargo/env" in my shell (or the fish version if using fish)
- Run echo "$PATH" | tr ':' '\n' | grep cargo to figure out which directories it added
- See that it says /Users/bork/.cargo/bin and shorten that to ~/.cargo/bin
- Add the directory ~/.cargo/bin to PATH (with the directions in this post)

I don’t think there’s anything wrong with doing what the tool suggests (it might be the “best way”!), but personally I usually use the second approach because I prefer knowing exactly what configuration I’m changing.

a note on `fish_add_path`

fish has a handy function called fish_add_path that you can run to add a directory to your PATH like this:

fish_add_path /some/directory

This is cool (it’s such a simple command!) but I’ve stopped using it for a couple of reasons:

Sometimes fish_add_path will update the PATH for every session in the future (with a “universal variable”) and sometimes it will update the PATH just for the current session and it’s hard for me to tell which one it will do. In theory the docs explain this but I could not understand them.
If you ever need to remove the directory from your PATH a few weeks or months later because maybe you made a mistake, it’s kind of hard to do (there are instructions in this comments of this github issue though).

that’s all

Hopefully this will help some people. Let me know (on Mastodon or Bluesky) if you there are other major gotchas that have tripped you up when adding a directory to your PATH, or if you have questions about this post!

Some terminal frustrations

Wed, 5 Feb 2025 16:57:00 +0000

A few weeks ago I ran a terminal survey (you can read the results here) and at the end I asked:

What’s the most frustrating thing about using the terminal for you?

1600 people answered, and I decided to spend a few days categorizing all the responses. Along the way I learned that classifying qualitative data is not easy but I gave it my best shot. I ended up building a custom tool to make it faster to categorize everything.

As with all of my surveys the methodology isn’t particularly scientific. I just posted the survey to Mastodon and Twitter, ran it for a couple of days, and got answers from whoever happened to see it and felt like responding.

Here are the top categories of frustrations!

I think it’s worth keeping in mind while reading these comments that

40% of people answering this survey have been using the terminal for 21+ years
95% of people answering the survey have been using the terminal for at least 4 years

These comments aren’t coming from total beginners.

Here are the categories of frustrations! The number in brackets is the number of people with that frustration. I’m mostly writing this up for myself because I’m trying to write a zine about the terminal and I wanted to get a sense for what people are having trouble with.

remembering syntax (115)

People talked about struggles remembering:

the syntax for CLI tools like awk, jq, sed, etc
the syntax for redirects
keyboard shortcuts for tmux, text editing, etc

One example comment:

There are just so many little “trivia” details to remember for full functionality. Even after all these years I’ll sometimes forget where it’s 2 or 1 for stderr, or forget which is which for > and >>.

switching terminals is hard (91)

People talked about struggling with switching systems (for example home/work computer or when SSHing) and running into:

OS differences in keyboard shortcuts (like Linux vs Mac)
systems which don’t have their preferred text editor (“no vim” or “only vim”)
different versions of the same command (like Mac OS grep vs GNU grep)
no tab completion
a shell they aren’t used to (“the subtle differences between zsh and bash”)

as well as differences inside the same system like pagers being not consistent with each other (git diff pagers, other pagers).

One example comment:

I got used to fish and vi mode which are not available when I ssh into servers, containers.

color (85)

Lots of problems with color, like:

programs setting colors that are unreadable with a light background color
finding a colorscheme they like (and getting it to work consistently across different apps)
color not working inside several layers of SSH/tmux/etc
not liking the defaults
not wanting color at all and struggling to turn it off

This comment felt relatable to me:

Getting my terminal theme configured in a reasonable way between the terminal emulator and fish (I did this years ago and remember it being tedious and fiddly and now feel like I’m locked into my current theme because it works and I dread touching any of that configuration ever again).

keyboard shortcuts (84)

Half of the comments on keyboard shortcuts were about how on Linux/Windows, the keyboard shortcut to copy/paste in the terminal is different from in the rest of the OS.

Some other issues with keyboard shortcuts other than copy/paste:

using Ctrl-W in a browser-based terminal and closing the window
the terminal only supports a limited set of keyboard shortcuts (no Ctrl-Shift-, no Super, no Hyper, lots of ctrl- shortcuts aren’t possible like Ctrl-,)
the OS stopping you from using a terminal keyboard shortcut (like by default Mac OS uses Ctrl+left arrow for something else)
issues using emacs in the terminal
backspace not working (2)

other copy and paste issues (75)

Aside from “the keyboard shortcut for copy and paste is different”, there were a lot of OTHER issues with copy and paste, like:

copying over SSH
how tmux and the terminal emulator both do copy/paste in different ways
dealing with many different clipboards (system clipboard, vim clipboard, the “middle click” clipboard on Linux, tmux’s clipboard, etc) and potentially synchronizing them
random spaces added when copying from the terminal
pasting multiline commands which automatically get run in a terrifying way
wanting a way to copy text without using the mouse

discoverability (55)

There were lots of comments about this, which all came down to the same basic complaint – it’s hard to discover useful tools or features! This comment kind of summed it all up:

How difficult it is to learn independently. Most of what I know is an assorted collection of stuff I’ve been told by random people over the years.

steep learning curve (44)

A lot of comments about it generally having a steep learning curve. A couple of example comments:

After 15 years of using it, I’m not much faster than using it than I was 5 or maybe even 10 years ago.

and

That I know I could make my life easier by learning more about the shortcuts and commands and configuring the terminal but I don’t spend the time because it feels overwhelming.

history (42)

Some issues with shell history:

history not being shared between terminal tabs (16)
limits that are too short (4)
history not being restored when terminal tabs are restored
losing history because the terminal crashed
not knowing how to search history

One example comment:

It wasted a lot of time until I figured it out and still annoys me that “history” on zsh has such a small buffer; I have to type “history 0” to get any useful length of history.

bad documentation (37)

People talked about:

documentation being generally opaque
lack of examples in man pages
programs which don’t have man pages

Here’s a representative comment:

Finding good examples and docs. Man pages often not enough, have to wade through stack overflow

scrollback (36)

A few issues with scrollback:

programs printing out too much data making you lose scrollback history
resizing the terminal messes up the scrollback
lack of timestamps
GUI programs that you start in the background printing stuff out that gets in the way of other programs’ outputs

One example comment:

When resizing the terminal (in particular: making it narrower) leads to broken rewrapping of the scrollback content because the commands formatted their output based on the terminal window width.

“it feels outdated” (33)

Lots of comments about how the terminal feels hampered by legacy decisions and how users often end up needing to learn implementation details that feel very esoteric. One example comment:

Most of the legacy cruft, it would be great to have a green field implementation of the CLI interface.

shell scripting (32)

Lots of complaints about POSIX shell scripting. There’s a general feeling that shell scripting is difficult but also that switching to a different less standard scripting language (fish, nushell, etc) brings its own problems.

Shell scripting. My tolerance to ditch a shell script and go to a scripting language is pretty low. It’s just too messy and powerful. Screwing up can be costly so I don’t even bother.

more issues

Some more issues that were mentioned at least 10 times:

(31) inconsistent command line arguments: is it -h or help or –help?
(24) keeping dotfiles in sync across different systems
(23) performance (e.g. “my shell takes too long to start”)
(20) window management (potentially with some combination of tmux tabs, terminal tabs, and multiple terminal windows. Where did that shell session go?)
(17) generally feeling scared/uneasy (“The debilitating fear that I’m going to do some mysterious Bad Thing with a command and I will have absolutely no idea how to fix or undo it or even really figure out what happened”)
(16) terminfo issues (“Having to learn about terminfo if/when I try a new terminal emulator and ssh elsewhere.”)
(16) lack of image support (sixel etc)
(15) SSH issues (like having to start over when you lose the SSH connection)
(15) various tmux/screen issues (for example lack of integration between tmux and the terminal emulator)
(15) typos & slow typing
(13) the terminal getting messed up for various reasons (pressing Ctrl-S, cating a binary, etc)
(12) quoting/escaping in the shell
(11) various Windows/PowerShell issues

n/a (122)

There were also 122 answers to the effect of “nothing really” or “only that I can’t do EVERYTHING in the terminal”

One example comment:

Think I’ve found work arounds for most/all frustrations

that’s all!

I’m not going to make a lot of commentary on these results, but here are a couple of categories that feel related to me:

remembering syntax & history (often the thing you need to remember is something you’ve run before!)
discoverability & the learning curve (the lack of discoverability is definitely a big part of what makes it hard to learn)
“switching systems is hard” & “it feels outdated” (tools that haven’t really changed in 30 or 40 years have many problems but they do tend to be always there no matter what system you’re on, which is very useful and makes them hard to stop using)

Trying to categorize all these results in a reasonable way really gave me an appreciation for social science researchers’ skills.

What's involved in getting a "modern" terminal setup?

Sat, 11 Jan 2025 09:46:01 +0000

Hello! Recently I ran a terminal survey and I asked people what frustrated them. One person commented:

There are so many pieces to having a modern terminal experience. I wish it all came out of the box.

My immediate reaction was “oh, getting a modern terminal experience isn’t that hard, you just need to….”, but the more I thought about it, the longer the “you just need to…” list got, and I kept thinking about more and more caveats.

So I thought I would write down some notes about what it means to me personally to have a “modern” terminal experience and what I think can make it hard for people to get there.

what is a “modern terminal experience”?

Here are a few things that are important to me, with which part of the system is responsible for them:

multiline support for copy and paste: if you paste 3 commands in your shell, it should not immediately run them all! That’s scary! (shell, terminal emulator)
infinite shell history: if I run a command in my shell, it should be saved forever, not deleted after 500 history entries or whatever. Also I want commands to be saved to the history immediately when I run them, not only when I exit the shell session (shell)
a useful prompt: I can’t live without having my current directory and current git branch in my prompt (shell)
24-bit colour: this is important to me because I find it MUCH easier to theme neovim with 24-bit colour support than in a terminal with only 256 colours (terminal emulator)
clipboard integration between vim and my operating system so that when I copy in Firefox, I can just press p in vim to paste (text editor, maybe the OS/terminal emulator too)
good autocomplete: for example commands like git should have command-specific autocomplete (shell)
having colours in ls (shell config)
a terminal theme I like: I spend a lot of time in my terminal, I want it to look nice and I want its theme to match my terminal editor’s theme. (terminal emulator, text editor)
automatic terminal fixing: If a programs prints out some weird escape codes that mess up my terminal, I want that to automatically get reset so that my terminal doesn’t get messed up (shell)
keybindings: I want Ctrl+left arrow to work (shell or application)
being able to use the scroll wheel in programs like less: (terminal emulator and applications)

There are a million other terminal conveniences out there and different people value different things, but those are the ones that I would be really unhappy without.

how I achieve a “modern experience”

My basic approach is:

use the fish shell. Mostly don’t configure it, except to:
- set the EDITOR environment variable to my favourite terminal editor
- alias ls to ls --color=auto
use any terminal emulator with 24-bit colour support. In the past I’ve used GNOME Terminal, Terminator, and iTerm, but I’m not picky about this. I don’t really configure it other than to choose a font.
use neovim, with a configuration that I’ve been very slowly building over the last 9 years or so (the last time I deleted my vim config and started from scratch was 9 years ago)
use the base16 framework to theme everything

A few things that affect my approach:

I don’t spend a lot of time SSHed into other machines
I’d rather use the mouse a little than come up with keyboard-based ways to do everything
I work on a lot of small projects, not one big project

some “out of the box” options for a “modern” experience

What if you want a nice experience, but don’t want to spend a lot of time on configuration? Figuring out how to configure vim in a way that I was satisfied with really did take me like ten years, which is a long time!

My best ideas for how to get a reasonable terminal experience with minimal config are:

shell: either fish or zsh with oh-my-zsh
terminal emulator: almost anything with 24-bit colour support, for example all of these are popular:
- linux: GNOME Terminal, Konsole, Terminator, xfce4-terminal
- mac: iTerm (Terminal.app doesn’t have 256-colour support)
- cross-platform: kitty, alacritty, wezterm, or ghostty
shell config:
- set the EDITOR environment variable to your favourite terminal text editor
- maybe alias ls to ls --color=auto
text editor: this is a tough one, maybe micro or helix? I haven’t used either of them seriously but they both seem like very cool projects and I think it’s amazing that you can just use all the usual GUI editor commands (Ctrl-C to copy, Ctrl-V to paste, Ctrl-A to select all) in micro and they do what you’d expect. I would probably try switching to helix except that retraining my vim muscle memory seems way too hard. Also helix doesn’t have a GUI or plugin system yet.

Personally I wouldn’t use xterm, rxvt, or Terminal.app as a terminal emulator, because I’ve found in the past that they’re missing core features (like 24-bit colour in Terminal.app’s case) that make the terminal harder to use for me.

I don’t want to pretend that getting a “modern” terminal experience is easier than it is though – I think there are two issues that make it hard. Let’s talk about them!

issue 1 with getting to a “modern” experience: the shell

bash and zsh are by far the two most popular shells, and neither of them provide a default experience that I would be happy using out of the box, for example:

you need to customize your prompt
they don’t come with git completions by default, you have to set them up
by default, bash only stores 500 (!) lines of history and (at least on Mac OS) zsh is only configured to store 2000 lines, which is still not a lot
I find bash’s tab completion very frustrating, if there’s more than one match then you can’t tab through them

And even though I love fish, the fact that it isn’t POSIX does make it hard for a lot of folks to make the switch.

Of course it’s totally possible to learn how to customize your prompt in bash or whatever, and it doesn’t even need to be that complicated (in bash I’d probably start with something like export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ ', or maybe use starship). But each of these “not complicated” things really does add up and it’s especially tough if you need to keep your config in sync across several systems.

An extremely popular solution to getting a “modern” shell experience is oh-my-zsh. It seems like a great project and I know a lot of people use it very happily, but I’ve struggled with configuration systems like that in the past – it looks like right now the base oh-my-zsh adds about 3000 lines of config, and often I find that having an extra configuration system makes it harder to debug what’s happening when things go wrong. I personally have a tendency to use the system to add a lot of extra plugins, make my system slow, get frustrated that it’s slow, and then delete it completely and write a new config from scratch.

issue 2 with getting to a “modern” experience: the text editor

In the terminal survey I ran recently, the most popular terminal text editors by far were vim, emacs, and nano.

I think the main options for terminal text editors are:

use vim or emacs and configure it to your liking, you can probably have any feature you want if you put in the work
use nano and accept that you’re going to have a pretty limited experience (for example I don’t think you can select text with the mouse and then “cut” it in nano)
use micro or helix which seem to offer a pretty good out-of-the-box experience, potentially occasionally run into issues with using a less mainstream text editor
just avoid using a terminal text editor as much as possible, maybe use VSCode, use VSCode’s terminal for all your terminal needs, and mostly never edit files in the terminal. Or I know a lot of people use code as their EDITOR in the terminal.

issue 3: individual applications

The last issue is that sometimes individual programs that I use are kind of annoying. For example on my Mac OS machine, /usr/bin/sqlite3 doesn’t support the Ctrl+Left Arrow keyboard shortcut. Fixing this to get a reasonable terminal experience in SQLite was a little complicated, I had to:

realize why this is happening (Mac OS won’t ship GNU tools, and “Ctrl-Left arrow” support comes from GNU readline)
find a workaround (install sqlite from homebrew, which does have readline support)
adjust my environment (put Homebrew’s sqlite3 in my PATH)

I find that debugging application-specific issues like this is really not easy and often it doesn’t feel “worth it” – often I’ll end up just dealing with various minor inconveniences because I don’t want to spend hours investigating them. The only reason I was even able to figure this one out at all is that I’ve been spending a huge amount of time thinking about the terminal recently.

A big part of having a “modern” experience using terminal programs is just using newer terminal programs, for example I can’t be bothered to learn a keyboard shortcut to sort the columns in top, but in htop I can just click on a column heading with my mouse to sort it. So I use htop instead! But discovering new more “modern” command line tools isn’t easy (though I made a list here), finding ones that I actually like using in practice takes time, and if you’re SSHed into another machine, they won’t always be there.

everything affects everything else

Something I find tricky about configuring my terminal to make everything “nice” is that changing one seemingly small thing about my workflow can really affect everything else. For example right now I don’t use tmux. But if I needed to use tmux again (for example because I was doing a lot of work SSHed into another machine), I’d need to think about a few things, like:

if I wanted tmux’s copy to synchronize with my system clipboard over SSH, I’d need to make sure that my terminal emulator has OSC 52 support
if I wanted to use iTerm’s tmux integration (which makes tmux tabs into iTerm tabs), I’d need to change how I configure colours – right now I set them with a shell script that I run when my shell starts, but that means the colours get lost when restoring a tmux session.

and probably more things I haven’t thought of. “Using tmux means that I have to change how I manage my colours” sounds unlikely, but that really did happen to me and I decided “well, I don’t want to change how I manage colours right now, so I guess I’m not using that feature!”.

It’s also hard to remember which features I’m relying on – for example maybe my current terminal does have OSC 52 support and because copying from tmux over SSH has always Just Worked I don’t even realize that that’s something I need, and then it mysteriously stops working when I switch terminals.

change things slowly

Personally even though I think my setup is not that complicated, it’s taken me 20 years to get to this point! Because terminal config changes are so likely to have unexpected and hard-to-understand consequences, I’ve found that if I change a lot of terminal configuration all at once it makes it much harder to understand what went wrong if there’s a problem, which can be really disorienting.

So I usually prefer to make pretty small changes, and accept that changes can might take me a REALLY long time to get used to. For example I switched from using ls to eza
a year or two ago and while I like it (because eza -l prints human-readable file sizes by default) I’m still not quite sure about it. But also sometimes it’s worth it to make a big change, like I made the switch to fish (from bash) 10 years ago and I’m very happy I did.

getting a “modern” terminal is not that easy

Trying to explain how “easy” it is to configure your terminal really just made me think that it’s kind of hard and that I still sometimes get confused.

I’ve found that there’s never one perfect way to configure things in the terminal that will be compatible with every single other thing. I just need to try stuff, figure out some kind of locally stable state that works for me, and accept that if I start using a new tool it might disrupt the system and I might need to rethink things.

"Rules" that terminal programs follow

Thu, 12 Dec 2024 09:28:22 +0000

Recently I’ve been thinking about how everything that happens in the terminal is some combination of:

Your operating system’s job
Your shell’s job
Your terminal emulator’s job
The job of whatever program you happen to be running (like top or vim or cat)

The first three (your operating system, shell, and terminal emulator) are all kind of known quantities – if you’re using bash in GNOME Terminal on Linux, you can more or less reason about how how all of those things interact, and some of their behaviour is standardized by POSIX.

But the fourth one (“whatever program you happen to be running”) feels like it could do ANYTHING. How are you supposed to know how a program is going to behave?

This post is kind of long so here’s a quick table of contents:

programs behave surprisingly consistently

As far as I know, there are no real standards for how programs in the terminal should behave – the closest things I know of are:

POSIX, which mostly dictates how your terminal emulator / OS / shell should work together. I think it does specify a few things about how core utilities like cp should work but AFAIK it doesn’t have anything to say about how for example htop should behave.
these command line interface guidelines

But even though there are no standards, in my experience programs in the terminal behave in a pretty consistent way. So I wanted to write down a list of “rules” that in my experience programs mostly follow.

these are meant to be descriptive, not prescriptive

My goal here isn’t to convince authors of terminal programs that they should follow any of these rules. There are lots of exceptions to these and often there’s a good reason for those exceptions.

But it’s very useful for me to know what behaviour to expect from a random new terminal program that I’m using. Instead of “uh, programs could do literally anything”, it’s “ok, here are the basic rules I expect, and then I can keep a short mental list of exceptions”.

So I’m just writing down what I’ve observed about how programs behave in my 20 years of using the terminal, why I think they behave that way, and some examples of cases where that rule is “broken”.

it’s not always obvious which “rules” are the program’s responsibility to implement

There are a bunch of common conventions that I think are pretty clearly the program’s responsibility to implement, like:

config files should go in ~/.BLAHrc or ~/.config/BLAH/FILE or /etc/BLAH/ or something
--help should print help text
programs should print “regular” output to stdout and errors to stderr

But in this post I’m going to focus on things that it’s not 100% obvious are the program’s responsibility. For example it feels to me like a “law of nature” that pressing Ctrl-D should quit a REPL, but programs often need to explicitly implement support for it – even though cat doesn’t need to implement Ctrl-D support, ipython does. (more about that in “rule 3” below)

Understanding which things are the program’s responsibility makes it much less surprising when different programs’ implementations are slightly different.

rule 1: noninteractive programs should quit when you press `Ctrl-C`

The main reason for this rule is that noninteractive programs will quit by default on Ctrl-C if they don’t set up a SIGINT signal handler, so this is kind of a “you should act like the default” rule.

Something that trips a lot of people up is that this doesn’t apply to interactive programs like python3 or bc or less. This is because in an interactive program, Ctrl-C has a different job – if the program is running an operation (like for example a search in less or some Python code in python3), then Ctrl-C will interrupt that operation but not stop the program.

As an example of how this works in an interactive program: here’s the code in prompt-toolkit (the library that iPython uses for handling input) that aborts a search when you press Ctrl-C.

rule 2: TUIs should quit when you press `q`

TUI programs (like less or htop) will usually quit when you press q.

This rule doesn’t apply to any program where pressing q to quit wouldn’t make sense, like tmux or text editors.

rule 3: REPLs should quit when you press `Ctrl-D` on an empty line

REPLs (like python3 or ed) will usually quit when you press Ctrl-D on an empty line. This rule is similar to the Ctrl-C rule – the reason for this is that by default if you’re running a program (like cat) in “cooked mode”, then the operating system will return an EOF when you press Ctrl-D on an empty line.

Most of the REPLs I use (sqlite3, python3, fish, bash, etc) don’t actually use cooked mode, but they all implement this keyboard shortcut anyway to mimic the default behaviour.

For example, here’s the code in prompt-toolkit that quits when you press Ctrl-D, and here’s the same code in readline.

I actually thought that this one was a “Law of Terminal Physics” until very recently because I’ve basically never seen it broken, but you can see that it’s just something that each individual input library has to implement in the links above.

Someone pointed out that the Erlang REPL does not quit when you press Ctrl-D, so I guess not every REPL follows this “rule”.

rule 4: don’t use more than 16 colours

Terminal programs rarely use colours other than the base 16 ANSI colours. This is because if you specify colours with a hex code, it’s very likely to clash with some users’ background colour. For example if I print out some text as #EEEEEE, it would be almost invisible on a white background, though it would look fine on a dark background.

But if you stick to the default 16 base colours, you have a much better chance that the user has configured those colours in their terminal emulator so that they work reasonably well with their background color. Another reason to stick to the default base 16 colours is that it makes less assumptions about what colours the terminal emulator supports.

The only programs I usually see breaking this “rule” are text editors, for example Helix by default will use a purple background which is not a default ANSI colour. It seems fine for Helix to break this rule since Helix isn’t a “core” program and I assume any Helix user who doesn’t like that colorscheme will just change the theme.

rule 5: vaguely support readline keybindings

Almost every program I use supports readline keybindings if it would make sense to do so. For example, here are a bunch of different programs and a link to where they define Ctrl-E to go to the end of the line:

ipython (Ctrl-E defined here)
atuin (Ctrl-E defined here)
fzf (Ctrl-E defined here)
zsh (Ctrl-E defined here)
fish (Ctrl-E defined here)
tmux’s command prompt (Ctrl-E defined here)

None of those programs actually uses readline directly, they just sort of mimic emacs/readline keybindings. They don’t always mimic them exactly: for example atuin seems to use Ctrl-A as a prefix, so Ctrl-A doesn’t go to the beginning of the line.

Also all of these programs seem to implement their own internal cut and paste buffers so you can delete a line with Ctrl-U and then paste it with Ctrl-Y.

The exceptions to this are:

some programs (like git, cat, and nc) don’t have any line editing support at all (except for backspace, Ctrl-W, and Ctrl-U)
as usual text editors are an exception, every text editor has its own approach to editing text

I wrote more about this “what keybindings does a program support?” question in entering text in the terminal is complicated.

rule 5.1: Ctrl-W should delete the last word

I’ve never seen a program (other than a text editor) where Ctrl-W doesn’t delete the last word. This is similar to the Ctrl-C rule – by default if a program is in “cooked mode”, the OS will delete the last word if you press Ctrl-W, and delete the whole line if you press Ctrl-U. So usually programs will imitate that behaviour.

I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!

rule 6: disable colours when writing to a pipe

Most programs will disable colours when writing to a pipe. For example:

rg blah will highlight all occurrences of blah in the output, but if the output is to a pipe or a file, it’ll turn off the highlighting.
ls --color=auto will use colour when writing to a terminal, but not when writing to a pipe

Both of those programs will also format their output differently when writing to the terminal: ls will organize files into columns, and ripgrep will group matches with headings.

If you want to force the program to use colour (for example because you want to look at the colour), you can use unbuffer to force the program’s output to be a tty like this:

unbuffer rg blah |  less -R

I’m sure that there are some programs that “break” this rule but I can’t think of any examples right now. Some programs have an --color flag that you can use to force colour to be on, in the example above you could also do rg --color=always | less -R.

rule 7: `-` means stdin/stdout

Usually if you pass - to a program instead of a filename, it’ll read from stdin or write to stdout (whichever is appropriate). For example, if you want to format the Python code that’s on your clipboard with black and then copy it, you could run:

pbpaste | black - | pbcopy

(pbpaste is a Mac program, you can do something similar on Linux with xclip)

My impression is that most programs implement this if it would make sense and I can’t think of any exceptions right now, but I’m sure there are many exceptions.

these “rules” take a long time to learn

These rules took me a long time for me to learn because I had to:

learn that the rule applied anywhere at all ("Ctrl-C will exit programs")
notice some exceptions (“okay, Ctrl-C will exit find but not less”)
subconsciously figure out what the pattern is ("Ctrl-C will generally quit noninteractive programs, but in interactive programs it might interrupt the current operation instead of quitting the program")
eventually maybe formulate it into an explicit rule that I know

A lot of my understanding of the terminal is honestly still in the “subconscious pattern recognition” stage. The only reason I’ve been taking the time to make things explicit at all is because I’ve been trying to explain how it works to others. Hopefully writing down these “rules” explicitly will make learning some of this stuff a little bit faster for others.

Why pipes sometimes get "stuck": buffering

Fri, 29 Nov 2024 08:23:31 +0000

Here’s a niche terminal problem that has bothered me for years but that I never really understood until a few weeks ago. Let’s say you’re running this command to watch for some specific output in a log file:

tail -f /some/log/file | grep thing1 | grep thing2

If log lines are being added to the file relatively slowly, the result I’d see is… nothing! It doesn’t matter if there were matches in the log file or not, there just wouldn’t be any output.

I internalized this as “uh, I guess pipes just get stuck sometimes and don’t show me the output, that’s weird”, and I’d handle it by just running grep thing1 /some/log/file | grep thing2 instead, which would work.

So as I’ve been doing a terminal deep dive over the last few months I was really excited to finally learn exactly why this happens.

why this happens: buffering

The reason why “pipes get stuck” sometimes is that it’s VERY common for programs to buffer their output before writing it to a pipe or file. So the pipe is working fine, the problem is that the program never even wrote the data to the pipe!

This is for performance reasons: writing all output immediately as soon as you can uses more system calls, so it’s more efficient to save up data until you have 8KB or so of data to write (or until the program exits) and THEN write it to the pipe.

In this example:

tail -f /some/log/file | grep thing1 | grep thing2

the problem is that grep thing1 is saving up all of its matches until it has 8KB of data to write, which might literally never happen.

programs don’t buffer when writing to a terminal

Part of why I found this so disorienting is that tail -f file | grep thing will work totally fine, but then when you add the second grep, it stops working!! The reason for this is that the way grep handles buffering depends on whether it’s writing to a terminal or not.

Here’s how grep (and many other programs) decides to buffer its output:

Check if stdout is a terminal or not using the isatty function
- If it’s a terminal, use line buffering (print every line immediately as soon as you have it)
- Otherwise, use “block buffering” – only print data if you have at least 8KB or so of data to print

So if grep is writing directly to your terminal then you’ll see the line as soon as it’s printed, but if it’s writing to a pipe, you won’t.

Of course the buffer size isn’t always 8KB for every program, it depends on the implementation. For grep the buffering is handled by libc, and libc’s buffer size is defined in the BUFSIZ variable. Here’s where that’s defined in glibc.

(as an aside: “programs do not use 8KB output buffers when writing to a terminal” isn’t, like, a law of terminal physics, a program COULD use an 8KB buffer when writing output to a terminal if it wanted, it would just be extremely weird if it did that, I can’t think of any program that behaves that way)

commands that buffer & commands that don’t

One annoying thing about this buffering behaviour is that you kind of need to remember which commands buffer their output when writing to a pipe.

Some commands that don’t buffer their output:

tail
cat
tee

I think almost everything else will buffer output, especially if it’s a command where you’re likely to be using it for batch processing. Here’s a list of some common commands that buffer their output when writing to a pipe, along with the flag that disables block buffering.

grep (--line-buffered)
sed (-u)
awk (there’s a fflush() function)
tcpdump (-l)
jq (-u)
tr (-u)
cut (can’t disable buffering)

Those are all the ones I can think of, lots of unix commands (like sort) may or may not buffer their output but it doesn’t matter because sort can’t do anything until it finishes receiving input anyway.

Also I did my best to test both the Mac OS and GNU versions of these but there are a lot of variations and I might have made some mistakes.

programming languages where the default “print” statement buffers

Also, here are a few programming language where the default print statement will buffer output when writing to a pipe, and some ways to disable buffering if you want:

C (disable with setvbuf)
Python (disable with python -u, or PYTHONUNBUFFERED=1, or sys.stdout.reconfigure(line_buffering=False), or print(x, flush=True))
Ruby (disable with STDOUT.sync = true)
Perl (disable with $| = 1)

I assume that these languages are designed this way so that the default print function will be fast when you’re doing batch processing.

Also whether output is buffered or not might depend on how you print, for example in C++ cout << "hello\n" buffers when writing to a pipe but cout << "hello" << endl will flush its output.

when you press `Ctrl-C` on a pipe, the contents of the buffer are lost

Let’s say you’re running this command as a hacky way to watch for DNS requests to example.com, and you forgot to pass -l to tcpdump:

sudo tcpdump -ni any port 53 | grep example.com

When you press Ctrl-C, what happens? In a magical perfect world, what I would want to happen is for tcpdump to flush its buffer, grep would search for example.com, and I would see all the output I missed.

But in the real world, what happens is that all the programs get killed and the output in tcpdump’s buffer is lost.

I think this problem is probably unavoidable – I spent a little time with strace to see how this works and grep receives the SIGINT before tcpdump anyway so even if tcpdump tried to flush its buffer grep would already be dead.

After a little more investigation, there is a workaround: if you find tcpdump’s PID and kill -TERM $PID, then tcpdump will flush the buffer so you can see the output. That’s kind of a pain but I tested it and it seems to work.

redirecting to a file also buffers

It’s not just pipes, this will also buffer:

sudo tcpdump -ni any port 53 > output.txt

Redirecting to a file doesn’t have the same “Ctrl-C will totally destroy the contents of the buffer” problem though – in my experience it usually behaves more like you’d want, where the contents of the buffer get written to the file before the program exits. I’m not 100% sure whether this is something you can always rely on or not.

a bunch of potential ways to avoid buffering

Okay, let’s talk solutions. Let’s say you’ve run this command:

tail -f /some/log/file | grep thing1 | grep thing2

I asked people on Mastodon how they would solve this in practice and there were 5 basic approaches. Here they are:

solution 1: run a program that finishes quickly

Historically my solution to this has been to just avoid the “command writing to pipe slowly” situation completely and instead run a program that will finish quickly like this:

cat /some/log/file | grep thing1 | grep thing2 | tail

This doesn’t do the same thing as the original command but it does mean that you get to avoid thinking about these weird buffering issues.

(you could also do grep thing1 /some/log/file but I often prefer to use an “unnecessary” cat)

solution 2: remember the “line buffer” flag to grep

You could remember that grep has a flag to avoid buffering and pass it like this:

tail -f /some/log/file | grep --line-buffered thing1 | grep thing2

solution 3: use awk

Some people said that if they’re specifically dealing with a multiple greps situation, they’ll rewrite it to use a single awk instead, like this:

tail -f /some/log/file |  awk '/thing1/ && /thing2/'

Or you would write a more complicated grep, like this:

tail -f /some/log/file |  grep -E 'thing1.*thing2'

(awk also buffers, so for this to work you’ll want awk to be the last command in the pipeline)

solution 4: use `stdbuf`

stdbuf uses LD_PRELOAD to turn off libc’s buffering, and you can use it to turn off output buffering like this:

tail -f /some/log/file | stdbuf -o0 grep thing1 | grep thing2

Like any LD_PRELOAD solution it’s a bit unreliable – it doesn’t work on static binaries, I think won’t work if the program isn’t using libc’s buffering, and doesn’t always work on Mac OS. Harry Marr has a really nice How stdbuf works post.

solution 5: use `unbuffer`

unbuffer program will force the program’s output to be a TTY, which means that it’ll behave the way it normally would on a TTY (less buffering, colour output, etc). You could use it in this example like this:

tail -f /some/log/file | unbuffer grep thing1 | grep thing2

Unlike stdbuf it will always work, though it might have unwanted side effects, for example grep thing1’s will also colour matches.

If you want to install unbuffer, it’s in the expect package.

that’s all the solutions I know about!

It’s a bit hard for me to say which one is “best”, I think personally I’m mostly likely to use unbuffer because I know it’s always going to work.

If I learn about more solutions I’ll try to add them to this post.

I’m not really sure how often this comes up

I think it’s not very common for me to have a program that slowly trickles data into a pipe like this, normally if I’m using a pipe a bunch of data gets written very quickly, processed by everything in the pipeline, and then everything exits. The only examples I can come up with right now are:

tcpdump
tail -f
watching log files in a different way like with kubectl logs
the output of a slow computation

what if there were an environment variable to disable buffering?

I think it would be cool if there were a standard environment variable to turn off buffering, like PYTHONUNBUFFERED in Python. I got this idea from a couple of blog posts by Mark Dominus in 2018. Maybe NO_BUFFER like NO_COLOR?

The design seems tricky to get right; Mark points out that NETBSD has environment variables called STDBUF, STDBUF1, etc which gives you a ton of control over buffering but I imagine most developers don’t want to implement many different environment variables to handle a relatively minor edge case.

I’m also curious about whether there are any programs that just automatically flush their output buffers after some period of time (like 1 second). It feels like it would be nice in theory but I can’t think of any program that does that so I imagine there are some downsides.

stuff I left out

Some things I didn’t talk about in this post since these posts have been getting pretty long recently and seriously does anyone REALLY want to read 3000 words about buffering?

the difference between line buffering and having totally unbuffered output
how buffering to stderr is different from buffering to stdout
this post is only about buffering that happens inside the program, your operating system’s TTY driver also does a little bit of buffering sometimes
other reasons you might need to flush your output other than “you’re writing to a pipe”

Importing a frontend Javascript library without a build system

Mon, 18 Nov 2024 09:35:42 +0000

I like writing Javascript without a build system and for the millionth time yesterday I ran into a problem where I needed to figure out how to import a Javascript library in my code without using a build system, and it took FOREVER to figure out how to import it because the library’s setup instructions assume that you’re using a build system.

Luckily at this point I’ve mostly learned how to navigate this situation and either successfully use the library or decide it’s too difficult and switch to a different library, so here’s the guide I wish I had to importing Javascript libraries years ago.

I’m only going to talk about using Javacript libraries on the frontend, and only about how to use them in a no-build-system setup.

In this post I’m going to talk about:

the three main types of Javascript files a library might provide (ES Modules, the “classic” global variable kind, and CommonJS)
how to figure out which types of files a Javascript library includes in its build
ways to import each type of file in your code

the three kinds of Javascript files

There are 3 basic types of Javascript files a library can provide:

the “classic” type of file that defines a global variable. This is the kind of file that you can just <script src> and it’ll Just Work. Great if you can get it but not always available
an ES module (which may or may not depend on other files, we’ll get to that)
a “CommonJS” module. This is for Node, you can’t use it in a browser at all without using a build system.

I’m not sure if there’s a better name for the “classic” type but I’m just going to call it “classic”. Also there’s a type called “AMD” but I’m not sure how relevant it is in 2024.

Now that we know the 3 types of files, let’s talk about how to figure out which of these the library actually provides!

where to find the files: the NPM build

Every Javascript library has a build which it uploads to NPM. You might be thinking (like I did originally) – Julia! The whole POINT is that we’re not using Node to build our library! Why are we talking about NPM?

But if you’re using a link from a CDN like https://cdnjs.cloudflare.com/ajax/libs/Chart.js/4.4.1/chart.umd.min.js, you’re still using the NPM build! All the files on the CDNs originally come from NPM.

Because of this, I sometimes like to npm install the library even if I’m not planning to use Node to build my library at all – I’ll just create a new temp folder, npm install there, and then delete it when I’m done. I like being able to poke around in the files in the NPM build on my filesystem, because then I can be 100% sure that I’m seeing everything that the library is making available in its build and that the CDN isn’t hiding something from me.

So let’s npm install a few libraries and try to figure out what types of Javascript files they provide in their builds!

example library 1: chart.js

First let’s look inside Chart.js, a plotting library.

$ cd /tmp/whatever
$ npm install chart.js
$ cd node_modules/chart.js/dist
$ ls *.*js
chart.cjs  chart.js  chart.umd.js  helpers.cjs  helpers.js

This library seems to have 3 basic options:

option 1: chart.cjs. The .cjs suffix tells me that this is a CommonJS file, for using in Node. This means it’s impossible to use it directly in the browser without some kind of build step.

option 2:chart.js. The .js suffix by itself doesn’t tell us what kind of file it is, but if I open it up, I see import '@kurkle/color'; which is an immediate sign that this is an ES module – the import ... syntax is ES module syntax.

option 3: chart.umd.js. “UMD” stands for “Universal Module Definition”, which I think means that you can use this file either with a basic <script src>, CommonJS, or some third thing called AMD that I don’t understand.

how to use a UMD file

When I was using Chart.js I picked Option 3. I just needed to add this to my code:

<script src="./chart.umd.js"> </script>

and then I could use the library with the global Chart environment variable. Couldn’t be easier. I just copied chart.umd.js into my Git repository so that I didn’t have to worry about using NPM or the CDNs going down or anything.

the build files aren’t always in the `dist` directory

A lot of libraries will put their build in the dist directory, but not always! The build files’ location is specified in the library’s package.json.

For example here’s an excerpt from Chart.js’s package.json.

  "jsdelivr": "./dist/chart.umd.js",
  "unpkg": "./dist/chart.umd.js",
  "main": "./dist/chart.cjs",
  "module": "./dist/chart.js",

I think this is saying that if you want to use an ES Module (module) you should use dist/chart.js, but the jsDelivr and unpkg CDNs should use ./dist/chart.umd.js. I guess main is for Node.

chart.js’s package.json also says "type": "module", which according to this documentation tells Node to treat files as ES modules by default. I think it doesn’t tell us specifically which files are ES modules and which ones aren’t but it does tell us that something in there is an ES module.

example library 2: `@atcute/oauth-browser-client`

@atcute/oauth-browser-client is a library for logging into Bluesky with OAuth in the browser.

Let’s see what kinds of Javascript files it provides in its build!

$ npm install @atcute/oauth-browser-client
$ cd node_modules/@atcute/oauth-browser-client/dist
$ ls *js
constants.js  dpop.js  environment.js  errors.js  index.js  resolvers.js

It seems like the only plausible root file in here is index.js, which looks something like this:

export { configureOAuth } from './environment.js';
export * from './errors.js';
export * from './resolvers.js';

This export syntax means it’s an ES module. That means we can use it in the browser without a build step! Let’s see how to do that.

how to use an ES module with importmaps

Using an ES module isn’t an easy as just adding a <script src="whatever.js">. Instead, if the ES module has dependencies (like @atcute/oauth-browser-client does) the steps are:

Set up an import map in your HTML
Put import statements like import { configureOAuth } from '@atcute/oauth-browser-client'; in your JS code
Include your JS code in your HTML like this: <script type="module" src="YOURSCRIPT.js"></script>

The reason we need an import map instead of just doing something like import { BrowserOAuthClient } from "./oauth-client-browser.js" is that internally the module has more import statements like import {something} from @atcute/client, and we need to tell the browser where to get the code for @atcute/client and all of its other dependencies.

Here’s what the importmap I used looks like for @atcute/oauth-browser-client:

<script type="importmap">
{
  "imports": {
    "nanoid": "./node_modules/nanoid/bin/dist/index.js",
    "nanoid/non-secure": "./node_modules/nanoid/non-secure/index.js",
    "nanoid/url-alphabet": "./node_modules/nanoid/url-alphabet/dist/index.js",
    "@atcute/oauth-browser-client": "./node_modules/@atcute/oauth-browser-client/dist/index.js",
    "@atcute/client": "./node_modules/@atcute/client/dist/index.js",
    "@atcute/client/utils/did": "./node_modules/@atcute/client/dist/utils/did.js"
  }
}
</script>

Getting these import maps to work is pretty fiddly, I feel like there must be a tool to generate them automatically but I haven’t found one yet. It’s definitely possible to write a script that automatically generates the importmaps using esbuild’s metafile but I haven’t done that and maybe there’s a better way.

I decided to set up importmaps yesterday to get github.com/jvns/bsky-oauth-example to work, so there’s some example code in that repo.

Also someone pointed me to Simon Willison’s download-esm, which will download an ES module and rewrite the imports to point to the JS files directly so that you don’t need importmaps. I haven’t tried it yet but it seems like a great idea.

problems with importmaps: too many files

I did run into some problems with using importmaps in the browser though – it needed to download dozens of Javascript files to load my site, and my webserver in development couldn’t keep up for some reason. I kept seeing files fail to load randomly and then had to reload the page and hope that they would succeed this time.

It wasn’t an issue anymore when I deployed my site to production, so I guess it was a problem with my local dev environment.

Also one slightly annoying thing about ES modules in general is that you need to be running a webserver to use them, I’m sure this is for a good reason but it’s easier when you can just open your index.html file without starting a webserver.

Because of the “too many files” thing I think actually using ES modules with importmaps in this way isn’t actually that appealing to me, but it’s good to know it’s possible.

how to use an ES module without importmaps

If the ES module doesn’t have dependencies then it’s even easier – you don’t need the importmaps! You can just:

put <script type="module" src="YOURCODE.js"></script> in your HTML. The type="module" is important.
put import {whatever} from "https://example.com/whatever.js" in YOURCODE.js

alternative: use esbuild

If you don’t want to use importmaps, you can also use a build system like esbuild. I talked about how to do that in Some notes on using esbuild, but this blog post is about ways to avoid build systems completely so I’m not going to talk about that option here. I do still like esbuild though and I think it’s a good option in this case.

what’s the browser support for importmaps?

CanIUse says that importmaps are in “Baseline 2023: newly available across major browsers” so my sense is that in 2024 that’s still maybe a little bit too new? I think I would use importmaps for some fun experimental code that I only wanted like myself and 12 people to use, but if I wanted my code to be more widely usable I’d use esbuild instead.

example library 3: `@atproto/oauth-client-browser`

Let’s look at one final example library! This is a different Bluesky auth library than @atcute/oauth-browser-client.

$ npm install @atproto/oauth-client-browser
$ cd node_modules/@atproto/oauth-client-browser/dist
$ ls *js
browser-oauth-client.js  browser-oauth-database.js  browser-runtime-implementation.js  errors.js  index.js  indexed-db-store.js  util.js

Again, it seems like only real candidate file here is index.js. But this is a different situation from the previous example library! Let’s take a look at index.js:

There’s a bunch of stuff like this in index.js:

__exportStar(require("@atproto/oauth-client"), exports);
__exportStar(require("./browser-oauth-client.js"), exports);
__exportStar(require("./errors.js"), exports);
var util_js_1 = require("./util.js");

This require() syntax is CommonJS syntax, which means that we can’t use this file in the browser at all, we need to use some kind of build step, and ESBuild won’t work either.

Also in this library’s package.json it says "type": "commonjs" which is another way to tell it’s CommonJS.

how to use a CommonJS module with esm.sh

Originally I thought it was impossible to use CommonJS modules without learning a build system, but then someone Bluesky told me about esm.sh! It’s a CDN that will translate anything into an ES Module. skypack.dev does something similar, I’m not sure what the difference is but one person mentioned that if one doesn’t work sometimes they’ll try the other one.

For @atproto/oauth-client-browser using it seems pretty simple, I just need to put this in my HTML:

<script type="module" src="script.js"> </script>

and then put this in script.js.

import { BrowserOAuthClient } from "https://esm.sh/@atproto/oauth-client-browser@0.3.0"

It seems to Just Work, which is cool! Of course this is still sort of using a build system – it’s just that esm.sh is running the build instead of me. My main concerns with this approach are:

I don’t really trust CDNs to keep working forever – usually I like to copy dependencies into my repository so that they don’t go away for some reason in the future.
I’ve heard of some issues with CDNs having security compromises which scares me.
I don’t really understand what esm.sh is doing.

esbuild can also convert CommonJS modules into ES modules

I also learned that you can also use esbuild to convert a CommonJS module into an ES module, though there are some limitations – the import { BrowserOAuthClient } from syntax doesn’t work. Here’s a github issue about that.

I think the esbuild approach is probably more appealing to me than the esm.sh approach because it’s a tool that I already have on my computer so I trust it more. I haven’t experimented with this much yet though.

summary of the three types of files

Here’s a summary of the three types of JS files you might encounter, options for how to use them, and how to identify them.

Unhelpfully a .js or .min.js file extension could be any of these 3 options, so if the file is something.js you need to do more detective work to figure out what you’re dealing with.

“classic” JS files
- How to use it:: <script src="whatever.js"></script>
- Ways to identify it:
  - The website has a big friendly banner in its setup instructions saying “Use this with a CDN!” or something
  - A .umd.js extension
  - Just try to put it in a <script src=... tag and see if it works
ES Modules
- Ways to use it:
  - If there are no dependencies, just import {whatever} from "./my-module.js" directly in your code
  - If there are dependencies, create an importmap and import {whatever} from "my-module"
    - or use download-esm to remove the need for an importmap
  - Use esbuild or any ES Module bundler
- Ways to identify it:
  - Look for an import or export statement. (not module.exports = ..., that’s CommonJS)
  - An .mjs extension
  - maybe "type": "module" in package.json (though it’s not clear to me which file exactly this refers to)
CommonJS Modules
- Ways to use it:
  - Use https://esm.sh to convert it into an ES module, like https://esm.sh/@atproto/oauth-client-browser@0.3.0
  - Use a build somehow (??)
- Ways to identify it:
  - Look for require() or module.exports = ... in the code
  - A .cjs extension
  - maybe "type": "commonjs" in package.json (though it’s not clear to me which file exactly this refers to)

it’s really nice to have ES modules standardized

The main difference between CommonJS modules and ES modules from my perspective is that ES modules are actually a standard. This makes me feel a lot more confident using them, because browsers commit to backwards compatibility for web standards forever – if I write some code using ES modules today, I can feel sure that it’ll still work the same way in 15 years.

It also makes me feel better about using tooling like esbuild because even if the esbuild project dies, because it’s implementing a standard it feels likely that there will be another similar tool in the future that I can replace it with.

A lot of the time when I talk about this stuff I get responses like “I hate javascript!!! it’s the worst!!!”. But my experience is that there are a lot of great tools for Javascript (I just learned about https://esm.sh yesterday which seems great! I love esbuild!), and that if I take the time to learn how things works I can take advantage of some of those tools and make my life a lot easier.

So the goal of this post is definitely not to complain about Javascript, it’s to understand the landscape so I can use the tooling in a way that feels good to me.

questions I still have

Here are some questions I still have, I’ll add the answers into the post if I learn the answer.

Is there a tool that automatically generates importmaps for an ES Module that I have set up locally? (apparently yes: jspm)
How can I convert a CommonJS module into an ES module on my computer, the way https://esm.sh does? (apparently esbuild can sort of do this, though named exports don’t work)
When people normally build CommonJS modules into regular JS code, what’s code is doing that? Obviously there are tools like webpack, rollup, esbuild, etc, but do those tools all implement their own JS parsers/static analysis? How many JS parsers are there out there?
Is there any way to bundle an ES module into a single file (like atcute-client.js), but so that in the browser I can still import multiple different paths from that file (like both @atcute/client/lexicons and @atcute/client)?

all the tools

Here’s a list of every tool we talked about in this post:

Simon Willison’s download-esm which will download an ES module and convert the imports to point at JS files so you don’t need an importmap
https://esm.sh/ and skypack.dev
esbuild
JSPM can generate importmaps

Writing this post has made me think that even though I usually don’t want to have a build that I run every time I update the project, I might be willing to have a build step (using download-esm or something) that I run only once when setting up the project and never run again except maybe if I’m updating my dependency versions.

that’s all!

Thanks to Marco Rogers who taught me a lot of the things in this post. I’ve probably made some mistakes in this post and I’d love to know what they are – let me know on Bluesky or Mastodon!

New microblog with TILs

Sat, 9 Nov 2024 09:24:29 +0000

I added a new section to this site a couple weeks ago called TIL (“today I learned”).

One kind of thing I like to post on Mastodon/Bluesky is “hey, here’s a cool thing”, like the great SQLite repl litecli, or the fact that cross compiling in Go Just Works and it’s amazing, or cryptographic right answers, or this great diff tool. Usually I don’t want to write a whole blog post about those things because I really don’t have much more to say than “hey this is useful!”

It started to bother me that I didn’t have anywhere to put those things: for example recently I wanted to use diffdiff and I just could not remember what it was called.

the solution: make a new section of this blog

So I quickly made a new folder called /til/, added some custom styling (I wanted to style the posts to look a little bit like a tweet), made a little Rake task to help me create new posts quickly (rake new_til), and set up a separate RSS Feed for it.

I think this new section of the blog might be more for myself than anything, now when I forget the link to Cryptographic Right Answers I can hopefully look it up on the TIL page. (you might think “julia, why not use bookmarks??” but I have been failing to use bookmarks for my whole life and I don’t see that changing ever, putting things in public is for whatever reason much easier for me)

So far it’s been working, often I can actually just make a quick post in 2 minutes which was the goal.

inspired by Simon Willison’s TIL blog

My page is inspired by Simon Willison’s great TIL blog, though my TIL posts are a lot shorter.

I don’t necessarily want everything to be archived

This came about because I spent a lot of time on Twitter, so I’ve been thinking about what I want to do about all of my tweets.

I keep reading the advice to “POSSE” (“post on your own site, syndicate elsewhere”), and while I find the idea appealing in principle, for me part of the appeal of social media is that it’s a little bit ephemeral. I can post polls or questions or observations or jokes and then they can just kind of fade away as they become less relevant.

I find it a lot easier to identify specific categories of things that I actually want to have on a Real Website That I Own:

blog posts here!
comics at https://wizardzines.com/comics/!
now TILs at https://jvns.ca/til/)

and then let everything else be kind of ephemeral.

I really believe in the advice to make email lists though – the first two (blog posts & comics) both have email lists and RSS feeds that people can subscribe to if they want. I might add a quick summary of any TIL posts from that week to the “blog posts from this week” mailing list.