alexalejandre.com

Janet on x32: 32-bit Pointers, 64-bit Speed, 25% Less RAM

alexalejandre.com by veqq — Tue, 6 Oct 2026 22:26:46 +0000

It’s possible to save a dramatic amount of memory (close to half for programs with pointer heavy heaps) and also speed up programs by a modest amount (by way of more data fitting in cache) by using 32 bit pointers on 64 bit systems.

The Linux x32 ABI lets you do just this, but it’s tragically underused and overlooked. It’s disabled by default on Debian (though you can enable it with a boot flag) and not even compiled in on Arch Linux. Most software compiles just fine for x32, but there isn’t much packaging for it so you do have to compile nearly everything yourself.

I experimented with deploying mastodon on x32 and it cut the app’s memory usage from 650mb to 350mb. There’s so much potential here, but the work is mostly of the thankless coordination and communication type and I don’t have the time or motivation to push it forward myself. - Hailey

How to Compile for 32 Bits?

Janet libraries like Spork supply their own build flags, but we can hijack them with a fake cc which starts the real one with the necessary -m32 -msse2 -mfpmath=sse flags. I modified the beginning of my default Janet build script:

#!/bin/sh
set -eu

TARGET="$PWD/janet32"
JANET="$TARGET/bin/janet"

mkdir -p "$TARGET/cc32"
printf '#!/bin/sh\nexec gcc -m32 -msse2 -mfpmath=sse "$@"\n' > "$TARGET/cc32/cc"
chmod +x "$TARGET/cc32/cc"
export PATH="$TARGET/cc32:$PATH"
export JANET_TOOLCHAIN=cc

unset JANET_PATH JANET_TREE

mkdir -p "$TARGET"
cd "$TARGET"
rm -rf build/spork janet

git clone https://github.com/janet-lang/janet
cd janet

Easy enough! Now let’s try it!

Failing on Arch

tl;dr: 20% memory reduction, 50% slower

See the Scripts and Results

This is on cachyos (arch, btw) with lib32-glibc lib32-gcc-libs. Adding this to a build script forces libraires like spork to build at 32 bit too!

For mem.janet:

(let [live @[]]
  (for i 0 2_000_000
    (array/push live [i (+ i 1)]))
  (gccollect)
  (print (length live)))

Note, these use Fish and I don’t want to rebuild my site with other language support:

sudo pacman -S time # different from just time
command time -v ./bin/janet mem.janet # command skips the fish version
command time -v ./janet32/bin/janet mem.janet

2000000
       Command being timed: "./bin/janet mem.janet"
       User time (seconds): 0.24
       System time (seconds): 0.05
       Percent of CPU this job got: 100%
       Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.30
       Average shared text size (kbytes): 0
       Average unshared data size (kbytes): 0
       Average stack size (kbytes): 0
       Average total size (kbytes): 0
       Maximum resident set size (kbytes): 145196
       Average resident set size (kbytes): 0
       Major (requiring I/O) page faults: 0
       Minor (reclaiming a frame) page faults: 33513
       Voluntary context switches: 1
       Involuntary context switches: 9

2000000
       Command being timed: "./janet32/bin/janet mem.janet"
       User time (seconds): 0.47
       System time (seconds): 0.03
       Percent of CPU this job got: 99%
       Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.50
       Average shared text size (kbytes): 0
       Average unshared data size (kbytes): 0
       Average stack size (kbytes): 0
       Average total size (kbytes): 0
       Maximum resident set size (kbytes): 113716
       Average resident set size (kbytes): 0
       Major (requiring I/O) page faults: 0
       Minor (reclaiming a frame) page faults: 25686
       Voluntary context switches: 1
       Involuntary context switches: 21

I added a 0 in the test

command time -v ./bin/janet mem.janet
                  command time -v ./janet32/bin/janet mem.janet
20000000
        Command being timed: "./bin/janet mem.janet"
        User time (seconds): 2.83
        System time (seconds): 0.46
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.31
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1410668
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 316820
        Voluntary context switches: 1
        Involuntary context switches: 164
        Exit status: 0
20000000
        Command being timed: "./janet32/bin/janet mem.janet"
        User time (seconds): 4.64
        System time (seconds): 0.35
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1098372
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 238633
        Voluntary context switches: 1
        Involuntary context switches: 193

For e.g. declarative-dsls/tests.janet

98 passed
        Command being timed: "./bin/janet ./lib/janet/bin/judge declarative-dsls/tests.janet"
        User time (seconds): 0.42
        System time (seconds): 0.02
        Percent of CPU this job got: 100%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.45
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 61900
declarative-dsls/tests.janet

98 passed
        Command being timed: "./bin/janet ./lib/janet/bin/judge declarative-dsls/tests.janet"
        User time (seconds): 0.70
        System time (seconds): 0.02
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.72
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 54108
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 12735

As Hailey said, Arch does not come with the x32 ABI compiled, so I had to use -m32 and while we do get a 20% memory savings, we execute 50% slower because half the registers aren’t used! -m32 is the old 32-bit x86 mode using only 8 registers. To see the speed-up, we need to use -mx32 which uses the 64-bit instruction set with shrunken 32-bit pointers. But I don’t want to recompile Linux…

Unfortunately, Ubuntu also dropped x32 support.

Winning on Deprecated Ubuntu

Luckily, lazily, I have access to some deprecated servers (of course not in prod…) and OS installs, wasting hard disk space:

Ubuntu 20.04 LTS Focal Fossa has reached its end of standard support on 31 May 2025.

Ubuntu, so we have bash again:

grep X86_X32 /boot/config-$(uname -r)  
CONFIG_X86_X32=y

Just what we need, although gcc version 9.4.0 is a bit old. We already have build-essential git gcc-multilib libc6-dev-x32. We are ready to soar! Here is our full 32janet.sh:

#!/bin/bash
set -eu

TARGET="$PWD/janetx32"
JANET="$TARGET/bin/janet"

mkdir -p "$TARGET/cc32"
# this forces Janet C libraries like Spork to build with 32bit also
# FFI build test fails on -mx32 build so -DJANET_NO_FFI
printf '#!/bin/sh\nexec gcc -mx32 -DJANET_NO_FFI "$@"\n' > "$TARGET/cc32/cc"
chmod +x "$TARGET/cc32/cc"
export PATH="$TARGET/cc32:$PATH"
export JANET_TOOLCHAIN=cc

unset JANET_PATH JANET_TREE
mkdir -p "$TARGET"
cd "$TARGET"

git clone https://github.com/janet-lang/janet
cd janet

export CFLAGS='-fPIC -O3 -flto -fno-semantic-interposition -march=native' # speed gains
PREFIX="$TARGET" make clean
PREFIX="$TARGET" make
PREFIX="$TARGET" make test
PREFIX="$TARGET" make install

cd ..
git clone --depth=1 https://github.com/janet-lang/spork build/spork
"$JANET" -e '(if (bundle/installed? "spork") (bundle/replace "spork" "build/spork") (bundle/install "build/spork"))'

# ### Packages
PM="$TARGET/lib/janet/bin/janet-pm"
git clone https://codeberg.org/veqq/declarative-dsls
"$JANET" "$PM" install file::declarative-dsls # also does https://codeberg.org/veqq/varray
git clone https://github.com/ianthehenry/judge # also does https://github.com/ianthehenry/cmd
"$JANET" "$PM" install file::judge

time "$JANET" "$TARGET/lib/janet/bin/judge" declarative-dsls/tests.janet # to show that it worked

And a `64janet.sh to compare it with:

#!/bin/bash
set -eu

TARGET="$PWD/janet64"
JANET="$TARGET/bin/janet"
unset JANET_PATH JANET_TREE
mkdir -p "$TARGET"
cd "$TARGET"

git clone https://github.com/janet-lang/janet
cd janet

export CFLAGS='-fPIC -O3 -flto -fno-semantic-interposition -march=native' # speed gains
PREFIX="$TARGET" make clean
PREFIX="$TARGET" make
PREFIX="$TARGET" make test
PREFIX="$TARGET" make install

cd ..
git clone --depth=1 https://github.com/janet-lang/spork build/spork
"$JANET" -e '(if (bundle/installed? "spork") (bundle/replace "spork" "build/spork") (bundle/install "build/spork"))'

# ### Packages
PM="$TARGET/lib/janet/bin/janet-pm"
git clone https://codeberg.org/veqq/declarative-dsls
"$JANET" "$PM" install file::declarative-dsls # also does https://codeberg.org/veqq/varray
git clone https://github.com/ianthehenry/judge # also does https://github.com/ianthehenry/cmd
"$JANET" "$PM" install file::judge

time "$JANET" "$TARGET/lib/janet/bin/judge" declarative-dsls/tests.janet # to show that it worked

Now we compare them:

sudo apt install time # different from just time
/usr/bin/time -f '%e s  %M KB' janet64/bin/janet janet64/lib/janet/bin/judge janet64/declarative-dsls/tests.janet
/usr/bin/time -f '%e s  %M KB' janetx32/bin/janet janetx32/lib/janet/bin/judge janetx32/declarative-dsls/tests.janet

2.42 s  61668 KB # 64bit
2.36 s  53472 KB # 32bit

But declarative-dsls is a worst case scenario, using typed c arrays for big speed ups, which don’t benefit much from this. Lets make some minimal benchmarks to really test the differences:

See the Benchmarks Scripts

words.janet builds a text, splits it into words, counts them and sorts on that:

(let [start (os/clock :monotonic)
      rng (math/rng 42)
      vocab (seq [i :range [0 5000]] (string "w" (math/rng-int rng 100000)))
      text @""
      counts @{}]
  (repeat 600000 (buffer/push text (in vocab (math/rng-int rng 5000)) " "))
  (each word (string/split " " text)
    (put counts word (+ 1 (get counts word 0))))
  (pp (take 3 (sort-by |(- (in $ 1)) (pairs counts))))
  (print (length counts) " distinct words")
  (print (- (os/clock :monotonic) start) " s"))

records.janet groups structs by department and summarizes those departments, with unused extra fields like real code:

(let [start (os/clock :monotonic)
      rng (math/rng 7)
      depts [:eng :ops :sales :legal :support]
      people (seq [i :range [0 300000]]
               {:id i
               # :name and :tags not used
                :name (string "person-" i)
                :dept (in depts (math/rng-int rng 5))
                :salary (+ 30000 (math/rng-int rng 90000))
                :tags [(math/rng-int rng 10) (math/rng-int rng 10)]})
      by-dept (group-by |(in $ :dept) people)]
  (each dept (sorted (keys by-dept))
    (let [staff (in by-dept dept)
          salaries (map |(in $ :salary) staff)]
      (pp [dept
           (length staff)
           (math/round (/ (sum salaries) (length staff)))
           (length (filter |(> $ 100000) salaries))])))
  (print (- (os/clock :monotonic) start) " s"))

parse.janet parses lines with a PEG and totals them:

(let [start (os/clock :monotonic)
      rng (math/rng 3)
      lines (seq [i :range [0 200000]]
              (string i "," (math/rng-int rng 1000) ",item-" (math/rng-int rng 500) "," (/ (math/rng-int rng 10000) 100)))
      row (peg/compile
            ~{:num (number (some (set "0123456789.")))
              :txt (capture (some (if-not "," 1)))
              :main (* :num "," :num "," :txt "," :num -1)})
      rows (map |(peg/match row $) lines)
      totals @{}]
  (each [id qty item price] rows
    (put totals item (+ (get totals item 0) (* qty price))))
  (print (length rows) " rows, " (length totals) " items, total " (math/round (sum (values totals))))
  (print (- (os/clock :monotonic) start) " s"))

tree.janet builds and walks binary trees, our pointer-heaviest example:

(let [start (os/clock :monotonic)
      make (fn make [depth]
             (if (= depth 0)
               [nil nil]
               [(make (- depth 1)) (make (- depth 1))]))
      check (fn check [node]
              (if (in node 0)
                (+ 1 (check (in node 0)) (check (in node 1)))
                1))
      long-lived (make 19)]
  (var total 0)
  (repeat 40 (+= total (check (make 14))))
  (print total " " (check long-lived))
  (print (- (os/clock :monotonic) start) " s"))

We benchmark:

for f in words records tree parse; do  
>   /usr/bin/time -f "$f 64:  %e s  %M KB" janet64/bin/janet $f.janet > /dev/null  
>   /usr/bin/time -f "$f x32: %e s  %M KB" janetx32/bin/janet $f.janet > /dev/null  
> done  
 
words 64:  0.58 s  41904 KB  
words x32: 0.67 s  31856 KB  
records 64:  6.40 s  138164 KB  
records x32: 5.80 s  126916 KB  
tree 64:  1.39 s  50504 KB  
tree x32: 1.50 s  35532 KB  
parse 64:  3.20 s  58504 KB  
parse x32: 3.19 s  44428 KB

words 64:  0.63 s  41984 KB  
words x32: 0.70 s  31976 KB  
records 64:  6.71 s  138240 KB  
records x32: 5.77 s  126844 KB  
tree 64:  1.40 s  50392 KB  
tree x32: 1.48 s  35304 KB  
parse 64:  3.37 s  58592 KB  
parse x32: 3.09 s  44496 KB

words 64:  0.59 s  41756 KB  
words x32: 0.67 s  31788 KB  
records 64:  6.50 s  138384 KB  
records x32: 5.80 s  126960 KB  
tree 64:  1.39 s  50288 KB  
tree x32: 1.43 s  35560 KB  
parse 64:  3.35 s  58520 KB  
parse x32: 3.24 s  44572 KB

words 64:  0.60 s  42012 KB  
words x32: 0.67 s  31896 KB  
records 64:  6.29 s  138448 KB  
records x32: 6.00 s  126832 KB  
tree 64:  1.38 s  50176 KB  
tree x32: 1.43 s  35392 KB  
parse 64:  3.36 s  58384 KB  
parse x32: 3.13 s  44364 KB

Conclusion

-mx32 cut RAM use by 21% (8-32%) with equal speed (-13% slower to 10% faster), while -x32 reduced speed by half. Janet is a worst case where nanboxing already packs values into 8 bytes so the main benefit’s smaller headers. I invite others to test their software and languages similarly. With continued x32 support, we could use this as a default for scripts, utilities, daemons and the like. I thus believe Linux distros should include -mx32 support and daemons and utilities should be compiled down for more efficient idling and day to day computing! It only costs a flag.

P.S. Implications for Janet

If Janet works with smaller object headers, why not shrink them in the implementation for 64-bit use? By my understanding, Janet heap objects use 16 bytes to help the garbage collector:

4 bytes for flags
4 bytes of padding
8 bytes linking to the next object

Reducing these would require a different garbage collection strategy e.g. our own allocator, which would hurt a core Janet use case: embedding in other projects.

Each type then adds:

string: 8 bytes for length and hash
tuple: 8 bytes for length and hash, 8 bytes for source line and column info
array: 8 bytes for count and capacity, 8 bytes for a pointer to the data
struct: 12 bytes for length, hash and capacity, 4 bytes padding and 8 bytes for its prototype’s pointer
table: 12 bytes for count, capacity and how many deleted, 4 bytes padding, 8 bytes for a pointer and 8 for the prototype’s pointer

For structs and tables in particular, we could regain 8 bytes through rearranging (not needing the later padding). Onlt tuples parsed from code need the source information (for errors); a flag could save normal tuples 8 bytes.

Unfortunately, glibc’s malloc adds 8 bytes and rounds to multiples of 16 so structs would remain the same. But tables and (some) tuples would shrunk more than the above indicates!

Down the Rabbit Hole

Kernels must be compiled with CONFIG_X86_X32_ABI providing syscall entry points on Linux (with no Windows and Mac equivalent) which the compilers expose via -mx32.

Good reading:

Questions:

can Zig via zig cc -target x86_64-linux-muslx32 crosscompile for easy distribution?
how much performance benefit remains when using a faster allocator like mimalloc? The points will still be faster, at any rate.

Lobsters Interview with Sjamaan

alexalejandre.com by veqq — Fri, 2 Oct 2026 22:01:27 +0000

Sep 2026 - Alex Alejandre

Peter Bex (website) is a CHICKEN scheme maintainer and professional Clojure developer. We got to know each other on IRC over a few months and discussed:

CHICKEN Scheme and its 6.0.0 release
Scheme’s community and standardization process
Postgres, the horrors of MySQL and (default) SQLite
Clojure

My ongoing core contributions mainly focus on the numerical tower code, keeping the in-core copy of the irregex library up-to-date with the upstream version (of which I’m a co-maintainer), squashing bugs and the odd security fix. I enjoy deep diving into odd corners of a code base to get a better understanding and then improve any weirdnesses I find. - His CHICKEN About

Beginnings

How’d you start computing?

I started computing when I got an old hand-me-down C64 which came with a “learn BASIC” book targeted at kids. Of course, I wanted to write my own computer games (which I never ended up doing). Later I got a real PC for my birthday and discovered my BASIC knowledge sort of transferred (with QBASIC in DOS).

Later, I learned about C, which I used for many years until at uni there was a course which taught Lisp (Scheme, really). We started with The Little Lisper/Schemer, and then SICP. I had a “functional programming course” before that which I almost flunked because it just didn’t make any sense (they used Concurrent Clean, because obviously teachers use their own language to teach regardless of its qualities - a common fault in academia). But the Little Schemer finally made it click. The teacher also showed how to implement objects using nothing but lambdas which I found awesome.

I actually studied AI before all the LLM bullshit. I much preferred the cleverness of the classical AI algorithms like A* search and genetic algorithms, but I haven’t really used them much in practice, only for my studies. I keep thinking I should use a GA for something, but no real use case so far. But even back then it was clear that neural networks were the future, though I found them boring because it’s just a bit of math (which I initially barely understood) and it’s basically a black box. I’m still grateful I studied AI because that’s the reason I came into contact with Lisp.

I might’ve looked into Lisp myself at some point (having been curious about it as one of the “foundational languages”), but I might not have had sufficient gumption to really dig in. After uni I got a job where they were using Rails in those somewhat early days (2006), so I learned Ruby as well. I liked learning Ruby and thought it was cool to see the power of Rails, but later got very frustrated with it because Rails really has strong opinions and my programming style didn’t seem fully compatible with it.

CHICKEN & Scheme

Why CHICKEN?

After that course, I tried using Lisp for every personal project. I first started with Scheme48 but it wasn’t very practical (though very elegant). I remember running into problems with the image not being big enough, running out of memory. I never really liked PLT Scheme (now Racket), coming into first contact with it through DrRacket, which felt very sluggish to me, though I do think DrRacket’s a cool alternative view of what an IDE could do. For example, when you hover over a variable and it shows you a line pointing to the origin of that variable.

CHICKEN was a practical and fast system, with a good community and acceptable license (I had quite a distaste for GPL at the time). CHICKEN is also one of two (as far as we know) implementations that use the Cheney on the MTA technique explained here. There were lots of rough edges, but I think wanting to address those is what enabled me to get so deep into the core. If everything is perfect, there’s not much to do, really!

So I started contributing to CHICKEN with some modest eggs (CHICKEN packages) at first, and eventually the core system. I mostly learned about Lisp internals by doing, mucking around with the CHICKEN core and trying things. I read Queinnec’s Lisp in Small Pieces, which deals with translation to C but leaves a lot undiscussed and distracts with OOP-heaviness. SICP has some good material. Then there’s Appel’s Compiling with Continuations, which is really short and to the point but still manages to be rather comprehensive; I love books like that.

In general, I just enjoy hacking on the core, even if I don’t have that much time to do so these days. I have to stress that I’m a slow learner. Building my understanding of CHICKEN was a process of many many years, and there are still parts of the system I’m not that familiar with (although I know my way around enough to get up to speed if needed).

What do those Lisp books lack?

Compiling with Continuations has only a brief section on the runtime system, so it doesn’t go very deeply into e.g. garbage collection and data representation. Lisp in Small Pieces doesn’t go into continuation passing style, IIRC. And its data representation isn’t very optimized. I like to blog about cool techniques that are undiscussed nowadays. For example implementing weak references and how to GC them efficiently. Other stuff I rarely see discussed is how to do FFI and cross-module optimizations, separate compilation and cross-compilation (which only a handful of Schemes even support) etc.

What’s Scheme to you?

In general, Scheme, to me, is a very clean language with a minimal core which facilitates experimentation. This is the fundamental tension of the standardization process - production-quality Schemes tend to grow in size, and there is value in standardizing that. But that also takes away the minimalism which makes experimental implementations possible. For example, Felix Winkelmann (@Bunny351, CHICKEN’s original author’s talk about Scheme implementation) once started a Common Lisp (subset) implementation where he experimented a lot with types and flow analysis, resulting in CHICKEN’s type stuff.

What are your thoughts on the overall Scheme ecosystem, R7RS, SRFIs vs. implementation-specific libraries etc.? A few implementations don’t seem to care anymore.

I think the split of R7RS into a small and large language was the right thing to do, as R6RS was reviled by minimalists and found lacking by maximalists. In general, I’m a bit sad for R7RS - if some of the bigger community Schemes are essentially completely ignoring it, they’re doing something wrong IMO. Many, maybe even most Scheme implementations are essentially one-man shows. I suppose CHICKEN is also turning that way again, since we have lost quite a few contributors (mostly due to changing life situations) and it’s hard to attract new ones.

At the same time, the R7RS “large” project sort of went off the deep end, doing its thing without really caring about community buy-in. The churn of the R7RS large is also a bit too fast to keep up with. They’ve pumped out tons of SRFIs in a few years (actually, looking back at it right now it doesn’t appear like it’s that much, but it is certainly a lot faster than SRFIs used to be.) The SRFI process is open to submissions from literally anyone, for better or worse. I’ve noticed there have been a few new contributors to CHICKEN who submitted implementations for some of the newer SRFIs, so that’s good and I’m happy at least some people are bothering to do this.

CHICKEN 6 is base R7RS, but older CHICKEN code will keep working. We still support the old module syntax (which is the “native” one). The R7RS library declaration is essentially syntactic sugar for the core module syntax. R7RS-small is almost fully backwards compatible with R5RS, so there is no conflict there. Porting an egg to CHICKEN 6 usually requires only a few small adjustments because some (non R5RS) identifiers moved around between modules to better fit the R7RS style.

What’s CHICKEN’s development process like?

CHICKEN 4 was “hygienic CHICKEN”, which introduced the module system (and required overhauling the expander). This was all Felix, requiring a lot of deep internal knowledge about how macros interact with modules etc.

CHICKEN 5 was a community effort through and through. It was mostly a sanity and cleanup release where we did a massive reorganization of the modules (what lives where) to make it logical and matching R7RS a bit better. We discussed this during an IRL meetup and continued for the months after. (Community is a strong advantage of CHICKEN!) We also added a numeric tower.

CHICKEN 6 was basically cut off from Felix’s branch to make UTF-8 handling sane and consistent (like Python 2 -> 3, but way less disruptive.) There’s a strict separation between strings and bytevectors, with changes to ports and other I/O as well. We took the opportunity to integrate the R7RS egg into core so it’s more “native”. Strings in the FFI should be more efficient because there’s no needless copying anymore.

CHICKEN 6.0.0 was held back by a bug causing heap corruptions (do view the patch’s description!). We had been looking in the completely wrong spot. These heap corruptions appeared somewhat randomly, but only with the CHICKEN wiki server, not with the plain web server serving simple files or even the entire Awful framework. We strongly suspected the Subversion client library (which the wiki uses as a backing store for the content), and we’d found other issues in there as well (it’s kind of hairy callback-heavy code due to the design of libsvn).

The wiki is a rather small program and the rest of the web stack seemed to be fine, so we suspected the svn client lib. But I whittled down the code of the wiki to almost nothing and it was still failing. When I commented out the URI normalization code (you get redirected when opening a page that’s behind a symlink, so as to get a canonical URL that points to the original file) it suddenly stopped crashing!

That normalization code didn’t appear to do all that much, so we quickly pinpointed it to be read-symbolic-link. A quick glance at the code in the core system confirmed it was totally borked because of a change made for CHICKEN 6’s UTF-8 transition.

Now that 6 has come out, what’s next?

Regarding the goals for CHICKEN, there are several things I’d like to work on. One idea I had is to teach the compiler about unsafe intrinsics using a “prelude”. Because Scheme is a safe but dynamically typed language, there’s some overhead in the intrinsics, say if you call “car” on a non-pair, it throws an exception. If the compiler can deduce that the object you pass in must be a pair (maybe because you checked it before with pair?, or called car or cdr on it before), it replaces the call to an unsafe, unchecked version. But this is all very ad-hoc, and not extensible by the user. My idea was to have a separate definition which splits the unsafe operation from the “typechecking prelude”, which can be inlined at the call site. This way, if multiple checks need to be done, it’s not all or nothing. We can elide the unnecessary checks and only do the necessary ones which might extend to user code, too.

Another idea relates to the way we handle dates and times - we have some stuff in core to access the POSIX functions but it’s messy and (IMO) mostly unusable. The alternative is SRFI-19, which is a beast because it has support for multiple calendar systems, localization etc. Might be nice to have something minimal (maybe English only) in core, so you have a common type that gets used everywhere (handy when sharing objects between libraries without building in a big dependency on SRFI-19). You can then use it for parsing timestamps in common protocols, say.

Postgres & SQLite

What domains do you like or know the most about?

Web stuff: I maintain the HTTP and URI implementations for CHICKEN
Some CHICKEN internals: GC, macro expander and Irregex implementation
Performance optimizations: though not an expert, I’ve done quite a bit and always thoroughly enjoy it
Postgres: Although I haven’t gone deep into the internals, I’m typically the go-to guy for (Postgre)SQL questions in companies I’ve worked at. Funny, because I initially flunked the DB/SQL course at uni and didn’t grok SQL at all
Distributed systems: though I’ve worked on them for 6 years at work, you’ll want to avoid them like the plague if at all possible. It can be hard to reason about the behaviour of the system at large, and you can’t really abstract it away

I’m trying really hard to think of something I’m truly excited about. The biggest positive I see right now is the push for digital sovereignty. I sincerely hope this will change how people deploy tech, maybe in a more mindful manner. More open source, less dependence on foreign (and hopefully big tech in general) products. But vested interests and inertia will be hard to overcome and really bum me out.

Why Postgres?

I properly learned about DBs at a calendar startup using Rails. We had instantiated repeated events in the DB and the event would sometimes need to be updated. At first, we were fetching models in a loop and updating them one by one, excruciatingly slow. We eventually discovered the bulk update (I think you even had to call into the DB directly because Rails didn’t offer that at the time). That made everything click for me - the importance of performance and the usefulness of SQL. At the time, MySQL was still Rails’ default and I got into MySQL character set hell a few times for a CMS we used. Later, another Rails project required such massive amounts of data to be stored (computational fluid dynamics simulation) that MySQL simply crashed every time I tried a bulk import.

Looking into alternatives, I found Postgres handled it without any problems. When I learned that Postgres doesn’t have any of those braindead misfeatures MySQL has. For instance, UTF-8 characters get verified on storage so you can’t get into character set hell like in MySQL so easily, and it actually allows DDL statements in a transaction, so you get transactional migrations that apply atomically. That was a real eye opener at the time. I was sold!

Postgres is a lot more regular and well-behaved on basically anything, and it has no strange limits (e.g. in MySQL, you can’t even put an index on text columns with indeterminate lengths). MySQL allows you to store an empty string in any non-nullable enum column. Makes no goddamn sense to me! The DB is full of footguns like that. I should stop ranting - talking about MySQL really makes my blood boil. Also, I’ve come to rely on more “advanced” features like LISTEN/NOTIFY, array storage, window functions, CTEs etc. However, I’ve found that stored procedures don’t really work that well - “real code” is more flexible as it doesn’t require finicky migrations to keep in sync.

I’m not a big fan of JSON in my relational DBs, but I have been known to use it when storing arbitrary data or actually putting JSON results (from APIs or other stuff) in the DB.

We use DataScript on the client via ClojureScript, but I don’t really grok it and don’t touch that part of the code often enough that it really sticks, so every time I have to deal with it it’s an exercise in frustration.

Even though I grok it nowadays, SQL is a really badly designed language. I’ve seen several projects that try to come up with a better query language, but I’ve given up hope that they will succeed as SQL is too entrenched to get rid of.

Why not SQLite?

I’ve used it a handful of times. The experience was always mostly one of frustration. It feels a lot like MySQL, with unsafe and stupid defaults.

IIRC it’s value-typed and (by default) doesn’t check types, so the type of a column is basically completely ignored. And you can’t alter a column, IIRC (or maybe only a few changes). I even remember a version (maybe WebSQL?) where you can’t even DROP a column. Anyway, it’s not worth any brain cycles for me to deal with that shit.

On REPLs

What do you think of Clojure?

Clojure’s influence is strong on things like Carp and Janet. I wrote about my impression of Clojure on my blog, but in a nutshell I don’t like the “everything is a map” approach and nil punning really turns me off as it makes bugs harder to find. I do like the fact that it has mostly purely functional data structures. I’m not sure I like the syntactic “heaviness” of Clojure - things like [] for vectors and {} for maps. The lack of cons cells is also a bit weird but I see how it simplifies list handling code (even though Clojure does not typically deal with lists much!) Most importantly, it revitalised the interest in Lisps!

In your article on Clojure:

never fully bought into the REPL style of developing. Sure, I experiment all the time in the REPL to try out a new API design or to quickly iterate on some function I’m writing, but my general development style tends more towards the “save and then run the test suite from an xterm”.

Normally, we hear such things from people who think using the REPL means typing into the little terminal box instead of sending code from files into the REPL with a hotkey, so it surprised me to read it from a veteran.

I do consider the REPL an essential tool for experimentation and debugging. But I struggle to keep track of what’s running in the system versus what I see in my editor. With Clojure, you can’t do without the REPL because it’s so doggone slow to start up that it would be impossible to just run something on the CLI over and over. So at work, I spend 100% of my time with the CIDER REPL. I do find myself closing and reconnecting several times a day though, because I can no longer trust the REPL state matches my editor buffers. One thing that gets me every time is if I delete a test from my buffer and then re-run the entire suite, it’s still there in the REPL (obviously), same thing with multimethod implementations. But overall, I really couldn’t live without a REPL.

What do you think of snapshot testing?

Snapshot testing’s an interesting approach. I think we have a few testcases in CHICKEN where we do something like that - we run the compiler and capture the output of the compiler (which is mostly type warnings) and check that it hasn’t changed with a simple diff on the output and expected output. I’ve also used something like this for great effect while refactoring and optimizing code - simply keep reference output in a file. For instance, in one case when working on a project which had no test suite, I used pg_dump to dump an “output table” as a reference and then went to town on the codebase, knowing I would immediately see if my optimized algorithm differed from the original. I also approach in another inherited project without test cases to refactor.

Real Life

What makes you happy?

I’ve mentioned before that I really enjoy performance optimizing code, but I also really enjoy refactoring and investigating vulnerabilities. Systems that are understandable and hackable make me happy.

Outside of programming (which more often frustrates me than makes me happy TBH), my family makes me happy. It’s a great source of joy to just relax and be with my wife and children. I’ve been trying to get back some balance in life, spending more time AFK, and I pay attention to my health a bit more as well.

I’m not super young anymore (43) and reading about age-related issues like sarcopenia made me realize we really tend to neglect our bodies with our sedentary lifestyle, especially us programmers. My mother has osteoporosis and I see how she struggles just doing basic things. I don’t want that for myself, so I’ve picked up weight lifting as a way to combat those age-related issues, so I can become old in a healthy way. The prognosis is for most of us to live up to 90 or so by now, so I’m not even at the halfway point. But these issues start cropping up at around 50-60.

How do you approach raising kids?

I’m still getting my bearings TBH. My kids are only 3 (going on 4) and 15 months. Raising kids is probably the hardest thing I’ve ever done.

I don’t really know yet if I want to teach them programming. I definitely want to raise them tech-sceptical, when they’re big enough, to teach them the dangers of social media and the importance of privacy. If they show an interest, obviously I’d teach them Scheme. It’s the perfect language for teaching! But maybe something like Logo first.