I like to pitch OccamsRecord as “the power tools ActiveRecord forgot”. It’s been rewarding both to build and use. A few other people/companies have used it, and that always blows my mind.

But this article isn’t about using a very cool library I wrote; it’s about testing it. TL;DR we’ll be talking about appraisal, Docker Compose, and bash scripting.

Don’t worry, I have unit tests!

OccamsRecord’s testing to library code ratio is about 2:1. Not bad. And I even ran them! But not against all supported ActiveRecord versions and databases. There were ways, but they were annoying and getting more annoying as versions of databases, database adapters, ActiveRecord, and Ruby itself marched on.

My one-person team

Surprise: it’s me, and my funding is $0.00. A few ideas and PRs have been submitted by others (and I’m grateful), but I’m directing, producing, and staring in 99.999% of this show.

Until recently I outsourced my test runs to a popular, free CI service (hint: it started with T-R-A-V-I-S). It was simple and took something off my plate. Recently they made some changes that deleted my account and put their free tier in question. I get it, “free” isn’t a great business model. No hard feelings, unnamed CI company.

But I faced a decision: figure out how to do much more thorough testing in some CI service while ensuring I didn’t go above my $0.00 budget and hoping they’d never change their tech or policies again…or…do it all myself. I chose myself. My new test matrix stack would be:

So many ActiveRecord versions!

Appraisal allows me to create different versions of my Gemfile, each with different versions of my dependencies, and trivially switch among them.

bundle install
bundle exec appraisal ar-7.0 bundle install
bundle exec appraisal ar-7.0 rake test

It’s great…until you start hitting gem versions that aren’t supported by the version of Ruby you have installed. First I tried my go-to Ruby version managers, ruby-install and chruby. But even those had trouble with older versions (hello, hand-compiling openssl!). Enter Docker Compose.

So many…Ruby versions?

OccamsRecord isn’t doing anything cutting edge in terms of Ruby language features or syntax, so there’s limited value in testing against multiple Ruby versions. But not every version of ActiveRecord that Occams supports runs on every version of Ruby that Occams supports. Datbase adapter gems (pg, sqlite3, mysql2) face the same problem.

I’ve been using “dev containers” for application development for years. It’s a fairly simple pattern: Docker Compose (or Podman) containers with system-level dependencies installed (ruby, make, libpq-dev, etc) but nothing else. Simply mount your source code, install gems onto a persistent volume, and you’ve got an OS-agnostic development environment! Why not do the same for library development?

Occam’s docker-compose.yml defines numerous Ruby containers that mount Occams’s source code at runtime. Dockerfiles are defined here.

So many databases!

Historically I’ve been terrible at regularly running the tests against all officially supported databases (PostgreSQL, MySQL, and SQLite). The in-memory SQLite DB is so easy to test against, and it covers 90% of cases.

Occam’s docker-compose.yml has a few containers and volumes for Postgres and Mysql. Pretty basic.

Bash and awk-powered matrix

Here’s my test matrix file. Each row defines a Ruby container, an appraisal ActiveRecord version, and one or more databases.

ruby-3.2  ar-7.0  sqlite3  postgres-14  mysql-8
ruby-3.2  ar-6.1  sqlite3  postgres-14  mysql-8
ruby-3.2  ar-6.0  sqlite3  postgres-14  mysql-8

ruby-3.1  ar-7.0  sqlite3  postgres-14  mysql-8
ruby-3.1  ar-6.1  sqlite3  postgres-14  mysql-8
ruby-3.1  ar-6.0  sqlite3  postgres-14  mysql-8

ruby-3.0  ar-7.0  sqlite3  postgres-14  mysql-8
ruby-3.0  ar-6.1  sqlite3  postgres-14  mysql-8
ruby-3.0  ar-6.0  sqlite3  postgres-14  mysql-8

ruby-2.7  ar-7.0  sqlite3  postgres-14  mysql-8
ruby-2.7  ar-6.1  sqlite3  postgres-14  mysql-8
ruby-2.7  ar-6.0  sqlite3  postgres-14  mysql-8
ruby-2.7  ar-5.2  sqlite3  postgres-14  mysql-8
ruby-2.7  ar-5.1  sqlite3  postgres-14  mysql-8
ruby-2.7  ar-5.0  sqlite3  postgres-14  mysql-8
ruby-2.7  ar-4.2  sqlite3  postgres-14  mysql-8

ruby-2.6  ar-6.1  sqlite3  postgres-14  mysql-8
ruby-2.6  ar-6.0  sqlite3  postgres-14  mysql-8
ruby-2.6  ar-5.2  sqlite3  postgres-14  mysql-8
ruby-2.6  ar-5.1  sqlite3  postgres-14  mysql-8
ruby-2.6  ar-5.0  sqlite3  postgres-14  mysql-8
ruby-2.6  ar-4.2  sqlite3  postgres-14  mysql-8

bin/testall

The matrix is interpreted and run by bin/testall. There is zero setup beyond having Docker/Podman installed:

# Test against all versions
bin/testall

# Only run matching rows
bin/testall ruby-3.0
bin/testall ruby-2.7 postgres-14

If all tests pass you’re greeted by an ASCII nyancat! (Weirdly, it’s colored in the terminal but not on the web.)

+      o     +              o
    +             o     +       +
o          +
    o  +           +        +
+        o     o       +        o
-_-_-_-_-_-_-_,------,      o
_-_-_-_-_-_-_-|   /\_/\
-_-_-_-_-_-_-~|__( ^ .^)  +     +
_-_-_-_-_-_-_-""  ""
    +      o         o   +       o
    +         +
o        o         o      o     +
    o           +
+      +     o        o      +

If any run has failures, it will stop. Fix them and keep trying until you see that cat!

NOTE The first run will be slow as it pulls down containers and installs gems.

bin/test

bin/testall calls out to bin/test. It builds the shell commands (bundle install, bundle exec appraisal ar-X rake test, etc) and hands them to a container.

If you want to test exactly one combination of things, you can call it yourself. Again, there is zero setup beyond having Docker installed:

bin/test ruby-3.2 ar-7.0 postgres-14

bin/run

Finally, bin/test calls out to bin/run. It’s a pretty thin wrapper around Docker Compose that runs a given series of commands in a given container.

Should everyone do this?

Not necessarily, particularly for large, well-funded projects. But for everyone else, maybe? Docker Compose and bash scripting are valuable skills, and it’s not much code to own.

It’s pretty portable, too. I wrote it on my MacBook Pro M1 and it ran perfectly on my Ubuntu Dell XPS. If I had a Windows machine, it would run fine in WSL. (That’s the promise of containers, after all). And if I ever want to, it should run fine on GitHub Actions.

Next time I write a gem that needs any kind of testing matrix, this is what I’ll reach for. Feel free to do so, too.

+      o     +              o
    +             o     +       +
o          +
    o  +           +        +
+        o     o       +        o
-_-_-_-_-_-_-_,------,      o
_-_-_-_-_-_-_-|   /\_/\
-_-_-_-_-_-_-~|__( ^ .^)  +     +
_-_-_-_-_-_-_-""  ""
    +      o         o   +       o
    +         +
o        o         o      o     +
    o           +
+      +     o        o      +