00Where this stands

This revision records what has actually been executed, with measured results. Figures throughout have been replaced with real ones; where a first-pass estimate turned out wrong, the correction is called out rather than quietly overwritten.

Corrections to revision 1

01What is running today

The instance is healthy and serving — /health returns 200 on ports 3000, 4000 and public HTTPS — but every layer of it is three years old, and the checkout is not on a release tag.

Other findings from the inspection

• There were no database backups at all. /var/backups held only routine OS state. addressed — one verified backup now exists, but there is still no recurring backup and no off-box copy.
• The media purge job has never run. The mastodon crontab contains 0 5 * * 1 purge-media.sh — a bare filename with no path, which cron cannot resolve. The script also calls bin/tootctl without the rbenv shims on PATH. Still unfixed; see section 07.
• nginx serves static assets straight off disk. /etc/nginx/conf.d/social.lein.us.conf sets root /home/mastodon/live/public and uses try_files $uri =404 for /assets/, /packs/, /emoji/, /sounds/, /avatars/, /headers/, /shortcuts/ and /sw.js. Those all need changing for a container deployment — see Phase 4.
• Sidekiq runs 25 threads on one core (-c 25, DB_POOL=25). Worth trimming as part of this work.
• The box reboots for kernel updates unattended — it moved from 6.8.0-1058-oracle to -1060 mid-session on 5 September. Anything long-running must survive that; run it under systemd-run, not a login shell.
• SSH is being brute-forced (invalid user testuser from 163.7.3.241). Ordinary background noise, but fail2ban is cheap.
• Ubuntu 22.04 standard support ends April 2027, so the OS itself is a fixed deadline sitting behind all of this.
• TLS is fine: Let’s Encrypt for social.lein.us, valid to 7 November 2026.

02The decision: yes, move to Docker Compose

You asked whether containerising would make upgrades easier. On this host it is not just easier — it is the difference between a routine upgrade and a source-compilation project.

Mastodon 4.6 dropped ImageMagick and made libvips mandatory, and raised the ffmpeg floor to 5.1. Mastodon 4.5 raised the Redis floor to 7.0. Ubuntu 22.04 cannot supply any of those three. Mastodon’s own Dockerfile resolves this by compiling libvips 8.18.5 and ffmpeg 9.0.1 from source inside the image.

So a native upgrade on this box means building Ruby 4.0.6, Node 24, libvips and ffmpeg from source, on a single ARM core with 5.8 GB of RAM, and then doing it again at every future release. The prebuilt image removes all of it — and ghcr.io/mastodon/mastodon:v4.7.1 and mastodon-streaming:v4.7.1 both publish linux/arm64 manifests, verified against the GHCR manifests.

The shape of the target stack

nginx and certbot stay on the host. Compose publishes on 127.0.0.1:3000 and 127.0.0.1:4000 — exactly where the existing backend and streaming upstreams already point, so the proxy blocks need no edits and TLS renewal keeps working untouched.

Postgres and Redis both move into containers. Leaving Postgres on the host would work, but it re-creates the problem this migration exists to solve: Mastodon would stay coupled to what the OS ships, and the April 2027 Ubuntu deadline would still be blocking. Redis has no choice — 6.0.16 is below 4.5’s floor and jammy has nothing newer.

Alternatives considered

03The upgrade chain

You cannot go 4.1 → 4.7 in one jump. Each release carries its own migrations and several carry hard gates. Take the latest patch of each branch in order, running db:migrate at every stop. Under Docker each hop is a tag change, so the chain is cheap.

The database is small — ~29,000 statuses, 28,650 accounts, 2.9 GB — and the measured dump/restore round trip was under six minutes, so the migrations themselves should be minutes rather than hours. On one core, still budget generously: plan a 4-hour window. The instance has four users, so a maintenance window is a social non-event.

04Execution

The reason the age-based passes found nothing is worth carrying forward: this instance’s cache is dominated by recent media. Attachments grew from 2.96 GB to 4.35 GB in a single evening of federation. A --days 7 cutoff has almost nothing to bite on, because anything older had already gone. Reclaiming that pool needs --days 1 or --days 3, at the cost of re-downloading as people browse.

# Swap — do this first; 5.8 GB with none is too tight for a restore + migrations
sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf
sudo sysctl -q vm.swappiness=10

# tootctl needs the rbenv shims on PATH — they are not there by default
TC="sudo -u mastodon env RAILS_ENV=production \
  PATH=/home/mastodon/.rbenv/shims:/usr/local/bin:/usr/bin:/bin \
  /home/mastodon/live/bin/tootctl"

$TC media usage                                     # baseline first
nice -n 15 ionice -c2 -n7 $TC media remove --days 7 --concurrency 2
nice -n 15 ionice -c2 -n7 $TC preview_cards remove --days 30 --concurrency 2
nice -n 15 ionice -c2 -n7 $TC accounts prune --concurrency 2
nice -n 15 ionice -c2 -n7 $TC media remove-orphans   # slowest, run last

sudo systemd-run --unit=mastodon-prune --collect --service-type=exec \
  bash -c '/path/to/script.sh > /home/ubuntu/prune.log 2>&1'

Custom emoji — the largest remaining pool

79,705 remote custom emoji, 1,865 MB, and zero local ones, so nothing of this instance’s own identity is at stake. None of the four commands above touch emoji; it needs its own pass:

$TC emoji purge --remote-only

Budget properly for it. Mastodon destroys each record individually along with its file, and the measured rate on this box is roughly 200 records per minute — about six hours for the full set. There is no faster safe path: a raw SQL delete would leave custom_emoji_categories and announcement_reactions references dangling. Remote emoji re-download as new posts reference them; older posts show the shortcode until then.

An unverified backup is not a backup. The restore test is the step that makes the rest of this plan safe to attempt — it is not optional garnish.

1 · Adam’s own exports, from the web UI

Log in as Adam at https://social.lein.us/settings/export and download the CSVs and the archive. This is the one step that needs his login. See section 05 for what those can and cannot restore. still outstanding

2 · Stop the services and dump

D=$(date +%F)
STAGE=/var/backups/mastodon
sudo mkdir -p $STAGE && sudo chown postgres:postgres $STAGE

# Record counts BEFORE stopping, as the verification baseline
sudo -u postgres psql -d mastodon_production -c \
  "SELECT count(*) FROM statuses WHERE account_id=110248000680880933;"

# Let Sidekiq drain — the Redis queue is not migrating
sudo systemctl stop mastodon-web mastodon-streaming
sleep 60
sudo systemctl stop mastodon-sidekiq

sudo -u postgres pg_dump -Fc -Z6 -d mastodon_production -f "$STAGE/mastodon_${D}.dump"

# Local media only — 7.01 MB of it. The remote cache is regenerable, skip it.
sudo tar czf "$STAGE/local-media_${D}.tar.gz" \
  -C /home/mastodon/live/public/system accounts media_attachments site_uploads

sudo systemctl start mastodon-web mastodon-sidekiq mastodon-streaming

3 · Prove the dump restores

sudo -u postgres createdb mastodon_restoretest
sudo -u postgres pg_restore -d mastodon_restoretest < "$STAGE/mastodon_${D}.dump"
sudo -u postgres psql -d mastodon_restoretest -c \
  "SELECT count(*) FROM statuses WHERE account_id=110248000680880933;"
# must return 941
sudo -u postgres dropdb mastodon_restoretest

04bWhat remains

# Docker CE from the official repo (arm64)
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu jammy stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io \
                        docker-buildx-plugin docker-compose-plugin

sudo mkdir -p /opt/mastodon/public/system
sudo cp -a /home/mastodon/live/public/system/. /opt/mastodon/public/system/
sudo chown -R 991:991 /opt/mastodon/public   # uid/gid the image runs as

That cp -a copies the whole media tree — currently ~7.7 GB and several hundred thousand files, so it is slow on this disk. Run it under systemd-run like everything else, and check df first: it roughly doubles media usage until the old tree is removed.

services:
  db:
    image: postgres:15-alpine
    restart: always
    shm_size: 256mb
    networks: [internal_network]
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'mastodon']
    volumes:
      - ./postgres15:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: mastodon
      POSTGRES_DB: mastodon_production
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

  redis:
    image: redis:7-alpine
    restart: always
    networks: [internal_network]
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    volumes:
      - ./redis:/data

  web:
    image: ghcr.io/mastodon/mastodon:v4.7.1
    restart: always
    env_file: .env.production
    command: bundle exec puma -C config/puma.rb
    networks: [external_network, internal_network]
    healthcheck:
      test: ['CMD-SHELL', "curl -s --noproxy localhost localhost:3000/health | grep -q 'OK' || exit 1"]
    ports: ['127.0.0.1:3000:3000']
    depends_on: [db, redis]
    volumes:
      - ./public/system:/mastodon/public/system

  streaming:
    image: ghcr.io/mastodon/mastodon-streaming:v4.7.1
    restart: always
    env_file: .env.production
    command: node ./streaming/index.js
    networks: [external_network, internal_network]
    healthcheck:
      test: ['CMD-SHELL', "curl -s --noproxy localhost localhost:4000/api/v1/streaming/health | grep -q 'OK' || exit 1"]
    ports: ['127.0.0.1:4000:4000']
    depends_on: [db, redis]

  sidekiq:
    image: ghcr.io/mastodon/mastodon:v4.7.1
    restart: always
    env_file: .env.production
    command: bundle exec sidekiq -c 8
    networks: [external_network, internal_network]
    depends_on: [db, redis]
    volumes:
      - ./public/system:/mastodon/public/system

networks:
  external_network:
  internal_network:
    internal: true

# --- CHANGE these ---
DB_HOST=db
DB_PORT=5432
DB_NAME=mastodon_production
DB_USER=mastodon
DB_PASS=<the same value as POSTGRES_PASSWORD>
REDIS_HOST=redis
REDIS_PORT=6379
# delete REDIS_PASSWORD — the container Redis is on an internal-only network
# --- KEEP byte-for-byte from the old file ---
LOCAL_DOMAIN=social.lein.us
SECRET_KEY_BASE=...
OTP_SECRET=...
VAPID_PRIVATE_KEY=...
VAPID_PUBLIC_KEY=...
# and the whole SMTP_* block, unchanged
# --- ADD in Phase 3, before the 4.3 hop ---
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=...
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=...
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=...

# --- sizing for one core ---
DB_POOL=10
WEB_CONCURRENCY=1
MAX_THREADS=5

cd /opt/mastodon
docker compose up -d db redis
# wait for the db healthcheck, then restore into the fresh volume
docker compose exec -T db pg_restore -U mastodon -d mastodon_production \
  --no-owner --role=mastodon < /home/ubuntu/mastodon-backup-2026-09-05/mastodon_2026-09-05.dump

Then hop the chain. For each tag in turn — v4.2.29, v4.3.23, v4.4.24, v4.5.17, v4.6.7, v4.7.1 — edit the three image tags and run:

docker compose pull
docker compose run --rm web bundle exec rails db:migrate

docker compose run --rm web bin/rails db:encryption:init

After the final hop, rebuild what Redis used to hold — the old queue and feed data did not come across:

docker compose run --rm web bin/tootctl cache clear
docker compose run --rm web bin/tootctl feeds build

Two edits to /etc/nginx/conf.d/social.lein.us.conf. The upstream and proxy_pass blocks stay exactly as they are — compose already listens on the ports they name.

1. Change both root /home/mastodon/live/public; lines to root /opt/mastodon/public;
2. Change every try_files $uri =404; to try_files $uri @proxy;. Assets now live inside the image, not on disk — without this, /assets/, /packs/, /emoji/, /sounds/, /avatars/, /headers/, /shortcuts/ and /sw.js all return 404 and you get an unstyled site. The image sets RAILS_SERVE_STATIC_FILES=true, so Puma serves them; nginx keeps its cache headers. The config file’s own comment at line 66 says to do exactly this.

sudo cp /etc/nginx/conf.d/social.lein.us.conf /root/nginx-preswitch.conf   # cheap undo
sudo sed -i 's#root /home/mastodon/live/public;#root /opt/mastodon/public;#' \
  /etc/nginx/conf.d/social.lein.us.conf
sudo sed -i 's#try_files $uri =404;#try_files $uri @proxy;#' \
  /etc/nginx/conf.d/social.lein.us.conf
sudo nginx -t && sudo systemctl reload nginx

Then disable the old units so nothing races for the ports on reboot — but leave them installed, they are your rollback:

sudo systemctl disable --now mastodon-web mastodon-sidekiq mastodon-streaming
docker compose up -d

Against the numbers from the verified backup, so a discrepancy is visible rather than assumed.

Leave the old systemd units and the host Postgres cluster in place for a fortnight before reclaiming the space.

05Adam’s account: backup, and the truth about restore

The account is Adam — capitalised, id 110248000680880933, admin role, created 23 April 2023. Mastodon resolves handles case-insensitively, so [email protected] is this account; there is no separate lowercase adam. It holds 941 statuses, 30 media attachments totalling 7.01 MB, 795 follows, 360 followers, 5 bookmarks, 2 favourites and 15 live OAuth tokens.

If a restore is actually needed

cd /opt/mastodon
docker compose stop web sidekiq streaming

docker compose exec -T db dropdb -U mastodon mastodon_production
docker compose exec -T db createdb -U mastodon mastodon_production
docker compose exec -T db pg_restore -U mastodon -d mastodon_production \
  --no-owner --role=mastodon < /home/ubuntu/mastodon-backup-2026-09-05/mastodon_2026-09-05.dump

sudo tar xzf /home/ubuntu/mastodon-backup-2026-09-05/local-media_2026-09-05.tar.gz \
  -C /opt/mastodon/public/system
sudo chown -R 991:991 /opt/mastodon/public/system

# the dump is 4.1-schema — re-walk the chain from P3 before starting web
docker compose run --rm web bundle exec rails db:migrate
docker compose run --rm web bin/tootctl feeds build
docker compose up -d

06Rollback

Available at every point up to and including Phase 4, and it is fast, because nothing in the plan writes to the host’s PostgreSQL cluster or to /home/mastodon/live.

cd /opt/mastodon && docker compose down
sudo cp /root/nginx-preswitch.conf /etc/nginx/conf.d/social.lein.us.conf
sudo nginx -t && sudo systemctl reload nginx
sudo systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming
# then wait — Puma needs ~50 s before /health answers

You are back on 4.1.2 with the original database, minus anything posted during the window. Keep /opt/mastodon/postgres15 so you can diagnose the failure before retrying.

07Afterwards — closing the gaps that got you here

The upgrade is the occasion; these are what stop the same three years happening again.

• Get the backup off the box, then automate it. One verified backup now exists, but it sits on the disk it protects and nothing recreates it tomorrow. A cron line does the job: docker compose exec -T db pg_dump -U mastodon -Fc mastodon_production to a dated file, shipped elsewhere. Keep .env.production alongside it — the dump is useless without the secrets.
• Fix the purge job. It has never once run. Absolute path, and rewrite the body for containers: 0 5 * * 1 cd /opt/mastodon && docker compose run --rm web bin/tootctl media remove --days 14. Drop the --days 0 --include-follows header purge from the old script; it re-downloads everything it deletes. Add an emoji purge --remote-only pass on a slower cadence — monthly is plenty, and it takes hours.
• Pruning is a holding action, not a fix. The cache regrew 1.4 GB in one evening. Whatever cadence you choose, expect to run it forever; the alternative is moving media to object storage.
• Keep the swap file. It cost 4 GB and was in use within the hour.
• Run long jobs under systemd-run. This box takes unattended kernel reboots — it did one mid-session — and login shells do not survive them.
• Upgrade cadence. Now that it is a tag change, take patch releases as they land. 4.2 onwards will tell you when one is available, and 4.7 warns when your version leaves support.
• The OS is still on the clock. Ubuntu 22.04 loses standard support in April 2027 — but with Mastodon in containers, that becomes an ordinary do-release-upgrade rather than a rebuild.

Decisions still open

1. Same host or new VM? The plan assumes same host. A new VM is cleaner and keeps the old box as a live rollback, at the cost of a second machine and a DNS cutover.
2. Where do backups go? Unanswered, and now the single largest risk — there is something worth protecting and nowhere off-box to put it.
3. Rotate the mastodon account password. It was shared in a chat transcript during this work and was never needed; passwordless sudo was already available.

00 Status

01 What runs now

Everything Mastodon lives under /opt/mastodon. nginx and certbot stay on the host and were never rebuilt — compose publishes on the same 127.0.0.1:3000 and :4000 the original upstreams already pointed at.

02 Operating it

Upgrading Mastodon

This is the whole point of the exercise. A release is now a one-line edit.

cd /opt/mastodon
sudo sed -i 's/^MASTODON_VERSION=.*/MASTODON_VERSION=v4.8.0/' .env
sudo docker compose pull
sudo docker compose run --rm web bundle exec rails db:migrate
sudo docker compose up -d
# then wait ~90 s and check
curl -s https://social.lein.us/api/v1/instance | grep -o '"version":"[^"]*"'

Rolling back is the same edit in reverse, provided the new version ran no migrations. Once migrations have applied, going back means restoring a dump — so take one first for a major release.

Backups

# database — takes ~2 min, no downtime needed (pg_dump is a consistent snapshot)
cd /opt/mastodon
sudo docker compose exec -T db pg_dump -U mastodon -Fc mastodon_production \
  > ~/mastodon_$(date +%F).dump

# local media — small; the remote cache is regenerable, skip it
sudo tar czf ~/local-media_$(date +%F).tar.gz \
  -C /opt/mastodon/public/system accounts media_attachments site_uploads

# the secrets, without which the dump is useless
sudo cp /opt/mastodon/.env.production ~/env.production_$(date +%F)

sudo docker compose exec -T db createdb -U mastodon restoretest
sudo docker compose exec -T db pg_restore -U mastodon -d restoretest < ~/mastodon_DATE.dump
sudo docker compose exec -T db psql -U mastodon -d restoretest -At \
  -c "SELECT count(*) FROM statuses WHERE account_id=110248000680880933;"
sudo docker compose exec -T db dropdb -U mastodon restoretest

Cache maintenance

Runs itself. /usr/local/bin/mastodon-maintenance.sh, logging to /var/log/mastodon-maintenance.log (rotated monthly), both entries flock-guarded so they cannot overlap.

Restoring

cd /opt/mastodon
sudo docker compose stop web sidekiq streaming
sudo docker compose exec -T db dropdb -U mastodon mastodon_production
sudo docker compose exec -T db createdb -U mastodon mastodon_production
sudo docker compose exec -T db pg_restore -U mastodon -d mastodon_production \
  --no-owner --role=mastodon < /path/to/mastodon_DATE.dump
sudo tar xzf /path/to/local-media_DATE.tar.gz -C /opt/mastodon/public/system
sudo chown -R 991:991 /opt/mastodon/public/system
sudo docker compose run --rm web bundle exec rails db:migrate
sudo docker compose run --rm web bin/tootctl feeds build
sudo docker compose up -d

Mail

Outbound goes through OCI Email Delivery, smtp.email.us-ashburn-1.oci.oraclecloud.com:587, STARTTLS, verified end to end on 7 Sep.

03 What was done

Chronological, with measured results. Times are UTC.

The six-hop chain

4.1 → 4.7 is not one jump. Each release carries migrations, and several carry hard gates. Under Docker each hop is a tag change, which is what made it 17 minutes instead of an evening.

Why containerising was the unblocker, not a preference

Mastodon 4.6 dropped ImageMagick and made libvips mandatory, and raised ffmpeg to ≥5.1. Mastodon 4.5 raised Redis to ≥7.0. Ubuntu 22.04 shipped libvips 8.12.1, ffmpeg 4.4.2 and Redis 6.0.16 — none of them qualified, and no newer versions existed in its repositories. Mastodon’s own Dockerfile resolves this by compiling libvips 8.18.5 and ffmpeg 9.0.1 from source inside the image.

A native upgrade meant building Ruby 4.0.6, Node 24, libvips and ffmpeg from source on a single ARM core — and repeating it at every release. The prebuilt image removed all of it. It also meant the 24.04 upgrade two days later was an ordinary do-release-upgrade that never took Mastodon down.

04 What surprised us

Kept because the reasoning matters more than the conclusions, and because several of these were wrong the first time.

05 Adam’s account, and the truth about restore

The account is Adam — capitalised, id 110248000680880933, admin. Mastodon resolves handles case-insensitively, so [email protected] is this account; there is no separate lowercase one. It came through the migration at 941 statuses, 30 attachments (7.01 MB), 795 following, 360 followers, and has since posted — 946 at the time of writing, which is the clearest evidence the new stack is genuinely in use.

06 Still open

The clock that is no longer ticking

The original deadline behind all of this was Ubuntu 22.04 losing standard support in April 2027. That is gone: 24.04 LTS is supported to 2029, and because Mastodon no longer depends on what the OS ships, the next release upgrade is an ordinary do-release-upgrade rather than a rebuild.