I made my own social media network federated via ActivityPub on the Fediverse running Mastodon 4.1 a few years ago. I did it with a script that installed everything on an Oracle Cloud VPS. I did not use the Docker method because I thought that might be extra RAM that could be used elsewhere. Then I became lazy about learning how to upgrade Mastodon on an installation built from source. Reading the release notes and documentation for each Mastodon upgrade involved lots of special tweaks and database migrations that sounded pretty dangerous. I was worried about breaking it completely, so I kept putting it off. I thought about paying a friend to help too. Then I decided to give Claude Code SSH access to the server and let the AI evaluate what needed to be done.
The Opus 5 AI model looked at my server, scanned to see how it was configured, and developed the below runbook as a plan of attack. After reading through it and deciding that it all sounded good, I decided to just let Claud Opus proceed. I was using the Claude Code plugin with Visual Studio Code and it all took quite a long time. Several times it seemed like Claude had stalled or frozen as there was no way for it to display progress on some of the procedures that it was running. So I had to “resume” the runbook process several times. This was totally fine as we had built out the runbook plan as an HTML web page ahead of time.
Lein Social Upgrade Runbook
A three-year-old source install is now Mastodon 4.7.1 in containers on Ubuntu 24.04 LTS. This revision is the record of how, and the reference for running it.
00 Status
4 m 05 s to take the verified backup · 10 m 50 s for the cutover to containers · ~90 s for the 24.04 reboot.
The six-hop upgrade chain and the whole OS release upgrade ran with the site live. That is the payoff from containerising first: the host was rewritten underneath a running Mastodon.
01 What runs now
Everything Mastodon lives under /opt/mastodon. nginx and certbot stay on the host and were never rebuilt — compose publishes on the same 127.0.0.1:3000 and :4000 the original upstreams already pointed at.
Host
Containers — /opt/mastodon
| Path | What it is |
|---|---|
| /opt/mastodon/docker-compose.yml | The stack. Image tags come from MASTODON_VERSION in .env. |
| /opt/mastodon/.env | MASTODON_VERSION and POSTGRES_PASSWORD. Mode 600, root. |
| /opt/mastodon/.env.production | Mastodon config and all secrets. Mode 600, root. |
| /opt/mastodon/postgres15/ | Database volume. |
| /opt/mastodon/public/system/ | Media. Owned 991:991 (the image’s uid). |
| /opt/mastodon/redis/ | Redis volume. |
| /usr/local/bin/mastodon-maintenance.sh | Cache maintenance, run by cron. |
| /home/ubuntu/mastodon-backup-2026-09-05/ | The verified backup. Also copied off-box. |
| /root/pre-noble-config/ | Config snapshot taken before the OS upgrade. |
Puma takes 60–105 s to preload. A health check straight after any restart returns 502 and looks broken. Wait for :3000/health before concluding anything. The compose healthcheck has start_period: 180s so it no longer reports false “unhealthy”.
nginx serves /system/ from disk and proxies everything else. All eight static blocks use try_files $uri @proxy. If they ever revert to =404 you get a site that loads with no CSS or JS.
Run long jobs under systemd-run, never a login shell. This box takes unattended kernel reboots and SSH sessions drop. Every long operation in this migration was wrapped: sudo systemd-run --unit=NAME --collect --service-type=exec bash -c '…'.
02 Operating it
Upgrading Mastodon
This is the whole point of the exercise. A release is now a one-line edit.
cd /opt/mastodon
sudo sed -i 's/^MASTODON_VERSION=.*/MASTODON_VERSION=v4.8.0/' .env
sudo docker compose pull
sudo docker compose run --rm web bundle exec rails db:migrate
sudo docker compose up -d
# then wait ~90 s and check
curl -s https://social.lein.us/api/v1/instance | grep -o '"version":"[^"]*"'Rolling back is the same edit in reverse, provided the new version ran no migrations. Once migrations have applied, going back means restoring a dump — so take one first for a major release.
Minimum PostgreSQL, Redis, Ruby and Node versions rise over time, and a release occasionally demands a new secret — 4.3 required three. The image carries Ruby, Node, libvips and ffmpeg, but PostgreSQL and Redis are yours to keep current: they are pinned in docker-compose.yml at postgres:15-alpine and redis:7-alpine.
Backups
# database — takes ~2 min, no downtime needed (pg_dump is a consistent snapshot)
cd /opt/mastodon
sudo docker compose exec -T db pg_dump -U mastodon -Fc mastodon_production \
> ~/mastodon_$(date +%F).dump
# local media — small; the remote cache is regenerable, skip it
sudo tar czf ~/local-media_$(date +%F).tar.gz \
-C /opt/mastodon/public/system accounts media_attachments site_uploads
# the secrets, without which the dump is useless
sudo cp /opt/mastodon/.env.production ~/env.production_$(date +%F)Prove it, into a scratch database, and count something you know:
sudo docker compose exec -T db createdb -U mastodon restoretest
sudo docker compose exec -T db pg_restore -U mastodon -d restoretest < ~/mastodon_DATE.dump
sudo docker compose exec -T db psql -U mastodon -d restoretest -At \
-c "SELECT count(*) FROM statuses WHERE account_id=110248000680880933;"
sudo docker compose exec -T db dropdb -U mastodon restoretestAnd get it off this machine. A backup on the disk it protects only survives the failures that were never going to matter.
Cache maintenance
Runs itself. /usr/local/bin/mastodon-maintenance.sh, logging to /var/log/mastodon-maintenance.log (rotated monthly), both entries flock-guarded so they cannot overlap.
| Schedule | Does | Why that cadence |
|---|---|---|
| 0 3 * * * | media remove --days 2preview_cards remove --days 7 | This instance ingests ~5.6 GB/day of remote media. At 2 days the cache settles near ~11 GB; at 3 days it was ~17 GB. |
| 0 4 1 * * | accounts prunestatuses remove --days 30media remove-orphans | Orphan scan is a full-disk walk (~10 min) that has found 0–1 files in practice, so it is monthly rather than nightly. |
emoji purge --remote-only. Measured 5 Sep: purging all 79,705 remote emoji reclaimed 1.82 GB — and it was fully back within 36 hours as they re-federated. It buys a day or two of space for the cost of re-downloading everything.
media remove --remove-headers --include-follows --days 0, which the original purge-media.sh ran. It deletes avatars and headers for accounts you actively follow, which Mastodon then immediately re-fetches. It spends bandwidth to reclaim space it gives straight back.
Restoring
cd /opt/mastodon
sudo docker compose stop web sidekiq streaming
sudo docker compose exec -T db dropdb -U mastodon mastodon_production
sudo docker compose exec -T db createdb -U mastodon mastodon_production
sudo docker compose exec -T db pg_restore -U mastodon -d mastodon_production \
--no-owner --role=mastodon < /path/to/mastodon_DATE.dump
sudo tar xzf /path/to/local-media_DATE.tar.gz -C /opt/mastodon/public/system
sudo chown -R 991:991 /opt/mastodon/public/system
sudo docker compose run --rm web bundle exec rails db:migrate
sudo docker compose run --rm web bin/tootctl feeds build
sudo docker compose up -d/home/ubuntu is mode 750, so a postgres user cannot traverse it and pg_restore /path/to.dump fails with a bare “Permission denied” that looks like a corrupt file. The < redirect runs as your shell and sidesteps it. This cost real time to diagnose.
Outbound goes through OCI Email Delivery, smtp.email.us-ashburn-1.oci.oraclecloud.com:587, STARTTLS, verified end to end on 7 Sep.
SMTP_SSL and SMTP_TLS must be false with SMTP_ENABLE_STARTTLS=always. The previous provider used 465 with SMTP_SSL=true; carrying that over to 587 silently breaks all mail. Also note OCI only accepts mail from an approved sender — authentication succeeding proves nothing about whether it will deliver.
03 What was done
Chronological, with measured results. Times are UTC.
| When | Step | Result |
|---|---|---|
| 5 Sep 16:56 | Inspection | Found 4.1.2 on main@9d75b03ba — a dev snapshot, not a release tag. No backups existed at all. |
| 5 Sep 18:24 | Swap + prune | 4 GB swap, swappiness=10. Accounts 55,482 → 31,294; disk 33 → 24 GB used. |
| 5 Sep 21:57 | Backup | 446 MB dump + 18 MB media. 4 m 05 s downtime. |
| 5 Sep 22:07 | Restore-verified | pg_restore rc=0, 904 TOC entries. 941/30/795/360 and 4 users exact; post text verbatim. |
| 5 Sep 23:08 | Emoji purge | 79,705 → 0 in 22 m 34 s. Zero local emoji existed to lose. |
| 5 Sep 23:19 | Docker installed | 29.8.0 + Compose v5.5.1, arm64. Container networking verified against ufw. |
| 6 Sep 00:47 | Restore into containers | 4 m 43 s. Counts matched exactly; schema at 4.1-era 20230215074423. |
| 6 Sep 00:48–01:05 | Six-hop chain | 17 minutes, site live throughout. Every hop gated on 941 statuses / 4 users. |
| 6 Sep 01:21 | Cutover | 10 m 50 s downtime. Web up 65 s after container start; public 200 immediately after. |
| 7 Sep 13:43 | Mail moved to OCI | Accepted by relay, delivered. |
| 7 Sep 14:55 | Old install removed | Host PG, Redis, Node 16, ffmpeg, ImageMagick, /home/mastodon, 3 units, the mastodon account. 7 health gates, no drop. |
| 7 Sep 15:12 | Ubuntu 24.04 | rc=0 in 12 minutes, Mastodon live throughout. |
| 7 Sep 15:20 | Reboot | Kernel 6.17.0-1020-oracle. Site answered 200 10 s after boot. 0 failed units. |
The six-hop chain
4.1 → 4.7 is not one jump. Each release carries migrations, and several carry hard gates. Under Docker each hop is a tag change, which is what made it 17 minutes instead of an evening.
| Hop | Schema after | Gate it carried |
|---|---|---|
| v4.2.29 | 20230907150100 | Ruby ≥3.0, Node ≥16, PG ≥10 |
| v4.3.23 | 20241007071624 | Three new encryption secrets. Image splits web/streaming. yarn 1 → 4. |
| v4.4.24 | 20250627132728 | Redis ≥6.2, PG ≥13. Redis namespaces dropped. |
| v4.5.17 | 20251023210145 | Redis ≥7.0, PG ≥14 |
| v4.6.7 | 20260611150940 | ImageMagick removed, libvips required, ffmpeg ≥5.1 |
| v4.7.1 | 20260812154114 | Long migrations: account uri uniqueness, ActivityPub identity rework |
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY, _DETERMINISTIC_KEY and _KEY_DERIVATION_SALT live in .env.production and are in the backup. Lose them and everything encrypted with them is unrecoverable, including users’ stored 2FA secrets. They were generated and backed up before the 4.3 migrations ran, deliberately in that order.
Why containerising was the unblocker, not a preference
Mastodon 4.6 dropped ImageMagick and made libvips mandatory, and raised ffmpeg to ≥5.1. Mastodon 4.5 raised Redis to ≥7.0. Ubuntu 22.04 shipped libvips 8.12.1, ffmpeg 4.4.2 and Redis 6.0.16 — none of them qualified, and no newer versions existed in its repositories. Mastodon’s own Dockerfile resolves this by compiling libvips 8.18.5 and ffmpeg 9.0.1 from source inside the image.
A native upgrade meant building Ruby 4.0.6, Node 24, libvips and ffmpeg from source on a single ARM core — and repeating it at every release. The prebuilt image removed all of it. It also meant the 24.04 upgrade two days later was an ordinary do-release-upgrade that never took Mastodon down.
04 What surprised us
Kept because the reasoning matters more than the conclusions, and because several of these were wrong the first time.
| Believed | Actually |
|---|---|
public/system held ~24 GB, largely orphaned files | A full remove-orphans scan found 3 files, 243 KB. It was all live remote cache. There was never an orphan problem. |
| Age-based pruning would reclaim the disk | --days 7 and even --days 2 removed zero bytes. Nothing is stale — the cache is entirely recent, because inflow is ~5.6 GB/day. The disk is simply smaller than the firehose. |
| Emoji purging is worth scheduling | 1.82 GB reclaimed, fully regrown in 36 hours. Dropped from the schedule. |
| The emoji purge would take ~6 hours | 22 minutes. The first estimate extrapolated from 50 seconds that were mostly Rails booting. |
| Gate the migration on >25 GB free | That number was sized against the mistaken 24 GB estimate. Phase 2/3 needed ~6 GB. The gate was wrong, not the outcome. |
The account was adam | Adam, capitalised. Mastodon matches handles case-insensitively, so both resolve — but SQL does not. |
| The 22.04 → 24.04 upgrade would take 45–90 min | 12 minutes, and it never interrupted Mastodon. |
do-release-upgrade uninstalled ufw and left INPUT policy: ACCEPT where it had been DROP. Exposure did not change in practice — only 22/80/443 listen externally — but default-deny was gone, and nothing announced it. Reinstalled and re-enabled from the preserved /etc/ufw rules. Check your firewall after any release upgrade. It also disabled the Docker apt repo (renamed to docker.list.distUpgrade), which would have quietly stopped Docker security updates.
05 Adam’s account, and the truth about restore
The account is Adam — capitalised, id 110248000680880933, admin. Mastodon resolves handles case-insensitively, so adam@social.lein.us is this account; there is no separate lowercase one. It came through the migration at 941 statuses, 30 attachments (7.01 MB), 795 following, 360 followers, and has since posted — 946 at the time of writing, which is the clearest evidence the new stack is genuinely in use.
Mastodon has no import for the account archive. The .tar.gz from “Request your archive” is a portability and readability format — outbox.json, actor.json, media. No version, 4.7.1 included, can load it back. There is also no tootctl command that restores one account.
So the only thing that brings back Adam’s posts is the full database restore. Per-account exports are a real but partial second net.
| Artefact | Restorable? |
|---|---|
| mastodon_2026-09-05.dump | Yes — the real path Everything: all accounts, posts, follows, tokens. |
| local-media_2026-09-05.tar.gz | Yes Untar into public/system, then chown 991:991. |
| following / lists / blocks / mutes / bookmarks .csv | Yes Via Import in the web UI. |
| archive .tar.gz | No import exists Readable reference only. |
| account-adam/*.csv | Manual Reconciliation aid, not a restore path. |
| Followers | Not exportable Held on remote servers; they re-federate over time. |
Post text contains newlines, so one record often spans several physical lines — statuses.csv reads as 1,061 lines for 941 posts. Use a CSV reader, or count in SQL.
06 Still open
One verified backup exists and has been copied off-box. Nothing recreates it tomorrow. The instance has been running two days past that snapshot. This is now the largest single risk, and it is one cron line plus somewhere to put the file — see section 02.
Nightly pruning at --days 2 should hold the cache near ~11 GB against ~5.6 GB/day of inflow. It is a treadmill: whatever cadence you pick, you run it forever, and a busier feed moves the equilibrium.
The durable fix is S3-compatible object storage for media, which Mastodon supports natively and which OCI provides. That removes the disk ceiling instead of managing it. The first genuinely productive nightly run is the night of 7–8 Sep; watch free space then to confirm it stabilises rather than climbs.
Adam’s own web-UI exports (Settings → Import and export) were never taken — the one step needing his login. Not gating anything, since the database restore is the real path.
fail2ban. SSH is being brute-forced from the open internet; ordinary background noise, but it is cheap insurance and key-only auth is already on.
Old kernels. 5.15.0-1030 and 6.8.0-1060 are still installed; autoremove will not take them while the linux-image-oracle meta-package holds a reference. Keeping one fallback kernel is sensible anyway.
The clock that is no longer ticking
The original deadline behind all of this was Ubuntu 22.04 losing standard support in April 2027. That is gone: 24.04 LTS is supported to 2029, and because Mastodon no longer depends on what the OS ships, the next release upgrade is an ordinary do-release-upgrade rather than a rebuild.
Not sure how I feel about this.