Gumbo Blog

Building an integrity-verified cloud backup ecosystem on Linux

I have a media library of about 2.5 TB and growing: lossless audio rips, high-resolution surround sound releases, DVD-Audio and SACD rips accumulated over many years. Losing it would be irreversible. Although most of it could relatively simple be re-acquired or downloaded from the original source, it would take way too much effort and time.

My local setup already has redundancy: masters live on /mnt/lotus and /mnt/titan, and lsyncd keeps one-way synced copies on /mnt/rhino and /mnt/oberon. These are all network files systems on seperate drives. But local redundancy only protects against disk failure. It does not protect against fire, theft, or the kind of catastrophic event that takes out an entire room. For that, you need offsite storage.

My requirements were deliberately narrow. I needed cloud storage with enough capacity for a large and growing media library, accessible via rclone so I could script the entire workflow myself. That was the starting point and the ending point of the provider requirements list.

What I explicitly did not want was a consumer backup product. Most cloud storage offerings come bundled with desktop clients, mobile apps, and browser-based interfaces designed around the assumption that the user will interact with their files manually on a Windows or Mac desktop, or on a smartphone. That model brings automatic syncing daemons that run in the background, proprietary clients that need to be installed and maintained on every device, and a layer of abstraction between you and what is actually happening to your data. For a home user backing up documents and photos from a single laptop, that model is convenient. For an automated server-side backup of a structured media library, it is the wrong tool entirely.

What I wanted was different: a server process that runs unattended, follows rules I define, and gives me full visibility into what it does and why. No GUI client to install, no daemon running on the desktop, no mobile app, no proprietary sync protocol. Just a remote storage endpoint that rclone can talk to, and scripts that implement exactly the backup semantics I need. The server handles everything; other devices on the network access the results through the FUSE mount or NFS if needed, without needing to know anything about the backup system itself.

This is a deliberate architectural choice rather than a constraint. Keeping the backup logic entirely on the server means there is exactly one place to monitor, one set of logs to read, and one codebase to maintain. It also means the system works correctly regardless of whether any other device on the network is powered on.

This post describes the cloud backup system I built on top of Internxt's 5TB Ultimate plan, running on a headless Devuan Daedalus server, which hostname is called 'mirage', with sysvinit. The system goes considerably beyond a simple scheduled rclone copy: it includes integrity verification, corruption detection, alert notifications, and a full audit trail of every file that has ever been backed up or deleted.

The difference between making a backup and syncing

Before going into the implementation, it is worth being precise about the distinction between backup and sync: two concepts that are routinely conflated, including by the tools themselves.

Syncing means keeping two locations in a consistent state. If you delete a file on the source, it gets deleted on the destination. If a file changes, the change propagates. The goal is identity: both sides should look the same. This is what lsyncd does for my local copies, and it is the right tool for that job: I want /mnt/rhino to be an exact mirror of /mnt/lotus, including deletions.

Backup means preserving a recoverable copy of a known-good state. The destination is not a mirror of the source: it is a record of what the source looked like at a point in time, or across multiple points in time. Crucially, a backup should be immune to what happens on the source. If you accidentally delete a directory, the backup still has it. If a file gets corrupted, the backup still has the uncorrupted version. The backup copy should require a deliberate, explicit action to modify: not an automatic propagation of whatever happened locally.

The failure mode of syncing-as-backup is well known but still catches people out: sync a corrupted or accidentally deleted file, and your "backup" now contains the same problem. You have two copies of the damage instead of one copy of the damage and one copy of the original.

What I built here is a backup system that uses copying mechanics: rclone transfers files from local to remote, but enforces backup semantics. Files on Internxt are only ever added or explicitly confirmed for update. They are never automatically overwritten because the local copy changed. They are never deleted because the local copy was deleted. Every deviation from the known-good state triggers a review process before the remote is touched. The local filesystem is the working copy; Internxt is the archive.

Why Internxt

The main reasons were straightforward: 5TB at a price point that made sense, end-to-end encryption by design (Internxt never holds your decryption keys), and native rclone support. rclone is the tool I already use for everything involving remote storage, so a provider with a first-class rclone backend was a requirement.

The E2E encryption deserves a mention. Internxt derives your encryption keys from a mnemonic: a sequence of words generated at account creation. rclone stores this mnemonic in its config file. If you lose the mnemonic and lose access to your account, your data is permanently unrecoverable. No support ticket will help. This is the correct design for privacy, but it means backup of the mnemonic itself is non-negotiable.

One thing Internxt does not support is server-side hashing. rclone cannot ask Internxt "what is the SHA256 of this file": Internxt simply does not expose that. This shaped several design decisions, as I'll explain.

Why not just rclone sync

The obvious starting point is rclone sync. Run it nightly, done. The problem is that, as stated before, sync is bidirectional in its damage potential: if a file gets corrupted locally - whether by a disk error, a botched metadata edit, or anything else - sync will faithfully overwrite the good remote copy with the corrupted local version. Your backup becomes a copy of the problem.

rclone copy is safer since it only transfers in one direction and never deletes from the destination. But it still has no concept of "this file changed unexpectedly" versus "this file was intentionally updated". It will overwrite the remote copy whenever the local file differs from the remote one, regardless of why it differs.

What I wanted was a system that:

The manifest

Since Internxt doesn't support server-side hashing, the only reliable way to know whether a backed-up file is intact is to maintain your own hash database locally. This became the manifest: a pipe-delimited text file with one line per backed-up file, storing the SHA256 hash, modification timestamp, file size, and absolute path.

c189d82f...|1668161796|10220293|/mnt/lotus/music/W/Wheel/Rumination/01-wheel.mp3

The manifest is the authoritative record of what every file looked like at the time it was last backed up. It serves three purposes:

  1. Change detection - the nightly backup compares local files against the manifest to find new files and detect suspicious changes
  2. Corruption detection - a weekly integrity check rehashes every local file and compares against the manifest, regardless of timestamp or size
  3. Restore verification - before trusting a file restored from Internxt, you can download it and verify its hash against the manifest

One manifest file per source directory, stored in /var/log/rclone/manifests/ and also backed up to Internxt itself.

The nightly backup flow

Each nightly run follows three steps for every source/destination pair defined in a CSV matrix file:

Pre-check: a Python script walks every local file and compares against the manifest. New files (not yet in the manifest) go onto an upload list. Files where both the modification time and hash have changed go onto a suspicious/exclude list - these are not uploaded, protecting the remote copy. Files where only the modification time changed but the content is identical are noted for a manifest-only update.

Upload: rclone runs with --include-from pointing at the upload list. It only ever sees new files. Suspicious files are invisible to it.

Post-update: after a successful upload, the manifest is updated. New files get their hash, timestamp and size recorded. Deleted local files are removed from the manifest and their paths are written to a separate deleted files log for later cleanup. Files with harmless timestamp changes get their stored timestamp updated.

This design means rclone never has to list the entire remote to decide what to upload, and it never touches a file the manifest hasn't blessed as new. On a typical night with a few new albums added, the entire process takes under 15 minutes including the uploads.

Detecting silent corruption

The nightly pre-check only catches changes that manifest as a modification time difference. There is a class of changes it cannot see: content that changes without the timestamp changing. The main real-world example in my library is whatlastgenre: a tool that writes genre tags directly into audio files. It modifies the file content but preserves the original modification time. And the file size isn't altered either: tags seem to be written in already reserved space in the file.

For this and other unintentional and therefor unwanted changes of files, a separate weekly integrity check script rehashes every local file from scratch and compares against the manifest. No timestamp comparison, no size comparison, just hash the file and see if it matches. This runs every Monday at 3am and takes about 6.5 hours for the full library.

When the integrity check finds a mismatch, it does not automatically update the manifest or trigger an upload. It adds the file to a review list and sends an alert email. The decision of what to do - confirm the change as intentional, or restore from backup - stays with the user.

The alert and review workflow

When either the nightly pre-check or the weekly integrity check flags a file, three things happen: the alert is written to a log, the filepath is added to a review list (/var/log/rclone/internxt-review.list), and an email is sent via msmtp.

The review list is the action queue. A dedicated script presents each file interactively:

[3/10] File: /mnt/titan/media/audio/surround/Steven Wilson/The Harmony Codex/03 What Is It Like to Be Something?.flac
  Size:         178538642 bytes
  Current hash: 58aaea01...

  [Y] Confirm change and re-upload  [N] Skip, preserve remote backup  [Q] Quit:

Answering Y updates the manifest with the new hash and re-uploads the file to Internxt. Answering N leaves the remote copy untouched. If you later restore the file from Internxt and run the integrity check again, it will find the hashes match and automatically remove the file from the review list.

There is also a --all flag for cases like a whatlastgenre run across a whole library, where you know all the changes are intentional and don't want to answer Y fifty times.

Redundant file cleanup

Files deleted locally after being backed up leave an orphan on Internxt. The system tracks these in a deleted files log, written automatically whenever the post-update step removes a file from the manifest. A separate script reads this log and produces a report of what can be cleaned up, with options for interactive or bulk deletion from Internxt.

Adding a new backup pair

Adding a new source directory to the backup is deliberately trivial. The entire system is driven by a CSV matrix file with one line per source/destination pair. Adding a new library folder means adding one line - source path, destination path - and saving the file. The next nightly run detects all files in that source as new, uploads them, and creates the manifest automatically. No script changes, no configuration files to touch, no service restarts. The same applies to temporarily disabling a backup pair: prefix the line with # and it is skipped until the comment is removed. For a system that will grow over time as new storage is added or libraries are reorganized, this matters.

The FUSE mount: a separate but complementary layer

The backup system described above communicates with Internxt entirely through rclone's transfer commands - it never goes through a mounted filesystem. The nightly backup, the integrity check, the confirm-change script, all of it talks to the Internxt API directly. This is intentional: direct API access is more reliable for automated operations, avoids FUSE caching complications, and works correctly regardless of whether the mount is up.

The FUSE mount - managed separately by a sysvinit service script - serves a different purpose. It makes the Internxt Drive appear as a local directory at /mnt/internxt, browsable like any other filesystem. This is where it becomes genuinely useful in practice:

For restore operations, being able to navigate the remote directory tree in a file manager, browse album folders, and drag individual files to a local location is considerably more convenient than constructing rclone copy commands by hand. The rclone-internxt-verify-file.py script handles hash verification after the fact, but the actual retrieval is much easier through the mount.

For inspection, the mount lets you confirm that a backup actually contains what you think it contains - that the folder structure is correct, that an album you uploaded last night is really there, that a specific file exists before you delete the local copy.

For manual one-off uploads, copying a file directly to /mnt/internxt works as expected, though files uploaded this way bypass the manifest system and will appear as "not in manifest" on the next integrity check.

The two layers - the automated backup ecosystem and the interactive FUSE mount - are independent enough that either can be down without affecting the other. A dropped mount (which happens when the OAuth token expires and is not refreshed in time) has no effect on the nightly backup. The backup running does not affect the mount. They share the same rclone config and the same credentials, but nothing else.

Technology choices

The rclone system user. The entire ecosystem runs under a dedicated system user named rclone, created with adduser --system --group. It has a real home directory (/home/rclone) for storing the rclone config and msmtp config, but no login shell - it cannot be logged into interactively. All scripts in /usr/local/bin/ are owned by rclone:rclone with permissions 750, meaning only root and the rclone user can read or execute them. Credentials never touch the root user or any interactive user account. When a command needs to be run manually - triggering the review workflow, running a targeted integrity check, inspecting the manifests - it is done via sudo -u rclone, which asks for the calling user's password rather than switching to a privileged shell. This separation keeps the attack surface small and makes the permission model easy to audit.

Python over bash for the complex logic. The manifest management started as bash and went through several painful iterations before being rewritten in Python. The specific problem was that bash's string splitting on delimiters breaks silently on filenames containing double spaces - a common pattern in music file naming. Python handles this correctly with no special effort. The remaining bash scripts (rclone-internxt-backup.sh, rclone-refresh-token.sh) are orchestration and system interaction, where bash is the right tool.

sysvinit for the FUSE mount service. mirage runs Devuan Daedalus with sysvinit by choice: no systemd. The rclone FUSE mount is managed by a traditional init script in /etc/init.d/, registered with update-rc.d. It works cleanly and predictably.

msmtp for email. A minimal SMTP client that relays through an existing mail account. No local mail server, no complexity. The rclone system user has its own ~/.msmtprc so alert emails work correctly from cron without any sudo involvement.

A dedicated system user. All rclone processes run as a system user named rclone (UID 110), with its own home directory holding the rclone config and msmtp config. The backup scripts, cron jobs, and service script all run under this user. This keeps credentials isolated and makes permission reasoning straightforward.

Process locking. A lock file at /tmp/rclone-internxt.lock prevents the nightly backup and the weekly integrity check from running simultaneously. Both scripts check for the lock on startup and exit gracefully if it's already held. The backup script also releases the lock on exit via a trap to handle crashes cleanly.

The cronjobs. All scheduled tasks run from the cron table of the rclone system user, not from root's crontab. Three jobs drive the system: the nightly backup at 02:00, a token refresh running eight times daily at odd hours that never coincide with the backup window, and the weekly integrity check every Monday at 03:00. Keeping everything under one user's crontab makes the schedule easy to inspect and reason about: sudo -u rclone crontab -l shows the complete picture in one place.

Transferability. One design decision worth highlighting separately is that the entire system is built around rclone's abstraction layer, not around Internxt specifically. The backup matrix, the manifest, the scripts: none of them contain anything Internxt-specific except the remote name in the config file. Switching to a different provider means reconfiguring the rclone remote and updating the destination paths in the matrix CSV. The manifest, the integrity check, the alert workflow, the review process: all of it transfers unchanged. This matters practically. Internxt is a relatively young company and pricing can change. I already have a planned migration path to Hetzner object storage when my current Internxt subscription ends. Hetzner's S3-compatible storage has one significant advantage over Internxt for this use case: rclone's S3 backend supports server-side checksums, meaning file integrity can be verified without downloading. The weekly integrity check would become dramatically faster. The migration itself would be: configure a new rclone remote, run the initial upload once, update the matrix. The rest of the ecosystem stays exactly as it is.

What this is not

This is not a disaster recovery system for the server itself: it backs up media files, not system configuration. It is not a versioned backup: if you confirm an intentional change and re-upload a file, the previous version is gone from Internxt. And it is not a replacement for local redundancy: the Internxt backup is a third copy for the scenario where both local copies are lost, not the primary safety net.

Scripts

All scripts are available on Pastebin:

A full operations manual covering setup, configuration, and day-to-day use is also available link

avatar

#implementation #linux