Gumbo Blog

Server: mirage (Devuan Daedalus, sysvinit)
Version: 1.1 — May 2026

Table of Contents

1. Goal
2. System Overview
3. Components
4. Directory and File Structure
5. The rclone Remote
6. The FUSE Mount
7. Token Management
8. The Backup Matrix
9. The Manifest System
10. Nightly Backup Flow
11. Alert and Review System
12. Periodic Integrity Check
13. File Verification at Restore Time
14. Confirming Intentional Changes
15. Redundant File Cleanup
16. Process Locking
17. Cron Schedule
18. Log Files
19. Logrotate
20. Scripts Reference
21. Routine Operations
22. Adding a New Backup Pair

1. Goal

Provide an automated, integrity-verified cloud backup of local media libraries on mirage to Internxt cloud storage (5TB Ultimate plan). The backup is a third copy — local masters are on /mnt/lotus and /mnt/titan, local backups are managed separately by lsyncd to /mnt/rhino and /mnt/oberon. The Internxt backup covers the scenario where both local copies are lost.

Key design principles:

• Files are never silently overwritten on the remote — any unexpected local change is flagged for manual review before the remote is touched
• A SHA256 manifest tracks the known-good state of every backed-up file
• Unexpected content changes (corruption, unintended modifications) trigger an alert email and block the upload
• Intentional changes (e.g. metadata updates via whatlastgenre) are confirmed interactively and re-uploaded
• The system is fully automated and runs unattended nightly

2. System Overview

Local libraries            rclone              Internxt cloud
/mnt/lotus/audio    -+
/mnt/lotus/music    -|   Pre-check (manifest)
/mnt/lotus/own      -+---------------------->  internxt:/Media/...
/mnt/lotus/purchased-|   Upload (new only)
/mnt/titan/media/   -+   Post-update (manifest)

Manifests: /var/log/rclone/manifests/*.sha256
           also backed up to internxt:/Manifests

The nightly backup flow is:

1. Pre-check: compare local files against manifest — identify new files and suspicious files
2. Upload: rclone copies only new files to Internxt; suspicious files are excluded
3. Post-update: manifest updated with newly uploaded files and any mtime-only changes
4. Manifest sync: updated manifests copied to Internxt

3. Components

4. Directory and File Structure

System user

User:  rclone (UID 110, GID 115)
Home:  /home/rclone
Shell: /usr/sbin/nologin

rclone configuration

/home/rclone/.config/rclone/rclone.conf   # Internxt remote credentials
/home/rclone/.msmtprc                      # SMTP config for alert emails

Mount points

/mnt/internxt        # FUSE mount of Internxt Drive (owned by rclone:rclone)

Scripts

/usr/local/bin/rclone-internxt-backup.sh              # Nightly backup orchestrator
/usr/local/bin/rclone-internxt-manifest.py            # Full manifest regeneration
/usr/local/bin/rclone-update-manifest.py              # Pre-check and post-update
/usr/local/bin/rclone-internxt-integrity-check.py     # Periodic full rehash check
/usr/local/bin/rclone-internxt-confirm-change.py      # Review and confirm changes
/usr/local/bin/rclone-internxt-verify-file.py         # Verify file against manifest
/usr/local/bin/rclone-internxt-bootstrap-deleted.py   # One-time bootstrap of deleted log
/usr/local/bin/rclone-internxt-redundant-list.py      # List and delete redundant remote files

Configuration

/etc/rclone/internxt-backup-matrix.csv   # Source/destination pairs
/etc/init.d/rclone-internxt              # Sysvinit service script
/etc/fuse.conf                           # Contains: user_allow_other
/etc/msmtprc                             # System SMTP config (root only)

Logs

/var/log/rclone/internxt.log                           # FUSE mount log
/var/log/rclone/internxt-media-backup.log              # Current nightly backup log
/var/log/rclone/internxt-media-backup-YYYYMMDD.log.gz  # Rotated backup logs
/var/log/rclone/internxt-token-refresh.log             # Token refresh log
/var/log/rclone/internxt-manifest.log                  # Manifest generation log
/var/log/rclone/internxt-integrity-check.log           # Integrity check log
/var/log/rclone/internxt-alerts.log                    # Alert log
/var/log/rclone/internxt-review.list                   # Files pending review
/var/log/rclone/internxt-deleted.log                   # Files deleted locally, still on Internxt
/var/log/rclone/internxt-redundant-YYYYMMDD.txt        # Redundant file reports
/var/log/rclone/msmtp.log                              # SMTP send log
/var/log/rclone/manifests/*.sha256                     # Manifest files

5. The rclone Remote

The Internxt remote is named internxt and configured in /home/rclone/.config/rclone/rclone.conf. It uses the native Internxt backend with email/password authentication. The config stores email, password (obfuscated), mnemonic, and an OAuth token.

The token expires approximately every 3 days. Token refresh is automated — see Section 7.

To verify the remote is working:

sudo -u rclone rclone listremotes
sudo -u rclone rclone lsd internxt:

6. The FUSE Mount

Internxt is mounted at /mnt/internxt via the sysvinit service script. The mount makes the Internxt Drive accessible as a local filesystem for browsing and manual operations. It is not used by the nightly backup script, which communicates with Internxt directly via rclone.

Service management

service rclone-internxt start
service rclone-internxt stop
service rclone-internxt restart
service rclone-internxt status

Mount options

--uid 1000 --gid 100          # Files appear as wouter:users
--file-perms 0664
--dir-perms 0775
--allow-other                 # Required for NFS and multi-user access
--vfs-cache-mode full
--vfs-cache-max-size 10G
--bwlimit 8M

Auto-start

The service is registered with update-rc.d and starts automatically at boot after network is available (Required-Start: $network $remote_fs).

7. Token Management

The Internxt OAuth token expires approximately every 3 days. The token refresh script runs 6 times daily and refreshes the token if it expires within 24 hours. It also monitors the FUSE mount health and remounts if needed.

Script

/usr/local/bin/rclone-refresh-token.sh — pastebin link here

Behavior

• Reads token expiry date from rclone.conf
• If expiry is within 24 hours: unmounts, reconnects, remounts
• If no refresh needed but mount is down: remounts
• If backup is running: skips refresh and logs a warning
• All actions logged to /var/log/rclone/internxt-token-refresh.log

Manual token refresh

sudo -u rclone rclone config reconnect internxt:

8. The Backup Matrix

/etc/rclone/internxt-backup-matrix.csv defines which local folders are backed up and where on Internxt.

Format

source,destination
/mnt/lotus/audio,internxt:/Media/lotus/audio
/mnt/lotus/music,internxt:/Media/lotus/music
/mnt/lotus/own,internxt:/Media/lotus/own
/mnt/lotus/purchased,internxt:/Media/lotus/purchased
/mnt/titan/media/audio,internxt:/Media/titan/media/audio

Lines starting with # are skipped, allowing pairs to be temporarily disabled.

9. The Manifest System

Each source folder has a corresponding SHA256 manifest file in /var/log/rclone/manifests/. The manifest is the authoritative record of what a file's content should look like — it is the reference for both corruption detection and restore verification.

Manifest format

One line per file:

<sha256hash>|<mtime>|<size>|<absolute_filepath>

Example:

e69513a1...fbc6|1668161796|10220293|/mnt/lotus/music/W/Wheel/Rumination/01-wheel.mp3

Manifest filenames

Source path is converted to filename by stripping the leading slash and replacing / with _:

/mnt/lotus/music  →  mnt_lotus_music.sha256
/mnt/titan/media/audio  →  mnt_titan_media_audio.sha256

Manifest regeneration

Full regeneration is only needed when the manifest format changes or after a disaster recovery. It takes approximately 6.5 hours for the full library. Run as:

sudo -u rclone python3 /usr/local/bin/rclone-internxt-manifest.py
tail -f /var/log/rclone/internxt-manifest.log

Manifests are also backed up to internxt:/Manifests automatically.

10. Nightly Backup Flow

The backup script runs nightly at 02:00 via the rclone user's crontab. For each source/destination pair in the matrix it performs three steps.

Step 1: Pre-check (before rclone)

rclone-update-manifest.py precheck walks all local files and compares against the manifest:

Step 2: Upload

If the upload list is non-empty, rclone copies only those files:

rclone copy <source> <destination> --include-from <upload_list>

Suspicious files are never uploaded, protecting the remote backup.

Step 3: Post-update (after successful rclone)

rclone-update-manifest.py postupdate updates the manifest:

• Adds newly uploaded files with their hash, mtime, and size
• Removes entries for files deleted locally
• Updates mtime/size for files where only the timestamp changed (content verified identical)

Final step: manifest sync

All updated manifest files are copied to internxt:/Manifests.

Rclone transfer settings

--bwlimit 8M
--transfers 2
--tpslimit 2
--tpslimit-burst 3
--retries 15
--retries-sleep 10s

11. Alert and Review System

Alert triggers

• File size changed without mtime change (nightly pre-check)
• File mtime and hash both changed (nightly pre-check)
• File hash changed regardless of mtime/size (integrity check)

Alert actions

1. Entry written to /var/log/rclone/internxt-alerts.log
2. Filepath added to /var/log/rclone/internxt-review.list
3. Alert email sent via msmtp to wouter@private-lotus.org
4. File excluded from upload (remote backup preserved as-is)

Reviewing alerts

sudo -u rclone rclone-internxt-confirm-change.py --review

This presents each file interactively with its current hash and size:

• Y — confirm change as intentional: manifest updated, file re-uploaded to Internxt, removed from review list
• N — skip: remote backup preserved, restore instructions shown
• Q — quit, remaining files stay in review list

To confirm all files at once without prompting:

sudo -u rclone rclone-internxt-confirm-change.py --all

Review list behavior

• Files are added automatically when alerts fire
• Files are removed when confirmed (Y) during review
• Files are removed automatically when the integrity check finds they match the manifest again (e.g. after a restore)

12. Periodic Integrity Check

The nightly backup detects changes by comparing mtime and size. It cannot detect silent corruption where file content changes without either value changing (e.g. whatlastgenre tag updates that preserve mtime, or disk-level corruption).

The integrity check rehashes every file and compares against the manifest, regardless of mtime or size.

Run on a specific folder (fast, for targeted checks)

sudo -u rclone python3 /usr/local/bin/rclone-internxt-integrity-check.py \
    /mnt/lotus/music/W/Wheel/Rumination

Run on all sources (slow, ~6.5 hours for full library)

sudo -u rclone python3 /usr/local/bin/rclone-internxt-integrity-check.py

Cron schedule

Runs every Monday at 03:00 (rclone user crontab):

0 3 * * 1 python3 /usr/local/bin/rclone-internxt-integrity-check.py

Lock behavior

The integrity check uses a lock file (/tmp/rclone-internxt.lock) to prevent conflicts with the nightly backup. If the backup is still running at 03:00, the integrity check exits cleanly and logs a message. Run it manually once the backup is complete:

sudo -u rclone python3 /usr/local/bin/rclone-internxt-integrity-check.py

Output

• Hash mismatches: files whose content has changed since last manifest update
• Missing files: in manifest but gone from disk
• Not in manifest: on disk but not yet backed up (picked up by next nightly run)

Corrupt and missing files are added to the review list and an alert email is sent.

After restoring a file from Internxt, the next integrity check will find it matches the manifest and automatically remove it from the review list.

13. File Verification at Restore Time

Before trusting a restored file, verify it against the manifest:

Single file

sudo -u rclone python3 /usr/local/bin/rclone-internxt-verify-file.py \
    "/mnt/lotus/music/A/Anouk/Hotel New York/01 - Hotel New York.flac"

Entire directory

sudo -u rclone python3 /usr/local/bin/rclone-internxt-verify-file.py \
    "/mnt/lotus/music/A/Anouk/Hotel New York"

The script downloads the file from Internxt to a temp location, hashes it, and compares against the manifest. Output is either OK or FAIL with hash details.

14. Confirming Intentional Changes

When a file is flagged as changed but the change is intentional (e.g. metadata updated by whatlastgenre, cover art replaced), use the confirm script to update the manifest and re-upload to Internxt.

Single file

sudo -u rclone python3 /usr/local/bin/rclone-internxt-confirm-change.py \
    "/mnt/lotus/music/W/Wheel/Rumination/01-wheel-blood_drinker.mp3"

Interactive review of all pending files

sudo -u rclone python3 /usr/local/bin/rclone-internxt-confirm-change.py --review

Confirm all pending files at once

sudo -u rclone python3 /usr/local/bin/rclone-internxt-confirm-change.py --all

15. Redundant File Cleanup

Files deleted locally after being backed up to Internxt are tracked in /var/log/rclone/internxt-deleted.log. This log is maintained automatically by the nightly backup's post-update step. The redundant list script reads this log to identify files that can be cleaned up on Internxt.

Deleted log format

<date>|<remote_destination>|<local_filepath>

Example:

2026-03-19|internxt:/Media/lotus/music|/mnt/lotus/music/A/Artist/OldAlbum/track.flac

One-time bootstrap

Run once after initial deployment to populate the deleted log with files already on Internxt but deleted locally before the system was in place:

sudo -u rclone python3 /usr/local/bin/rclone-internxt-bootstrap-deleted.py
tail -f /var/log/rclone/internxt-bootstrap-deleted.log

Show redundant files report

sudo -u rclone python3 /usr/local/bin/rclone-internxt-redundant-list.py

Produces a dated report file at /var/log/rclone/internxt-redundant-YYYYMMDD.txt grouped by destination.

Interactive deletion

sudo -u rclone python3 /usr/local/bin/rclone-internxt-redundant-list.py --delete

For each file:

• D — delete from Internxt, remove from deleted log
• S — skip, keep on Internxt
• Q — quit, remaining files stay in deleted log

Delete all without prompting

sudo -u rclone python3 /usr/local/bin/rclone-internxt-redundant-list.py --delete-all

Asks for confirmation (yes) before proceeding.

16. Process Locking

A lock file at /tmp/rclone-internxt.lock prevents the nightly backup and the weekly integrity check from running simultaneously, which could cause manifest corruption and disk I/O contention.

Behavior

• The backup script creates the lock on start and removes it on exit (including on error via trap)
• The integrity check creates the lock on start and releases it in a finally block
• If either script finds the lock already exists, it exits cleanly with a log message
• The token refresh script checks for a running backup via pgrep and skips the reconnect if one is found, but does not use the lock file

Checking lock status

ls -la /tmp/rclone-internxt.lock

Removing a stale lock

If mirage crashed or was rebooted while a script was running, the lock file may remain. Remove it manually:

rm /tmp/rclone-internxt.lock

17. Cron Schedule

Rclone user crontab (sudo -u rclone crontab -e):

# Nightly backup at 02:00
0 2 * * * /usr/local/bin/rclone-internxt-backup.sh

# Token refresh at 01:00, 04:00, 07:00, 10:00, 13:00, 16:00, 19:00, 22:00
0 1,4,7,10,13,16,19,22 * * * /usr/local/bin/rclone-refresh-token.sh

# Weekly integrity check every Monday at 03:00
0 3 * * 1 python3 /usr/local/bin/rclone-internxt-integrity-check.py

18. Log Files

Checking the current backup log

cat /var/log/rclone/internxt-media-backup.log

Checking for alerts

cat /var/log/rclone/internxt-alerts.log
cat /var/log/rclone/internxt-review.list

Checking recent backup archive

zcat /var/log/rclone/internxt-media-backup-20260319.log.gz | grep -E "ALERT|ERROR|WARNING|Manifest updated"

Checking redundant files

cat /var/log/rclone/internxt-deleted.log

19. Logrotate

Configuration at /etc/logrotate.d/rclone-internxt:

/var/log/rclone/internxt.log
/var/log/rclone/internxt-token-refresh.log
/var/log/rclone/internxt-alerts.log
/var/log/rclone/msmtp.log
/var/log/rclone/internxt-integrity-check.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 640 rclone rclone
}

/var/log/rclone/internxt-deleted.log {
    monthly
    rotate 12
    compress
    missingok
    copytruncate
}

The nightly backup log is self-managed by the backup script and not included in logrotate. The review list and redundant reports are not rotated.

20. Scripts Reference

21. Routine Operations

Check last night's backup

cat /var/log/rclone/internxt-media-backup.log

Check if anything needs review

cat /var/log/rclone/internxt-review.list

Review pending files interactively

sudo -u rclone rclone-internxt-confirm-change.py --review

Confirm all pending files at once

sudo -u rclone rclone-internxt-confirm-change.py --all

Run integrity check on a specific folder

sudo -u rclone python3 /usr/local/bin/rclone-internxt-integrity-check.py \
    /mnt/lotus/music/W/Wheel/Rumination

Verify a file before restoring

sudo -u rclone python3 /usr/local/bin/rclone-internxt-verify-file.py \
    "/mnt/lotus/music/A/Artist/Album/track.flac"

Check redundant files on Internxt

sudo -u rclone python3 /usr/local/bin/rclone-internxt-redundant-list.py

Interactively delete redundant files

sudo -u rclone python3 /usr/local/bin/rclone-internxt-redundant-list.py --delete

Check mount status

service rclone-internxt status
mountpoint /mnt/internxt

Check token expiry

sudo -u rclone grep -o '"expiry":"[^"]*"' /home/rclone/.config/rclone/rclone.conf

Manually refresh token

sudo -u rclone rclone config reconnect internxt:
service rclone-internxt restart

Remove stale lock file after crash

rm /tmp/rclone-internxt.lock

22. Adding a New Backup Pair

1. Add a line to /etc/rclone/internxt-backup-matrix.csv:

   /mnt/newdisk/folder,internxt:/Media/newdisk/folder
   

2. The next nightly run will treat all files in the new source as new, upload them all, and create a new manifest file automatically. No manual steps needed.

3. Optionally run the backup immediately for the new pair only by temporarily commenting out the other pairs in the matrix and running the script manually:

   sudo -u rclone /usr/local/bin/rclone-internxt-backup.sh
   

4. If the initial upload is very large, run it manually with nohup to survive session disconnect:

   nohup sudo -u rclone rclone --config /home/rclone/.config/rclone/rclone.conf \
       copy /mnt/newdisk/folder internxt:/Media/newdisk/folder \
       --bwlimit 8M --transfers 2 --tpslimit 2 --tpslimit-burst 3 \
       --retries 15 --retries-sleep 10s \
       --log-file /var/log/rclone/internxt-initial-backup.log \
       --log-level INFO > /tmp/rclone-initial.out 2>&1 &
   

   Then add the pair to the matrix and let the nightly run handle the manifest creation.

Manual version 1.1 — May 2026