Using `httm` and ZFS to detect file modifications for `dano` (or getting a little sleazy to do a little good)

2022.10.10 [Updated: 2022.10.10] :: {programming, misc.} :: #httm #ZFS #dano #checksum #hash #flac #ffmpeg

One thing ZFS doesn't do (and thank goodness it doesn't?), it doesn't detect out-of-band changes to files. If a program has the permissions to modify a file, ZFS happily modifies that file. But, for some files (and for the truly paranoid), we may want to keep additional metadata about file integrity. For instance, FLAC files keep checksums of its music streams. A program I wrote, dano, provides a way to do the same for all FFMPEG compatible media streams.

And perhaps you want to read back and verify such media bitstreams occasionally as one might in a zpool scrub, of course dano lets you do that too:

% find /tank/Media -type f | dano --test

But doesn't verifying all those checksums take a long time? And don't you already do this when you do a zpool scrub? Good point! Wouldn't it be nice to only test those files which ZFS tells us have changed (if you trust ZFS)? Is it wrong/sleazy to not verify these checksums if we know the underlying bits haven't changed? Maybe httm can help us be a little sleazy:

% httm --num-versions ~/program/httm/*
"/srv/program/httm/Cargo.lock" : 3 Versions available.
"/srv/program/httm/Cargo.toml" : 4 Versions available.
"/srv/program/httm/LICENSE" : 1 Version available.
"/srv/program/httm/README.md" : 2 Versions available.
"/srv/program/httm/httm.1" : 3 Versions available.
"/srv/program/httm/packaging" : 1 Version available.
"/srv/program/httm/scripts" : 1 Version available.
"/srv/program/httm/src" : 6 Versions available.
"/srv/program/httm/target" : 1 Version available

Of course, this is really great for determining, at a glace, "Wow, some of these files in /etc are seeing lot's of churn. I have more questions...". But maybe what I really want, right now, is files with a single version available but which have been around long enough to have at least one snapshot available, httm has you covered:

% httm --num-versions=single-with-snap ~/program/httm/*
"/srv/program/httm/LICENSE"
"/srv/program/httm/packaging"
"/srv/program/httm/scripts"
"/srv/program/httm/target"

And we might now also rewrite our little dano script to verify only those files with multiple versions that ZFS detected:

% find /tank/Media -type f | httm --num-versions=multiple | dano --test

Available in httm version 0.15.2.