New ways to make ZFS/btrfs snapshots UNIX-y (again?)
One thing I didn't understand when I started using ZFS is why there wasn't a standard periodic snapshot tool packaged with the filesystem. Given the triumph of engineering that ZFS is, and the good taste shown in the design of the zfs and zpool utilities, it was hard for me to understand why a standard set of snapshot scripts wasn't included everywhere ZFS was used.
One can only guess at a few reasons why. Maybe the ZFS developers simply wanted a thousand flowers to bloom. They didn't want to impose their strict idea of what a periodic snapshot tool should look like. Maybe they were simply too busy shipping an OS, and they didn't really know, at the time, how snapshots would be used by their consumers. Perhaps significantly, ZFS was not a "desktop" filesystem until only recently. Apple missed their opportunity (at eternal glory!). Ubuntu is now the first mainstream OS most will encounter with ZFS on root.
Of course, not including standard scripts has probably had some positive results. Through trial and error, I/we have created or found snapshot, pruning, replication tools, etc., with which I am/we are comfortable. An ecosystem was allowed to develop in the absence of strict direction. An ecosystem that developed good tools for one, or a set of jobs, rather than all or too many.
But I think it has had some negative results, as well. Some very beautiful UNIX-y abstractions have been allowed to languish -- many of us have loads of snapshots, and many of us seem to never use them, except perhaps when we do the occasional rollback, or go hunting for an older version of a file.
Of course, ZFS loves that "most things are a file" for certain operations:
zfs send rpool/scratch@autosnap_2022-10-31_17:01:20_hourly | gzip > /srv/tmp/scratch.gz
But it seems the ZFS/btrfs communities still haven't wrapped their collective heads around the idea that snapshot datasets can be useful everywhere, especially at the file (read: not dataset) level. This is the reason why Jim Salter's findoid was such a revelation for me, and such a huge inspiration for httm. It was the first CLI tool I saw which made an attempt to deal with snapshots on a file level.
And httm attempts to take that very broad idea to its logical conclusion. See:
Of course, httm does lots of neat stuff on its own, but I won't describe all that stuff here. If you're interested in the specifics of httm, please take a look at the README. I could spend all day (okay, maybe an hour) talking about the things httm does simply by itself. But what is of most interest is to me is the ways it could be used. I've tried to spark some curiosity by showing the very UNIX-y ways I use httm, see, for instance, to create tar and git archives. However, shell snippets are just proofs of concept. I think what people (like me) need are fully working scripts/models of how httm could be used to do new and interesting and UNIX-y things.
So -- the following are two fully formed (that is, at the limits of my bash scripting abilities!) examples of how you might use httm to create other new and interesting tools:
ounce (codename: "dimebag") is a wrapper script for httm which gives you no mental overhead, non-periodic dynamic snapshots.
When I type ounce nano /etc/samba/smb.conf (I actually alias 'nano'='ounce nano'), ounce knows that it's smart and I'm dumb, and it sees that I just edited /etc/samba/smb.conf a few short minutes ago and it takes a snapshot before I edit the file again because it wants me to have snapshots of all my file changes.
Of course, we can check with httm:
➜ httm /etc/samba/smb.conf
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Tue Aug 09 16:46:14 2022 17.6 KiB "/.zfs/snapshot/autosnap_2022-10-19_20:00:22_hourly/etc/samba/smb.conf"
Fri Oct 21 09:11:17 2022 17.6 KiB "/.zfs/snapshot/snap_2022-10-21-09:11:18_ounceSnapFileMount/etc/samba/smb.conf"
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Fri Oct 21 09:11:25 2022 17.6 KiB "/etc/samba/smb.conf"
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
bowie is a wrapper script for httm which quickly displays the difference (with colors) between unique snapshot versions and the live file.
A search, in the default, last change mode:
➜ bowie `which bowie`
/usr/bin/bowie
__
Files /.zfs/snapshot/snap_2022-10-31-08:34:09_ounceSnapFileMount/usr/bin/bowie and /usr/bin/bowie differ
69c69
< while read line; do
---
> while read -r line; do
Each script contains more options and features than can be described here. I'm interested in your thoughts on what new options to add, or other potential scripts ideas. But I'm even more interested in your scripts, and what new features you'd like me to add to httm to make new scripts possible/better.
ounce and bowie are available in httm version 0.16.7.