kimono koans' blog!

A cross platform, filesystem independent Time Machine (which directly supports your TM backups)

2023-12-15T00:00:00+00:00

You can have it all -- the convenience of Time Machine backups, and the speed and flexibility of httm</code>.</p>

Of course, one could always use httm</code> to browse all files in a folder backed up via rsync</code> to a remote ZFS or btrfs share, if that share has a hidden snapshot directory:</p>

# mount the share
➜ open smb://<your name>@<your remote share>.local/Home
# execute httm 
➜ httm -b -R /Volumes/Home
</code></pre>
Or one could use httm</code> to browse all files in your local MacOS home directory, if you mapped that directory to a remote dataset. Note: The difference from above is, here, you would be browsing files from a "live" directory:</p>
# mount the share
➜ open smb://<your name>@<your remote share>.local/Home
# execute httm
➜ httm -b -R --map-aliases /Users/<your name>:/Volumes/Home
</code></pre>
Now, httm</code> supports your Time Machine backups directly, when those backups are mounted</a>:</p>
➜ httm .zshrc
─────────────────────────────────────────────────────────────────────────────────────
Tue May 09 22:57:09 2023  6.7 KiB  "/Volumes/.timemachine/842A693F-CB54-4C5A-9AB1-C73681D4DFCD/2023-11-08-212757.backup/2023-11-08-212757.backup/Data/Users/kimono/.zshrc"
Sun Nov 12 20:29:57 2023  6.7 KiB  "/Volumes/.timemachine/842A693F-CB54-4C5A-9AB1-C73681D4DFCD/2023-11-18-011056.backup/2023-11-18-011056.backup/Data/Users/kimono/.zshrc"
Sun Nov 26 02:14:56 2023  6.7 KiB  "/Volumes/.timemachine/842A693F-CB54-4C5A-9AB1-C73681D4DFCD/2023-12-13-054342.backup/2023-12-13-054342.backup/Data/Users/kimono/.zshrc"
─────────────────────────────────────────────────────────────────────────────────────
Sun Nov 26 02:14:56 2023  6.7 KiB  "/Users/kimono/.zshrc"
─────────────────────────────────────────────────────────────────────────────────────
</code></pre>
httm</a> prints the size, date and corresponding locations of available unique versions (deduplicated by modify time and size) of files residing on snapshots, but can also be used interactively to select and restore files, even snapshot mounts by file! httm might change the way you use snapshots (because ZFS/BTRFS/NILFS2 aren't designed for finding for unique file versions) or the Time Machine concept (because httm is very fast!).</p>
For more info, see the README</a>.</p>



All your snapshots, at your fingertips, enables NILFS2-inspired ZFS snapshots
2023-12-04T00:00:00+00:00
Of course, ZFS loves that "most things are a file" for certain operations:</p>
zfs send rpool/scratch@autosnap_2022-10-31_17:01:20_hourly | gzip > /srv/tmp/scratch.gz
</code></pre>
However, I don't think the ZFS/btrfs/NILFS2 communities have wrapped their collective heads around the idea that snapshot datasets can be useful everywhere, especially at the file (read: not dataset) level.</p>
This is the reason why Jim Salter's findoid</a> was such an inspiration for httm</code> (along with fzf</code>, zsh</code> key bindings, and rg</code>).  It was the first CLI tool I saw which made an attempt to deal with snapshots on a file level.  One could always go looking for file versions, but findoid</code> made an effort to consolidate useful snapshot information about a specific file.</p>
httm</a> attempts to take Jim's very broad idea to its logical conclusion.  httm</code> is an interactive, file-level Time Machine-like tool for ZFS/btrfs/NILFS2.  See:</p>
</a></p>
Put simply -- httm</code> aims to make all your snapshot data more easily accessible to all your UNIX tools, specifically, at the file level, because that's where UNIX tools live, so you can create new and interesting things for yourself.</p>
So, what, in practice, does this mean?  It means -- as a ZFS guy, one NILFS2 feature that always impressed me was how it seems to "take" continuous snapshots.  In addition to checkpoints, one can stop a NILFS2 filesystem from garbage collecting, roll back to a point in time, and recover a file that was once there. And, this seems great, if you can take the performance hit of a log-structured file system (which is substantial).  Just as I don't understand how people live without snapshots, I didn't want to live without "continuous" snapshots.  With httm</code>'s help, my ZFS alternative to NILFS2 is simple -- instead of continuous snapshots, why don't we simply take a snapshot, if we need to, before we do something silly.</p>
ounce</a> (codename: "dimebag") is a wrapper script for httm</a> which gives you no mental overhead, non-periodic dynamic snapshots.</p>
When I type ounce nano /etc/samba/smb.conf</code> (I actually alias $EDITOR='ounce --trace $EDITOR'</code>), ounce</code> knows that it's smart and I'm dumb, and it sees that I just edited /etc/samba/smb.conf</code> a few short minutes ago, because it uses seccomp and eBPF to trace all the file open calls my $EDITOR makes, and uses httm</code> to check whether I have any un-snapshot-ed changes.  If I do, it takes a snapshot before I edit the file again.</p>
Instead of an auto-save, you have auto-snapshot. Of course, we can confirm the snapshot was taken with httm</code>:</p>
➜ httm /etc/samba/smb.conf
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Tue Aug 09 16:46:14 2022  17.6 KiB  "/.zfs/snapshot/autosnap_2022-10-19_20:00:22_hourly/etc/samba/smb.conf"
Fri Oct 21 09:11:17 2022  17.6 KiB  "/.zfs/snapshot/snap_2022-10-21-09:11:18_ounceSnapFileMount/etc/samba/smb.conf"
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Fri Oct 21 09:11:25 2022  17.6 KiB  "/etc/samba/smb.conf"
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
</code></pre>
ounce</code> is just one example of a new use enabled by snapshot data at your finger tips.  I hope you'll find inspiration to use it to create something new and interesting for yourself too.</p>


Why the Rust Trademark Policy was such a problem...
2023-04-11T00:00:00+00:00
The YouTubers had a very good time telling the Foundation/project it "doesn't understand the internet"</a> and other such nonsense.  The Zig VP of Community was able to gloat</a>.  Notwithstanding, I happen to think the trademark policy itself will be resolved with, relative to the heated rhetoric right now, little controversy.  The Foundation asked for comment.  It got comments.  It will adjust.  We can stop the rending of our garments.</p>
However, there is an established narrative about the Rust project, which like it or not, fair or not, is: "These people don't just want to control how I use memory" (this is a joke</em>).  They have opinions about what are appropriate politics.  And now they think their trademark policy</em> is going to prevent malware.</p>
Of course, this narrative isn't the problem.  The problem is actually trying to control anything and everything</em> Rust related.  I know it can be hard to watch your FOSS baby used in a way you didn't intend, but very clearly I think the community, when given the choice between being rigidly controlled (which may have some speculative virtues), and being diverse and vibrant and noisy (which may have some speculative vices), will choose diverse and vibrant and noisy.  The Rust project and the Foundation can bristle at that.  And maybe even fight it.  And they may even think a trademark policy is a good way to do it, but Tide and Clorox don't have a community, they have customers.  The reason why the Rust mark has any value is that there is a community of people who love using it.</p>
This is self-evidently a fissure between the community, and the Foundation and project.  That's not a problem, if our leadership works through it.  Yet the difficulty of not having a BDFL, but having a nameless/faceless Executive Director of the Foundation, is, right now, in this moment, there appear to be no leaders within Rust to say the measured and reasonable thing.  Something like: "This missed the mark.  We're going to consider changes to the policy itself, but also to our processes for consideration of such a policy."  For one thing, it appears as if no one sent comments to the attorney asking: "Why is this necessary?  Is there a lighter touch way to achieve X and Y goals?  What are the difficulties with the Python model?"  Staff of nine, and board of twelve, and no one asking these questions, and, significantly, no one available to explain the policy</em>, is a failure of leadership.</p>
I want to be very clear -- I'm not saying heads should roll.  I'm saying someone needs to take the wheel.</p>


Use `httm` to convert your ZFS/btrfs snapshots to `git` archives
2023-01-31T00:00:00+00:00
Introducing nicotine</a>, the perfect complement to tar</code>.  (Yes, I do need a better name for this script.)  nicotine</code> is a wrapper script for httm</a>1</a></sup> which converts unique file versions on snapshots to git</code> archives.</p>
Although a proof of concept script which accomplishes something similar has been included in the httm</code> README.md</a> for awhile, if you need your snapshots on the go, and if you can't just send and receive the bits, you shouldn't have to copy and paste a script.  Perhaps you also deserve some polish and some basic sanity checks!</p>
First, create a snapshot archive:</p>
➜ nicotine ~/program/httm/*
nicotine archive created successfully: /home/kimono/Cargo.lock-snapshot-archive.tar.gz
nicotine archive created successfully: /home/kimono/Cargo.toml-snapshot-archive.tar.gz
/srv/program/httm/debian is an empty directory. Skipping.
nicotine archive created successfully: /home/kimono/httm.1-snapshot-archive.tar.gz
nicotine archive created successfully: /home/kimono/LICENSE-snapshot-archive.tar.gz
nicotine archive created successfully: /home/kimono/packaging-snapshot-archive.tar.gz
nicotine archive created successfully: /home/kimono/README.md-snapshot-archive.tar.gz
nicotine archive created successfully: /home/kimono/scripts-snapshot-archive.tar.gz
nicotine archive created successfully: /home/kimono/src-snapshot-archive.tar.gz
</code></pre>
Next, expand that archive wherever you need it:</p>
➜ tar zxvf httm.1-snapshot-archive.tar.gz
./httm.1/
./httm.1/httm.1
./httm.1/.git/
...
</code></pre>
Then, use it as you would any git repo.  For instance, view changes to files via git log -p</code>:</p>
➜  cd httm.1; git log -p
commit c2cd0f4c7728b3f3dcd7d87d7c24334b0d5b5d41 (HEAD -> master)
Author: ...
Date:   Mon Jan 30 14:53:30 2023 -0600

    httm commit from ZFS snapshot

diff --git a/httm.1 b/httm.1
index 3831a10..e895f53 100644
--- a/httm.1
+++ b/httm.1
@@ -1,9 +1,9 @@
    .\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.49.3.
-.TH HTTM "1" "January 2023" "httm 0.20.0" "User Commands"
+.TH HTTM "1" "January 2023" "httm 0.20.1" "User Commands"
    .SH NAME
-httm \- manual page for httm 0.20.0
+httm \- manual page for httm 0.20.1
    .SH DESCRIPTION
-httm 0.20.0
+httm 0.20.1
...
</code></pre>
^1</sup>httm</a> prints the size, date and corresponding locations of available unique versions (deduplicated by modify time and size) of files residing on snapshots, and can also be used interactively</em> to select, restore, and snapshot files.</p>
</div>


A Somewhat Opinionated Guide to Effective ZFS Snapshots
2022-12-14T00:00:00+00:00
Much too long ago, I was asked by a user</a> of httm</a> to describe the way I (over)use snapshots, and, though I said I would explain, I never thought I had anything important enough to say about best practices.</p>
However, I'm increasingly seeing posts on r/ZFS requesting instruction of how best to use the snapshot mechanisms of ZFS, and though I can only share my opinions</em> regarding my perhaps very idiosyncratic</em> snapshot setup, maybe these opinions could be helpful for those just starting out.</p>
Of course, this Getting Started guide will make some assumptions that may not apply to your setup.  This guide assumes you're using:</p>

ZFS for a data and a root pool</li>
Running Ubuntu</li>
</ul>
But I'm certain at least some of this advice will be useful even if you're on a different setup.</p>
I believe an effective snapshot scheme is composed of possibly three types of snapshots:</p>

Periodic Snapshots</a></li>
Triggered Snapshots</a></li>
Dynamic Snapshots</a></li>
</ol>
Let's discuss each.</p>
Periodic Snapshots§</a>
</h2>
Periodic snapshots, or snapshots taken at a regular intervals, should be considered the base of any good snapshot scheme, and I wholeheartedly recommend sanoid</a> as a periodic snapshot tool.  For those that don't already know, sanoid</code> describes itself as "a policy-driven snapshot management tool for ZFS filesystems".  Although other tools exist and perhaps deserve mention, sanoid</code> and syncoid</code>, its replication tools, have made periodic snapshots and replication simple for me.</p>
But note -- like any good tool -- sanoid</code> isn't a perfect fit for every use case (and shouldn't be, see later zsys</code> and the "Curse of Trying to Do Too Much").  But what sanoid</code> is</em> is simple and composable.</p>
To my mind, what makes sanoid</code> great is that it doesn't force you into using all its features, all the time.  It doesn't force you into a complicated policy/scheme (even I can do it!), and since it includes such good documentation</a>, I won't waste time discussing how to initially configure your pools here.</p>
Instead, I'd like to highlight, how, in contrast to some other tools, it allows you to compose it with your other Linux utilities, and doesn't break when you do something a little different.  For instance, suppose you want to sleep your NAS drives occasionally and not invoke sanoid</code> when your drives are sleeping.  Here's how this user solved this particular problem.</p>
First, you'll need to check if any of the spinning rust drives are sleeping:</p>
find /dev/disk/by-id/ -type l | \
grep -v -e part -e wwn | \
while read disk; do
 if [[ $(lsblk -o rota "$disk" | grep -c "1") -gt 0 ]]; then
  smartctl -d sat --nocheck=standby "$disk"
 fi
done
</code></pre>
Then, it becomes simple to check that condition whenever you invoke sanoid</code>:</p>
...
[[ $( /usr/local/sbin/checkHDstatus | grep -i 'Device' | /usr/bin/grep -i -c 'STANDBY' ) -gt 0 ]] || \
/usr/sbin/sanoid --prune-snapshots --verbose --configdir=/etc/sanoid/
...
</code></pre>
Now imagine you want to use an alternate configuration file when that spinning rust is asleep.  Just change the configuration file path:</p>
...
if [[ $( /usr/local/sbin/checkHDstatus | grep -i 'Device' | /usr/bin/grep -i -c 'STANDBY' ) -gt 0 ]]; then
 /usr/sbin/sanoid --take-snapshots --verbose --configdir=/etc/sanoid/awake/
else
 /usr/sbin/sanoid --take-snapshots --verbose --configdir=/etc/sanoid/sleep/
fi
...
</code></pre>
sanoid</code> handles all this in stride.  It doesn't panic when you skip a few hours worth of snapshots on a certain pool.  It doesn't need to control heaven and earth.  It just keeps trucking.</p>
Replicating§</a>
</h3>
Pop Quiz Hotshot: now that you've made a few snapshots, how would you replicate your rpool</code> to your local datapool</code> using your own custom zfs send/recv</code> options?</p>
Easy peasy, you say. Just like sanoid</code>, syncoid</code> has simple options, sane defaults, and an ability to cut a rug when you need to.</p>
/usr/sbin/syncoid -r --sendoptions="L ec" --recvoptions="o recordsize=1M o compression=zstd" \
--force-delete --exclude=scratch --exclude=test --exclude=tmp rpool datapool/rpool 2>&1 | logger -t syncoid
</code></pre>
Triggered Snapshots§</a>
</h2>
Canonical's zsys</code> promised snapshots of every system update and seamless rollback on boot as well as</em> periodic snapshots of significant datasets/directories.  It's unfortunate zsys</code> wasn't ready for the Ubuntu 22.04 release, and may never be ready.  I won't rehash "Why?" here, but one basic zsys</code> premise is sound: Periodic Snapshots are not enough</em>.</p>
You may ask -- why?  Because, for me, it's sometimes important to know that a snapshot was triggered on a date and time certain.  Let's discuss a few examples of triggers you might like to use for a snapshot and how you might take those snapshots.</p>
Before a System Upgrade§</a>
</h3>
The first triggers we might consider are snapshots upon apt upgrade</code> and kernel updates.</p>
First, you'll need a snapshot script to execute (perhaps called /usr/local/sbin/snapPrepApt</code>):</p>
DATE="$( /bin/date +%F-%T )"
# FYI a user helpfully notes there may be some issue with snapshot-ing a bpool and GRUB
# See: https://github.com/kimono-koans/httm/issues/11#issuecomment-1860329869
#zfs snapshot -r bpool@snap_"$DATE"_prepApt
zfs snapshot -r bpool/BOOT@snap_"$DATE"_prepApt
zfs snapshot rpool@snap_"$DATE"_prepApt
zfs snapshot -r rpool/ROOT@snap_"$DATE"_prepApt
zfs snapshot -r rpool/USERDATA@snap_"$DATE"_prepApt
</code></pre>
Next, you'll need to execute such a script automatically upon apt upgrade</code>.  A simple script in /etc/apt/apt.conf.d</code> will suffice:</p>
// Takes a snapshot of the system before package changes.
DPkg::Pre-Invoke {"[ -x /usr/local/sbin/snapPrepApt ] && /usr/local/sbin/snapPrepApt || true";};
</code></pre>
And you will also probably want to execute a script each time you update your kernel.  A script invoked from /etc/kernel/preinst.d</code> might look something like:</p>
[ -x /usr/local/sbin/snapPrepApt ] && /usr/local/sbin/snapPrepApt || true
</code></pre>
Before Service Launch§</a>
</h3>
Sometimes you will want to take a snapshot when a service starts up or shuts down.  For instance perhaps you have a service, with a database, that needs to be cleanly shutdown so that its state can also be cleanly snapshot-ed.</p>
Just add a little script to execute before or after start up via systemctl edit</code>:</p>
...
[Service]
ExecStartPre=/bin/bash -c "/usr/local/sbin/snapDataService"
...
</code></pre>
On Network Mount§</a>
</h3>
Perhaps you want to take a snapshot every time a network drive is mounted or unmounted.  So, when you or a program deletes something over the network, you have snapshot of the state just prior to mount or just after unmount.</p>
Your smb.conf</code> allows you to execute scripts just like this:</p>
...
[TM Volume]
path = "/srv/timemachine"
valid users = timemachine
read only = no
wide links = no
create mask = 0740
directory mask = 0750
root preexec = "/usr/local/sbin/TMpre"
root postexec = "/usr/local/sbin/TMpost"
...
</code></pre>
Watch a Directory§</a>
</h3>
Maybe you only want to take a snapshot of a folder when new files are added to it.  inotifywait</code> is a wonderful tool for just that use case (and many others!):</p>
inotifywait -m -e moved_to "/srv/downloads/" | while read -r line; do
 snapDownloads
done
</code></pre>
Cleanup§</a>
</h3>
Now that you've made these snapshots, if you're anything like me, you must</em> clean them up once they aren't needed anymore.  I suggest zfs-prune-snapshots</a> for this task.</p>
Just run as a cron script daily to cleanup any triggered or dynamic snapshots that have outlived their usefulness:</p>
/usr/local/sbin/zfs-prune-snapshots -s '_prepApt' 2w 2>&1 | logger -t sanoid
</code></pre>
Dynamic Snapshots§</a>
</h2>
"Dynamic Snapshots" is a term I'll credit myself for, at least, popularizing.  Dynamic Snapshots are very similar to Triggered Snapshots.  In fact, I think it is appropriate to think of Dynamic Snapshots as simply a breed of Triggered Snapshots.  The key difference is how ad-hoc a Dynamic Snapshot feels</em>.</p>
I'll try to give a few examples.</p>
Imagine -- you're in a folder and you realize you're about to change a bunch of files, and you want a snapshot of the state of the folder before you make any edits.  You don't know precisely which dataset your working directory resides.  And you're not really in the mood to think about it.  Of course, you could just copy that folder, rename it, and make edits in the new folder. Or you could do the same with each file as you go.  Or you could determine the dataset upon which this folder is located and manually do a snapshot yourself.</p>
All of these things you could do, feel</em> like leg work.  A Dynamic Snapshot shouldn't feel like leg work.  A Dynamic Snapshot should just work by determining the ZFS mount of the $PWD</code> and taking a snapshot immediately.</p>
For instance, with httm</code>, you might invoke a Dynamic Snapshot like so:</p>
➜ httm -S .
httm took a snapshot named: rpool/ROOT/ubuntu_tiebek@snap_2022-12-14-12:31:41_httmSnapFileMount
</code></pre>
In order for the above to work you will need permissions.  The most elegant way to give a user ZFS permissions is now through the new zfs allow</code> function.  If you have httm</code> installed on Ubuntu, running ounce --give-priv</code>, as an ordinary user, will execute something like the block below to give an ordinary user (you!) permissions to snapshot on all pools.</p>
for pool in "$( sudo zpool list -o name | grep -v -e NAME )"; do
 sudo zfs allow "$( whoami )" mount,snapshot "$pool"
done
</code></pre>
Speaking of ounce</a>, ounce</code> is a script I wrote which wraps a target executable, can trace its system calls, and will execute snapshots before</em> you do something silly.  ounce</code> is my canonical example of a dynamic snapshot script.  When I type ounce nano /etc/samba/smb.conf</code> (I actually alias 'nano'='ounce --trace nano'</code>), ounce</code> knows that it's smart and I'm dumb, so -- it traces each file open call, sees that I just edited /etc/samba/smb.conf</code> a few short minutes ago.  Once ounce</code> sees I have no snapshot of those file changes, it takes a snapshot of the dataset upon which /etc/samba/smb.conf</code> is located, before I edit and save the file again.</p>
We can check that ounce</code> worked as advertised via httm</code>:</p>
➜ httm /etc/samba/smb.conf
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Fri Dec 09 07:45:41 2022  17.6 KiB  "/.zfs/snapshot/autosnap_2022-12-13_18:00:27_hourly/etc/samba/smb.conf"
Wed Dec 14 12:58:10 2022  17.6 KiB  "/.zfs/snapshot/snap_2022-12-14-12:58:18_ounceSnapFileMount/etc/samba/smb.conf""
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Wed Dec 14 12:58:10 2022  17.6 KiB  "/etc/samba/smb.conf"
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
</code></pre>
Conclusion§</a>
</h2>
Good luck with your future snapshot adventures!  I'd love to know your thoughts on how you do it better/more cleverly!</p>


New ways to make ZFS/btrfs snapshots UNIX-y (again?)
2022-10-31T00:00:00+00:00
One thing I didn't understand when I started using ZFS is why there wasn't a standard periodic snapshot tool packaged with the filesystem.  Given the triumph of engineering that ZFS is, and the good taste shown in the design of the zfs</code> and zpool</code> utilities, it was hard for me to understand why a standard set of snapshot scripts wasn't included everywhere ZFS was used.</p>
One can only guess at a few reasons why.  Maybe the ZFS developers simply wanted a thousand flowers to bloom.  They didn't want to impose their strict idea of what a periodic snapshot tool should look like.  Maybe they were simply too busy shipping an OS, and they didn't really know, at the time, how snapshots would be used by their consumers.  Perhaps significantly, ZFS was not a "desktop" filesystem until only recently.  Apple missed their opportunity (at eternal glory!).  Ubuntu is now the first mainstream OS most will encounter with ZFS on root.</p>
Of course, not including standard scripts has probably had some positive results.  Through trial and error, I/we have created or found snapshot, pruning, replication tools, etc., with which I am/we are comfortable.  An ecosystem was allowed to develop in the absence of strict direction.  An ecosystem that developed good tools for one, or a set of jobs, rather than all or too many.</p>
But I think it has had some negative results, as well.  Some very beautiful UNIX-y abstractions have been allowed to languish -- many of us have loads of snapshots, and many of us seem to never use them, except perhaps when we do the occasional rollback, or go hunting for an older version of a file.</p>
Of course, ZFS loves that "most things are a file" for certain operations:</p>
zfs send rpool/scratch@autosnap_2022-10-31_17:01:20_hourly | gzip > /srv/tmp/scratch.gz
</code></pre>
But it seems the ZFS/btrfs communities still haven't wrapped their collective heads around the idea that snapshot datasets can be useful everywhere, especially at the file (read: not dataset) level.  This is the reason why Jim Salter's findoid</code> was such a revelation for me, and such a huge inspiration for httm</code>.  It was the first CLI tool I saw which made an attempt to deal with snapshots on a file level.</p>
And httm</code> attempts to take that very broad idea to its logical conclusion.  See:</p>
</a></p>
Of course, httm</code> does lots of neat stuff on its own, but I won't describe all that stuff here.  If you're interested in the specifics of httm</code>, please take a look at the README</a>.  I could spend all day (okay, maybe an hour) talking about the things httm</code> does simply by itself.  But what is of most interest is to me is the ways it could be used</em>.  I've tried to spark some curiosity by showing the very UNIX-y ways I use httm</code>, see, for instance, to create tar</code> and git</code> archives</a>.  However, shell snippets are just proofs of concept.  I think what people (like me) need are fully working scripts/models of how httm</code> could be used to do new and interesting and UNIX-y things.</p>
So -- the following are two fully formed (that is, at the limits of my bash</code> scripting abilities!) examples of how you might use httm</code> to create other new and interesting tools:</p>
ounce</a> (codename: "dimebag") is a wrapper script for httm</a> which gives you no mental overhead, non-periodic dynamic snapshots.</p>
When I type ounce nano /etc/samba/smb.conf</code> (I actually alias 'nano'='ounce nano'</code>), ounce</code> knows that it's smart and I'm dumb, and it sees that I just edited /etc/samba/smb.conf</code> a few short minutes ago and it takes a snapshot before I edit the file again because it wants me to have snapshots of all my file changes.</p>
Of course, we can check with httm</code>:</p>
➜ httm /etc/samba/smb.conf
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Tue Aug 09 16:46:14 2022  17.6 KiB  "/.zfs/snapshot/autosnap_2022-10-19_20:00:22_hourly/etc/samba/smb.conf"
Fri Oct 21 09:11:17 2022  17.6 KiB  "/.zfs/snapshot/snap_2022-10-21-09:11:18_ounceSnapFileMount/etc/samba/smb.conf"
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Fri Oct 21 09:11:25 2022  17.6 KiB  "/etc/samba/smb.conf"
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
</code></pre>
bowie</a> is a wrapper script for httm</a> which quickly displays the difference (with colors) between unique snapshot versions and the live file.</p>
A search, in the default, last change mode:</p>
➜ bowie `which bowie`
/usr/bin/bowie
__
Files /.zfs/snapshot/snap_2022-10-31-08:34:09_ounceSnapFileMount/usr/bin/bowie and /usr/bin/bowie differ
69c69
<		while read line; do
---
>		while read -r line; do
</code></pre>
Each script contains more options and features than can be described here.  I'm interested in your thoughts on what new options to add, or other potential scripts ideas.  But I'm even more interested</em> in your scripts</em>, and what new features you'd like me to add to httm</code> to make new scripts possible/better.</p>
ounce</code> and bowie</code> are available in httm</code> version 0.16.7</a>.</p>


ounce (codename: 'dimebag'): Dynamic File-Level ZFS Snapshots
2022-10-21T00:00:00+00:00
ounce</a> (codename: "dimebag") is a wrapper script for httm</a> which give you no mental overhead, non-periodic dynamic snapshots.</p>
On Hacker News</a>, there was an article about how NILFS seems to "take" continuous snapshots.  In addition to checkpoints, one can stop a NILFS filesystem from garbage collecting, roll back to a point in time, and recover a file that was once there.  And, this seems great, if you can take the performance hit of a log-structured file system (which is substantial).</p>
But perhaps like my boy Ben Franklin (or was it my mother, or Ben Franklin's mother?) said "an ounce of prevention is worth a pound of cure"?  Why not just take a snapshot before you do something silly?</p>
For awhile, one has been able to just feed httm</code> file names and have it figure out which dataset those files resided on.  Like so:</p>
➜  printf "/var/log/syslog\n/etc/passwd\n" | sudo httm --snap
httm took a snapshot named: rpool/var/log@snap_2022-10-21-08:57:04_httmSnapFileMount
httm took a snapshot named: rpool@snap_2022-10-21-08:57:04_httmSnapFileMount
</code></pre>
But I don't know about you, I don't usually feed file names into a pipe before I type Name of $EDITOR redacted to prevent flame war</code> or rm</code>?</p>
When I type:</p>
# nano used here to frustrate just you
➜  nano /etc/samba/smb.conf
</code></pre>
If there is not a snapshot that matches the live version, I would like my computer to snapshot /etc/samba/smb.conf</code> before I do something stupid.  ounce</code> does this for me and now for you.</p>
# why do you hate nano so much?
➜  ounce nano /etc/samba/smb.conf
</code></pre>
ounce</code> knows that it's smart and I'm dumb, and sees that I just edited /etc/samba/smb.conf</code> a few short minutes ago and takes a snapshot before I edit the file again because it wants me to have snapshots of all my file changes.  You can check:</p>
➜  httm /etc/samba/smb.conf
─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Tue Aug 09 16:46:14 2022  17.6 KiB  "/.zfs/snapshot/autosnap_2022-10-19_20:00:22_hourly/etc/samba/smb.conf"
Fri Oct 21 09:11:17 2022  17.6 KiB  "/.zfs/snapshot/snap_2022-10-21-09:11:18_httmSnapFileMount/etc/samba/smb.conf"
─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Fri Oct 21 09:11:25 2022  17.6 KiB  "/etc/samba/smb.conf"
─────────────────────────────────────────────────────────────────────────────────────────────────────────────
</code></pre>
For more fun, simply alias to the commands you make modifications with:</p>
# don't get triggered but...
alias nano="ounce nano"
alias rm="ounce rm"
</code></pre>
Now available in httm</code> version 0.16.1</a>.</p>


Imagine You're a Goofball: Dynamic Preventative ZFS Snapshots
2022-10-19T00:00:00+00:00
On Hacker News</a>, there is an article about how NILFS seems to "take" continuous snapshots.  In addition to checkpoints, one can stop a NILFS filesystem from garbage collecting, roll back to a point in time, and recover a file that was once there.  And, this seems great, if you can take the performance hit of a log-structured file system (which is substantial).</p>
But perhaps like our mother's said "an ounce of prevention is worth a pound of cure"?  Why not just take a snapshot before you do something silly?  With httm</a>, one can now take these sorts of no mental overhead, non-periodic dynamic snapshots, which can complement</em> your current periodic snapshot scheme.</p>
Let me reiterate -- one can and should be able to</em> take a snapshot without needing to specify a dataset or snapshot name. Just feed httm</code> a filename and httm</code> will figure out the rest.</p>
See, for example:</p>
#!/bin/bash
function ounce_of_prevention {
    for a; do
        # is the argument a file?
        if [[ -f "$a" ]]; then
            local LIVE_FILE="$a"
        else
            continue
        fi

        # get last snap version of the live file?
        local LAST_SNAP="$(httm -l "$LIVE_FILE")"

        # check whether to take snap - do we have a snap of the live file already?
        # 1) if empty, live file does not have a snapshot, then take snap, or
        # 2) if live file is not the same as the last snap, then take snap
        if [[ -z "$LAST_SNAP" ]] || \
           [[ ! -z "$LAST_SNAP" && "$(stat -c %Y "$LIVE_FILE")" -ne "$(stat -c %Y "$LAST_SNAP")" ]]
        then
            # httm will dynamically determine the location of
            # the file's ZFS dataset and snapshot that mount
            sudo httm --snap "$LIVE_FILE" > /dev/null &
        fi
    done
}

ounce_of_prevention "$@"
# expressly used `nano` instead of `vim` or `emacs` to avoid a unholy war
/usr/bin/nano "$@"
</code></pre>
First, we check whether we even need to take a snapshot:</p>

The 0th condition checks whether an argument is a file.</li>
The command httm -l $LIVE_FILE</code> requests the last snapshot for the file, allowing you to test the 1st condition, which checks whether that file has a snapshot, and,</li>
The 2nd condition checks whether the live file is not the same as the last snapshot version.</li>
</ol>
Next, httm</code> makes certain that it doesn't matter whether you are editing ~/.zshrc</code> or /etc/samba/smb.conf</code>.  httm</code> dynamically determines and snapshots the appropriate mount (before you modify/screw up the file!).</p>
Now, one will have to also set up a bash alias (alias nano=nano-with-prevention</code>), and a sudoers policy (kimono ALL=(ALL) NOPASSWD: /usr/bin/httm --snap</code>), which allows these types of dynamic snapshots, but this pattern does seem to work pretty well.</p>
Does anyone else have any other fancy dynamic snapshot tricks I could add to my arsenal?</p>
The latest release of httm</a> is 0.16.0</a>. </p>


Using `httm` and ZFS to detect file modifications for `dano` (or getting a little sleazy to do a little good)
2022-10-10T00:00:00+00:00
One thing ZFS doesn't do (and thank goodness it doesn't?), it doesn't detect out-of-band changes to files.  If a program has the permissions to modify a file, ZFS happily modifies that file.  But, for some files (and for the truly paranoid), we may want to keep additional metadata about file integrity.  For instance, FLAC files keep checksums of its music streams. A program I wrote, dano</a>, provides a way to do the same for all FFMPEG compatible media streams.</p>
And perhaps you want to read back and verify such media bitstreams occasionally as one might in a zpool scrub</code>, of course dano</code> lets you do that too:</p>
% find /tank/Media -type f | dano --test
</code></pre>
But doesn't verifying all those checksums</em> take a long time?  And don't you already do this when you do a zpool scrub</code>?  Good point!  Wouldn't it be nice to only test those files which ZFS tells us have changed (if you trust ZFS)?  Is it wrong/sleazy to not verify these checksums if we know the underlying bits haven't changed?  Maybe httm</a> can help us be a little sleazy:</p>
% httm --num-versions ~/program/httm/*
"/srv/program/httm/Cargo.lock" : 3 Versions available.
"/srv/program/httm/Cargo.toml" : 4 Versions available.
"/srv/program/httm/LICENSE" : 1 Version available.
"/srv/program/httm/README.md" : 2 Versions available.
"/srv/program/httm/httm.1" : 3 Versions available.
"/srv/program/httm/packaging" : 1 Version available.
"/srv/program/httm/scripts" : 1 Version available.
"/srv/program/httm/src" : 6 Versions available.
"/srv/program/httm/target" : 1 Version available
</code></pre>
Of course, this is really great for determining, at a glace, "Wow, some of these files in /etc</code> are seeing lot's of churn.  I have more questions...".  But maybe what I really want, right now, is files with a single version available but which have been around long enough to have at least one snapshot available, httm</code> has you covered:</p>
% httm --num-versions=single-with-snap ~/program/httm/*
"/srv/program/httm/LICENSE"
"/srv/program/httm/packaging"
"/srv/program/httm/scripts"
"/srv/program/httm/target"
</code></pre>
And we might now also rewrite our little dano</code> script to verify only those files with multiple versions that ZFS detected:</p>
% find /tank/Media -type f | httm --num-versions=multiple | dano --test
</code></pre>
Available in httm</code> version 0.15.2</a>.</p>


Fun with ZFS, httm, and inotifywait
2022-06-25T00:00:00+00:00
If you've never used inotifywait</code>, it's pretty handy for when you want to snapshot a directory after a file is accessed, or written to, or moved to a watched directory, etc., like so:</p>
inotifywait -r -m --exclude "/srv/downloads/incomplete/" -e moved_to "/srv/downloads/" | while read -r line; do
	sudo /usr/sbin/zfs snapshot rpool/downloads@snap_"$( /bin/date +%F-%T )"_completionSnap
done
</code></pre>
However, imagine you're watching a large directory tree where there may be many ZFS datasets, how would you know which dataset to snapshot?</p>
httm</a> has your back!</p>
httm -m <INPUT_FILE></code> will detect and report back the mount of the ZFS dataset upon which that watched file resides, so you can use that mount/dataset in your snapshot scripts:</p>
inotifywait -r -m --format %w%f --exclude "/srv/downloads/incomplete/" -e moved_to "/srv/downloads/" | while read -r line; do
	filemount="$(httm -m $line)" 
	filedataset="$(mount | grep $filemount | cut -f1 -d' ')" 
	sudo /usr/sbin/zfs snapshot $filedataset@snap_"$( /bin/date +%F-%T )"_completionSnap 
done
</code></pre>
If you're comfortable with httm</code> making a few choices for you, httm</code> can also take a snapshot from using just the filename as input:</p>
% sudo httm --snap /var/log/syslog
httm took a snapshot named: rpool/ROOT/ubuntu_wi01vc/var/log@snap_Jun-25-2022-23:11:50_httmSnapFileMount
</code></pre>
This last feature is ZFS only for right now.</p>
Available now for 0.12.6</a> and later editions.</p>


Re: packaging, FOSS user expectations are out of alignment
2022-06-07T00:00:00+00:00
Usually when asked to build a package for your particular distribution, I say, "I'm afraid I just don't have the time, but PRs for your distribution are welcome."  But my real talk, B-side opinion on FOSS packaging, is -- when you ask a dev to package software in a specific way for you, in the FOSS world, that's entitlement, because you can do it yourself.</p>
Of course, you may refuse to use</em> packages that aren't natively packaged just the way you like. But -- for me -- it crosses a line when one whines about the packages that are available. "Ugh, only Flatpak (or deb, or rpm), how dare you, the dev, not pay attention to my tastes after I've done nothing</em>, contributed nothing</em>."</p>
Because one should build a package</em> or two before they are entitled to an opinion about the packaging a dev uses for their personal project.  Why?  Because packaging sucks. It's the last thing anyone wants to do. And when you do make a package, and no one is happy, it sucks even more.</p>
A few common refrains and my feelings --</p>

"What no pkgsrc?" "Where's my PKGBUILD?"</p>
</blockquote>
I use/like pkgsrc and have used PKGBUILD and think it's pretty neat, but I think we all know these are fairly niche systems.  People/users who use fairly niche systems should be the first to take responsibility for them, and not expect someone else to do it for them.</p>

"Just figure out my preferred packaging and distribution system (I have no idea but I imagine it's easy)."</p>
</blockquote>
Why?  The software builds on my system just fine, and there are instructions for you to build it yourself.  "But you want me as a user..."  Explain this to me -- do I want anyone as a user who is not willing to build their own package for their distribution/OS?  Because that user tends to be</em> more of a consumer (who doesn't pay) than a valued community member.</p>

"The package you distribute doesn't work on my slightly different system."</p>
</blockquote>
I totally sympathize, but this is the problem with packaging.  You ship one binary with one dependency, say libc, and guess what?  Debian 10 or whatever decides to use a version that is way older than what Ubuntu uses.  I guess we need 3 more Debian packages then...  Yeesh.</p>

"Snap/Flatpak/AppImage sucks."</p>
</blockquote>
Maybe.  At least they allow you to easily ship your dependencies.  The concept seems quite nice when: 1) there are multiple packaging models to be covered, 2) distribution maintainers are overworked and aren't packaging your software for you, and 3) their distribution's users aren't helping you or them out either.</p>


Workflow: ZFS snapshot-ed file versions to tar and git archives with httm
2022-05-07T00:00:00+00:00
For most non-programming tasks, no one actually wants to work within a git repo.  While git repos offer many benefits, including interoperating with the git ecosystem, most of the time, they're just too heavy weight</em> for most system administration tasks.</p>
Imagine you could have many of those same benefits, but only when you need them</em>.</p>
Maybe you already know what to do when you want to send all snapshot-ed versions of your syslog</code> via email:</p>
httm -n /var/log/syslog | tar -zcvf all-versions-syslog.tar.gz -T -
</code></pre>
But imagine something a little more difficult.  Imagine you've misconfigured your sysctl.conf</code> and you have just the pal you know knows how to fix it, but she's a half-a-world away.  You think -- who wants to dig through 10+ file versions</em> to identify the source of the error?  Wouldn't it be nice if she could see your changes as you made them incrementally?</p>
Use httm</code> and this just works:</p>
# create variable for file name
file="/etc/sysctl.conf"
# create git repo
mkdir ./archive-git; cd ./archive-git; git init
# copy each file version to repo and commit after each copy
for version in $(httm -n $file); do 
	cp "$version" ./ 
	git add "./$(basename $version)" 
	git commit -m "$(stat -c %y $version)"
done
# create git tar.gz archive in the parent directory
git archive --format=tar.gz -o "../archive-git-$(basename $file).tar.gz" master; cd ../
</code></pre>
And to view the your changes incrementally:</p>
tar zxvf archive-git-sysctl.conf.tar.gz 
cd ./archive-git
git log -p
</code></pre>
Is there a Unix-y workflow you'd like to see covered by httm</code>?  Let me know!</p>