Some new servers arrived. They work, and now I have a plan.
I've been worried about single points of failure -- using local disk only for VMs seems so weird now that I've been spoiled by the cloud. But also I don't want to run Ceph because it's a bit of a mare, especially at small scale.
I thought about storage servers but unless you have two, then that's a SPOF. I just want to be able to live-migrate VMs because I need to do maintenance, or dead-migrate them because a machine failed.
I think that I've arrived at a bit of an answer? DRBD.
Basically, take my two similar HP DL3x0 Gen8 machines and configure their disks identically as single-disk RAID0 arrays(2x300GiB SAS, 1x500GiB SATA SSD). Create volume groups from each disk and use DRBD for the actual block device used by the VMs. This way the primary can be defined on a per-DRBD basis, and ahead of live migration a resource can be put into primary-primary mode temporarily.
I look forward to making it work. I'll also need to reinstall both machines and take advantage of the onboard-USB drive slot to move the OS to a pen drive instead of booting from the SAS controller.
The Cisco 1U server that I ordered is pretty great and arrived in okay state. I was pleased to find three PCIe slots -- a half height, a full height and a little dedicated one for the RAID controller. On my HP the P420i takes up a whole slot.
Then I couldn't get the PCIe SSD to show up. Weird.
And then I read the inside of the case, where the instructions say that if you don't have a second CPU installed (and I don't) then the second PCIe slot is inactive. Womp! I'm still happy with the machine, and this makes sense, but it's a little disappointing.
Continued to think about auth because I'm also thinking about Cognito, OneLogin and Auth0 for work. I think a PocketID could be all that Colocataires needs for its customer auth, freeing us from the burden of writing our own login and signup systems.