metznet-channel/metznet/machines/README.md

23 KiB

metznet/machines

One file per host. Each file defines an operating-system and ends with that value as its last expression, so you can pass the file straight to guix system (guix system reconfigure -L <checkout> metznet/machines/ldap.scm). You can also reference the exported variable, e.g. (@ (metznet machines ldap) ldap.metznet.ca).

Everything shared lives in metznet/system/base-system.scm:

Base Used by Provides
%metznet-base-server-system base-server GRUB EFI, labelled disks (guix-data = /, guix-boot = /boot/efi), serial console, %metznet-server-services
%metznet-base-aws-system ldap, kerberos, vpn Server base plus aws-service-type (%metznet-aws-server-services): the EC2 launch key pair for the aws account, see below
%metznet-base-desktop-system otto Server base plus SDDM/i3, NetworkManager, nonguix substitutes, desktop setuid programs

Machines that set their own services must append to the base's list (%metznet-server-services or %metznet-aws-server-services); otherwise they silently lose the base's services.

AWS (metznet/aws.scm). The aws-pubkey one-shot fetches the instance's launch key pair from the EC2 metadata service at boot and writes it to /etc/ssh/authorized_keys.d/aws, which sshd reads for the local aws account. That login keeps working when LDAP is down, so it is the way back into the AWS hosts. The service also adds le-certs and nss-certs to the system profile.

Every machine (server or desktop) gets %metznet-services:

  • sssd reads users and groups from ldaps://ldap.metznet.ca and binds as uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca.
  • krb5 + pam_krb5 use realm METZNET.CA, with the KDC and kadmin at kerberos.metznet.ca.
  • openssh has passwords disabled. Its AuthorizedKeysCommand looks up SSH public keys in LDAP.
  • openvpn client connects to vpn.metznet.ca.
  • pam_mkhomedir and metznet-service-type (shell symlinks, le-certs).

Service dependency graph

ldap.metznet.ca  ◄── sssd / ssh pubkey lookup on every machine
      ▲
      │ ldaps:// (KDC database lives in LDAP under cn=kerberos,dc=metznet,dc=ca)
kerberos.metznet.ca  ◄── krb5 / pam_krb5 on every machine

ldap is the root of the tree. If it is down, nobody can log in on any machine except as root or from cached sssd credentials, and the KDC stops working too. Plan any LDAP work with that in mind.

Build-time environment variables

base-system.scm reads some values with getenv when the system is evaluated, not at boot. They must be set in the environment of guix system build|reconfigure|image:

Variable Effect If unset
GUIX_ROOT_PW root password (crypted with a fixed salt) root login disabled (!)
LDAP_AUTHTOK sssd bind password and ssh pubkey-command bind password sssd binds with no password and LDAP user lookups fail
LDAP_BIND_DN sssd bind DN uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca
LDAP_AUTHTOK_TYPE sssd authtok type password

⚠️ These values end up in the world-readable /gnu/store (in sssd.conf and the pubkey-command script). This is a known problem and is planned to move to metznet/services/secrets.scm.

scripts/qemu.sh and docker-compose.yml also export DOMAIN_CAPS/DOMAIN_NAME.

Out-of-band state (not in this repo)

Guix rebuilds everything under /gnu/store and /etc, but these paths hold state that must be provisioned or migrated by hand:

Path / thing Machines What it is
/var/lib/openvpn/{ca.crt,client.crt,client.key,ta.key} all VPN client credentials
/etc/letsencrypt/ ldap, kerberos certbot certificates (re-issuable)
/var/lib/slapd/{cert,chain,privkey}.pem ldap the copies of the certbot certificate slapd actually serves (olcTLS*), owned by slapd. The certbot deploy hook refreshes them and restarts slapd after each renewal
/var/lib/slapd/slapd.d/ ldap slapd cn=config (OLC) database: schema, ACLs, TLS paths, root DN
/var/lib/slapd/data/ ldap MDB data directory (the actual directory tree)
/var/lib/slapd/.last_slapadd_restore ldap timestamp of the last seed restore (see below)
EC2 key pair ldap, kerberos, vpn the launch key pair aws-pubkey installs for the aws account; managed in AWS
/var/lib/openvpn/{dh2048.pem,ipp.txt} + server cert/key vpn OpenVPN server state
/var/lib/kerberos/stash kerberos KDC master key stash. Without it the KDC cannot decrypt principals.
TPM persistent handle 0x81010001 kerberos sealed age key for secrets/secrets.enc.yaml

Day-to-day management

Test locally

On macOS, use Docker (see docker/README.md):

scripts/guix-docker.sh image metznet/machines/ldap.scm
scripts/guix-docker.sh load-run metznet/machines/ldap.scm

On Linux with KVM, use a VM on a tap bridge:

sudo scripts/network.sh start_bridge <uplink-iface>
sudo scripts/network.sh add_tap tap0
scripts/qemu.sh start metznet/machines/ldap.scm tap0 52:54:00:12:34:56

Deploy a change to a real host

On the host, from a checkout of this repo:

git pull
guix pull -C channels.scm            # only when channels.d/ changed or to update Guix
export LDAP_AUTHTOK=... GUIX_ROOT_PW=...   # see table above
sudo -E guix system reconfigure -L . metznet/machines/<host>.scm

-L . builds from the working tree, so you can deploy uncommitted edits without pushing them through the channel first. Check the result with sudo herd status (and herd status slapd / herd status kdc).

Roll back

sudo guix system roll-back            # or pick an older generation in GRUB
guix system list-generations

A roll-back only reverts the system generation. It does not touch the out-of-band state listed above.

Logs

Services log via shepherd and syslog: /var/log/messages and /var/log/shepherd.log. slapd runs in the foreground with -d <log-level>, so its output goes to shepherd. The KDC logs to SYSLOG:DEBUG:DAEMON.


Machines

base-server.scm — base-server.metznet.ca

The bare server base with only a host name. It runs no services beyond %metznet-server-services. Use it to bring up and smoke-test new hardware (disk labels, EFI boot, sssd/krb5/VPN client), or copy it as the template for a new server.

ldap.scm — ldap.metznet.ca

OpenLDAP (openldap-slapd from metznet/packages/slapd.scm, which ships the pw-totp module) plus certbot for ldap.metznet.ca. slapd listens on ldap:// and ldaps:// (no ldapi://).

slapd runs with -F /var/lib/slapd/slapd.d, so the schema, ACLs, TLS cert paths, root DN and database definitions live in the on-disk cn=config database. Its contents are described in Scheme (%ldap-slapd-config in ldap.scm), which the service's config field points at. A host with no slapd.d builds it from that. An existing slapd.d is never touched, so a change to a running server is applied with a change file (see Changing cn=config), and the live server is checked against the Scheme. Only root over ldapi:// can write cn=config, and slapd doesn't listen on ldapi://, so changes are made offline. {1}mdb's root DN (uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca) can read it over ldaps://. On every activation the service:

  1. Creates /var/run/slapd, /var/lib/slapd/{slapd.d,modules,data}, all owned by slapd.
  2. Copies pw-totp.so into /var/lib/slapd/modules (so olcModulePath can point at a stable path rather than a store path). The other modules in olcModuleLoad are built into slapd.
  3. If slapd.d has no cn=config.ldif, slapadd -n 0s the generated config into it, owned by slapd (slapd-init). Data is never loaded.
  4. Runs the seed check (see Seeding below).

Admin login. uid=admin is {1}mdb's root DN, so it bypasses the data ACLs, but there is no olcRootPW (removed by ldap/changes/0004). It logs in with its own userPassword, which is data, so it comes back with every restore and cn=config holds no secret. On a brand-new directory with no backup to restore, set it offline:

herd stop slapd
printf 'dn: uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca\nchangetype: modify\nreplace: userPassword\nuserPassword: %s\n' \
  "$(slappasswd)" | sudo -u slapd slapmodify -F /var/lib/slapd/slapd.d -n 1
herd start slapd

Tree layout that other machines rely on:

  • dc=metznet,dc=ca: search base for sssd (rfc2307bis, posixGroupAux, member)
  • ou=system,ou=accounts,dc=metznet,dc=ca: service bind accounts (uid=guix, uid=kerberos)
  • cn=kerberos,dc=metznet,dc=ca: Kerberos realm container

Seeding (backups / backup-date)

slapd-configuration takes an ordered list of LDIF files in backups. Entry i is slapadd-ed into database number i. So the list is (config.ldif data.ldif): -n 0 is cn=config, -n 1 is the first data DB. The seeds are only applied when /var/lib/slapd/.last_slapadd_restore is missing or older than backup-date. After a successful restore it writes backup-date into that file.

Caveats:

  • backup-date defaults to (current-date) at evaluation time. If you leave seeds configured without pinning backup-date, every reconfigure looks "newer" and slapd tries to slapadd again. That fails because the entries already exist, and activation errors out. Always set backup-date to when the dump was taken, or remove backups once the restore is done.
  • Seed LDIFs become local-files in the world-readable store. They contain password hashes, so prefer the manual restore below for production data.
  • backups and config can't both be set, since both would load -n 0.

Runbook: migrate LDAP to a new instance

This moves ldap.metznet.ca to new hardware or a new VM while keeping the host name, so no client config changes are needed.

1. Check the old instance matches the repo. The new host's cn=config is built from the repo, not copied, so anything live that the repo lacks would be lost. Run the two exact checks from step 6 against the old host. If the config differs, fix %ldap-slapd-config or apply the missing change files first.

2. Bring up the new host without data. Partition with labels guix-data/guix-boot, install, and reconfigure with metznet/machines/ldap.scm. Activation builds slapd.d from the generated config. slapd won't start yet because it has no TLS certificate (step 5). That is expected. Copy the /var/lib/openvpn/* credentials across if the host needs the VPN.

3. Dump the old instance. MDB supports consistent online slapcat, but stopping slapd first guarantees nothing changes after the dump:

# on the OLD host
sudo herd stop slapd
sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 0 -l /root/config.ldif   # for the record
sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 1 -l /root/data.ldif
sudo herd start slapd      # keep serving until cutover, but treat it as read-only from here

slapcat and slapadd are in the system profile via openldap-slapd in %metznet-base-packages. Check which database numbers exist first (grep '^dn: olcDatabase' config.ldif). If there is more than one data DB, dump each by its number.

4. Restore the data on the new host. Move data.ldif over with scp. Keep it out of /gnu/store. Only the data is restored; config.ldif stays with the backups.

# on the NEW host
sudo herd stop slapd
sudo rm -rf /var/lib/slapd/data/*
sudo -u slapd slapadd -F /var/lib/slapd/slapd.d -n 1 -l data.ldif
sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u    # sanity-check config

tests/ldap/first-boot.sh rehearses steps 2 and 4 in the container.

5. TLS. slapd serves copies of the certificate in /var/lib/slapd/{cert,chain,privkey}.pem, not certbot's files. Copy those three from the old host and make them readable by slapd. certbot on the new host can only complete the ACME challenge once ldap.metznet.ca resolves to it, so renewal there starts working after cutover. After each renewal, certbot's deploy hook (certificate-copy-deploy-hook in metznet/services/certificates.scm) copies the new files over and restarts slapd. To copy the current certificate by hand, run the hook itself: sudo $(guix gc -R /run/current-system | grep 'ldap.metznet.ca-certificate-copy-hook$'). The KDC validates the cert (SSL_CERT_DIR=/etc/ssl/certs), so a self-signed or expired cert will break Kerberos even though sssd (ldap_tls_reqcert never) keeps working.

6. Start and verify before cutover.

sudo herd start slapd && sudo herd status slapd
ldapsearch -H ldaps://<new-host-ip> -x -D uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca -W \
  -b dc=metznet,dc=ca '(uid=<some-user>)'
ldapsearch ... -b cn=kerberos,dc=metznet,dc=ca -s one dn     # realm container is present

Compare entry counts against the old host (grep -c '^dn:' data.ldif), and check the schema and the entries other machines depend on:

docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh   # builds build/*.ldif
LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps://<new-host-ip> -D <admin-dn> -y pwfile \
  -b dc=metznet,dc=ca -X -e build/ldap-structure.ldif
LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps://<new-host-ip> -D <admin-dn> -y pwfile \
  -b cn=config -X -e build/slapd-config.ldif

The first checks the directory structure (%ldap-structure in ldap.scm: containers, realm, policy, groups and the accounts this repo binds as) exactly while ignoring data; the second, the config (%ldap-slapd-config). Run them against the old host first so you know what a passing run looks like.

7. Cut over. Repoint ldap.metznet.ca DNS at the new host, then stop slapd on the old host so nothing can write to it. Clients resolve by name, so no reconfigure is needed. sssd will reconnect, and the KDC reconnects (ldap_conns_per_server = 5) or can be kicked with sudo herd restart kdc kadmind on kerberos.metznet.ca.

8. Verify from clients.

getent passwd <user>        # sssd (may need: sudo sss_cache -E)
kinit <user>                # KDC → LDAP backend
ssh <user>@<some-host>      # AuthorizedKeysCommand → LDAP

9. Decommission the old host after a soak period. Keep config.ldif/ data.ldif somewhere safe and encrypted. They are a complete backup.

Rollback: until step 7, nothing has changed for clients. After step 7, point DNS back and herd start slapd on the old host. Any writes made on the new instance in between are lost unless you slapcat them back.

Changing cn=config

The config is %ldap-slapd-config in ldap.scm (records from metznet/services/slapd-config.scm). New hosts are built from it, but a running server's slapd.d is never rebuilt, so a change is made in two places, in the same commit:

  1. Edit %ldap-slapd-config.
  2. Write the same change as a numbered LDIF modify in ldap/changes/, to apply to the running server.

Check the Scheme builds and behaves, and rehearse the change file against a dump of the live config (slapcat -n 0 output; a redacted olcRootPW is fine). Pass the number of the first change the dump doesn't have yet. The rehearsal fails unless the dump plus those changes equals the generated config exactly:

docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh
docker compose run --rm -T guix bash -lc 'tests/ldap/apply-changes.sh config.ldif 0004'

Then apply it with slapd stopped. slapmodify needs OpenLDAP 2.5 or later:

# on ldap.metznet.ca, as root
cp -a /var/lib/slapd/slapd.d /root/slapd.d.$(date +%F)      # backup
herd stop slapd
sudo -u slapd slapmodify -F /var/lib/slapd/slapd.d -n 0 -l ldap/changes/NNNN-name.ldif
sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u
herd start slapd

Then check the live server matches the generated config exactly (generated-config.sh writes build/slapd-config.ldif):

scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \
  -D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \
  -b cn=config -X -e build/slapd-config.ldif

Schema definitions aren't compared textually (the live server stores them reformatted); the schema check against cn=Subschema covers what's used.

If something goes wrong, stop slapd and restore the copy of slapd.d.

Changing the directory structure

Containers, groups (and their gidNumbers), the realm, the policy and the accounts this repo binds as are %ldap-structure in ldap.scm; everything under a (children-data? #t) container is data and is managed live as usual. To add or change a structure entry, edit %ldap-structure, make the same change on the server with ldapadd/ldapmodify as uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca, then rebuild and check:

docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh
scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \
  -D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \
  -b dc=metznet,dc=ca -X -e build/ldap-structure.ldif

An account's primary gid comes from the group record it names, and every DN from its parent, so those can't drift. Passwords are never part of the structure.

Plain backups

Take backups the same way as step 2 (slapcat -n 0 and -n 1). Restore them the same way as step 4.


kerberos.scm — kerberos.metznet.ca

MIT Kerberos KDC and kadmind (mit-krb5-ldap) for realm METZNET.CA, plus certbot and secrets-service-type.

  • The database lives in LDAP (ldap_kerberos_container_dn = cn=kerberos,dc=metznet,dc=ca) and is reached over ldaps://ldap.metznet.ca. It binds as uid=kerberos,ou=system,ou=accounts,dc=metznet,dc=ca for both KDC and kadmind operations.
  • That DN's password is the ldap_service_password_file /run/secrets/kerberos-ldap-bind-password. At activation, metznet/services/secrets.scm decrypts it from secrets/secrets.enc.yaml using an age key unsealed from TPM handle 0x81010001, with PCR policy sha256:7. The file is owned by kerberos:kerberos with mode 0400.
  • kadm5.acl grants */admin@METZNET.CA full rights.
  • New principals default to +preauth.
  • /etc/kdc.conf is generated. The KDC and kadmind run with KRB5_KDC_PROFILE=/etc/kdc.conf.

⚠️ The committed secrets/secrets.enc.yaml and .sops.yaml recipient are a test fixture. They hold a placeholder password encrypted to a key that is not in any real TPM. Do the provisioning below before relying on it.

Provision secrets on real hardware

# on kerberos.metznet.ca
guix shell sops age -- scripts/seal-age-key.sh 0x81010001          # optional 2nd arg: PCR selection

seal-age-key.sh needs tpm2-tools 5.8 or later (metznet-tpm2-tools). Guix's own 5.7 fails against OpenSSL 3.

  1. Put the printed age1… recipient in .sops.yaml, replacing the fixture key.
  2. Run sops updatekeys secrets/secrets.enc.yaml, then sops secrets/secrets.enc.yaml and set kerberos-ldap-bind-password to the real password of uid=kerberos,…. Updating keys needs a key that can currently decrypt the file. For the fixture, it may be simpler to recreate the file from scratch.
  3. If you chose a different handle or PCRs, update %kerberos-secrets-tpm-handle / pcr-policy. The sha256:7 default has not yet been checked against this machine's measured boot. Firmware or Secure Boot changes alter PCR 7 and make unsealing fail, so you will need to re-seal after them.
  4. Reconfigure, then check /run/secrets/kerberos-ldap-bind-password exists.

First-time KDC setup

There is no automatic stash yet (see the TODO in metznet/services/kdc.scm). On a brand-new realm, create the realm in LDAP with kdb5_ldap_util ... create and then run:

sudo env KRB5_KDC_PROFILE=/etc/kdc.conf kdb5_util stash

For an existing realm, run only the stash step. You will be prompted for the master password.

Migrate the KDC to a new instance

Principals live in LDAP, so the KDC host itself is mostly stateless.

  1. Bring up the new host with metznet/machines/kerberos.scm.
  2. Copy /var/lib/kerberos/stash (mode 0600, root-only), or re-create it with kdb5_util stash and the master password.
  3. New hardware means a new TPM. Run seal-age-key.sh there, add the new recipient to .sops.yaml (keep the old one until cutover is done), run sops updatekeys, and update the handle if it changed.
  4. Reconfigure, then check that herd status kdc kadmind shows both running and /run/secrets/kerberos-ldap-bind-password is present.
  5. Repoint kerberos.metznet.ca DNS. Clients find the KDC through krb5.conf by name. Test with kinit and kadmin -p <you>/admin.
  6. Remove the old host's recipient from .sops.yaml and run sops updatekeys again.

vpn.scm — vpn.metznet.ca

OpenVPN server on 10.0.80.0/24, plus certbot for vpn.metznet.ca. It drops the OpenVPN client every other machine runs. The file was restored from commit 425b173^, where system configs moved to a separate repo, so check it against the live host before the first reconfigure from here. Its key and certificates live in /var/lib/openvpn/ (it reuses the client.crt/client.key names).


otto.scm — otto

Noah's desktop/laptop, built on %metznet-base-desktop-system.

  • Disks: uses fixed UUIDs (EFI 6E88-FE62, root ba93a043-…), not the server labels. Update these if you reinstall.
  • Users: local nmetz (zsh; groups wheel netdev audio video usb kvm) in addition to LDAP users via sssd.
  • Hardware/desktop: nouveau is blacklisted. The NVIDIA driver setup is commented out. CUPS has its web UI on, SANE is enabled, and the USB udev rule is applied.
  • Hosts: a static /etc/hosts entry for geofff.homelab (192.168.86.2).
  • Packages: qemu, texlive-scheme-full, gcc-toolchain, vulkan, etc. firefox and the ARM embedded toolchain are currently left out because they are impractical to build (see docker/README.md → Known gotchas).
  • Requires the nonguix channel (channels.d/nonguix.scm, declared in .guix-channel). Its substitute key is authorized by the desktop base.

Reconfigure with sudo -E guix system reconfigure -L . metznet/machines/otto.scm.


Adding a new machine

  1. Copy base-server.scm to <name>.scm. Rename the module to (metznet machines <name>), because it must match the path or guix pull fails to load the channel. Set the host name, and make the file's last expression the operating-system value.
  2. Add services in front of %metznet-server-services, following the <name>-services + append pattern used in ldap.scm/kerberos.scm.
  3. If it needs secrets: seal a key on the machine, add its recipient to .sops.yaml, run sops updatekeys, and add a secrets-service-type like kerberos.scm does.
  4. Test with scripts/guix-docker.sh image metznet/machines/<name>.scm.
  5. On the hardware, label partitions guix-data/guix-boot (or override file-systems like otto.scm does), install, provision the out-of-band state, and reconfigure.