# metznet/machines One file per host. Each file defines an `operating-system` and ends with that value as its last expression, so you can pass the file straight to `guix system` (`guix system reconfigure -L metznet/machines/ldap.scm`). You can also reference the exported variable, e.g. `(@ (metznet machines ldap) ldap.metznet.ca)`. Everything shared lives in `metznet/system/base-system.scm`: | Base | Used by | Provides | |---------------------------------|----------------------------------|----------| | `%metznet-base-server-system` | `base-server` | GRUB EFI, labelled disks (`guix-data` = `/`, `guix-boot` = `/boot/efi`), serial console, `%metznet-server-services` | | `%metznet-base-aws-system` | `ldap`, `kerberos`, `vpn` | Server base plus `aws-service-type` (`%metznet-aws-server-services`): the EC2 launch key pair for the `aws` account, see below | | `%metznet-base-desktop-system` | `otto` | Server base plus SDDM/i3, NetworkManager, nonguix substitutes, desktop setuid programs | Machines that set their own `services` must append to the base's list (`%metznet-server-services` or `%metznet-aws-server-services`); otherwise they silently lose the base's services. **AWS (`metznet/aws.scm`).** The `aws-pubkey` one-shot fetches the instance's launch key pair from the EC2 metadata service at boot and writes it to `/etc/ssh/authorized_keys.d/aws`, which sshd reads for the local `aws` account. That login keeps working when LDAP is down, so it is the way back into the AWS hosts. The service also adds `le-certs` and `nss-certs` to the system profile. Every machine (server or desktop) gets `%metznet-services`: - **sssd** reads users and groups from `ldaps://ldap.metznet.ca` and binds as `uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca`. - **krb5 + pam_krb5** use realm `METZNET.CA`, with the KDC and kadmin at `kerberos.metznet.ca`. - **openssh** has passwords disabled. Its `AuthorizedKeysCommand` looks up SSH public keys in LDAP. - **openvpn client** connects to `vpn.metznet.ca`. - **pam_mkhomedir** and **metznet-service-type** (shell symlinks, `le-certs`). ## Service dependency graph ``` ldap.metznet.ca ◄── sssd / ssh pubkey lookup on every machine ▲ │ ldaps:// (KDC database lives in LDAP under cn=kerberos,dc=metznet,dc=ca) kerberos.metznet.ca ◄── krb5 / pam_krb5 on every machine ``` `ldap` is the root of the tree. If it is down, nobody can log in on any machine except as root or from cached sssd credentials, and the KDC stops working too. Plan any LDAP work with that in mind. ## Build-time environment variables `base-system.scm` reads some values with `getenv` when the system is **evaluated**, not at boot. They must be set in the environment of `guix system build|reconfigure|image`: | Variable | Effect | If unset | |---------------------|--------|----------| | `GUIX_ROOT_PW` | root password (crypted with a fixed salt) | root login disabled (`!`) | | `LDAP_AUTHTOK` | sssd bind password and ssh pubkey-command bind password | sssd binds with no password and LDAP user lookups fail | | `LDAP_BIND_DN` | sssd bind DN | `uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca` | | `LDAP_AUTHTOK_TYPE` | sssd authtok type | `password` | > ⚠️ These values end up in the world-readable `/gnu/store` (in `sssd.conf` > and the pubkey-command script). This is a known problem and is planned to > move to `metznet/services/secrets.scm`. `scripts/qemu.sh` and `docker-compose.yml` also export `DOMAIN_CAPS`/`DOMAIN_NAME`. ## Out-of-band state (not in this repo) Guix rebuilds everything under `/gnu/store` and `/etc`, but these paths hold state that must be provisioned or migrated by hand: | Path / thing | Machines | What it is | |---|---|---| | `/var/lib/openvpn/{ca.crt,client.crt,client.key,ta.key}` | all | VPN client credentials | | `/etc/letsencrypt/` | `ldap`, `kerberos` | certbot certificates (re-issuable) | | `/var/lib/slapd/{cert,chain,privkey}.pem` | `ldap` | the copies of the certbot certificate slapd actually serves (`olcTLS*`), owned by `slapd`. The certbot deploy hook refreshes them and restarts slapd after each renewal | | `/var/lib/slapd/slapd.d/` | `ldap` | slapd `cn=config` (OLC) database: schema, ACLs, TLS paths, root DN | | `/var/lib/slapd/data/` | `ldap` | MDB data directory (the actual directory tree) | | `/var/lib/slapd/.last_slapadd_restore` | `ldap` | timestamp of the last seed restore (see below) | | EC2 key pair | `ldap`, `kerberos`, `vpn` | the launch key pair `aws-pubkey` installs for the `aws` account; managed in AWS | | `/var/lib/openvpn/{dh2048.pem,ipp.txt}` + server cert/key | `vpn` | OpenVPN server state | | `/var/lib/kerberos/stash` | `kerberos` | KDC master key stash. **Without it the KDC cannot decrypt principals.** | | TPM persistent handle `0x81010001` | `kerberos` | sealed age key for `secrets/secrets.enc.yaml` | ## Day-to-day management ### Test locally On macOS, use Docker (see `docker/README.md`): ```sh scripts/guix-docker.sh image metznet/machines/ldap.scm scripts/guix-docker.sh load-run metznet/machines/ldap.scm ``` On Linux with KVM, use a VM on a tap bridge: ```sh sudo scripts/network.sh start_bridge sudo scripts/network.sh add_tap tap0 scripts/qemu.sh start metznet/machines/ldap.scm tap0 52:54:00:12:34:56 ``` ### Deploy a change to a real host On the host, from a checkout of this repo: ```sh git pull guix pull -C channels.scm # only when channels.d/ changed or to update Guix export LDAP_AUTHTOK=... GUIX_ROOT_PW=... # see table above sudo -E guix system reconfigure -L . metznet/machines/.scm ``` `-L .` builds from the working tree, so you can deploy uncommitted edits without pushing them through the channel first. Check the result with `sudo herd status` (and `herd status slapd` / `herd status kdc`). ### Roll back ```sh sudo guix system roll-back # or pick an older generation in GRUB guix system list-generations ``` A roll-back only reverts the system generation. It does not touch the out-of-band state listed above. ### Logs Services log via shepherd and syslog: `/var/log/messages` and `/var/log/shepherd.log`. slapd runs in the foreground with `-d `, so its output goes to shepherd. The KDC logs to `SYSLOG:DEBUG:DAEMON`. --- ## Machines ### `base-server.scm` — `base-server.metznet.ca` The bare server base with only a host name. It runs no services beyond `%metznet-server-services`. Use it to bring up and smoke-test new hardware (disk labels, EFI boot, sssd/krb5/VPN client), or copy it as the template for a new server. ### `ldap.scm` — `ldap.metznet.ca` OpenLDAP (`openldap-slapd` from `metznet/packages/slapd.scm`, which ships the `pw-totp` module) plus certbot for `ldap.metznet.ca`. slapd listens on `ldap://` and `ldaps://` (no `ldapi://`). slapd runs with `-F /var/lib/slapd/slapd.d`, so the schema, ACLs, TLS cert paths, root DN and database definitions live in the on-disk `cn=config` database. Its contents are described in Scheme (`%ldap-slapd-config` in `ldap.scm`), which the service's `config` field points at. A host with no `slapd.d` builds it from that. An existing `slapd.d` is never touched, so a change to a running server is applied with a change file (see *Changing cn=config*), and the live server is checked against the Scheme. Only root over `ldapi://` can write `cn=config`, and slapd doesn't listen on `ldapi://`, so changes are made offline. `{1}mdb`'s root DN (`uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca`) can read it over `ldaps://`. On every activation the service: 1. Creates `/var/run/slapd`, `/var/lib/slapd/{slapd.d,modules,data}`, all owned by `slapd`. 2. Copies `pw-totp.so` into `/var/lib/slapd/modules` (so `olcModulePath` can point at a stable path rather than a store path). The other modules in `olcModuleLoad` are built into slapd. 3. If `slapd.d` has no `cn=config.ldif`, `slapadd -n 0`s the generated config into it, owned by `slapd` (`slapd-init`). Data is never loaded. 4. Runs the seed check (see *Seeding* below). **Admin login.** `uid=admin` is `{1}mdb`'s root DN, so it bypasses the data ACLs, but there is no `olcRootPW` (removed by `ldap/changes/0004`). It logs in with its own `userPassword`, which is data, so it comes back with every restore and `cn=config` holds no secret. On a brand-new directory with no backup to restore, set it offline: ```sh herd stop slapd printf 'dn: uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca\nchangetype: modify\nreplace: userPassword\nuserPassword: %s\n' \ "$(slappasswd)" | sudo -u slapd slapmodify -F /var/lib/slapd/slapd.d -n 1 herd start slapd ``` Tree layout that other machines rely on: - `dc=metznet,dc=ca`: search base for sssd (rfc2307bis, `posixGroupAux`, `member`) - `ou=system,ou=accounts,dc=metznet,dc=ca`: service bind accounts (`uid=guix`, `uid=kerberos`) - `cn=kerberos,dc=metznet,dc=ca`: Kerberos realm container #### Seeding (`backups` / `backup-date`) `slapd-configuration` takes an ordered list of LDIF files in `backups`. Entry *i* is `slapadd`-ed into database number *i*. So the list is `(config.ldif data.ldif)`: `-n 0` is `cn=config`, `-n 1` is the first data DB. The seeds are only applied when `/var/lib/slapd/.last_slapadd_restore` is missing **or** older than `backup-date`. After a successful restore it writes `backup-date` into that file. Caveats: - `backup-date` defaults to `(current-date)` **at evaluation time**. If you leave seeds configured without pinning `backup-date`, every reconfigure looks "newer" and slapd tries to `slapadd` again. That fails because the entries already exist, and activation errors out. Always set `backup-date` to when the dump was taken, or remove `backups` once the restore is done. - Seed LDIFs become `local-file`s in the world-readable store. They contain password hashes, so prefer the manual restore below for production data. - `backups` and `config` can't both be set, since both would load `-n 0`. --- ### Runbook: migrate LDAP to a new instance This moves `ldap.metznet.ca` to new hardware or a new VM while keeping the host name, so no client config changes are needed. **1. Check the old instance matches the repo.** The new host's `cn=config` is built from the repo, not copied, so anything live that the repo lacks would be lost. Run the two exact checks from step 6 against the old host. If the config differs, fix `%ldap-slapd-config` or apply the missing change files first. **2. Bring up the new host without data.** Partition with labels `guix-data`/`guix-boot`, install, and reconfigure with `metznet/machines/ldap.scm`. Activation builds `slapd.d` from the generated config. slapd won't start yet because it has no TLS certificate (step 5). That is expected. Copy the `/var/lib/openvpn/*` credentials across if the host needs the VPN. **3. Dump the old instance.** MDB supports consistent online `slapcat`, but stopping slapd first guarantees nothing changes after the dump: ```sh # on the OLD host sudo herd stop slapd sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 0 -l /root/config.ldif # for the record sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 1 -l /root/data.ldif sudo herd start slapd # keep serving until cutover, but treat it as read-only from here ``` `slapcat` and `slapadd` are in the system profile via `openldap-slapd` in `%metznet-base-packages`. Check which database numbers exist first (`grep '^dn: olcDatabase' config.ldif`). If there is more than one data DB, dump each by its number. **4. Restore the data on the new host.** Move `data.ldif` over with `scp`. Keep it out of `/gnu/store`. Only the data is restored; `config.ldif` stays with the backups. ```sh # on the NEW host sudo herd stop slapd sudo rm -rf /var/lib/slapd/data/* sudo -u slapd slapadd -F /var/lib/slapd/slapd.d -n 1 -l data.ldif sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u # sanity-check config ``` `tests/ldap/first-boot.sh` rehearses steps 2 and 4 in the container. **5. TLS.** slapd serves copies of the certificate in `/var/lib/slapd/{cert,chain,privkey}.pem`, not certbot's files. Copy those three from the old host and make them readable by `slapd`. certbot on the new host can only complete the ACME challenge once `ldap.metznet.ca` resolves to it, so renewal there starts working after cutover. After each renewal, certbot's deploy hook (`certificate-copy-deploy-hook` in `metznet/services/certificates.scm`) copies the new files over and restarts slapd. To copy the current certificate by hand, run the hook itself: `sudo $(guix gc -R /run/current-system | grep 'ldap.metznet.ca-certificate-copy-hook$')`. The KDC validates the cert (`SSL_CERT_DIR=/etc/ssl/certs`), so a self-signed or expired cert will break Kerberos even though sssd (`ldap_tls_reqcert never`) keeps working. **6. Start and verify before cutover.** ```sh sudo herd start slapd && sudo herd status slapd ldapsearch -H ldaps:// -x -D uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca -W \ -b dc=metznet,dc=ca '(uid=)' ldapsearch ... -b cn=kerberos,dc=metznet,dc=ca -s one dn # realm container is present ``` Compare entry counts against the old host (`grep -c '^dn:' data.ldif`), and check the schema and the entries other machines depend on: ```sh docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh # builds build/*.ldif LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps:// -D -y pwfile \ -b dc=metznet,dc=ca -X -e build/ldap-structure.ldif LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps:// -D -y pwfile \ -b cn=config -X -e build/slapd-config.ldif ``` The first checks the directory structure (`%ldap-structure` in `ldap.scm`: containers, realm, policy, groups and the accounts this repo binds as) exactly while ignoring data; the second, the config (`%ldap-slapd-config`). Run them against the old host first so you know what a passing run looks like. **7. Cut over.** Repoint `ldap.metznet.ca` DNS at the new host, then stop slapd on the old host so nothing can write to it. Clients resolve by name, so no reconfigure is needed. sssd will reconnect, and the KDC reconnects (`ldap_conns_per_server` = 5) or can be kicked with `sudo herd restart kdc kadmind` on `kerberos.metznet.ca`. **8. Verify from clients.** ```sh getent passwd # sssd (may need: sudo sss_cache -E) kinit # KDC → LDAP backend ssh @ # AuthorizedKeysCommand → LDAP ``` **9. Decommission** the old host after a soak period. Keep `config.ldif`/ `data.ldif` somewhere safe and encrypted. They are a complete backup. **Rollback:** until step 7, nothing has changed for clients. After step 7, point DNS back and `herd start slapd` on the old host. Any writes made on the new instance in between are lost unless you `slapcat` them back. #### Changing cn=config The config is `%ldap-slapd-config` in `ldap.scm` (records from `metznet/services/slapd-config.scm`). New hosts are built from it, but a running server's `slapd.d` is never rebuilt, so a change is made in two places, in the same commit: 1. Edit `%ldap-slapd-config`. 2. Write the same change as a numbered LDIF modify in `ldap/changes/`, to apply to the running server. Check the Scheme builds and behaves, and rehearse the change file against a dump of the live config (`slapcat -n 0` output; a redacted `olcRootPW` is fine). Pass the number of the first change the dump doesn't have yet. The rehearsal fails unless the dump plus those changes equals the generated config exactly: ```sh docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh docker compose run --rm -T guix bash -lc 'tests/ldap/apply-changes.sh config.ldif 0004' ``` Then apply it with slapd stopped. `slapmodify` needs OpenLDAP 2.5 or later: ```sh # on ldap.metznet.ca, as root cp -a /var/lib/slapd/slapd.d /root/slapd.d.$(date +%F) # backup herd stop slapd sudo -u slapd slapmodify -F /var/lib/slapd/slapd.d -n 0 -l ldap/changes/NNNN-name.ldif sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u herd start slapd ``` Then check the live server matches the generated config exactly (`generated-config.sh` writes `build/slapd-config.ldif`): ```sh scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \ -D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \ -b cn=config -X -e build/slapd-config.ldif ``` Schema definitions aren't compared textually (the live server stores them reformatted); the schema check against `cn=Subschema` covers what's used. If something goes wrong, stop slapd and restore the copy of `slapd.d`. #### Changing the directory structure Containers, groups (and their gidNumbers), the realm, the policy and the accounts this repo binds as are `%ldap-structure` in `ldap.scm`; everything under a `(children-data? #t)` container is data and is managed live as usual. To add or change a structure entry, edit `%ldap-structure`, make the same change on the server with `ldapadd`/`ldapmodify` as `uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca`, then rebuild and check: ```sh docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \ -D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \ -b dc=metznet,dc=ca -X -e build/ldap-structure.ldif ``` An account's primary gid comes from the group record it names, and every DN from its parent, so those can't drift. Passwords are never part of the structure. #### Plain backups Take backups the same way as step 2 (`slapcat -n 0` and `-n 1`). Restore them the same way as step 4. --- ### `kerberos.scm` — `kerberos.metznet.ca` MIT Kerberos KDC and kadmind (`mit-krb5-ldap`) for realm `METZNET.CA`, plus certbot and `secrets-service-type`. - The **database lives in LDAP** (`ldap_kerberos_container_dn = cn=kerberos,dc=metznet,dc=ca`) and is reached over `ldaps://ldap.metznet.ca`. It binds as `uid=kerberos,ou=system,ou=accounts,dc=metznet,dc=ca` for both KDC and kadmind operations. - That DN's password is the **`ldap_service_password_file`** `/run/secrets/kerberos-ldap-bind-password`. At activation, `metznet/services/secrets.scm` decrypts it from `secrets/secrets.enc.yaml` using an age key unsealed from TPM handle `0x81010001`, with PCR policy `sha256:7`. The file is owned by `kerberos:kerberos` with mode `0400`. - `kadm5.acl` grants `*/admin@METZNET.CA` full rights. - New principals default to `+preauth`. - `/etc/kdc.conf` is generated. The KDC and kadmind run with `KRB5_KDC_PROFILE=/etc/kdc.conf`. > ⚠️ The committed `secrets/secrets.enc.yaml` and `.sops.yaml` recipient > are a **test fixture**. They hold a placeholder password encrypted to a key > that is not in any real TPM. Do the provisioning below before relying on > it. #### Provision secrets on real hardware ```sh # on kerberos.metznet.ca guix shell sops age -- scripts/seal-age-key.sh 0x81010001 # optional 2nd arg: PCR selection ``` `seal-age-key.sh` needs `tpm2-tools` 5.8 or later (`metznet-tpm2-tools`). Guix's own 5.7 fails against OpenSSL 3. 1. Put the printed `age1…` recipient in `.sops.yaml`, replacing the fixture key. 2. Run `sops updatekeys secrets/secrets.enc.yaml`, then `sops secrets/secrets.enc.yaml` and set `kerberos-ldap-bind-password` to the real password of `uid=kerberos,…`. Updating keys needs a key that can currently decrypt the file. For the fixture, it may be simpler to recreate the file from scratch. 3. If you chose a different handle or PCRs, update `%kerberos-secrets-tpm-handle` / `pcr-policy`. The `sha256:7` default has not yet been checked against this machine's measured boot. Firmware or Secure Boot changes alter PCR 7 and make unsealing fail, so you will need to re-seal after them. 4. Reconfigure, then check `/run/secrets/kerberos-ldap-bind-password` exists. #### First-time KDC setup There is no automatic stash yet (see the TODO in `metznet/services/kdc.scm`). On a brand-new realm, create the realm in LDAP with `kdb5_ldap_util ... create` and then run: ```sh sudo env KRB5_KDC_PROFILE=/etc/kdc.conf kdb5_util stash ``` For an existing realm, run only the stash step. You will be prompted for the master password. #### Migrate the KDC to a new instance Principals live in LDAP, so the KDC host itself is mostly stateless. 1. Bring up the new host with `metznet/machines/kerberos.scm`. 2. Copy `/var/lib/kerberos/stash` (mode `0600`, root-only), or re-create it with `kdb5_util stash` and the master password. 3. **New hardware means a new TPM.** Run `seal-age-key.sh` there, add the new recipient to `.sops.yaml` (keep the old one until cutover is done), run `sops updatekeys`, and update the handle if it changed. 4. Reconfigure, then check that `herd status kdc kadmind` shows both running and `/run/secrets/kerberos-ldap-bind-password` is present. 5. Repoint `kerberos.metznet.ca` DNS. Clients find the KDC through `krb5.conf` by name. Test with `kinit` and `kadmin -p /admin`. 6. Remove the old host's recipient from `.sops.yaml` and run `sops updatekeys` again. --- ### `vpn.scm` — `vpn.metznet.ca` OpenVPN server on `10.0.80.0/24`, plus certbot for `vpn.metznet.ca`. It drops the OpenVPN client every other machine runs. The file was restored from commit `425b173^`, where system configs moved to a separate repo, so check it against the live host before the first reconfigure from here. Its key and certificates live in `/var/lib/openvpn/` (it reuses the `client.crt`/`client.key` names). --- ### `otto.scm` — `otto` Noah's desktop/laptop, built on `%metznet-base-desktop-system`. - **Disks:** uses fixed UUIDs (EFI `6E88-FE62`, root `ba93a043-…`), not the server labels. Update these if you reinstall. - **Users:** local `nmetz` (zsh; groups `wheel netdev audio video usb kvm`) in addition to LDAP users via sssd. - **Hardware/desktop:** `nouveau` is blacklisted. The NVIDIA driver setup is commented out. CUPS has its web UI on, SANE is enabled, and the USB udev rule is applied. - **Hosts:** a static `/etc/hosts` entry for `geofff.homelab` (`192.168.86.2`). - **Packages:** qemu, texlive-scheme-full, gcc-toolchain, vulkan, etc. `firefox` and the ARM embedded toolchain are currently left out because they are impractical to build (see `docker/README.md` → Known gotchas). - Requires the **nonguix** channel (`channels.d/nonguix.scm`, declared in `.guix-channel`). Its substitute key is authorized by the desktop base. Reconfigure with `sudo -E guix system reconfigure -L . metznet/machines/otto.scm`. --- ## Adding a new machine 1. Copy `base-server.scm` to `.scm`. Rename the module to `(metznet machines )`, because it must match the path or `guix pull` fails to load the channel. Set the host name, and make the file's last expression the `operating-system` value. 2. Add services in front of `%metznet-server-services`, following the `-services` + `append` pattern used in `ldap.scm`/`kerberos.scm`. 3. If it needs secrets: seal a key on the machine, add its recipient to `.sops.yaml`, run `sops updatekeys`, and add a `secrets-service-type` like `kerberos.scm` does. 4. Test with `scripts/guix-docker.sh image metznet/machines/.scm`. 5. On the hardware, label partitions `guix-data`/`guix-boot` (or override `file-systems` like `otto.scm` does), install, provision the out-of-band state, and reconfigure.