metznet-channel/metznet/machines/README.md

511 lines
22 KiB
Markdown

# metznet/machines
One file per host. Each file defines an `operating-system` and ends with
that value as its last expression, so you can pass the file straight to
`guix system` (`guix system reconfigure -L <checkout> metznet/machines/ldap.scm`).
You can also reference the exported variable, e.g.
`(@ (metznet machines ldap) ldap.metznet.ca)`.
Everything shared lives in `metznet/system/base-system.scm`:
| Base | Used by | Provides |
|---------------------------------|----------------------------------|----------|
| `%metznet-base-server-system` | `base-server` | GRUB EFI, labelled disks (`guix-data` = `/`, `guix-boot` = `/boot/efi`), serial console, `%metznet-server-services` |
| `%metznet-base-aws-system` | `ldap`, `kerberos`, `vpn` | Server base plus `aws-service-type` (`%metznet-aws-server-services`): the EC2 launch key pair for the `aws` account, see below |
| `%metznet-base-desktop-system` | `otto` | Server base plus SDDM/i3, NetworkManager, nonguix substitutes, desktop setuid programs |
Machines that set their own `services` must append to the base's list
(`%metznet-server-services` or `%metznet-aws-server-services`); otherwise they
silently lose the base's services.
**AWS (`metznet/aws.scm`).** The `aws-pubkey` one-shot fetches the instance's
launch key pair from the EC2 metadata service at boot and writes it to
`/etc/ssh/authorized_keys.d/aws`, which sshd reads for the local `aws`
account. That login keeps working when LDAP is down, so it is the way back
into the AWS hosts. The service also adds `le-certs` and `nss-certs` to the
system profile.
Every machine (server or desktop) gets `%metznet-services`:
- **sssd** reads users and groups from `ldaps://ldap.metznet.ca` and binds as
`uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca`.
- **krb5 + pam_krb5** use realm `METZNET.CA`, with the KDC and kadmin at
`kerberos.metznet.ca`.
- **openssh** has passwords disabled. Its `AuthorizedKeysCommand` looks up SSH
public keys in LDAP.
- **openvpn client** connects to `vpn.metznet.ca`.
- **pam_mkhomedir** and **metznet-service-type** (shell symlinks, `le-certs`).
## Service dependency graph
```
ldap.metznet.ca ◄── sssd / ssh pubkey lookup on every machine
▲
│ ldaps:// (KDC database lives in LDAP under cn=kerberos,dc=metznet,dc=ca)
kerberos.metznet.ca ◄── krb5 / pam_krb5 on every machine
```
`ldap` is the root of the tree. If it is down, nobody can log in on any
machine except as root or from cached sssd credentials, and the KDC stops
working too. Plan any LDAP work with that in mind.
## Build-time environment variables
`base-system.scm` reads some values with `getenv` when the system is
**evaluated**, not at boot. They must be set in the environment of
`guix system build|reconfigure|image`:
| Variable | Effect | If unset |
|---------------------|--------|----------|
| `GUIX_ROOT_PW` | root password (crypted with a fixed salt) | root login disabled (`!`) |
| `LDAP_AUTHTOK` | sssd bind password and ssh pubkey-command bind password | sssd binds with no password and LDAP user lookups fail |
| `LDAP_BIND_DN` | sssd bind DN | `uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca` |
| `LDAP_AUTHTOK_TYPE` | sssd authtok type | `password` |
> ⚠️ These values end up in the world-readable `/gnu/store` (in `sssd.conf`
> and the pubkey-command script). This is a known problem and is planned to
> move to `metznet/services/secrets.scm`.
`scripts/qemu.sh` and `docker-compose.yml` also export `DOMAIN_CAPS`/`DOMAIN_NAME`.
## Out-of-band state (not in this repo)
Guix rebuilds everything under `/gnu/store` and `/etc`, but these paths
hold state that must be provisioned or migrated by hand:
| Path / thing | Machines | What it is |
|---|---|---|
| `/var/lib/openvpn/{ca.crt,client.crt,client.key,ta.key}` | all | VPN client credentials |
| `/etc/letsencrypt/` | `ldap`, `kerberos` | certbot certificates (re-issuable) |
| `/var/lib/slapd/{cert,chain,privkey}.pem` | `ldap` | the copies of the certbot certificate slapd actually serves (`olcTLS*`), owned by `slapd`. The certbot deploy hook refreshes them and restarts slapd after each renewal |
| `/var/lib/slapd/slapd.d/` | `ldap` | slapd `cn=config` (OLC) database: schema, ACLs, TLS paths, root DN |
| `/var/lib/slapd/data/` | `ldap` | MDB data directory (the actual directory tree) |
| `/var/lib/slapd/.last_slapadd_restore` | `ldap` | timestamp of the last seed restore (see below) |
| EC2 key pair | `ldap`, `kerberos`, `vpn` | the launch key pair `aws-pubkey` installs for the `aws` account; managed in AWS |
| `/var/lib/openvpn/{dh2048.pem,ipp.txt}` + server cert/key | `vpn` | OpenVPN server state |
| `/var/lib/kerberos/stash` | `kerberos` | KDC master key stash. **Without it the KDC cannot decrypt principals.** |
| TPM persistent handle `0x81010001` | `kerberos` | sealed age key for `secrets/secrets.enc.yaml` |
## Day-to-day management
### Test locally
On macOS, use Docker (see `docker/README.md`):
```sh
scripts/guix-docker.sh image metznet/machines/ldap.scm
scripts/guix-docker.sh load-run metznet/machines/ldap.scm
```
On Linux with KVM, use a VM on a tap bridge:
```sh
sudo scripts/network.sh start_bridge <uplink-iface>
sudo scripts/network.sh add_tap tap0
scripts/qemu.sh start metznet/machines/ldap.scm tap0 52:54:00:12:34:56
```
### Deploy a change to a real host
On the host, from a checkout of this repo:
```sh
git pull
guix pull -C channels.scm # only when channels.d/ changed or to update Guix
export LDAP_AUTHTOK=... GUIX_ROOT_PW=... # see table above
sudo -E guix system reconfigure -L . metznet/machines/<host>.scm
```
`-L .` builds from the working tree, so you can deploy uncommitted edits
without pushing them through the channel first. Check the result with
`sudo herd status` (and `herd status slapd` / `herd status kdc`).
### Roll back
```sh
sudo guix system roll-back # or pick an older generation in GRUB
guix system list-generations
```
A roll-back only reverts the system generation. It does not touch the
out-of-band state listed above.
### Logs
Services log via shepherd and syslog: `/var/log/messages` and
`/var/log/shepherd.log`. slapd runs in the foreground with `-d <log-level>`,
so its output goes to shepherd. The KDC logs to `SYSLOG:DEBUG:DAEMON`.
---
## Machines
### `base-server.scm` — `base-server.metznet.ca`
The bare server base with only a host name. It runs no services beyond
`%metznet-server-services`. Use it to bring up and smoke-test new hardware
(disk labels, EFI boot, sssd/krb5/VPN client), or copy it as the template
for a new server.
### `ldap.scm` — `ldap.metznet.ca`
OpenLDAP (`openldap-slapd` from `metznet/packages/slapd.scm`, which ships
the `pw-totp` module) plus certbot for `ldap.metznet.ca`. slapd listens on
`ldap://` and `ldaps://` (no `ldapi://`).
`metznet/services/slapd.scm` does **not** generate slapd's configuration.
slapd runs with `-F /var/lib/slapd/slapd.d`, so the schema, ACLs, TLS cert
paths, root DN and database definitions all live in the on-disk
`cn=config` database. Its intended contents are described in Scheme
(`%ldap-slapd-config` in `ldap.scm`), and the live server is checked against
that (see *Changing cn=config*). Only root over `ldapi://` can write `cn=config`, and slapd
doesn't listen on `ldapi://`, so changes are made offline (see *Changing
cn=config* below). `{1}mdb`'s root DN (`uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca`)
can read it over `ldaps://`. On every activation the service:
1. Creates `/var/run/slapd`, `/var/lib/slapd/{slapd.d,modules,data}`, all
owned by `slapd`.
2. Copies `pw-totp.so` into `/var/lib/slapd/modules` (so `olcModulePath`
can point at a stable path rather than a store path).
3. Runs the seed check (see *Seeding* below).
Tree layout that other machines rely on:
- `dc=metznet,dc=ca`: search base for sssd (rfc2307bis, `posixGroupAux`, `member`)
- `ou=system,ou=accounts,dc=metznet,dc=ca`: service bind accounts
(`uid=guix`, `uid=kerberos`)
- `cn=kerberos,dc=metznet,dc=ca`: Kerberos realm container
#### Seeding (`backups` / `backup-date`)
`slapd-configuration` takes an ordered list of LDIF files in `backups`.
Entry *i* is `slapadd`-ed into database number *i*. So the list is
`(config.ldif data.ldif)`: `-n 0` is `cn=config`, `-n 1` is the first data
DB. The seeds are only applied when
`/var/lib/slapd/.last_slapadd_restore` is missing **or** older than
`backup-date`. After a successful restore it writes `backup-date` into that
file.
Caveats:
- `backup-date` defaults to `(current-date)` **at evaluation time**. If you
leave seeds configured without pinning `backup-date`, every reconfigure
looks "newer" and slapd tries to `slapadd` again. That fails because the
entries already exist, and activation errors out. Always set
`backup-date` to when the dump was taken, or remove `backups` once the
restore is done.
- Seed LDIFs become `local-file`s in the world-readable store. They contain
password hashes (including `olcRootPW`), so prefer the manual restore
below for production data.
---
### Runbook: migrate LDAP to a new instance
This moves `ldap.metznet.ca` to new hardware or a new VM while keeping the
host name, so no client config changes are needed.
**1. Bring up the new host without data.** Partition with labels
`guix-data`/`guix-boot`, install, and reconfigure with
`metznet/machines/ldap.scm` (no `backups`). slapd will fail to start
because `slapd.d` is empty. That is expected. Copy the
`/var/lib/openvpn/*` credentials across if the host needs the VPN.
**2. Dump the old instance.** MDB supports consistent online `slapcat`,
but stopping slapd first guarantees nothing changes after the dump:
```sh
# on the OLD host
sudo herd stop slapd
sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 0 -l /root/config.ldif
sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 1 -l /root/data.ldif
sudo herd start slapd # keep serving until cutover, but treat it as read-only from here
```
`slapcat` and `slapadd` are in the system profile via `openldap-slapd` in
`%metznet-base-packages`. Check which database numbers exist first
(`grep '^dn: olcDatabase' config.ldif`). If there is more than one data DB,
dump each by its number.
**3. Review `config.ldif` for host-specific paths.** Look at
`olcDbDirectory` (should be `/var/lib/slapd/data`), `olcModulePath`
(`/var/lib/slapd/modules`), `olcTLSCertificateFile`/`KeyFile`/`CACertificateFile`
(`/var/lib/slapd/{cert,privkey,chain}.pem`), and any
`/gnu/store/...` paths. Store paths from the old host will not exist on the
new one, so point them at stable locations.
**4. Restore on the new host.** Move the dumps over with `scp`. Keep them out
of `/gnu/store`.
```sh
# on the NEW host
sudo herd stop slapd
sudo rm -rf /var/lib/slapd/slapd.d/* /var/lib/slapd/data/*
sudo -u slapd slapadd -F /var/lib/slapd/slapd.d -n 0 -l config.ldif
sudo -u slapd slapadd -F /var/lib/slapd/slapd.d -n 1 -l data.ldif
sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u # sanity-check config
```
**5. TLS.** slapd serves copies of the certificate in
`/var/lib/slapd/{cert,chain,privkey}.pem`, not certbot's files. Copy those
three from the old host and make them readable by `slapd`. certbot on the
new host can only complete the ACME challenge once `ldap.metznet.ca`
resolves to it, so renewal there starts working after cutover. After each
renewal, certbot's deploy hook (`certificate-copy-deploy-hook` in
`metznet/services/certificates.scm`) copies the new files over and restarts
slapd. To copy the current certificate by hand, run the hook itself:
`sudo $(guix gc -R /run/current-system | grep 'ldap.metznet.ca-certificate-copy-hook$')`.
The KDC validates the cert (`SSL_CERT_DIR=/etc/ssl/certs`), so
a self-signed or expired cert will break Kerberos even though sssd
(`ldap_tls_reqcert never`) keeps working.
**6. Start and verify before cutover.**
```sh
sudo herd start slapd && sudo herd status slapd
ldapsearch -H ldaps://<new-host-ip> -x -D uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca -W \
-b dc=metznet,dc=ca '(uid=<some-user>)'
ldapsearch ... -b cn=kerberos,dc=metznet,dc=ca -s one dn # realm container is present
```
Compare entry counts against the old host (`grep -c '^dn:' data.ldif`),
and check the schema and the entries other machines depend on:
```sh
docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh # builds build/*.ldif
LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps://<new-host-ip> -D <admin-dn> -y pwfile \
-b dc=metznet,dc=ca -X -e build/ldap-structure.ldif
LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps://<new-host-ip> -D <admin-dn> -y pwfile \
-b cn=config -X -e build/slapd-config.ldif
```
The first checks the directory structure (`%ldap-structure` in `ldap.scm`:
containers, realm, policy, groups and the accounts this repo binds as)
exactly while ignoring data; the second, the config (`%ldap-slapd-config`).
Run them against the old host first so you know what a passing run looks
like.
**7. Cut over.** Repoint `ldap.metznet.ca` DNS at the new host, then stop
slapd on the old host so nothing can write to it. Clients resolve by name,
so no reconfigure is needed. sssd will reconnect, and the KDC reconnects
(`ldap_conns_per_server` = 5) or can be kicked with
`sudo herd restart kdc kadmind` on `kerberos.metznet.ca`.
**8. Verify from clients.**
```sh
getent passwd <user> # sssd (may need: sudo sss_cache -E)
kinit <user> # KDC → LDAP backend
ssh <user>@<some-host> # AuthorizedKeysCommand → LDAP
```
**9. Decommission** the old host after a soak period. Keep `config.ldif`/
`data.ldif` somewhere safe and encrypted. They are a complete backup.
**Rollback:** until step 7, nothing has changed for clients. After step 7,
point DNS back and `herd start slapd` on the old host. Any writes made on
the new instance in between are lost unless you `slapcat` them back.
#### Changing cn=config
The intended config is `%ldap-slapd-config` in `ldap.scm` (records from
`metznet/services/slapd-config.scm`). Until slapd is started from it, a
change is made in two places, in the same commit:
1. Edit `%ldap-slapd-config`.
2. Write the same change as a numbered LDIF modify in `ldap/changes/`, to
apply to the running server.
Check the Scheme builds and behaves, and rehearse the change file against a
dump of the live config (`slapcat -n 0` output; a redacted `olcRootPW` is
fine). The rehearsal fails unless the dump plus the change files equals the
generated config exactly:
```sh
docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh
docker compose run --rm -T guix bash -lc 'tests/ldap/apply-changes.sh config.ldif'
```
Then apply it with slapd stopped. `slapmodify` needs OpenLDAP 2.5 or later:
```sh
# on ldap.metznet.ca, as root
cp -a /var/lib/slapd/slapd.d /root/slapd.d.$(date +%F) # backup
herd stop slapd
sudo -u slapd slapmodify -F /var/lib/slapd/slapd.d -n 0 -l ldap/changes/NNNN-name.ldif
sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u
herd start slapd
```
Then check the live server matches the generated config exactly
(`generated-config.sh` writes `build/slapd-config.ldif`):
```sh
scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \
-D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \
-b cn=config -X -e build/slapd-config.ldif
```
Schema definitions aren't compared textually (the live server stores them
reformatted); the schema check against `cn=Subschema` covers what's used.
If something goes wrong, stop slapd and restore the copy of `slapd.d`.
#### Changing the directory structure
Containers, groups (and their gidNumbers), the realm, the policy and the
accounts this repo binds as are `%ldap-structure` in `ldap.scm`; everything
under a `(children-data? #t)` container is data and is managed live as
usual. To add or change a structure entry, edit `%ldap-structure`, make the
same change on the server with `ldapadd`/`ldapmodify` as
`uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca`, then rebuild and check:
```sh
docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh
scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \
-D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \
-b dc=metznet,dc=ca -X -e build/ldap-structure.ldif
```
An account's primary gid comes from the group record it names, and every
DN from its parent, so those can't drift. Passwords are never part of the
structure.
#### Plain backups
Take backups the same way as step 2 (`slapcat -n 0` and `-n 1`). Restore
them the same way as step 4.
---
### `kerberos.scm` — `kerberos.metznet.ca`
MIT Kerberos KDC and kadmind (`mit-krb5-ldap`) for realm `METZNET.CA`, plus
certbot and `secrets-service-type`.
- The **database lives in LDAP** (`ldap_kerberos_container_dn =
cn=kerberos,dc=metznet,dc=ca`) and is reached over `ldaps://ldap.metznet.ca`.
It binds as `uid=kerberos,ou=system,ou=accounts,dc=metznet,dc=ca` for both
KDC and kadmind operations.
- That DN's password is the **`ldap_service_password_file`**
`/run/secrets/kerberos-ldap-bind-password`. At activation,
`metznet/services/secrets.scm` decrypts it from `secrets/secrets.enc.yaml`
using an age key unsealed from TPM handle `0x81010001`, with PCR policy
`sha256:7`. The file is owned by `kerberos:kerberos` with mode `0400`.
- `kadm5.acl` grants `*/admin@METZNET.CA` full rights.
- New principals default to `+preauth`.
- `/etc/kdc.conf` is generated. The KDC and kadmind run with
`KRB5_KDC_PROFILE=/etc/kdc.conf`.
> ⚠️ The committed `secrets/secrets.enc.yaml` and `.sops.yaml` recipient
> are a **test fixture**. They hold a placeholder password encrypted to a key
> that is not in any real TPM. Do the provisioning below before relying on
> it.
#### Provision secrets on real hardware
```sh
# on kerberos.metznet.ca
guix shell sops age -- scripts/seal-age-key.sh 0x81010001 # optional 2nd arg: PCR selection
```
`seal-age-key.sh` needs `tpm2-tools` 5.8 or later (`metznet-tpm2-tools`).
Guix's own 5.7 fails against OpenSSL 3.
1. Put the printed `age1…` recipient in `.sops.yaml`, replacing the
fixture key.
2. Run `sops updatekeys secrets/secrets.enc.yaml`, then
`sops secrets/secrets.enc.yaml` and set `kerberos-ldap-bind-password` to
the real password of `uid=kerberos,…`. Updating keys needs a key that can
currently decrypt the file. For the fixture, it may be simpler to recreate
the file from scratch.
3. If you chose a different handle or PCRs, update
`%kerberos-secrets-tpm-handle` / `pcr-policy`. The `sha256:7` default
has not yet been checked against this machine's measured boot. Firmware
or Secure Boot changes alter PCR 7 and make unsealing fail, so you will
need to re-seal after them.
4. Reconfigure, then check `/run/secrets/kerberos-ldap-bind-password` exists.
#### First-time KDC setup
There is no automatic stash yet (see the TODO in `metznet/services/kdc.scm`).
On a brand-new realm, create the realm in LDAP with
`kdb5_ldap_util ... create` and then run:
```sh
sudo env KRB5_KDC_PROFILE=/etc/kdc.conf kdb5_util stash
```
For an existing realm, run only the stash step. You will be prompted for the
master password.
#### Migrate the KDC to a new instance
Principals live in LDAP, so the KDC host itself is mostly stateless.
1. Bring up the new host with `metznet/machines/kerberos.scm`.
2. Copy `/var/lib/kerberos/stash` (mode `0600`, root-only), or re-create it
with `kdb5_util stash` and the master password.
3. **New hardware means a new TPM.** Run `seal-age-key.sh` there, add the
new recipient to `.sops.yaml` (keep the old one until cutover is done),
run `sops updatekeys`, and update the handle if it changed.
4. Reconfigure, then check that `herd status kdc kadmind` shows both
running and `/run/secrets/kerberos-ldap-bind-password` is present.
5. Repoint `kerberos.metznet.ca` DNS. Clients find the KDC through
`krb5.conf` by name. Test with `kinit` and `kadmin -p <you>/admin`.
6. Remove the old host's recipient from `.sops.yaml` and run
`sops updatekeys` again.
---
### `vpn.scm` — `vpn.metznet.ca`
OpenVPN server on `10.0.80.0/24`, plus certbot for `vpn.metznet.ca`. It drops
the OpenVPN client every other machine runs. The file was restored from
commit `425b173^`, where system configs moved to a separate repo, so check it
against the live host before the first reconfigure from here. Its key and
certificates live in `/var/lib/openvpn/` (it reuses the `client.crt`/`client.key`
names).
---
### `otto.scm` — `otto`
Noah's desktop/laptop, built on `%metznet-base-desktop-system`.
- **Disks:** uses fixed UUIDs (EFI `6E88-FE62`, root
`ba93a043-…`), not the server labels. Update these if you reinstall.
- **Users:** local `nmetz` (zsh; groups `wheel netdev audio video usb
kvm`) in addition to LDAP users via sssd.
- **Hardware/desktop:** `nouveau` is blacklisted. The NVIDIA driver setup is
commented out. CUPS has its web UI on, SANE is enabled, and the USB udev
rule is applied.
- **Hosts:** a static `/etc/hosts` entry for `geofff.homelab`
(`192.168.86.2`).
- **Packages:** qemu, texlive-scheme-full, gcc-toolchain, vulkan, etc. `firefox` and the ARM
embedded toolchain are currently left out because they are impractical to build (see
`docker/README.md` → Known gotchas).
- Requires the **nonguix** channel (`channels.d/nonguix.scm`, declared in
`.guix-channel`). Its substitute key is authorized by the desktop base.
Reconfigure with
`sudo -E guix system reconfigure -L . metznet/machines/otto.scm`.
---
## Adding a new machine
1. Copy `base-server.scm` to `<name>.scm`. Rename the module to
`(metznet machines <name>)`, because it must match the path or
`guix pull` fails to load the channel. Set the host name, and make
the file's last expression the `operating-system` value.
2. Add services in front of `%metznet-server-services`, following the
`<name>-services` + `append` pattern used in `ldap.scm`/`kerberos.scm`.
3. If it needs secrets: seal a key on the machine, add its recipient to
`.sops.yaml`, run `sops updatekeys`, and add a `secrets-service-type`
like `kerberos.scm` does.
4. Test with `scripts/guix-docker.sh image metznet/machines/<name>.scm`.
5. On the hardware, label partitions `guix-data`/`guix-boot` (or override
`file-systems` like `otto.scm` does), install, provision the
out-of-band state, and reconfigure.