|
|
||
|---|---|---|
| .. | ||
| README.md | ||
| base-server.scm | ||
| kerberos.scm | ||
| ldap.scm | ||
| otto.scm | ||
| vpn.scm | ||
README.md
metznet/machines
One file per host. Each file defines an operating-system and ends with
that value as its last expression, so you can pass the file straight to
guix system (guix system reconfigure -L <checkout> metznet/machines/ldap.scm).
You can also reference the exported variable, e.g.
(@ (metznet machines ldap) ldap.metznet.ca).
Everything shared lives in metznet/system/base-system.scm:
| Base | Used by | Provides |
|---|---|---|
%metznet-base-server-system |
base-server |
GRUB EFI, labelled disks (guix-data = /, guix-boot = /boot/efi), serial console, %metznet-server-services |
%metznet-base-aws-system |
ldap, kerberos, vpn |
Server base plus aws-service-type (%metznet-aws-server-services): the EC2 launch key pair for the aws account, see below |
%metznet-base-desktop-system |
otto |
Server base plus SDDM/i3, NetworkManager, nonguix substitutes, desktop setuid programs |
Machines that set their own services must append to the base's list
(%metznet-server-services or %metznet-aws-server-services); otherwise they
silently lose the base's services.
AWS (metznet/aws.scm). The aws-pubkey one-shot fetches the instance's
launch key pair from the EC2 metadata service at boot and writes it to
/etc/ssh/authorized_keys.d/aws, which sshd reads for the local aws
account. That login keeps working when LDAP is down, so it is the way back
into the AWS hosts. The service also adds le-certs and nss-certs to the
system profile.
Every machine (server or desktop) gets %metznet-services:
- sssd reads users and groups from
ldaps://ldap.metznet.caand binds asuid=guix,ou=system,ou=accounts,dc=metznet,dc=ca. - krb5 + pam_krb5 use realm
METZNET.CA, with the KDC and kadmin atkerberos.metznet.ca. - openssh has passwords disabled. Its
AuthorizedKeysCommandlooks up SSH public keys in LDAP. - openvpn client connects to
vpn.metznet.ca. - pam_mkhomedir and metznet-service-type (shell symlinks,
le-certs).
Service dependency graph
ldap.metznet.ca ◄── sssd / ssh pubkey lookup on every machine
▲
│ ldaps:// (KDC database lives in LDAP under cn=kerberos,dc=metznet,dc=ca)
kerberos.metznet.ca ◄── krb5 / pam_krb5 on every machine
ldap is the root of the tree. If it is down, nobody can log in on any
machine except as root or from cached sssd credentials, and the KDC stops
working too. Plan any LDAP work with that in mind.
Build-time environment variables
base-system.scm reads some values with getenv when the system is
evaluated, not at boot. They must be set in the environment of
guix system build|reconfigure|image:
| Variable | Effect | If unset |
|---|---|---|
GUIX_ROOT_PW |
root password (crypted with a fixed salt) | root login disabled (!) |
LDAP_AUTHTOK |
sssd bind password and ssh pubkey-command bind password | sssd binds with no password and LDAP user lookups fail |
LDAP_BIND_DN |
sssd bind DN | uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca |
LDAP_AUTHTOK_TYPE |
sssd authtok type | password |
⚠️ These values end up in the world-readable
/gnu/store(insssd.confand the pubkey-command script). This is a known problem and is planned to move tometznet/services/secrets.scm.
scripts/qemu.sh and docker-compose.yml also export DOMAIN_CAPS/DOMAIN_NAME.
Out-of-band state (not in this repo)
Guix rebuilds everything under /gnu/store and /etc, but these paths
hold state that must be provisioned or migrated by hand:
| Path / thing | Machines | What it is |
|---|---|---|
/var/lib/openvpn/{ca.crt,client.crt,client.key,ta.key} |
all | VPN client credentials |
/etc/letsencrypt/ |
ldap, kerberos |
certbot certificates (re-issuable) |
/var/lib/slapd/{cert,chain,privkey}.pem |
ldap |
the copies of the certbot certificate slapd actually serves (olcTLS*), owned by slapd. The certbot deploy hook refreshes them and restarts slapd after each renewal |
/var/lib/slapd/slapd.d/ |
ldap |
slapd cn=config (OLC) database: schema, ACLs, TLS paths, root DN |
/var/lib/slapd/data/ |
ldap |
MDB data directory (the actual directory tree) |
/var/lib/slapd/.last_slapadd_restore |
ldap |
timestamp of the last seed restore (see below) |
| EC2 key pair | ldap, kerberos, vpn |
the launch key pair aws-pubkey installs for the aws account; managed in AWS |
/var/lib/openvpn/{dh2048.pem,ipp.txt} + server cert/key |
vpn |
OpenVPN server state |
/var/lib/kerberos/stash |
kerberos |
KDC master key stash. Without it the KDC cannot decrypt principals. |
TPM persistent handle 0x81010001 |
kerberos |
sealed age key for secrets/secrets.enc.yaml |
Day-to-day management
Test locally
On macOS, use Docker (see docker/README.md):
scripts/guix-docker.sh image metznet/machines/ldap.scm
scripts/guix-docker.sh load-run metznet/machines/ldap.scm
On Linux with KVM, use a VM on a tap bridge:
sudo scripts/network.sh start_bridge <uplink-iface>
sudo scripts/network.sh add_tap tap0
scripts/qemu.sh start metznet/machines/ldap.scm tap0 52:54:00:12:34:56
Deploy a change to a real host
On the host, from a checkout of this repo:
git pull
guix pull -C channels.scm # only when channels.d/ changed or to update Guix
export LDAP_AUTHTOK=... GUIX_ROOT_PW=... # see table above
sudo -E guix system reconfigure -L . metznet/machines/<host>.scm
-L . builds from the working tree, so you can deploy uncommitted edits
without pushing them through the channel first. Check the result with
sudo herd status (and herd status slapd / herd status kdc).
Roll back
sudo guix system roll-back # or pick an older generation in GRUB
guix system list-generations
A roll-back only reverts the system generation. It does not touch the out-of-band state listed above.
Logs
Services log via shepherd and syslog: /var/log/messages and
/var/log/shepherd.log. slapd runs in the foreground with -d <log-level>,
so its output goes to shepherd. The KDC logs to SYSLOG:DEBUG:DAEMON.
Machines
base-server.scm — base-server.metznet.ca
The bare server base with only a host name. It runs no services beyond
%metznet-server-services. Use it to bring up and smoke-test new hardware
(disk labels, EFI boot, sssd/krb5/VPN client), or copy it as the template
for a new server.
ldap.scm — ldap.metznet.ca
OpenLDAP (openldap-slapd from metznet/packages/slapd.scm, which ships
the pw-totp module) plus certbot for ldap.metznet.ca. slapd listens on
ldap:// and ldaps:// (no ldapi://).
metznet/services/slapd.scm does not generate slapd's configuration.
slapd runs with -F /var/lib/slapd/slapd.d, so the schema, ACLs, TLS cert
paths, root DN and database definitions all live in the on-disk
cn=config database. Its intended contents are described in Scheme
(%ldap-slapd-config in ldap.scm), and the live server is checked against
that (see Changing cn=config). Only root over ldapi:// can write cn=config, and slapd
doesn't listen on ldapi://, so changes are made offline (see Changing
cn=config below). {1}mdb's root DN (uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca)
can read it over ldaps://. On every activation the service:
- Creates
/var/run/slapd,/var/lib/slapd/{slapd.d,modules,data}, all owned byslapd. - Copies
pw-totp.sointo/var/lib/slapd/modules(soolcModulePathcan point at a stable path rather than a store path). - Runs the seed check (see Seeding below).
Tree layout that other machines rely on:
dc=metznet,dc=ca: search base for sssd (rfc2307bis,posixGroupAux,member)ou=system,ou=accounts,dc=metznet,dc=ca: service bind accounts (uid=guix,uid=kerberos)cn=kerberos,dc=metznet,dc=ca: Kerberos realm container
Seeding (backups / backup-date)
slapd-configuration takes an ordered list of LDIF files in backups.
Entry i is slapadd-ed into database number i. So the list is
(config.ldif data.ldif): -n 0 is cn=config, -n 1 is the first data
DB. The seeds are only applied when
/var/lib/slapd/.last_slapadd_restore is missing or older than
backup-date. After a successful restore it writes backup-date into that
file.
Caveats:
backup-datedefaults to(current-date)at evaluation time. If you leave seeds configured without pinningbackup-date, every reconfigure looks "newer" and slapd tries toslapaddagain. That fails because the entries already exist, and activation errors out. Always setbackup-dateto when the dump was taken, or removebackupsonce the restore is done.- Seed LDIFs become
local-files in the world-readable store. They contain password hashes (includingolcRootPW), so prefer the manual restore below for production data.
Runbook: migrate LDAP to a new instance
This moves ldap.metznet.ca to new hardware or a new VM while keeping the
host name, so no client config changes are needed.
1. Bring up the new host without data. Partition with labels
guix-data/guix-boot, install, and reconfigure with
metznet/machines/ldap.scm (no backups). slapd will fail to start
because slapd.d is empty. That is expected. Copy the
/var/lib/openvpn/* credentials across if the host needs the VPN.
2. Dump the old instance. MDB supports consistent online slapcat,
but stopping slapd first guarantees nothing changes after the dump:
# on the OLD host
sudo herd stop slapd
sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 0 -l /root/config.ldif
sudo -u slapd slapcat -F /var/lib/slapd/slapd.d -n 1 -l /root/data.ldif
sudo herd start slapd # keep serving until cutover, but treat it as read-only from here
slapcat and slapadd are in the system profile via openldap-slapd in
%metznet-base-packages. Check which database numbers exist first
(grep '^dn: olcDatabase' config.ldif). If there is more than one data DB,
dump each by its number.
3. Review config.ldif for host-specific paths. Look at
olcDbDirectory (should be /var/lib/slapd/data), olcModulePath
(/var/lib/slapd/modules), olcTLSCertificateFile/KeyFile/CACertificateFile
(/var/lib/slapd/{cert,privkey,chain}.pem), and any
/gnu/store/... paths. Store paths from the old host will not exist on the
new one, so point them at stable locations.
4. Restore on the new host. Move the dumps over with scp. Keep them out
of /gnu/store.
# on the NEW host
sudo herd stop slapd
sudo rm -rf /var/lib/slapd/slapd.d/* /var/lib/slapd/data/*
sudo -u slapd slapadd -F /var/lib/slapd/slapd.d -n 0 -l config.ldif
sudo -u slapd slapadd -F /var/lib/slapd/slapd.d -n 1 -l data.ldif
sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u # sanity-check config
5. TLS. slapd serves copies of the certificate in
/var/lib/slapd/{cert,chain,privkey}.pem, not certbot's files. Copy those
three from the old host and make them readable by slapd. certbot on the
new host can only complete the ACME challenge once ldap.metznet.ca
resolves to it, so renewal there starts working after cutover. After each
renewal, certbot's deploy hook (certificate-copy-deploy-hook in
metznet/services/certificates.scm) copies the new files over and restarts
slapd. To copy the current certificate by hand, run the hook itself:
sudo $(guix gc -R /run/current-system | grep 'ldap.metznet.ca-certificate-copy-hook$').
The KDC validates the cert (SSL_CERT_DIR=/etc/ssl/certs), so
a self-signed or expired cert will break Kerberos even though sssd
(ldap_tls_reqcert never) keeps working.
6. Start and verify before cutover.
sudo herd start slapd && sudo herd status slapd
ldapsearch -H ldaps://<new-host-ip> -x -D uid=guix,ou=system,ou=accounts,dc=metznet,dc=ca -W \
-b dc=metznet,dc=ca '(uid=<some-user>)'
ldapsearch ... -b cn=kerberos,dc=metznet,dc=ca -s one dn # realm container is present
Compare entry counts against the old host (grep -c '^dn:' data.ldif),
and check the schema and the entries other machines depend on:
docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh # builds build/*.ldif
LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps://<new-host-ip> -D <admin-dn> -y pwfile \
-b dc=metznet,dc=ca -X -e build/ldap-structure.ldif
LDAPTLS_REQCERT=never scripts/ldap-verify.sh -H ldaps://<new-host-ip> -D <admin-dn> -y pwfile \
-b cn=config -X -e build/slapd-config.ldif
The first checks the directory structure (%ldap-structure in ldap.scm:
containers, realm, policy, groups and the accounts this repo binds as)
exactly while ignoring data; the second, the config (%ldap-slapd-config).
Run them against the old host first so you know what a passing run looks
like.
7. Cut over. Repoint ldap.metznet.ca DNS at the new host, then stop
slapd on the old host so nothing can write to it. Clients resolve by name,
so no reconfigure is needed. sssd will reconnect, and the KDC reconnects
(ldap_conns_per_server = 5) or can be kicked with
sudo herd restart kdc kadmind on kerberos.metznet.ca.
8. Verify from clients.
getent passwd <user> # sssd (may need: sudo sss_cache -E)
kinit <user> # KDC → LDAP backend
ssh <user>@<some-host> # AuthorizedKeysCommand → LDAP
9. Decommission the old host after a soak period. Keep config.ldif/
data.ldif somewhere safe and encrypted. They are a complete backup.
Rollback: until step 7, nothing has changed for clients. After step 7,
point DNS back and herd start slapd on the old host. Any writes made on
the new instance in between are lost unless you slapcat them back.
Changing cn=config
The intended config is %ldap-slapd-config in ldap.scm (records from
metznet/services/slapd-config.scm). Until slapd is started from it, a
change is made in two places, in the same commit:
- Edit
%ldap-slapd-config. - Write the same change as a numbered LDIF modify in
ldap/changes/, to apply to the running server.
Check the Scheme builds and behaves, and rehearse the change file against a
dump of the live config (slapcat -n 0 output; a redacted olcRootPW is
fine). The rehearsal fails unless the dump plus the change files equals the
generated config exactly:
docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh
docker compose run --rm -T guix bash -lc 'tests/ldap/apply-changes.sh config.ldif'
Then apply it with slapd stopped. slapmodify needs OpenLDAP 2.5 or later:
# on ldap.metznet.ca, as root
cp -a /var/lib/slapd/slapd.d /root/slapd.d.$(date +%F) # backup
herd stop slapd
sudo -u slapd slapmodify -F /var/lib/slapd/slapd.d -n 0 -l ldap/changes/NNNN-name.ldif
sudo -u slapd slaptest -F /var/lib/slapd/slapd.d -u
herd start slapd
Then check the live server matches the generated config exactly
(generated-config.sh writes build/slapd-config.ldif):
scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \
-D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \
-b cn=config -X -e build/slapd-config.ldif
Schema definitions aren't compared textually (the live server stores them
reformatted); the schema check against cn=Subschema covers what's used.
If something goes wrong, stop slapd and restore the copy of slapd.d.
Changing the directory structure
Containers, groups (and their gidNumbers), the realm, the policy and the
accounts this repo binds as are %ldap-structure in ldap.scm; everything
under a (children-data? #t) container is data and is managed live as
usual. To add or change a structure entry, edit %ldap-structure, make the
same change on the server with ldapadd/ldapmodify as
uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca, then rebuild and check:
docker compose run --rm -T guix bash -lc tests/ldap/generated-config.sh
scripts/ldap-verify.sh -H ldaps://ldap.metznet.ca \
-D uid=admin,ou=sudoers,ou=accounts,dc=metznet,dc=ca -y pwfile \
-b dc=metznet,dc=ca -X -e build/ldap-structure.ldif
An account's primary gid comes from the group record it names, and every DN from its parent, so those can't drift. Passwords are never part of the structure.
Plain backups
Take backups the same way as step 2 (slapcat -n 0 and -n 1). Restore
them the same way as step 4.
kerberos.scm — kerberos.metznet.ca
MIT Kerberos KDC and kadmind (mit-krb5-ldap) for realm METZNET.CA, plus
certbot and secrets-service-type.
- The database lives in LDAP (
ldap_kerberos_container_dn = cn=kerberos,dc=metznet,dc=ca) and is reached overldaps://ldap.metznet.ca. It binds asuid=kerberos,ou=system,ou=accounts,dc=metznet,dc=cafor both KDC and kadmind operations. - That DN's password is the
ldap_service_password_file/run/secrets/kerberos-ldap-bind-password. At activation,metznet/services/secrets.scmdecrypts it fromsecrets/secrets.enc.yamlusing an age key unsealed from TPM handle0x81010001, with PCR policysha256:7. The file is owned bykerberos:kerberoswith mode0400. kadm5.aclgrants*/admin@METZNET.CAfull rights.- New principals default to
+preauth. /etc/kdc.confis generated. The KDC and kadmind run withKRB5_KDC_PROFILE=/etc/kdc.conf.
⚠️ The committed
secrets/secrets.enc.yamland.sops.yamlrecipient are a test fixture. They hold a placeholder password encrypted to a key that is not in any real TPM. Do the provisioning below before relying on it.
Provision secrets on real hardware
# on kerberos.metznet.ca
guix shell sops age -- scripts/seal-age-key.sh 0x81010001 # optional 2nd arg: PCR selection
seal-age-key.sh needs tpm2-tools 5.8 or later (metznet-tpm2-tools).
Guix's own 5.7 fails against OpenSSL 3.
- Put the printed
age1…recipient in.sops.yaml, replacing the fixture key. - Run
sops updatekeys secrets/secrets.enc.yaml, thensops secrets/secrets.enc.yamland setkerberos-ldap-bind-passwordto the real password ofuid=kerberos,…. Updating keys needs a key that can currently decrypt the file. For the fixture, it may be simpler to recreate the file from scratch. - If you chose a different handle or PCRs, update
%kerberos-secrets-tpm-handle/pcr-policy. Thesha256:7default has not yet been checked against this machine's measured boot. Firmware or Secure Boot changes alter PCR 7 and make unsealing fail, so you will need to re-seal after them. - Reconfigure, then check
/run/secrets/kerberos-ldap-bind-passwordexists.
First-time KDC setup
There is no automatic stash yet (see the TODO in metznet/services/kdc.scm).
On a brand-new realm, create the realm in LDAP with
kdb5_ldap_util ... create and then run:
sudo env KRB5_KDC_PROFILE=/etc/kdc.conf kdb5_util stash
For an existing realm, run only the stash step. You will be prompted for the master password.
Migrate the KDC to a new instance
Principals live in LDAP, so the KDC host itself is mostly stateless.
- Bring up the new host with
metznet/machines/kerberos.scm. - Copy
/var/lib/kerberos/stash(mode0600, root-only), or re-create it withkdb5_util stashand the master password. - New hardware means a new TPM. Run
seal-age-key.shthere, add the new recipient to.sops.yaml(keep the old one until cutover is done), runsops updatekeys, and update the handle if it changed. - Reconfigure, then check that
herd status kdc kadmindshows both running and/run/secrets/kerberos-ldap-bind-passwordis present. - Repoint
kerberos.metznet.caDNS. Clients find the KDC throughkrb5.confby name. Test withkinitandkadmin -p <you>/admin. - Remove the old host's recipient from
.sops.yamland runsops updatekeysagain.
vpn.scm — vpn.metznet.ca
OpenVPN server on 10.0.80.0/24, plus certbot for vpn.metznet.ca. It drops
the OpenVPN client every other machine runs. The file was restored from
commit 425b173^, where system configs moved to a separate repo, so check it
against the live host before the first reconfigure from here. Its key and
certificates live in /var/lib/openvpn/ (it reuses the client.crt/client.key
names).
otto.scm — otto
Noah's desktop/laptop, built on %metznet-base-desktop-system.
- Disks: uses fixed UUIDs (EFI
6E88-FE62, rootba93a043-…), not the server labels. Update these if you reinstall. - Users: local
nmetz(zsh; groupswheel netdev audio video usb kvm) in addition to LDAP users via sssd. - Hardware/desktop:
nouveauis blacklisted. The NVIDIA driver setup is commented out. CUPS has its web UI on, SANE is enabled, and the USB udev rule is applied. - Hosts: a static
/etc/hostsentry forgeofff.homelab(192.168.86.2). - Packages: qemu, texlive-scheme-full, gcc-toolchain, vulkan, etc.
firefoxand the ARM embedded toolchain are currently left out because they are impractical to build (seedocker/README.md→ Known gotchas). - Requires the nonguix channel (
channels.d/nonguix.scm, declared in.guix-channel). Its substitute key is authorized by the desktop base.
Reconfigure with
sudo -E guix system reconfigure -L . metznet/machines/otto.scm.
Adding a new machine
- Copy
base-server.scmto<name>.scm. Rename the module to(metznet machines <name>), because it must match the path orguix pullfails to load the channel. Set the host name, and make the file's last expression theoperating-systemvalue. - Add services in front of
%metznet-server-services, following the<name>-services+appendpattern used inldap.scm/kerberos.scm. - If it needs secrets: seal a key on the machine, add its recipient to
.sops.yaml, runsops updatekeys, and add asecrets-service-typelikekerberos.scmdoes. - Test with
scripts/guix-docker.sh image metznet/machines/<name>.scm. - On the hardware, label partitions
guix-data/guix-boot(or overridefile-systemslikeotto.scmdoes), install, provision the out-of-band state, and reconfigure.