Digest: teaching lab machines to trust our CA

This is a decision digest: sanitized to design level. Addresses and internal names are deliberately absent.

The situation

The lab runs its own certificate authority (step-ca) with a companion web UI. Issuing a certificate is solved. What was not solved: making every other machine in the environment trust that authority and enroll against it without an operator shuttling files around.

The problem

Root distribution has a chicken-and-egg shape. The natural instinct is to fetch the root from the CA’s own TLS endpoint — but the whole point is that the client does not trust the server yet, so the fetch needs an insecure-mode flag, which trains exactly the habit PKI exists to break. Alternatively, operators copy root files by hand into trust stores per machine: works, scales terribly, leaves no record of who trusts what, and drifts the moment a machine is rebuilt.

Enrollment had a fork to pick too: ACME where the client speaks it, or token-based one-shot commands for everything else.

The decision

1. The companion UI serves the root chain itself, unauthenticated, over plain HTTP on its LAN listener. Trust bootstrap is therefore possible before any trust exists — no insecure flags, no pre-seeded secrets. The missing TLS is compensated out-of-band: the UI displays the root’s SHA-256 fingerprint, the installing operator verifies it after download, and the fingerprint is what actually anchors trust — not the transport.
2. Fingerprints are shown in the lowercase-hex form the current CLI tooling requires, alongside a colon-separated display form for humans. We verified that the documented base64url form fails against this CA build; the page shows only what demonstrably works.
3. ACME first for capable services (ingresses, proxies, appliances): point them at the directory URL once; issue and renewal are then theirs.
4. Token flow for everything else: bootstrap the host once with the pinned fingerprint, then mint single-use enrollment tokens in the UI. Keys are always generated on the target host and never transit the network; the token authorizes, it does not carry secrets.
5. Renewal belongs to the client. ACME clients renew themselves; token hosts run the renewal command from a timer.

Alternatives considered

• Distribute the root via configuration management/SSH loops: no chicken-and-egg, but opaque, unrecorded, and hostile to rebuilds.
• Serve the root only over the CA’s TLS endpoint: perpetuates the insecure-flag habit we are trying to eliminate.
• DNS-based trust distribution: elegant, disproportionate effort for this environment.

Consequences

Plain HTTP for one well-known path is acceptable because the fingerprint check carries the security weight; if that check is skipped, nothing else in the design saves you, so the page leads with it. The hex/base64 divergence is now documented in-repo so future sessions do not re-derive it from error messages. And because trust bootstrap and enrollment live on one page with copy-paste commands, “add a machine to the PKI” stopped being knowledge in someone’s head and became a link.