durpdeploy Ships the P1 Feature Suite
This was a heavy week for durpdeploy. Four P1 features landed back-to-back on July 17-18, followed by the backup-restore runbook on July 19. The full P1 list – per-project authorization, admin user management, encryption at rest, runner sandboxing, and log redaction – is now either shipped or documented. Here is what each commit actually changed, pulled from the diffs.
Admin User Management and Per-Project Authorization (P1-1/P1-2)
Commit 2840e49f on July 17 is the largest single change to the codebase this week, touching 5,421 additions and 818 deletions across 6,239 lines. It delivers two P1 items: admin user management and per-project authorization.
User CRUD operations. The users.sql query file gained four new queries: DeleteUser, UpdateUser, UpdateUserPassword, and DeleteSessionsByUser. The corresponding Go methods were generated by sqlc. Users can now be created, edited (name and role), and deleted through the admin UI, with sessions cleaned up on delete via the DeleteSessionsByUser query which cascades through the foreign key relationship. The UpdateUser query sets updated_at = unixepoch() automatically so the timestamp is never stale.
The admin nav dropdown and users management page are the visible UI. The page lists all users with their email, name, role, and timestamps. An admin can change a user’s role between admin, deployer, and viewer, or delete the user entirely. The CLI durpdeploy admin create command, which previously created only admin users, now takes the same code path.
Project membership system. A new project_members table was added with AddProjectMember, RemoveProjectMember, GetProjectMember, IsProjectMember, ListProjectMembers, and ListProjectsForUser queries. The AddProjectMember query uses ON CONFLICT ... DO UPDATE SET role = excluded.role so re-inviting a user updates their role rather than failing.
The ProjectMember model in internal/db/models.go captures project_id, user_id, role, and created_at. The ListProjectMembers query joins with users to return email and name alongside the member record.
RequireProjectAccess middleware. New file internal/auth/projectaccess.go implements two middleware functions. RequireProjectAccess resolves the {id} chi URL parameter as a project ID, then checks if the authenticated user is a member (or a global admin, who bypasses the check). If the user is not an admin and the project does not exist, it returns 404 to hide the project’s existence. Non-members get a styled 403 page or an HTMX toast depending on the request type.
RequireDeploymentProjectAccess does the same for deployment routes, resolving the project through the deployment -> release -> project chain. Both middleware inject the project ID into the request context via a typed context key so handlers can use ProjectIDFromContext instead of re-parsing the URL parameter.
Viewer role UI gating. The CSRF middleware was extended to reject state-changing requests from viewer-role users. A RenderUnauthorized function in csrf.go handles the response: HTMX requests get an HX-Trigger with a makeToast event (danger level), while non-HTMX requests get a full styled 403 page with a “Back to projects” link.
On the templ side, a pages.CanWrite(ctx) helper (and its components.canWrite counterpart) gates all write affordances. Every form page (New Project, New Environment, New Lifecycle, New Template, Deploy, Edit, etc.) wraps its form body in if CanWrite(ctx) { @FormBody(...) } else { @ViewerForbiddenMessage(...) }. The ViewerForbiddenMessage component renders a permission-denied explanation with a back link. This means a viewer never sees a form they cannot submit, and never gets a useless error toast from clicking a button that was hidden.
The RequireRole middleware in internal/auth/role.go was also updated to use RenderUnauthorized instead of a bare http.Error with “Forbidden”, so admin-only routes (like /admin/*) also get proper styled responses.
Encryption at Rest (P1-3)
Commit f87676f5 on July 18, titled “Encryption at Rest”, adds 1,297 lines and removes 32. This is the AES-256-GCM encryption layer for release_variables.value.
Secret key management. A new internal/secret/ package handles key generation, parsing, and rotation. The CLI gained a secret-key rotate subcommand that generates a new 32-byte key, base64-encodes it, and prints it to stdout (or writes it to a file with --plaintext). The key is stored as the DURPDEPLOY_SECRET_KEY environment variable in the systemd unit.
The cmd/server/main.go file gained the secret-key case in the top-level command switch and the runSecretKey function. The help text was updated to show: Usage: durpdeploy [admin create --email X --password Y] [secret-key rotate [--plaintext]] [version] [help].
Encryption/decryption at the repository layer. The repository layer’s variable operations now encrypt value on insert and decrypt on read. This is transparent to the handlers and the runner – the runner’s ListReleaseVariablesByRelease still receives plaintext values because the decryption happens at the repository boundary. The encryption uses AES-256-GCM with a random 12-byte nonce per value, producing a ciphertext that is base64-encoded for storage. The key is never written to the database.
Database schema note. The release_variables.value column stores the base64 ciphertext. No schema migration was needed because the column is already TEXT. Old plaintext values will fail to decrypt on first read, so the deployment runbook includes a one-time migration step (not needed for greenfield deployments like mine).
Security documentation. The docs/security.md file was updated to document the encryption architecture: each variable gets a unique nonce, the ciphertext includes the nonce (prefixed), and the repository layer never exposes the key to calling code. The known-gaps table at the bottom now marks “Secret encryption at rest” as shipped (P1-3) and moves “Log redaction” to P1-5 (next).
Runner Sandboxing (P1-4)
Commit 8ec6b612 on July 18, titled “Feature/croot”, adds 399 lines and removes 35. This is the runner sandboxing feature that isolates deployment step execution from the server process.
Dedicated runner user. A new low-privileged durpdeploy-runner system user was created. Steps no longer run as the durpdeploy user, which means a malicious or buggy step script cannot read durpdeploy.db, the secret key, or modify the host filesystem. The docs/deploy.md was updated with provisioning instructions for the durpdeploy-runner user (system account, no home directory, /usr/sbin/nologin shell).
Environment sanitization. A new baseStepEnv() function in internal/runner/runner.go replaces the previous approach of inheriting the server’s os.Environ(). The step environment now includes only PATH, HOME=/nonexistent, USER=durpdeploy-runner, LOGNAME=durpdeploy-runner, TERM=xterm, and LANG (inherited from the server). This prevents leaks of DURPDEPLOY_DB, DURPDEPLOY_SECRET_KEY, and any other server-side environment variables.
Cgroup v2 resource limits. Per-deployment cgroups are created under /sys/fs/cgroup/durpdeploy/. The parent cgroup directory must be provisioned on the host (documented in the deploy runbook). The durpdeploy systemd unit uses Delegate=yes to let the process manage its own cgroup sub-tree. ProtectControlGroups was intentionally removed (previously set to true), and RestrictNamespaces was removed because the sandbox needs mount namespace operations.
The capabilities CAP_SETUID, CAP_SETGID, and CAP_SYS_ADMIN are granted in the systemd unit’s AmbientCapabilities and CapabilityBoundingSet so the server can switch to the durpdeploy-runner user and perform bind-mount/chroot operations. NoNewPrivileges still blocks privilege escalation via execve.
Process group management. The setPgid function in procgroup_unix.go was fixed to not panic when cmd.SysProcAttr is nil. It now initializes a new SysProcAttr if needed, then sets Setpgid = true. This is a safety fix – previously, if a caller didn’t set SysProcAttr, the function would panic on nil dereference.
Cgroup directory setup. The deploy documentation now includes a cgroup provisioning step: creating /sys/fs/cgroup/durpdeploy, setting ownership to durpdeploy:durpdeploy, and enabling +cpu +memory +pids controllers via cgroup.subtree_control. Since cgroupfs is virtual and does not survive reboot, I need to add a systemd-tmpfiles rule or a oneshot unit to automate this on boot. For now, the server degrades gracefully (logs a warning, skips resource limiting) if the directory is missing.
Log Redaction (P1-5)
Commit 56aef4d8 on July 18, titled “Feature/logging”, adds 273 lines and removes 23. This is the log redaction scrubber for deployment logs.
The Scrubber type. New file internal/runner/scrubber.go defines a Scrubber struct with a single pre-compiled regex. The NewScrubber function takes a slice of literal secret values, sorts them longest-first (so a superstring secret is redacted whole rather than leaving a suffix), escapes them with regexp.QuoteMeta, then combines them with a set of common credential patterns into one (?s)(...) alternation.
The common patterns include:
Bearer\s+[A-Za-z0-9._~+/-]+=*– Bearer tokens in Authorization headersghp_[A-Za-z0-9]{36,}– GitHub personal access tokensAKIA[0-9A-Z]{16}– AWS access key IDsxox[bap]-[A-Za-z0-9-]+– Slack bot/user/app tokens(?i:\b(password|token|key)\s*=\s*[^\s\"']+)– inline credential assignments
Extra patterns can be added via the DURPDEPLOY_EXTRA_SCRUB_PATTERNS environment variable (comma-separated). These are loaded at init time and appended to the common patterns list.
Buffered scrubbing. The broadcastWriter.Write method was rewritten. Previously, it processed one line at a time (split on \n) and applied strings.ReplaceAll for each secret. Now it scrubs everything up to the last newline in its buffer, which catches secrets that span multiple Write calls or contain embedded newlines (like a multi-line SSH key). The Flush method also uses the scrubber for any remaining buffered data.
Unit tests. New file internal/runner/scrubber_test.go has 12 test cases covering literal secret redaction, multiline secrets, longest-first matching, overlapping secrets, empty secrets, bearer tokens, GitHub PATs, AWS keys, Slack tokens, assignment patterns, no false positives, and custom patterns. Each test is self-contained and uses the exported NewScrubber and Scrub API.
Documentation updates. The docs/security.md file was updated with a full “Log redaction (P1-5)” section describing the architecture. The known-gaps table now marks “Log redaction hardening” as shipped. The AGENTS.md file was updated to note that naive string replacement was replaced by the regex-based scrubber, and the docs/deploy.md references were updated.
Backup and Restore Runbook (P1-6)
Commit c2672007 on July 19 adds 488 lines and removes 6. This is the backup and restore documentation and supporting files.
Litestream setup. The 216-line docs/backup-restore.md file documents two backup strategies. The recommended option is Litestream, which tails the SQLite WAL and streams changed pages to S3-compatible storage (AWS S3, MinIO, Cloudflare R2, Backblaze B2). The documentation covers installation (pinning version v0.5.14), configuration template with Litestream YAML, systemd service setup, verification via litestream ltx, and restore procedures with point-in-time recovery using -timestamp.
The systemd/litestream.service file uses the same hardening profile as durpdeploy.service (ProtectSystem=strict, NoNewPrivileges=true, PrivateTmp=true, etc.), running as the durpdeploy user after network-online.target and durpdeploy.service. The systemd/litestream.yml config template includes placeholders for bucket, endpoint, and credentials, with comments explaining each setting.
The cron fallback option uses sqlite3 .backup in a daily cron job with 14-day local retention and offsite rsync. This is documented as the fallback for environments where S3-compatible storage is not available.
Acceptance test. The scripts/test-backup-restore.sh script automates a full disaster-recovery drill against a local filesystem replica (no S3 bucket required). It builds the server, seeds an admin user, starts the server, starts Litestream replication, creates 100 deployments via the API, waits for replication, simulates disk failure by killing the server and deleting the database, restores from the replica, and verifies all 100 deployments are present. The script uses mktemp for isolation and a trap for cleanup.
Documentation cross-references. The docs/deploy.md file was updated with cross-references to docs/backup-restore.md and scripts/test-backup-restore.sh. The systemd/durpdeploy.service unit file was updated with a comment pointing to the Litestream service.
GitOps: GitLab Runner DMZ Fix and Traefik Multi-Cluster Applications
GitLab Runner DMZ Configuration Fix
Commit d04de820 by Hermes on July 18 fixes three issues in the dmz-talos environment’s GitLab Runner deployment, merged via MR !13.
Secret name mismatch. The ExternalSecret created a secret named gitlab-secret-personal, but the GitLab Runner chart was configured to look for gitlab-secret. This mismatch meant the runner pod never had a registration token, causing a panic on URL validation. The fix aligns the chart’s runners.secret reference to gitlab-secret-personal, matching what ExternalSecret actually produces.
HostAliases for GitLab reachability. Build pods need to reach gitlab.durp.info to clone repositories and post build status. On the Talos DMZ cluster, the MetalLB IP (192.168.98.141) is not routable from within build pods due to Talos’s routing architecture. The fix adds a hostAliases entry mapping gitlab.durp.info to 10.109.156.7, the ClusterIP of the Traefik ingress controller in the DMZ. ClusterIP is always reachable from any pod in the cluster, so this works without depending on external DNS or MetalLB.
PodSecurity standards. The GitLab Runner namespace was previously unlabeled for Pod Security Standards. Build pods require privileged: true for Docker-in-Docker and other build operations. The managedNamespaceMetadata block was added to label the namespace with pod-security.kubernetes.io/enforce=privileged, matching what the runner chart’s runners.privileged=true expects.
Traefik ArgoCD Application Split
Commit 0456dce1 on July 16 adds 45 lines and removes 3 in infra-talos/argocd/templates/traefik.yaml. Two new ArgoCD Application resources were created: traefik-dmz (pointing to dmz-talos/traefik) and traefik-dev (pointing to dev/traefik). The existing traefik-infra Application had its managedNamespaceMetadata labels cleaned up, removing the istio-injection: enabled label.
This split is part of the ongoing multi-cluster Traefik rollout. Each environment now has its own Traefik Application in ArgoCD, pointing to its respective directory in the gitops repo (infra-talos/, dmz-talos/, dev/). The istio-injection label removal suggests that the previous attempt to run Istio alongside Traefik on the infra-talos cluster has been walked back – the managedNamespaceMetadata label block was removed entirely rather than switching to a different value.
Ansible Inventory Cleanup
Commit eafc533a on July 16 removes 75 lines from invintory.yaml in the runbooks/ansible repo. The file previously held Ansible inventory definitions for the full cluster topology: infra-master, infra-node, dmz-master, dmz-node, prd-master, prd-node, dev-master, and dev-node host groups, each with their respective IP addresses. Also removed were the cluster group definitions (infra-cluster, dmz-cluster, dev-cluster, prd-cluster), the composite master and node groups, and an openvpn host entry.
The inventory was stripping out old cluster definitions as those environments are decommissioned or migrated to Talos-based clusters. The developer cluster (dev-master/dev-node) had already been commented out in a previous iteration; this commit removes those comments entirely along with all the other stale group definitions. The only remaining entry in the file is the hermes host (the management VM at 192.168.12.50).
This is consistent with the infrastructure migration pattern seen in the gitops repo, where the infra, dmz, and dev environments now have Talos counterparts (infra-talos, dmz-talos) and the legacy cluster definitions are being cleaned up.
Looking Ahead
The P1 feature suite for durpdeploy is effectively complete for this cycle. The next areas I want to address:
Audit log retention. The audit_log table has no retention policy or tamper-proofing. I want to add a scheduled cleanup job (DELETE older than N days) and consider append-only semantics or a separate audit database for compliance-sensitive deployments.
Per-project role enforcement. The RequireProjectAccess middleware currently treats any project_members row as authorized regardless of the per-project role. Finer-grained enforcement (project-admin, project-deployer, project-viewer) would let organizations delegate project management without granting global admin.