Homelab Weekly: dmz-talos Environment Launches, ArgoCD Ingress, and SecretStore Consolidation

Kevn Lukritz | Jul 14, 2026 min read

dmz-talos Environment Ships

The headliner this week is the creation of the dmz-talos environment – a full Talos Linux-based DMZ cluster that mirrors the existing dmz environment running on the older infrastructure. Commit bf62068c on July 10 landed 20 new files bootstrapping the entire service stack for this cluster. This is a significant milestone because it means both the DMZ and the internal infrastructure layers now have Talos-based counterparts, bringing us closer to decommissioning the original cluster.

What the dmz-talos Stack Includes

The environment follows the same architecture as dmz but rewritten for the Talos context. Each service has its own directory under dmz-talos/ with a Helm Chart.yaml, values.yaml, and ExternalSecret templates where secrets are needed.

cert-manager. The cert-manager deployment includes a ClusterIssuer for Let’s Encrypt production (letsencrypt-production) configured with the ACME v2 endpoint at https://acme-v02.api.letsencrypt.org/directory. The issuer uses Vault-backed Kubernetes auth for authentication. Interestingly, the initial version of the issuer template used a legacy ServiceAccount-based token approach (a vault ServiceAccount with an explicit token secret reference), which was later cleaned up in commit d0ba12cb (July 14) where the auth method was switched to a secretRef-based token flow using vault-issuer-token with the role pki-issuer-role. The cert-manager values pin image pull policy to Always and set hostAliases pointing at the Vault server at 192.168.12.141 – the Talos cluster’s load-balanced VIP.

external-secrets. The external-secrets deployment includes a CA certificate ConfigMap (vault.pem) extracted from the internal PKI, and a full values.yaml with debug-level logging enabled. The ExternalSecret templates pull credentials from Vault at various kv/ paths for services like cert-manager, crowdsec, external-dns, and gitlab-runner. An important detail: the initial secretStore.yaml in dmz-talos/external-secrets/templates/ referenced a vault store, which was later updated in commit 12aa37cd (July 14) to reference vault-backend and use a dedicated service account external-secrets-dmz instead of the generic external-secrets. Then in commit ccdc09ae (also July 14), a separate ClusterSecretStore named vault-backend was created specifically for the dmz-talos cluster, pointing at https://vault.infra.durp.info with the CA from the ConfigMap.

crowdsec. The crowdsec deployment mirrors the existing dmz configuration with an ExternalSecret for the enrollment key pulled from Vault. The values.yaml configures log format for containerd (CRI format, since Talos uses containerd as its container runtime) and sets up the acquisition configuration for Kubernetes audit logs.

external-dns. The external-dns deployment includes a Cloudflare API token ExternalSecret and values.yaml with registry.durp.info as the global image registry and insecure images allowed (necessary for the local registry which doesn’t have a trusted certificate chain).

gitlab-runner. The gitlab-runner deployment includes an ExternalSecret for the GitLab registration token and a Helm chart skeleton. The registration secret is pulled from kv/gitlab-runner in Vault.

authentik. A remote-cluster authentik chart was added to enable the authentik proxy to route authentication for services on the dmz-talos cluster. The ingress template references the remote cluster’s authentik service.

Terraform CI/CD Pipeline

The dmz-talos/.gitlab/.gitlab-ci.yml file adds a full OpenTofu (the open-source Terraform fork) pipeline with plan, apply, and destroy stages. The pipeline uses registry.durp.info/opentofu/opentofu:latest as the execution image and configures the GitLab-managed Terraform state backend for the dmz workspace. Key pipeline jobs include:

  • format: Runs tofu fmt -diff -check -write=false on changes to dmz/terraform/*.tf files.
  • validate: Runs tofu validate on the same trigger path.
  • plan-dmz-infrastructure: Generates a plan and converts it to JSON using jq for MR comments via the GitLab API. It uses tofu show --json to extract create/update/delete counts and posts them as a formatted MR note.
  • apply-dmz-infrastructure: Applies the plan only on the default branch (main), with a needs dependency on the plan job.
  • destroy: A manual job for tearing down the environment.

Why dmz-talos Exists

The original DMZ cluster runs on an older Kubernetes distribution provisioned with k3s. The Talos Linux migration is driven by several factors: Talos offers a minimal attack surface (no SSH, API-only access), atomic upgrades, and etcd management baked into the OS. By creating dmz-talos as a parallel environment, I can migrate services one at a time with zero downtime – the dmz internalproxy routes control which cluster receives traffic for each service, and updating a route is a single YAML change that ArgoCD picks up automatically.

Infra-Talos Cluster: OpenClarity and Nebula-Sync

The infra-talos cluster (the internal infrastructure Talos cluster) got two major deployments this week.

OpenClarity Vulnerability Scanner

Commit 5ffce137 on July 12 added the full OpenClarity deployment to infra-talos. OpenClarity is a CNCF sandbox project that provides SBOM generation, CVE scanning, malware detection, and exploit analysis for container images and running pods. The deployment consists of eight microservices managed by a single Helm chart from oci://ghcr.io/openclarity/charts version 1.1.3.

The values.yaml is comprehensive – 916 lines of configuration – so I’ll highlight the most architecturally significant decisions:

Security posture. Every container across all eight services runs with a hardened security context: readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, all Linux capabilities dropped. Most services run as non-root with UID/GID 1001. This is consistent with the security baseline I’m applying to all workloads on the Talos clusters. The only exceptions are Swagger UI (runs as root due to the upstream image) and the CR discovery server (runs without security context restrictions because it needs host filesystem access for container runtime socket discovery).

Scanner configuration. The orchestrator is configured with deleteJobPolicy: Always, meaning completed scan jobs are automatically cleaned up. The scanner image uses the OpenClarity CLI from ghcr.io/openclarity/openclarity-cli. The provider is set to kubernetes, which means the orchestrator will scan running pods in the cluster. Supporting services include:

  • Trivy server (aquasec/trivy:0.69.3) for CVE scanning
  • Grype server (openclarity/grype-server:v0.7.5) for additional vulnerability matching
  • Exploit DB server (openclarity/exploit-db-server:v0.3.0) for exploit lookup
  • Freshclam mirror (openclarity/freshclam-mirror:v0.3.1) for ClamAV signature distribution
  • YARA rule server (openclarity/yara-rule-server:v0.3.0) for malware signature matching

Database. PostgreSQL uses Bitnami’s legacy image (bitnamilegacy/postgresql:16.6.0-debian-12-r1) pinned to a specific tag. The auth secret is managed by external-secrets, pulling credentials from Vault at kv/openclarity/postgres. The database is configured with a “small” resource preset.

ArgoCD Application. The ArgoCD Application manifest points at infra-talos/openclarity with prune and selfHeal enabled, and CreateNamespace=true so the openclarity namespace is created automatically on first sync.

Nebula-Sync Migration

Nebula-sync, the Pi-hole replication tool, was migrated from the old infra environment to infra-talos in two steps. Commit 0c7bb3f3 (July 12) created the ArgoCD Application in infra-talos/argocd/templates/nebula-sync.yaml pointing at path infra-infra/nebula-sync – still referencing the old chart location. Commit 30b05177 (also July 12) corrected the path to infra-talos/nebula-sync, completing the migration.

The nebula-sync deployment template defines a single-replica Deployment using registry.durp.info/lovelaze/nebula-sync:latest with two environment variables (PRIMARY and REPLICAS) pulled from a Vault-backed ExternalSecret. The pod mounts the Pi-hole custom.list file via a hostPath volume from /etc/pihole/custom.list. The ExternalSecret was updated from the old vault store to vault-backend ClusterSecretStore in commit d4dcafac (July 12), and the standalone vault ServiceAccount that was embedded in the templates was removed.

Internalproxy ArgoCD Route

Commit 6bbfc67c (July 13) added an ArgoCD Application for the internalproxy service itself (dmz/internalproxy) to the infra-talos cluster. This is the service that routes external traffic to the correct backend cluster. The Application targets the dmz destination cluster (the existing DMZ, not the new dmz-talos – the internalproxy still runs on the original DMZ to avoid circular dependencies during migration).

ArgoCD Ingress Goes Live

Commit bdd34f28 on July 12 added a proper ingress for the ArgoCD web UI at argocd.infra.durp.info. This was the last piece needed to make ArgoCD accessible without port-forwarding or kubectl proxy.

The configuration uses a Traefik IngressRoute (Custom Resource Definition) with the websecure entrypoint, meaning TLS is enforced at the ingress layer. The route matches Host(\argocd.infra.durp.info`)and forwards to theargocd-server` service on port 443 using HTTPS scheme (since ArgoCD’s server terminates its own TLS).

A cert-manager Certificate resource requests a TLS certificate from the vault-issuer ClusterIssuer for the domain argocd.infra.durp.info. The certificate is stored in a Secret named argocd-tls in the same namespace, which the IngressRoute references in its tls.secretName field.

There’s a commented-out middleware section for a whitelist – the #middlewares: block references a whitelist middleware in the traefik namespace that I haven’t created yet. The commented middleware suggests this ingress should eventually be restricted to internal network ranges, but for now it’s accessible from anywhere with a valid DNS resolution. I’ll add the whitelist middleware once the initial setup is confirmed working.

Talos Endpoint and Service Routing

Commit 7757fd2d on July 11 added a generic infra-talos-cluster Endpoints and Service resource to the internalproxy manifests. The Endpoints point at 192.168.12.141:443 – the load-balanced VIP for the new Talos cluster’s ingress controller.

This was followed by route updates in commit 196c7211 (July 12) which changed the ArgoCD internalproxy route from infra-cluster to infra-talos-cluster, and commit a8a9dbef (also July 12) which did the same for the bitwarden route. The authentik route was already updated as part of the initial 7757fd2d commit, changing the backend service from infra-cluster to infra-talos-cluster.

The pattern is clean: the internalproxy acts as a centralized routing layer where backend references are plain Kubernetes Service names. Changing the backend for any service is a one-line diff in the internalproxy template. No DNS changes, no load balancer reconfiguration – ArgoCD syncs the new route and traffic flows to the new cluster.

MetalLB Pod Security Standards

Commit c38ad498 on July 13 added Pod Security Standards labels to the MetalLB system namespace in the infra-talos cluster. The managedNamespaceMetadata block in the ArgoCD Application template now sets:

managedNamespaceMetadata:
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged

MetalLB requires the privileged level because it manipulates kernel networking (ARP/NDP for Layer 2 mode, BGP for Layer 3 mode). The enforce label means the kubelet will reject any pod in this namespace that doesn’t meet the privileged policy level. This is correctly permissive for MetalLB’s requirements while making the security posture explicit in the manifest.

SecretStore Consolidation: vault-backend

A recurring pattern across this week’s commits is the consolidation of SecretStore references from the old vault store to the new vault-backend ClusterSecretStore. This affected at least four application templates:

  • nebula-sync (infra-talos/nebula-sync/templates/secrets.yaml): Store name changed from vault to vault-backend.
  • bitwarden (infra-talos/bitwarden/templates/secrets.yaml): Store reference updated in commit 3583ecdc (July 10), along with removal of a standalone vault ServiceAccount that was apparently left over from an earlier architecture.
  • cert-manager (dev/cert-manager/templates/secretvault.yaml): Same change in commit b92c99ff (July 14), again with a vault ServiceAccount removal.
  • OpenClarity (infra-talos/openclarity/templates/secret.yaml): Created from scratch with vault-backend as the store reference.

The vault-backend ClusterSecretStore uses Vault AppRole authentication, which provides finer-grained access control and audit trail compared to the older Kubernetes-native auth that used per-namespace ServiceAccount tokens. The old vault store that was being removed was a ClusterSecretStore resource that used the kubernetes auth method with a service account that had broad access to Vault paths. The new approach centralizes credential management: the vault-backend store is defined once at the cluster level, and each application’s ExternalSecret specifies only the Vault path and keys it needs.

Cert-Manager Vault IP Updates and Cleanup

Commit 62593cb9 (July 14) updated the hostAliases IP for Vault in the dmz-talos cert-manager values.yaml files. The dmz-talos values changed from 192.168.12.130 to 192.168.12.141 (the Talos VIP). This swap reflects the fact that each cluster’s cert-manager should talk to the Vault instance co-located on its own infrastructure, rather than crossing cluster boundaries for certificate issuance.

Commit 8b4b88f4 (July 14) removed a legacy ServiceAccount and its associated token Secret from dmz-talos/cert-manager/templates/issuer.yaml. The deleted resources were a v1/ServiceAccount named issuer and a manually-managed kubernetes.io/service-account-token Secret. In Kubernetes 1.24+, service account tokens are no longer auto-generated, and the token Secret approach was deprecated in favor of the TokenRequest API. The cert-manager issuer on dmz-talos had already been switched to use vault-issuer-token with a secretRef in commit d0ba12cb (also July 14), so this ServiceAccount was dead configuration.

durpdeploy: Attack Drill and Roles Documentation

The durpdeploy repository had one commit this week (11e80ae5, July 9) that added two documentation files and updated the e2e test suite.

Attack Drill Documentation

The docs/attack-drill.md file is a five-minute hands-on walkthrough of the three most likely attacks against a deployed durpdeploy instance, designed to be run as a quarterly exercise after every deploy to a new VM. Each drill includes the exact curl command to simulate the attack, the expected behavior, what defends against it, and how to detect it in production.

Drill 1: Password guessing (brute force). The drill simulates 20 login attempts with wrong passwords against the login endpoint. Every attempt should return HTTP 422 (not 401 – the distinction matters: 401 means “authenticate,” while 422 means “your request was well-formed but I can’t process it”). Each wrong-password guess takes roughly 100ms of server time because HashPassword in internal/auth/auth.go uses argon2id with time=2, memory=64MB, threads=2. At 10 attempts per second, one vCPU is fully saturated. The drill notes that failed logins are deliberately not written to the audit_log table to prevent user enumeration attacks, but they appear in the structured slog output and can be alerted on via journalctl:

sudo journalctl -u durpdeploy --since "5 min ago" | grep '"path":"/login".*"status":422'

Drill 2: CSRF via stolen cookie. An attacker who has obtained a valid session cookie (via XSS, network interception, or physical access) tries to make a state-changing POST. Without the csrf_token parameter or X-CSRF-Token header, the CSRF middleware returns 403 before any handler logic runs. The same request with a valid CSRF token returns a 303 redirect. This validates that the CSRF protection is working at the middleware layer and not accidentally bypassed.

Drill 3: Direct database read. If an attacker obtains a copy of the SQLite database file (backup leak, compromised admin machine, misconfigured backup job), the password hashes are argon2id-encoded with per-user random salts. The drill demonstrates that the hash format is $argon2id$v=19$m=65536,t=2,p=2$<salt>$<hash> and that each user has a unique salt. The recovery procedure includes rotating all passwords if a leak is suspected, checking the audit_log for any anomalous activity, and rotating the SESSION_SECRET environment variable to invalidate all existing sessions.

The drill document has a pragmatic design philosophy: it’s written as a single bash block per drill that a teammate can copy-paste, with no dependencies beyond curl, sudo, and journalctl. The expected output is clearly documented for each step, so even someone unfamiliar with the codebase can verify the defenses are working.

Roles Documentation

The docs/roles.md file documents the three-user role model: admin (everything + audit log access), deployer (same writes as admin but no audit log), and viewer (read-only, every state-changing request returns 403). The document explains where the gates live in the codebase:

  1. Auth middleware (internal/auth/middleware.go) – session verification, same for all roles.
  2. CSRF middleware (internal/auth/csrf.go) – the coarse “can write at all” gate for viewer users.
  3. RequireRole("admin") middleware – gates only the /admin/* sub-group.

A notable caveat documented in roles.md: any deployer can deploy to any project today. There’s no project_members table yet, so the practical “least privilege” is to make non-admins viewer until per-project authorization ships.

E2E Test Suite Updates

The e2e_test.sh script was substantially reworked (the diff is ~30KB). Based on the auth system that shipped last week, the e2e tests now exercise the full authentication flow: admin user creation via CLI, login with session cookie retrieval, CSRF token extraction from the login response, and authenticated API calls with proper CSRF headers. The test also validates that the session recovery mechanism works after a server restart (relaunching pending deployments from the deployments table).

Looking Ahead

On the infra-talos side, OpenClarity is deployed but I haven’t triggered the first vulnerability scan. The deleteJobPolicy: Always setting means I need to catch a scan job before it gets cleaned up to verify the pipeline works end-to-end. I also need to configure OpenClarity to push scan results somewhere actionable – right now a scan that finds CVEs produces results that nobody sees. The current plan is to have a Hermes cron job poll the OpenClarity API for recent findings and include a summary in the weekly roundup.

On the durpdeploy front, the attack drill document is written and the roles model is documented, but the per-project authorization model (project_members table) is still a gap. Without it, any deployer can deploy to any project, which doesn’t scale past a single team. That’s the next P1 feature for the application.

The SecretStore consolidation is nearly complete. I should audit all environments to make sure no application is still referencing the old vault store, then delete it to prevent accidental re-introduction. A grep for name: vault$ in the gitops repo would catch any remaining references.