This week was about making operational boundaries explicit and giving the build system enough dedicated capacity to be dependable. The work crossed the GitOps, infrastructure-as-code, deployment-server, deployment-agent, Terraform-module, and runbook repositories. The activity was broad, but it had a coherent shape: the DMZ GitLab Runner estate became a purpose-built pool rather than an opportunistic workload, private ingress policy became more systematic, storage and module-proxy services gained clearer contracts, and DurpDeploy received further work on execution safety, authorization, diagnostics, and release controls.
I am deliberately describing this as work merged to each repository’s default branch, not as a claim that every resulting workload has already reconciled and completed rollout. Git is the desired state and the review record. Argo CD, GitLab CI, Terraform and the services themselves still provide the evidence for a completed deployment. That distinction matters most for networking, storage and worker-capacity changes, where a correct-looking diff is only the first part of the operation.
The platform work this week also reinforces a recurring lesson. Most difficult incidents do not come from one enormous design error. They come from a small ambiguity at a boundary: a private route that happens to be reachable from the wrong source network, a runner that competes with ordinary workloads, a storage configuration that depends on an untracked secret shape, or a deployment result that reports raw and potentially sensitive error text. The useful response is not more machinery everywhere. It is to make the contract at the boundary narrow, visible and tested.
Dedicated DMZ Capacity for GitLab CI
The most concrete infrastructure change was the creation and refinement of a dedicated DMZ GitLab Runner worker pool. The Terraform configuration defines three runner nodes, runner01-dmz, runner02-dmz, and runner03-dmz, distributed across the existing Proxmox hosts. Their configuration is intentionally separate from the regular DMZ workers. That preserves the existing general-worker addresses and ordering instead of turning a capacity change into an accidental cluster-identity migration.
The pool started as three 8-vCPU, 16 GiB RAM, 20 GB-disk nodes. During the week I increased their disks to 100 GB and then increased each runner to 12 vCPUs. Those are practical adjustments for CI rather than generic scaling. Container-image builds, package caches, OCI layouts, test databases and build artifacts consume disk quickly, while concurrent compilation and test work benefits from CPU. Keeping the runner nodes separate lets me tune those resources for that workload without inflating every general Kubernetes worker.
The GitLab Runner chart now schedules both the runner manager and its job pods onto the dedicated nodes. This needs two related settings rather than a chart-level label alone. The manager deployment uses the workload=gitlab-runner node selector and tolerates the dedicated=gitlab-runner:NoSchedule taint. The runner’s TOML configuration repeats the selector and toleration for job pods, helpers and service containers. If only one of those paths were configured, CI work could still leak back onto the general pool or the manager could schedule while its jobs cannot. The accompanying documentation calls out that split so a later chart change does not accidentally erase half of the placement policy.
I also raised the runner concurrency from 10 to 30. That is paired with the dedicated node capacity rather than treated as a standalone number. Increasing concurrency without CPU, memory, disk and placement capacity turns queue time into contention and flaky jobs. In the other direction, provisioned runner nodes that remain underutilized do not reduce delivery latency. The intended result is a bounded execution environment: CI jobs get a known pool, regular workloads do not compete with image builds, and the runner configuration documents where the work is expected to land.
The helper image is now pulled through the existing Harbor GitLab proxy cache and is pinned to the matching GitLab Runner version. Helper containers are not incidental. They perform clone, artifact and service operations in Kubernetes-executed jobs, so an unplanned image pull can become the reason a job fails. Using the existing proxy cache reduces dependence on an external registry during repeated CI activity while retaining an explicit version contract. The configuration check verifies the helper image rather than relying on a YAML edit remaining correct by inspection.
The DMZ workers also received an opt-in user.max_user_namespaces sysctl. The setting is scoped through the Talos module’s worker configuration rather than applied by hand after boot. This supports rootless Podman workloads in CI, where user namespaces are required for the isolation model. It is a good example of keeping a hardware and operating-system prerequisite next to the workload that needs it. A future node replacement or Terraform recreate receives the same declared setting instead of rediscovering it during a failed job.
Right-Sizing the Infra Cluster
The infrastructure worker pool moved in the opposite direction: its count was reduced from six workers to three, while the remaining workers’ disks were doubled from 120 GiB to 240 GiB. The resulting pool still spans the three Proxmox hosts. This is not simply a reduction in resources. It is a consolidation of compute into fewer, roomier nodes while retaining host-level distribution.
The choice reflects the different failure and utilization profiles. Dedicated CI nodes now take the bursty build workload that can otherwise pressure the platform cluster. The infra workers can focus on the controllers, storage-adjacent services and platform applications they exist to run. Larger disks give those nodes more useful local capacity for their actual workload, while fewer nodes reduce the routine operational surface: fewer Talos machines to update, fewer scheduler targets to reason about and fewer places for a marginally used workload to end up.
This type of change needs post-apply verification rather than confidence from Terraform alone. The important checks are that the intended three workers exist, workloads have rescheduled as designed, PodDisruptionBudgets and replica counts remain compatible with the smaller pool, Longhorn replicas and storage health are acceptable, and node pressure stays comfortably below the available capacity. The configuration makes the desired topology explicit. It does not remove the need to observe the topology under real demand.
The shared Talos module also grew support for per-worker configuration patches and opt-in worker sysctls. That is a useful boundary in the module design. A cluster consumer can state a targeted exception for the worker that requires it without forking the entire machine configuration or making every worker look like a CI host. Module work is valuable when it removes duplicated, error-prone configuration while keeping environment-specific policy in the calling root.
Ingress Policy: Default Private, Explicit Exceptions
The GitOps repository saw a substantial pass over DMZ ingress policy. The policy is based on a simple default: HTTP IngressRoute objects are internal-only unless they explicitly declare the public internet-access middleware. A mutating admission policy prepends the default access chain, and a validating policy ensures routes begin with the expected Traefik middleware. This is a better failure mode than treating exposure as the default and hoping every service author remembers to add a whitelist.
I added CI validation for the policy and refined the routing declarations to make public exceptions explicit. The result is reviewable in the route itself. A reviewer can see that a public endpoint intentionally begins with the public middleware, while an unannotated management or internal route inherits the private chain. The configuration check also makes admission-default behavior testable in Git, rather than requiring someone to infer it from a running cluster after a route has already changed.
One special case was the private Athens Go module proxy. GitLab Runner pods on the dedicated DMZ nodes retain pod source addresses that the normal LAN/VLAN allowlist does not cover. Initially that was represented as a proxy-specific exception. The final shape broadens the shared internal allowlist to include the three runner pod CIDRs, while retaining the important constraints: it does not trust all pod CIDRs and it does not trust the proxy transit address. The values are documented as tied to the known runner nodes and require review when those nodes are replaced.
That is a deliberate tradeoff. It grants the CI workers access to private routes required by their builds, including the module proxy, without converting the entire cluster pod network into a trusted client network. It is also more maintainable than a growing series of one-off access chains. The ceiling is clear: if runner networks or trust requirements diverge further, the policy should return to narrower scoped middleware rather than quietly adding broad ranges.
I also configured Traefik’s HTTP entry point to issue a permanent redirect to HTTPS. This is a basic edge contract, but it is worth declaring and testing at the shared entry point. Individual applications should not need to implement their own redirect behavior merely to ensure clients use the encrypted route. A single Traefik setting makes the policy consistent for routes passing through that entry point.
Argo CD sync options were updated for Traefik and SearXNG to use server-side apply, alongside the existing admission-aware comparison behavior. Admission controllers can add or mutate fields that client-side apply and ordinary diff handling do not model cleanly. Server-side apply improves ownership tracking and reduces drift caused by a controller fighting a mutation it does not understand. The repository test checks these options on the applications that need them, which is more useful than a comment claiming that they are present.
Athens, Garage and Longhorn
Athens was added as a Go module proxy with Garage-backed storage. The service is deployed as an Argo CD application in its own namespace, stores its S3 connection details through an ExternalSecret backed by the existing Vault cluster secret store, and uses go.durp.info as its hostname. The DMZ route preserves the host header and connects to the infra cluster over HTTPS using a dedicated Traefik ServersTransport with the expected server name. The infra-side route terminates its own TLS and forwards to Athens on port 3000.
The two-layer route is intentional. The DMZ proxy and the infra service each have a clear responsibility, and the backend TLS relationship is configured instead of assumed. SNI, hostname, certificate and upstream server name must agree. When one of those details is implicit, a route can look healthy while clients fail with a certificate or protocol error. The Athens documentation records the path and its private-access requirement, which is especially useful for a service that CI depends on but users may not visit directly.
Garage gained a separate private management UI route at garage.internal.durp.info. It proxies to Unraid’s HTTP management port, uses a certificate managed by cert-manager, and requires the Authentik forward-auth middleware. The route is explicitly not the S3 API. That distinction avoids mixing an operator-facing UI with a data-plane endpoint and gives each one an appropriate access contract. The policy documentation counts it as an internal route, not a new public exception.
Longhorn backup configuration was moved to Garage using the s3://longhorn-backups@garage/ target. The backup credential remains represented as an ExternalSecret from the Vault backend store rather than a manually created Kubernetes secret. A small Python validation script verifies the target, secret store reference and generated secret shape in CI. Storage backups are a poor place to rely on memory or an old commented-out manifest. The useful outcome is a declarative backup destination that can be reviewed, recreated and checked before it is needed for recovery.
I also restored the Longhorn backend ingress configuration. The important operational distinction is between adding a path to Git and asserting that it has already been exercised under failure. The follow-up is to validate certificate issuance, proxy connectivity and Longhorn UI behavior from the intended client network, then verify that backup creation and restore workflows actually reach Garage using the configured credentials.
DurpDeploy: Execution Boundaries and Operator Clarity
DurpDeploy and durp-deploy-agent continued to evolve together. Both sides changed their default dedicated agent communication port from 10943 to 23187, with the development container mapping adjusted separately to 23188. The server charts, listener defaults, Dockerfile exposure, Compose examples, agent defaults, documentation and tests were updated as one contract. A port change is deceptively easy to make only halfway. If the server listens on the new port but the pairing client, service, firewall or published image still expects the old one, the failure is a connection problem that appears far from the original diff.
The server now exposes the agent port in its image and chart configuration, while the agent keeps its pairing and service contract aligned. The relevant tests cover both defaults and explicit overrides. That matters because operator-selected ports remain supported. The default should be coherent, but a deployment that has an established firewall policy or a shared host still needs a deliberate override path.
The agent added stronger release-package handling. Container jobs import a package into a bounded temporary filesystem volume, retain a read-only copy for the job, and remove the writable importer. Agent-side staging does not become a shared cache across claimed steps. Retries within one job can reuse already verified contents, but separate steps retain their own boundary. ZIP validation now rejects unsafe tree collisions, duplicate entries and conflicting implicit or explicit directories. Those conditions are not theoretical: archive extraction is parsing untrusted structure, and a conflict that is harmless in a happy-path archive can otherwise create surprising overwrite behavior.
Container execution gained a check for PID-controller support before each container attempt. Some runtimes can appear available while silently ignoring a process limit without the needed cgroup controller. Checking at attempt time covers changed runtime conditions and retries without blocking agent startup or polling merely because a future container step might be impossible. The retry behavior also distinguishes a short probe timeout from the whole step deadline, preventing a transient runtime probe from consuming the job’s entire execution budget.
The agent’s terminal reporting now returns bounded, classified failure diagnostics instead of untrusted raw error text. Examples include execution failures, step timeouts, log-delivery failures and interrupted execution recovery. The tests specifically ensure the bounded diagnostic does not expose private values from underlying errors. This is an operational and security improvement. Operators need a meaningful category in the result, but a failure result is not a safe channel for arbitrary command, variable or credential-bearing error strings.
Step logs now identify the execution location before script output: the selected agent, physical host where available, host or container mode, interpreter, attempt number, and for containers the runtime, container name and resolved image ID. This is the right kind of additional observability. It helps connect a failed step to its actual execution boundary without pretending the log itself is a source of secrets or configuration. A socket-free agent container can accept an operator-supplied physical hostname without being granted wider host access.
On the server side, DurpDeploy gained a Maintainer role and more precise project-scoped API token permissions. The authorization code separates readers, deployers and content authors rather than treating any authenticated user who can deploy as someone who can also modify project configuration. The project-role tests include the case where a global deployer has a project-admin membership but remains restricted from authoring. That is a meaningful least-privilege boundary, not just another role label in the UI.
The UI also received inline project-member role editing, clearer deployment error toasts, support for agents with artifact approval, and a change to keep step panels collapsed. Live deployment-log selection now excludes timestamps. These are smaller user-facing changes, but they reduce operator friction during the exact situations where people are reading quickly, comparing output or correcting project membership. The server also includes build numbers in the application and versioned image identities, making it easier to connect a running artifact, a visible version and its CI output.
Finally, the DurpDeploy delivery pipeline was tightened. The application added Go linting and expanded production image and deployment security scans. Pull requests build and scan images before release promotion, while the agent repository migrated its CI and publication flow to GitLab and Harbor. The agent’s pipeline rules suppress duplicate feature-branch pipelines when the merge-request pipeline is the intended validation path, while retaining default-branch and tag handling. CI is part of the product boundary here: it should test the artifact that will be promoted, avoid duplicate work, and publish only through the explicit pipeline path.
Looking Ahead
The immediate work is verification and measured follow-through. For the runner fleet, I will watch queue time, job scheduling, node utilization, disk consumption, image-cache behavior and the behavior of rootless Podman jobs. The goal is not simply to see three new nodes registered. It is to confirm that the dedicated pool removes contention without becoming an undersized bottleneck of its own.
For the reduced infra worker pool, I will validate placement, disruption tolerance, storage health and node pressure under normal platform activity. For GitOps networking, I will check Argo CD reconciliation, HTTPS redirects, certificate issuance, private-route access from the intended LAN and runner networks, and denial from networks that should remain outside the boundary. Athens, Garage and Longhorn all need end-to-end checks, including a real module fetch, UI authentication behavior, backup creation and a recovery-oriented restore test.
For DurpDeploy, the next checks focus on upgrade and failure paths: agent port migration with preserved pairing state, host and container package execution, invalid archive rejection, runtime PID capability failures, recovery after interrupted work, and role or token restrictions at real API boundaries. The desired end state remains conservative. CI has dedicated capacity, private routes are private by default, storage contracts are expressed in Git, and deployment tooling reports enough context to operate without turning its failure paths into a source of sensitive data.