Controller Fails Open Instead of Hard-Erroring When CODER_URL Is Unavailable During Worker Provisioning

Summary

When the AgentFlow controller attempts to provision a Coder workspace for a worker (forge, sentinel, or lore), it silently returns Ok(None) if CODER_URL and its associated token cannot be resolved, logs a WARN, retries, and eventually escalates the ticket to human intervention. The underlying cause is that coder_client_from_store treats a missing URL/token as a recoverable non-error rather than a hard configuration fault. The practical impact is that workers are never provisioned, tickets are silently reassigned, and operators have no actionable failure signal.

Root-Cause Analysis

Confirmed Evidence

1. coder_client_from_store fails open. In crates/agent-nexus/src/lib.rs, the function resolves CODER_URL from two sources in order:

• The Redis store key coder_url
• The CODER_URL environment variable

When neither is present, it returns Ok(None). The caller logs a WARN ("Coder workspace requested but CODER_URL/token are unavailable") and returns without provisioning. This is a fail-open path: a configuration fault is treated as a graceful no-op.

1. The Redis coder_url key is never written. No code path in the codebase populates the coder_url Redis key at bootstrap or at tenant provisioning time. The function therefore always falls through to the environment variable.

1. Bootstrap/run masks the misconfiguration with a silent default. In binary/src/bin/agentflow.rs:150, the bootstrap path resolves CODER_URL with a default of http://localhost:7080. This means bootstrap completes without error on a fresh system where CODER_URL is not set, giving the operator a false signal that the system is correctly configured.

1. The controller cannot distinguish "intentionally no Coder" from "misconfigured Coder." Because Ok(None) is returned in both cases, the controller has no way to know whether Coder integration was intentionally disabled or simply misconfigured. Retry logic fires, escalation fires, and the ticket is reassigned — all without a human-visible error.

Reasonable Inference

• The coder_url Redis key appears to have been designed for runtime reconfiguration (so operators could update the URL without restarting the process), but the write side was never implemented. The read side is therefore dead code under normal operation.
• The localhost:7080 bootstrap default was likely added as a convenience for local development but was not guarded behind a dev-mode flag, causing it to silently suppress misconfiguration in production-like environments.

Alternative Causes (if evidence is incomplete)

• It is possible CODER_URL is set but the token is missing. The issue groups URL and token together; if only one is absent, the failure mode and fix path differ slightly. Operators should verify both are present and non-empty.

Resolution Steps

1. Change coder_client_from_store to return a hard error when provisioning is required and configuration is absent.

Replace the Ok(None) return path with a typed error (e.g., Err(AgentNexusError::CoderNotConfigured(...))) that surfaces a human-readable message such as:

> CODER_URL is not configured. Set CODER_URL and CODER_TOKEN in your environment or .env file and restart the controller.

The caller should propagate this error rather than swallow it, so it surfaces in the ticket's failure record and in structured logs at ERROR level.

1. Populate the Redis coder_url key during bootstrap and tenant add.

In binary/src/bin/agentflow.rs (and any tenant provisioning path), after validating CODER_URL, write it to the Redis store key coder_url. This closes the gap between the intended design (runtime-reconfigurable URL via Redis) and the current reality (key never written). This also makes the runtime behavior self-consistent: coder_client_from_store will find the key regardless of whether the env var is still present.

1. Remove or guard the localhost:7080 bootstrap default.

The silent default in binary/src/bin/agentflow.rs:150 must not be used in production. Options:

• Require CODER_URL explicitly and fail bootstrap with a clear error if it is absent.
• Allow the default only when an explicit --dev flag is passed, and emit a prominent WARN that workspace provisioning will target localhost.

1. Audit all callers of coder_client_from_store.

Identify every call site. For each:

• If Coder provisioning is genuinely optional at that call site, document the intent explicitly in code and handle None with a debug-level log, not a warn.
• If Coder provisioning is required (as it is for worker provisioning), propagate the error.

1. Add a startup configuration check in the controller.

At controller startup (before accepting any tickets), validate that CODER_URL is reachable and that the token is non-empty. Fail the startup probe if either check fails. This converts a runtime provisioning failure into a deployment-time failure, which is far easier to diagnose.

CLI Commands

Verify whether CODER_URL is set in the running environment:

printenv CODER_URL
printenv CODER_TOKEN

Check whether the Redis coder_url key is populated:

redis-cli -h <REDIS_HOST> -p <REDIS_PORT> GET coder_url

If the key is missing, populate it manually as a stopgap until the bootstrap write is implemented:

redis-cli -h <REDIS_HOST> -p <REDIS_PORT> SET coder_url "<CODER_URL_VALUE>"

Confirm the controller is picking up CODER_URL from the environment (substitute the correct process name or container):

# Kubernetes
kubectl exec -n <NAMESPACE> <POD_NAME> -- env | grep CODER

# Docker
docker exec <CONTAINER_NAME> env | grep CODER

# systemd
systemctl show-environment | grep CODER

Tail controller logs for the warn/error path to confirm current behavior before applying fixes:

# Kubernetes
kubectl logs -n <NAMESPACE> <POD_NAME> --since=1h | grep -i coder

# journald
journalctl -u <SERVICE_NAME> --since "1 hour ago" | grep -i coder

Configuration Snippets

Minimum required environment configuration for the controller:

CODER_URL=https://<CODER_HOST>
CODER_TOKEN=<CODER_API_TOKEN>

Example .env entry with a comment that prevents the silent-default trap:

# Required. No default. Controller will hard-fail if unset.
CODER_URL=https://coder.internal.example.com

# Required. Generate via: coder tokens create
CODER_TOKEN=<TOKEN>

Pseudocode showing the intended Rust change to fail closed (for code review reference):

// crates/agent-nexus/src/lib.rs
pub async fn coder_client_from_store(store: &Store) -> Result<CoderClient, AgentNexusError> {
    let url = store.get("coder_url").await?
        .or_else(|| std::env::var("CODER_URL").ok())
        .ok_or_else(|| AgentNexusError::CoderNotConfigured(
            "CODER_URL is not set. Configure it in your environment or .env file \
             and restart the controller.".into()
        ))?;

    let token = std::env::var("CODER_TOKEN")
        .map_err(|_| AgentNexusError::CoderNotConfigured(
            "CODER_TOKEN is not set. Generate a token via `coder tokens create`.".into()
        ))?;

    CoderClient::new(url, token)
}

Verification

After applying the fix:

1. Controller refuses to start with missing configuration:

# Unset CODER_URL and start the controller
unset CODER_URL
cargo run --bin agentflow
# Expected: immediate error at startup, non-zero exit code, message referencing CODER_URL

2. Controller starts successfully with correct configuration:

CODER_URL=https://<CODER_HOST> CODER_TOKEN=<TOKEN> cargo run --bin agentflow
# Expected: startup completes, no WARN about Coder configuration

3. Redis key is written during bootstrap:

redis-cli -h <REDIS_HOST> -p <REDIS_PORT> GET coder_url
# Expected: returns the configured CODER_URL value, not (nil)

4. Worker provisioning succeeds end-to-end:

Submit a ticket that triggers worker provisioning. Confirm:

• No WARN log line matching "Coder workspace requested but CODER_URL/token are unavailable"
• A Coder workspace appears in the Coder UI for the provisioned worker
• The ticket progresses past the provisioning state without escalation

Rollback indicator: If after the fix the controller emits CoderNotConfigured errors on every ticket, the environment variables are still absent in the deployment. Recheck secret/ConfigMap injection and restart the pod/service.

Prevention

1. Startup configuration validation. Add a dedicated configuration validation step at process startup that checks all required external service credentials (Coder URL reachable, token non-empty, Redis writable). Fail fast with a structured error before accepting any work.

1. Bootstrap writes all derived config to the store. Establish the invariant that every key coder_client_from_store reads from Redis must be written by bootstrap or tenant add. Enforce this with an integration test that runs bootstrap and asserts the key exists in Redis afterward.

1. Remove silent defaults for external service URLs. Audit all unwrap_or("http://localhost:...") patterns in non-dev code paths. Replace with explicit required-or-error resolution. If a localhost default is needed for development, gate it on an explicit --dev / APP_ENV=development flag.

1. CI integration test for missing CODER_URL. Add a test that starts the controller without CODER_URL set and asserts it exits with a non-zero code and a message matching the expected configuration error. This prevents regression to fail-open behavior.

1. Alerting on WARN log volume. Until code changes are deployed, configure a log-based alert that fires when the "Coder workspace requested but CODER_URL/token are unavailable" warn message appears more than once in a rolling 5-minute window. This provides an operational signal while the fix is in flight.

1. Document the required environment variables. Add CODER_URL and CODER_TOKEN to the project's deployment checklist, runbook, and any Helm chart or Kubernetes manifest templates as explicitly required (not optional) values, with no default substitution.