MOCK_MODE=true Ignored on SDK Vision Path Causes Live Anthropic API Billing

Summary

When MOCK_MODE=true and ANTHROPIC_API_KEY are both set, callVisionAgent bypasses all mock guards and routes directly to callClaudeSDK, sending vision critic requests to the live Anthropic API and incurring real charges. The assertNotAutomated() and isMockMode() guards exist exclusively in callClaudeCLI, leaving the SDK vision path entirely unprotected. This also defeats the CI fast-fail behavior: GitHub Actions runs with MOCK_MODE=true silently replay text agents from fixtures while billing for vision agents.

Root-Cause Analysis

Confirmed Evidence

The routing logic in vision-router.js:67-74 gates on two conditions:

hasApiKey() && imageCount > 0  →  callClaudeSDK(...)
otherwise                       →  callClaudeCLI(...)

This branch executes before any fixture or mock check. Because ANTHROPIC_API_KEY is typically present in developer environments and CI secrets, the condition is almost always true when images are involved.

The mock and automation guards live exclusively in claude-cli.js:121-126:

// claude-cli.js (lines 121-126) — only guard in the codebase
assertNotAutomated();
if (isMockMode()) {
  return nextFixture(prompt);
}

claude-sdk.js:84-116 (callClaudeSDK) has no equivalent guard. There is no call to assertNotAutomated(), isMockMode(), nextFixture(), or recordFixture() anywhere on the SDK path.

Confirmed Impact

| Scenario | Text agents | Vision agents | |—|—|—| | MOCK_MODE=true, no API key | Fixture replay | Fixture replay (CLI fallback) | | MOCK_MODE=true, API key set | Fixture replay | Live API — billed | | CI (MOCK_MODE=true, API key injected as secret) | Fixture replay | Live API — billed; no fast-fail |

Reasonable Inference

The SDK vision path was added after the CLI guard was written. The guard was not ported to callClaudeSDK or callVisionAgent, an omission consistent with the issue author's reference to this being a recurrence of issue #220 on a second code path.

Alternative Causes

It is theoretically possible that an outer caller is expected to check isMockMode() before invoking callVisionAgent, but the issue report and the file references indicate no such caller-side check exists.

Resolution Steps

1. Add assertNotAutomated() at the top of callVisionAgent (in vision-router.js) so both routing branches are covered by the CI refusal before any routing decision is made.

1. Add the mock-mode short-circuit in callClaudeSDK (in claude-sdk.js) immediately after argument validation, before the Anthropic client is constructed or any network call is made:

   if (isMockMode()) {
     return nextFixture(prompt);
   }

1. Instrument recordFixture on the SDK path so that live vision responses captured during fixture-recording runs are written to the same fixture store the CLI path reads from. This ensures parity between the two paths during fixture regeneration.

1. Remove or demote the guard in callClaudeCLI from being the sole enforcement point. It can remain as a defense-in-depth check, but it must no longer be the only gate.

1. Audit related issues #220 and the referenced %o2/%o3 items to confirm no third call path (e.g., a future embedding or audio agent route) bypasses the same guards.

CLI Commands

Locate all current call sites of assertNotAutomated and isMockMode to confirm no additional gaps exist:

grep -rn "assertNotAutomated\|isMockMode\|nextFixture\|recordFixture" scripts/utils/

Confirm which callers invoke callVisionAgent to assess blast radius:

grep -rn "callVisionAgent\|callClaudeSDK" scripts/ --include="*.js"

Check out the affected commit for local inspection:

git checkout f6d3078
grep -n "hasApiKey\|callClaudeSDK\|isMockMode" scripts/utils/vision-router.js
grep -n "isMockMode\|assertNotAutomated\|nextFixture\|recordFixture" scripts/utils/claude-sdk.js

Configuration Snippets

scripts/utils/vision-router.js — add guard before routing

// vision-router.js  (callVisionAgent, before line 67)
import { assertNotAutomated, isMockMode } from './mock-utils.js';
import { nextFixture } from './fixtures.js';

export async function callVisionAgent(prompt, images) {
  assertNotAutomated();           // ← NEW: fail fast in CI regardless of path

  if (isMockMode()) {             // ← NEW: honor MOCK_MODE before routing
    return nextFixture(prompt);
  }

  if (hasApiKey() && images.length > 0) {
    return callClaudeSDK(prompt, images);
  }
  return callClaudeCLI(prompt);
}

scripts/utils/claude-sdk.js — add defense-in-depth guard

// claude-sdk.js  (callClaudeSDK, top of function body — before line 84)
import { isMockMode, recordFixture, nextFixture } from './fixtures.js';

export async function callClaudeSDK(prompt, images) {
  if (isMockMode()) {             // ← NEW: defense-in-depth; should not reach here after router guard
    return nextFixture(prompt);
  }

  // ... existing Anthropic client construction and call ...

  const response = await client.messages.create({ /* ... */ });

  if (process.env.RECORD_FIXTURES === 'true') {
    recordFixture(prompt, response); // ← NEW: fixture parity with CLI path
  }

  return response;
}

Verification

Confirm mock mode is now respected on the vision path

Set both variables in a local shell and invoke a vision critic task. No API charge should occur and the response should come from fixtures:

MOCK_MODE=true ANTHROPIC_API_KEY=sk-ant-test-placeholder node scripts/run-critics.js --screenshot screenshot.png

Expected: process exits successfully, fixture data is returned, no HTTP request reaches api.anthropic.com.

Confirm assertNotAutomated fires in simulated CI

CI=true MOCK_MODE=true ANTHROPIC_API_KEY=sk-ant-test-placeholder node scripts/run-critics.js --screenshot screenshot.png

Expected: process exits with a non-zero code and a message referencing the automation guard, before any API call is made.

Confirm fixture recording writes SDK responses

RECORD_FIXTURES=true ANTHROPIC_API_KEY=<REAL_KEY> node scripts/run-critics.js --screenshot screenshot.png

Verify new fixture files are written for vision critic prompts alongside the existing CLI fixture files:

ls -lt fixtures/ | head -20

Rollback indicator

If the guard is incorrectly placed and breaks legitimate live runs, the symptom will be vision agents returning fixture data in production. Monitor for stale or truncated vision responses and revert the vision-router.js change if observed.

Prevention

1. Enforce guard co-location via lint rule or test: Add a unit test that instantiates callClaudeSDK with MOCK_MODE=true and asserts it never constructs the Anthropic client (spy/mock the constructor). This test will fail if the guard is removed or bypassed in future refactors.

1. Abstract the mock-guard preamble: Extract assertNotAutomated(); if (isMockMode()) return nextFixture(prompt); into a single checkMockMode(prompt) helper. Any new agent call path that omits it becomes visibly inconsistent during code review.

1. Block API key + MOCK_MODE combination in CI startup: Add a preflight check in the CI entry-point script that hard-fails if both MOCK_MODE=true and a real ANTHROPIC_API_KEY (non-placeholder) are detected simultaneously, making the misconfiguration immediately visible rather than silently expensive.

   if [[ "$MOCK_MODE" == "true" && "$ANTHROPIC_API_KEY" == sk-ant-api* ]]; then
     echo "ERROR: MOCK_MODE=true but a real ANTHROPIC_API_KEY is set. Unset the key or disable MOCK_MODE." >&2
     exit 1
   fi

1. Fixture parity requirement: Require that any new agent call path that supports live API calls also implements nextFixture / recordFixture hooks. Codify this as a PR checklist item or architecture decision record (ADR).

1. Cost alerting: Configure an Anthropic usage alert threshold so unexpected spend from CI pipelines is caught within minutes rather than at end-of-billing-period.