Token Counts Redacted by Tracking Sanitizer · Issue #642 · AISecurityLab/hackagent · GitHub
Skip to content

Token Counts Redacted by Tracking Sanitizer #642

Open

Description

@hassen8

Problem

hackagent/router/tracking/utils.py:24 marks any dict key containing token as sensitive:

_SENSITIVE_SUBSTRINGS = ("key", "token", "secret", "password")

sanitize_for_json() in hackagent/router/tracking/utils.py:85-86 then replaces
matching values with ***REDACTED***. This redacts operational metrics stored in the hackagent-db
via Tracker/StepTracker such as prompt_tokens, completion_tokens,
total_tokens, input_tokens, output_tokens, cached_tokens, and
token_count (e.g. response.usage in hackagent/router/router.py:895-897,
hackagent/server/api/models.py:435-443, hackagent/cli/tui/views/results/formatters/http.py:290-309).
Counts are not secrets and must remain visible in traces/results.

Sensitive keys like api_key, api_token, access_token, secret, password
must still be redacted.

Minimal Fix

In hackagent/router/tracking/utils.py add an exact-match allowlist checked
before the sensitive-substring test and keep the substring check for everything
else:

_TOKEN_COUNT_ALLOWLIST = frozenset({
    "prompt_tokens", "completion_tokens", "total_tokens",
    "input_tokens", "output_tokens", "cached_tokens",
    "token_count", "prompt_tokens_details", "completion_tokens_details",
})
# in sanitize_for_json, if k.lower() in allowlist → recurse, don't redact

Files To Change

  • hackagent/router/tracking/utils.py -> add allowlist and guard in sanitize_for_json.
  • tests/unit/router/tracking/test_tracker.py -> add regression: usage token counts preserved while api_token/api_key remain redacted (also cover nested prompt_tokens_details.cached_tokens).

Verification

uv run ruff format .
uv run ruff check .
env -u OLLAMA_HOST -u OLLAMA_BASE_URL -u OLLAMA_API_BASE -u OLLAMA_MODELS \
  uv run pytest tests/unit/router/tracking/ tests/unit/router/test_envelope.py tests/unit/cli/tui/test_tui_results_and_logs.py

Expected result: token-count fields persist through sanitize_for_json; secret/token keys still redacted; existing tracking tests remain green.

Activity

  1. franconicola commented on Sep 25, 2026

    @franconicola
    Member

    Hold on this until the restructure lands first (#640 / refactor/main). Reopen or retarget a fix after that lands so we don’t fight path/layout churn.

  2. franconicola commented on Oct 19, 2026

    @franconicola
    Member

    Partly fixed by the restructure: sanitize_for_json (now hackagent/tracking/serialize.py) keeps the usage counts prompt_tokens, completion_tokens, total_tokens, input_tokens, output_tokens, cached_tokens, reasoning_tokens and the other *_tokens counters.

    Still redacted on current main:

    sanitize_for_json({'usage': {'token_count': 3, 'prompt_tokens_details': {'cached_tokens': 1}}})
    # {'usage': {'token_count': '***REDACTED***', 'prompt_tokens_details': '***REDACTED***'}}

    So token_count and the *_tokens_details dicts still need to be exempted. The *_tokens_details dicts should be recursed into rather than kept whole.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions