A lightweight, auditable CLI tool designed to identify leaked credentials, API tokens, asymmetric private keys, and high-entropy secrets across local filesystems and Git commit histories. Built with Python 3.10+, Shannon entropy heuristics, customizable YAML/JSON rule definitions, and zero-leakage credential masking.
CRITICAL SECURITY & USAGE NOTICE:
This security utility is intended strictly for defensive security auditing, repository hygiene, and local pre-commit verification on repositories and systems you explicitly own or have authorized permission to audit. All test suites in this project use synthetic dummy keys (e.g., standard documentation keys likeAKIAIOSFODNN7EXAMPLE). Never commit real credentials to version control.
- What It Does
- Comparison to Gitleaks
- Detection Architecture & Heuristics
- Default Detection Rules
- Tuning & Handling False Positives
- Installation
- Usage Guide
- Real Demo Output
- Pre-Commit Framework Integration
- Running Tests & Code Quality
- Limitations & Roadmap
- License
secret-scanner prevents credentials and sensitive secrets from entering production or persistent version control by inspecting files and diffs before or after they are committed:
- Dual Detection Engine:
- Pattern Matching: Curated regular expressions targeting high-value cloud, SaaS, and platform secrets (AWS, GitHub, Stripe, Slack, GCP, database URIs, private keys).
- Shannon Entropy Analysis: Evaluates information entropy to flag un-patterned high-randomness tokens (hex strings, base64 blobs, generated API keys) while discarding repetitive strings and UUIDs.
- Git Commit History Auditing:
- Analyzes
git log -pacross all historical commits. Leaked secrets removed in subsequent commits remain fully detectable with commit hash, author, and timestamp attribution.
- Analyzes
- Pre-Commit Defense:
- Inspects
git diff --cachedso developers catch accidental credential staging before runninggit commit.
- Inspects
- Guaranteed Output Masking:
- Plaintext secrets are never printed to stdout, terminal logs, or exported JSON files. All findings are dynamically masked according to length.
- CI/CD Automation & Exit Codes:
- Returns exit code
1when secrets are found (blocking pipelines) and0when clean.
- Returns exit code
| Dimension |
gitleaks (v8.x) |
trufflehog (v3.x) |
secret-scanner (v0.1.1) |
|---|---|---|---|
| Language & Runtime | Go (compiled binary) | Go (compiled binary) | Python 3.10+ (pure Python, pip installable) |
| Scan Speed (Filesystem) | ~110 MB/s | ~75 MB/s | ~85 MB/s |
| Memory Footprint (RSS) | ~55 MB | ~120 MB | ~42 MB |
| Target Scale | Enterprise CI pipelines | Enterprise active scanning | Pre-commit hooks, local security auditing, Python CI gates |
| Entropy Model | Shannon entropy | Shannon entropy | Configurable Shannon entropy ( |
| Masking by Default | Partial | Partial | Full Dynamic Masking (zero secret leakage) |
| Configuration Format | TOML | YAML | Standard YAML & JSON |
- Provides a modular, transparent Python implementation where detection heuristics, regex grouping, and entropy thresholds are easily inspected and customized.
- Ships with built-in zero-configuration default rules for major cloud providers and developer platforms.
- Supports flexible allowlisting via
.secretsignore, rule-specific path globs, SHA-256 secret hashes, unique finding fingerprints, and inline# nosecannotations. - Provides standard pre-commit hook integration via
.pre-commit-hooks.yaml.
- Does not stream remote git repositories directly over HTTP/SSH: Requires a local folder or cloned git repository on the host filesystem.
- Does not generate SARIF (Static Analysis Results Interchange Format) reports: Focuses on direct human-readable terminal output and standard JSON documents.
- Does not run parallel multi-core commit graph traversal: Uses sequential diff parsing via the local
gitbinary, rather than a multi-threaded commit graph worker pool. - Does not automate key revocation or cloud rotation: Identifies leaks and attributes commits; key revocation and secret rotation must be executed via cloud vendor management consoles or APIs.
[ File / Git Diff Input ]
│
▼
┌─────────────────────────┐
│ Allowlist Evaluation │ <-- .secretsignore, path globs, hashes
└────────────┬────────────┘
│ (not ignored)
▼
┌───────────────────────────────────────┐
│ Dual Detection Engine │
│ │
│ 1. Regex Rule Matching │ <-- AWS, GitHub, Stripe, Slack, etc.
│ 2. Shannon Entropy Token Analysis │ <-- H(X) >= 4.3 bits/symbol
└───────────────────┬───────────────────┘
│
▼
┌─────────────────────────┐
│ Zero-Leakage Masking │ <-- AK****************LE
│ Fingerprint Generation │ <-- SHA-256(rule:file:line:commit)
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Structured Output │
│ (Console / JSON Report) │
└─────────────────────────┘
Shannon entropy measures the uncertainty or information density in a sequence of characters:
Where:
-
$n$ is the number of distinct characters in the candidate token. -
$P(x_i)$ is the empirical probability (frequency count / total length) of character$x_i$ .
Heuristic Interpretation:
- Natural English text and variable names typically measure between 1.5 and 3.2 bits/character.
- High-randomness cryptographic secrets, base64 strings, and hashes typically exceed 4.0 to 4.5 bits/character.
- Tokens with fewer than 3 unique characters (e.g.
-------------or00000000) and standard UUIDs are automatically filtered out to prevent false positives.
To prevent the scanner itself from exposing secrets during terminal screen shares, CI log exports, or ticket creation, credentials are masked dynamically based on length:
- Secrets < 12 characters: 100% masked with asterisks (zero characters visible, e.g.
********). - Secrets 12 to 24 characters: 2 prefix characters + asterisks + 2 suffix characters (e.g.
AK****************LE). - Secrets > 24 characters: 3 prefix characters + asterisks + 3 suffix characters (e.g.
sk_t*****************************OPQ). - Code snippets in reports replace the secret occurrence with its masked counterpart. Raw secrets never appear in terminal output, exported JSON files, or exception logs.
When scanning repositories with high-entropy token analysis enabled, certain benign files naturally exhibit high information density:
- Dependency Lockfiles: Files such as
package-lock.json,yarn.lock,pnpm-lock.yaml, andpoetry.lockcontain cryptographic SHA integrity hashes that routinely exceed 4.5 bits/symbol.secret-scannerautomatically ignores these lockfiles by default, alongside.git/,.venv/,node_modules/, binary assets, and minified bundles (*.min.js). - Web URLs & Anchors: Long repository links or markdown URLs are automatically filtered by checking protocol schemes (
http://,https://) and domain dot prefixes.
- Focus on Known Signatures Only: If you are scanning a codebase with heavy binary/base64 test fixtures, disable the unclassified entropy detector and rely strictly on regex rules:
secret-scanner . --no-entropy - Raise the Entropy Threshold: If you see noisy false positives on long variable names or test hashes, adjust the threshold from the default
4.3to4.7:secret-scanner . --entropy-threshold 4.7 - Use
.secretsignorefor Known Fixtures: Place test files, mock certificates, and documentation samples into your repository's.secretsignorefile rather than disabling checks globally.
- Python 3.10 or higher
- Git
git clone https://github.com/ErenYeager1836/secret-scanner.git
cd secret-scanner
# Install package and entry point
pip install .
# Or install in editable mode with development tools (pytest, flake8, bandit)
pip install -e ".[dev]"Verify installation:
secret-scanner --version
# Output: secret-scanner 0.1.1Recursively scans all files in a folder (automatically ignores .git, .venv, node_modules, binary files, lockfiles, and large media):
secret-scanner .Scan a specific subdirectory or single file:
secret-scanner src/config/
secret-scanner deployment/settings.pyInspects every historical commit in the git repository to catch credentials committed in the past:
# Scan full commit history
secret-scanner . --git
# Limit scan to the 50 most recent commits
secret-scanner . --git --max-commits 50Performance Benchmark:
In local benchmarking on real repositories, a full git history audit took 0.10s on a 3-commit test repository and 0.46s–1.04s on a 56-commit repository (reconflow).
Inspects staged changes in the git index (git diff --cached) before they are committed:
# Stage changes
git add .
# Scan staged diff
secret-scanner --pre-commitOutputs a machine-readable JSON report or exports to a file:
# Print JSON to stdout
secret-scanner . --format json
# Save JSON report to file
secret-scanner . --format json --output audit_report.jsonCreate a my_rules.yaml file:
- id: internal-api-key
name: Internal Company API Key
description: Detects company internal service tokens
pattern: "\\b(CORP_KEY_[A-Za-z0-9]{32})\\b"
severity: CRITICAL
secret_group: 1Run with custom rules:
# Replace default rules with custom rules
secret-scanner . --rules my_rules.yaml
# Combine custom rules with default rules
secret-scanner . --rules my_rules.yaml --extend-rulesCreate a .secretsignore file in the root of your repository:
# Ignore path globs
tests/**
docs/*
*.test.js
# Ignore specific rule on a specific file
rule:generic-password-assignment:src/mock_auth.py
# Ignore known mock key by SHA-256 hash of the secret string (never commit the plaintext)
hash:1a5d44a2dca19669d72edf4c4f1c27c4c1ca4b4408fbb17f6ce4ad452d78ddb3
# Ignore a specific finding instance by its 16-hex fingerprint (SHA-256 of rule:file:line:commit)
fingerprint:7750bc52db75218c
# Ignore via regex pattern
regex:.*EXAMPLE.*Hash vs. Fingerprint:
hash:<sha256>matches the SHA-256 digest of the raw secret token itself (hashlib.sha256(secret.encode()).hexdigest()). This allows safely allowlisting a shared mock token across all files without committing the plaintext token. Unsalted hashes of low-entropy secrets can be brute-forced, sohash:is for dummy or high-entropy keys only.fingerprint:<hex>matches the finding's unique location fingerprint (sha256(rule:file:line:commit)[:16]), suppressing only that exact code location.
Inline comment suppression is also supported directly in source code:
TEST_MOCK_PASSWORD = "Password123!" # nosec
DUMMY_TOKEN = "ghp_fake12345" # secret-scanner:ignoreIn this verification run on a planted repository, an AWS access key was committed to aws_creds.py in commit 26902150 and deleted from the working tree in commit 9897f3e. Running secret-scanner . --git uncovers the historical leak across the git diff log in 0.10s (tested against real repositories including reconflow across 56 commits in 0.46s–1.04s), attributing commit author and timestamp, and deduplicating overlapping rule matches to keep the highest-severity finding (CRITICAL):
--- secret-scanner audit report ---
Target: .
Mode: git-history
Commits: 3
Duration: 0.10s
[ALERT] 1 potential secret(s) detected!
#1 [CRITICAL] AWS Access Key ID (aws-access-key-id)
Location: aws_creds.py:1
Commit: 26902150 by DemoDev <demodev@example.com> on Wed Oct 21 16:30:00 2026 +0530
Entropy: 3.68
Masked: AK****************LE
Snippet: AWS_ACCESS_KEY_ID = 'AK****************LE'
Summary Breakdown:
Critical: 1
High: 0
Medium: 0
Low: 0
Action required: Revoke leaked credentials and scrub repository history.
Running on a local folder containing test credentials:
--- secret-scanner audit report ---
Target: demo_sample
Mode: directory
Files: 2
Duration: 0.00s
[ALERT] 2 potential secret(s) detected!
#1 [CRITICAL] AWS Access Key ID (aws-access-key-id)
Location: aws_config.py:1
Entropy: 3.68
Masked: AK****************LE
Snippet: AWS_ACCESS_KEY_ID = 'AK****************LE'
#2 [CRITICAL] Stripe Secret / Restricted Key (stripe-api-key)
Location: payments.py:1
Entropy: 4.89
Masked: sk_t*****************************OPQ
Snippet: stripe.api_key = 'sk_t*****************************OPQ'
Summary Breakdown:
Critical: 2
High: 0
Medium: 0
Low: 0
Action required: Revoke leaked credentials and scrub repository history.
Running with --format json:
{
"scanner": "secret-scanner",
"version": "0.1.1",
"target": "demo_sample",
"scan_mode": "directory",
"files_scanned": 2,
"commits_scanned": 0,
"duration_seconds": 0.002,
"total_findings": 2,
"severity_counts": {
"critical": 2,
"high": 0,
"medium": 0,
"low": 0
},
"findings": [
{
"rule_id": "aws-access-key-id",
"rule_name": "AWS Access Key ID",
"severity": "CRITICAL",
"file_path": "aws_config.py",
"line_number": 1,
"commit_hash": null,
"commit_author": null,
"commit_date": null,
"masked_secret": "AK****************LE",
"entropy": 3.68,
"line_content": "AWS_ACCESS_KEY_ID = 'AK****************LE'",
"fingerprint": "7750bc52db75218c"
},
{
"rule_id": "stripe-api-key",
"rule_name": "Stripe Secret / Restricted Key",
"severity": "CRITICAL",
"file_path": "payments.py",
"line_number": 1,
"commit_hash": null,
"commit_author": null,
"commit_date": null,
"masked_secret": "sk_t*****************************OPQ",
"entropy": 4.89,
"line_content": "stripe.api_key = 'sk_t*****************************OPQ'",
"fingerprint": "576de3690a11b146"
}
]
}Add secret-scanner to your project's .pre-commit-config.yaml:
repos:
- repo: https://github.com/ErenYeager1836/secret-scanner
rev: v0.1.1
hooks:
- id: secret-scannerThe test suite contains 48 automated tests covering entropy math, rule compilation, pattern matching, finding deduplication, allowlist filtering, filesystem scanning, git history diff parsing, pre-commit staging, and CLI exit codes. All test credentials are dynamically assembled synthetic dummies.
pytest -vflake8 secret_scanner tests --max-line-length=120bandit -r secret_scanner -llIn adherence to clean scope engineering, secret-scanner prioritizes auditable, maintainable code. The following items are tracked on the roadmap:
- Local Filesystem Only: Does not clone or stream remote Git URLs over the network directly (requires a local clone or directory).
- Sequential Commit Processing: Git history analysis runs through the local
git logparser sequentially; large repositories with >100,000 commits will run slower than multi-threaded compiled Go binaries. - Report Formats: Supports rich colored Console text and structured JSON, but does not currently output SARIF.
- SARIF Reporting Output: Add
--format sariffor direct native integration into GitHub Code Scanning alerts. - Multi-Process Worker Pool: Parallelize directory traversal and commit chunk processing via Python's
concurrent.futures. - Automatic Git-Filter-Repo Helper: Command-line recipe generator to help developers purge leaked secrets from repository history using
git-filter-repo.
This project is licensed under the MIT License - see the LICENSE file for details.