daymade
776b760c10
代码（审阅 HIGH，已实跑复现）： - 全部 4 个脚本的 13 处 subprocess text=True 解码统一补 errors="replace" ——blob 通道（git grep, grep_all_commits）至今 strict 解码，GBK 编码 源文件含命中行时 verify/scan 照样 UnicodeDecodeError 崩掉无报告； Lesson 9 的处方此前只落在 message 通道，现对该类整体闭环 - 验证：GBK 源文件含泄漏 → FAILED 且 blob+message 双通道各自定位 commit hash（此前崩溃点）；scan_repo 同仓 exit 0；GBK message 不崩 文档（审阅 LOW×2 + INFO×1）： - tooling_notes/SKILL.md/Lesson 7 的 git log 字面命令与实际 --format=%H%x1f%B%x1e 不符，改为行为描述（hash 标注记录格式） - tooling_notes/Lesson 9/CHANGELOG 补 commit_message_commits 前 10 截断说明 - CHANGELOG 修正 #328 的 --yes 修复面（Step 4 两块 + reference 节一块） - Lesson 9 补记 blob 通道同类缺陷的发现与闭环 + 跨编码检测边界 （errors=replace 防崩不让 UTF-8 pattern 命中 GBK 字节，归 Layer 4） Co-authored-by: Claude <noreply@anthropic.com>
3.2 KiB
Tooling Notes for GitHub Sensitive Data Cleanup
git-filter-repo vs BFG Repo-Cleaner
git-filter-repo (recommended default)
- Pros: Modern, actively maintained, Python-based, flexible
--replace-text, readable docs, safer defaults thangit-filter-branch. - Cons: Requires a "fresh enough" clone (no multiple remotes, no stale refs). Slower than BFG on very large repositories.
- Install:
brew install git-filter-repo
BFG Repo-Cleaner
- Pros: Very fast on large repos. Good for removing large files or
converting files to
git-lfs. - Cons: Requires Java. Less flexible than git-filter-repo for arbitrary string replacement. Slightly more complex for private-domain replacement.
- Install: Download the JAR from the official repo.
Rule of thumb: Use git-filter-repo for private-domain / secret-string
replacement. Use BFG if the repo is huge and you are mainly removing large
files.
Common git-filter-repo Errors
"Need a fresh clone"
Error: Need a fresh clone to operate on. Please clone with `git clone --mirror ...`
Cause: The repo has multiple remotes, stale refs, or was not cloned normally.
Fix:
git clone --mirror /path/to/repo /tmp/repo-mirror.git
cd /tmp/repo-mirror.git
# run rewrite_history.py here
"Cannot combine --force with ..."
git-filter-repo has strict option validation. Read the error and adjust the
command. The bundled rewrite_history.py uses the minimal safe set of flags.
gitleaks Allowlist Patterns
If gitleaks flags test fixtures or documentation examples, add an allowlist rather than bypassing the hook.
Example .gitleaks.toml:
title = "Repo allowlist"
[allowlist]
paths = [
'''tests/fixtures/secrets.json''',
'''docs/examples.md''',
]
regexes = [
'''sk-kimi-REDACTED''',
]
Never use --no-verify to suppress a real finding.
Replacement File Syntax
git-filter-repo --replace-text accepts a file with one replacement per line:
literal:old-string==>new-string
regex:old-pattern==>new-string
Use literal: for exact strings. Use regex: only when necessary and test
thoroughly, because a bad regex can corrupt many commits.
Checking Whether a String Is in History
git log --all --pickaxe-regex -S 'your-pattern' --pretty=format:'%H %s'
Use this pickaxe form for a manual "when did this string enter/leave" check.
verify_cleanup.py itself checks each pattern over both channels: blob
content (git grep across every commit's tree) and commit messages
(git log over all refs with a hash-annotated record format, decoded with
errors="replace" so legacy-encoded messages cannot crash verification).
A FAILED message check lists the offending commit hashes
(commit_message_commits, first 10), not just a count.
Git Bundle Backups
A bundle is a file that contains a complete copy of the repository refs. It can be cloned or fetched from later:
# Create
git bundle create backup.bundle --all
# Verify
git bundle verify backup.bundle
# Restore
git clone backup.bundle restored-repo
GitHub Support
For severe leaks (live production secrets, PII), contact GitHub Support after rotating credentials. They can remove cached views of sensitive data and assist with repository-level cleanup.