Codebase Q&A

Codebase Q&A is a CLI tool that answers plain-English questions about a local codebase (“where’s auth handled?”, “what changed in the last commit that broke this?”) by giving Claude scoped, instrumented access to the repo through a custom MCP server. Every answer comes back with exact file:line citations, a confidence rating, and a full audit trail of every file Claude opened to produce it.

The project’s actual focus is the explainability layer built on top of that Q&A; the Q&A part alone is comparatively easy. An attribution checker verifies every cited file:line range against the real filesystem and flags anything that doesn’t hold up. A scope-transparency panel reports exactly what was and wasn’t read. Confidence and uncertainty type (epistemic versus aleatoric) are self-rated by Claude, then checked in code against what actually happened during the run. A separate faithfulness pass has a second, independent Claude call re-read the cited files straight off disk and score each claim. A stability metric reruns a question multiple times and measures Jaccard similarity across the sets of files Claude reads each run, an approach adapted from explanation-stability research in the SHAP and feature-attribution literature.