v1.10.0 Post-Release Evaluation

Date: 2026-05-02 Daemon: codelens-mcp v1.10.0 (HTTP, readonly profile, port 7839) Method: live native MCP calls + curl direct dispatch + grep/Read cross-check Scope: rate efficiency, accuracy, latency, token efficiency, native-MCP integration Bias note: this is a self-eval. Numbers below are reproducible from a clean clone — see "Reproduction" at the bottom.

Summary

Findings (5)

F1 — get_callers misses function-reference patterns (severity: high)

Symptom

mcp__codelens__get_callers(function_name="build_tools")
→ callers: 0, count: 0, confidence: 0.35, confidence_basis: "unresolved_only"

Reality (grep cross-check, crates/codelens-mcp/src/tool_defs/build.rs:13)

static TOOLS: LazyLock<Vec<Tool>> = LazyLock::new(build_tools);

build_tools has exactly 1 caller: it is passed as a function pointer to LazyLock::new. get_callers returned 0.

Root cause The tree-sitter call-graph extractor matches call expressions (callee(args)) but does not match function-reference expressions where the function is passed as a value (closure-construction, callback registration, builder accumulators). Concretely affected idioms:

• Rust: LazyLock::new(fn), OnceCell::get_or_init(fn), Box::new(fn as fn() -> T), iter.map(fn).collect() (when fn is a path expression, not an inline closure), event-bus register("name", fn).
• Go: defer fn() is fine, but go fn() and time.AfterFunc(d, fn) may also slip if the AST node kind differs.
• TypeScript: setTimeout(fn, 100), arr.map(fn), bus.on("evt", fn) when fn is a bare identifier (not a function () { ... } literal).

Severity High because get_callers returning 0 is a correctness signal a user may treat as "this is dead code". The handler does flag confidence: 0.35 and confidence_basis: "unresolved_only" — that is honest, but a model agent reading the JSON will still see count: 0 first.

Recommended fix Add a "function reference" extraction pass in crates/codelens-engine/src/call_graph/:

1. Walk identifier nodes whose parent is a call expression argument list and whose name matches a known function definition in the same project.
2. Tag the edge with resolution: "function_reference" and a separate confidence_basis: "structural_reference".
3. Surface both in confidence_basis so users can filter (include_function_references: bool, default true).

This matches the ROI/risk of the existing import_map resolution: a structural signal that's wrong sometimes (for non-call passes like type-annotation references) but right almost always for the patterns above.

Tracked: separate PR after v1.10.1.

F2 — claude-code host deferred surface is 11 tools, not 112 (severity: high)

Symptom

• get_current_config reports tool_count: 112, surface: preset:full.
• But claude-code's deferred tool registration (the surface a Claude Code session can actually call without ToolSearch keyword round-trips) is only 11: activate_project, get_analysis_job, get_analysis_section, get_callees, get_callers, get_current_config, get_ranked_context, set_preset, set_profile, start_analysis_job, verify_change_readiness
• find_symbol, get_symbols_overview, find_referencing_symbols, semantic_search, review_changes, plan_safe_refactor, explore_codebase etc. are not in deferred surface. ToolSearch select:mcp__codelens__find_symbol returns "No matching deferred tools found".
• The user can still reach them via ToolSearch keyword retrieval if the keyword happens to match the description, or via curl, or via host-side adapter rewiring — but none of these are in the native flow.

Root cause The claude-code host adapter (crates/codelens-mcp/src/dispatch/session.rs + src/protocol.rs::tool_anthropic_always_load) marks only the control-plane tools as always_load=true. tool_anthropic_search_hint is set on a wider list, but search hints rely on keyword similarity — which fails when the user's request doesn't share tokens with the description (e.g. asking for "signature of build_tools" won't match find_symbol's description).

Severity High for UX. The slogan "112 tools, workflow-first" is contradicted by "11 tools you can actually call right now". The product's primary affordance — workflow-first composite tools (explore_codebase, review_changes, plan_safe_refactor, etc.) — is invisible in the default host surface.

Recommended fix (this PR) Add the 7 workflow-first aliases + 6 high-value primitives to tool_anthropic_always_load for the claude-code host, raising the deferred surface from 11 to 24:

24 deferred entries is still well within the budget Anthropic recommends (the host adapter docs cite 30–50 as a typical cap). The remaining 88 tools stay reachable via ToolSearch keyword and direct dispatch.

F3 — get_ranked_context over-aggressive query expansion + 8K default budget (severity: medium)

Symptom

• Input query (5 words): "tool registry annotation bindings post-build pass"
• Effective query after expansion: "... tool_registry registry_annotation annotation_bindings bindings_post-build post-build_pass tool_registry_annotation registry_annotation_bindings annotation_bindings_post-build bindings_post-b..." — snake_case + camelCase + cartesian products of adjacent tokens.
• Result: 173 candidate symbols, Stage 5 truncation triggered (payload 15,318 tokens vs effective budget 8,000), only 3 symbols returned.
• Truncation hint correctly reported: compression_stage: 5, recovery_hint: "Response over budget; structured arrays clipped to 3 items each."

Root cause

1. expand_query_tokens in retrieval pipeline aggressively adds n-gram joins (a_b, b_c, aB, bC, etc.) — useful for partial-name matching, noisy for natural-language queries.
2. Default max_tokens for get_ranked_context is 8000 (defined in tools.toml). Hybrid retrieval (semantic + sparse + structural evidence) plus 173 symbol candidates routinely exceeds this.

Severity medium (not wrong, just truncated; users reading truncation_warning can recover).

Recommended fix (this PR)

1. Raise default max_tokens for get_ranked_context from 8000 → 16384 (still well below the 200K MCP cap).
2. Add expand_query: bool = true parameter; when false, use the raw query verbatim. Default stays true to preserve current behaviour for existing harnesses.
3. Document in tool description: "For natural-language queries, set expand_query=false; for partial-identifier search use the default."

F4 — verify_change_readiness top-findings are generic cues (severity: low)

Symptom For a change "Add a new tool category 'visualization' to tools.toml", top_findings were:

- "add_import_tool: verify crates/codelens-mcp/src/tools/mutation.rs first"
- "Tool: verify crates/codelens-mcp/src/protocol.rs first"
- "with_audit_category: verify crates/codelens-mcp/src/protocol.rs first"

These are file references derived from the ranked-files lane. Useful as cues, but they read as boilerplate ("verify X first"). A user who supplies the actual changed-files list expects task-specific evidence (e.g., "tools.toml schema_version is v1; new categories must register a generated function in build.rs and re-export in mod.rs").

Root cause verify_change_readiness blends ranked_context output with verifier signal labels. The blend is correct but generic: it lacks task-shape awareness ("are these changes co-evolutive?", "do they touch a known invariant set?").

Severity low — the 4-axis readiness check (diagnostic_ready, reference_safety, test_readiness, mutation_ready) is the actually useful part, and the blocker count (0) was correct.

Recommended fix (deferred, post-1.10.1) Add task-templates that pattern-match common change shapes (new tool category, rename across project, add feature flag, migrate ADR-X) and emit task-specific guidance instead of generic file cues. Out of scope for this patch.

F5 — Daemon boot fails with default-features build after ADR-0012 (severity: medium)

Symptom After ADR-0012 (default = []), running the launchd-loaded HTTP daemon binary fails with:

Error: HTTP/HTTPS transport requires the `http` feature. Rebuild with: cargo build --features http

Discovered on v1.10.0 release: the daemon was rebuilt with default features and refused to bind, causing both 7838/7839 ports to drop until rebuilt with --features http,semantic.

Root cause

• cargo install codelens-mcp (default-features) does not enable http (per ADR-0012, intentional).
• The launchd .plist files installed for the HTTP daemon were not updated to require the right binary, and the install docs do not say "if you want the HTTP daemon, build with --features http".

Severity medium — affects anyone running the daemon stack the way this repo's plists do; doesn't affect stdio MCP users.

Recommended fix (this PR)

1. Add an OperationalRequirements block to README and docs/release-verification.md documenting the feature-flag matrix (stdio default-features OK; HTTP daemon requires http; semantic search requires semantic).
2. Add a comment block to the launchd .plist files (dev.codelens.mcp-readonly.plist, dev.codelens.mcp-mutation.plist) noting the required build command. Plists don't render comments at runtime, but they survive in source as documentation.
3. (Optional, follow-up) Add a startup banner to the binary: when --transport http is requested but the binary lacks the http feature, fail with a user-actionable error pointing to the rebuild command — already done by ADR-0012, but the message could include cargo install --features http,semantic as the ready-to-paste form.

Reproduction

All findings reproducible from codelens-mcp v1.10.0 daemon:

# Boot a v1.10.0 HTTP daemon
cargo build --release --features http,semantic
target/release/codelens-mcp $(pwd) --transport http --profile reviewer-graph --port 7839 &

# F1 — call-graph callers miss
curl -s http://127.0.0.1:7839/mcp -X POST -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_callers","arguments":{"function_name":"build_tools"}}}'
grep -n "build_tools" crates/codelens-mcp/src/tool_defs/build.rs   # shows 3 lines incl. LazyLock::new

# F3 — Stage 5 truncation
curl -s http://127.0.0.1:7839/mcp -X POST -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_ranked_context","arguments":{"query":"tool registry annotation bindings post-build pass"}}}' \
  | python3 -c 'import json,sys; r=json.loads(sys.stdin.read())["result"]["content"][0]["text"]; d=json.loads(r); print(d.get("data",{}).get("truncation_warning"))'

Decision

v1.10.1 patch (this evaluation → PR fix branch):

• F2: extend claude-code host adapter tool_anthropic_always_load to 24 entries.
• F3: raise get_ranked_context.max_tokens default to 16384, add expand_query param.
• F5: feature-flag matrix in README + release-verification + plist comments.

Deferred to v1.11.0 (separate PRs):

• F1: tree-sitter call-graph function-reference extractor pass.
• F4: task-template-aware verify_change_readiness summaries.

The deferred items are larger surgery on the engine; bundling them into a patch release would inflate scope and dilute the value of the immediate UX fix (F2).

v1.11.0 follow-up — F1 closure (2026-05-02)

F1 shipped in v1.11.0:

• Extended RUST_CALL_QUERY with (arguments (identifier) @callee) and (arguments (scoped_identifier name: (identifier) @callee)) patterns to capture function-reference arguments (LazyLock::new(fn), iter.map(parse_line), callback registration).
• The 6-stage resolution cascade in resolve_call_edges already filters extracted candidates against the symbol DB: variable arguments (f(local_var)) where the name isn't a known function fall to unresolved and are dropped at confidence 0. Genuine function references resolve via Stage 5 (unique_name) at confidence 0.5 — honest, lower than import_map but no longer silently 0.
• Two regression tests pin the canonical cliff: extracts_rust_function_reference_arguments reproduces the build_tools / LazyLock::new shape; function_reference_extraction_is_resilient_to_variable_arguments ensures variable arguments don't break direct-call extraction.

Other languages (TS/JS callbacks, Python register("evt", fn), Java method references already partially covered) are still on the deferred list — each language grammar needs its own argument-position pattern. Tracked separately.

F4 (task-template-aware verify_change_readiness) remains deferred. The UX work needs task-shape pattern-matching infrastructure large enough to warrant its own ADR.

v1.12.0 follow-up — multi-language F1 closure + accuracy bench (2026-05-02)

After v1.11.x extended function-reference call-graph extraction to JS/TS, Python, Go, and Java/Kotlin (matching the Rust closure in v1.11.0), v1.12.0 adds a regression-gated accuracy bench so future refactors of the extractor cannot silently regress.

• Fixture set: benchmarks/call-graph-accuracy/fixtures/{rust,js,python,go,java}/ — one source file per language, mixing direct calls, method chains, macros / decorators, and the v1.11.x function-reference patterns.
• Ground truth: benchmarks/call-graph-accuracy/expected.json — declarative (caller, callee) set per fixture, plus an overall f1_threshold (0.85 in v1.12.0).
• Runner: crates/codelens-engine/tests/call_graph_accuracy.rs — an integration test, so cargo test -p codelens-engine already gates CI; a named CI step (call-graph accuracy bench) surfaces it as a discrete check with per-fixture diff in stdout.

• Initial measurement (v1.12.0 release):

  Fixture	TP	FP	FN	P	R	F1
  rust/registry.rs	5	2	0	0.714	1.000	0.833
  js/callbacks.js	6	1	1	0.857	0.857	0.857
  python/registry.py	5	0	0	1.000	1.000	1.000
  go/server.go	6	0	0	1.000	1.000	1.000
  java/Service.java	5	0	0	1.000	1.000	1.000
  OVERALL	27	3	1	0.900	0.964	0.931

Threshold 0.85, observed 0.931 — passing with headroom. The Rust / JS spurious edges are raw extract_calls artifacts that the runtime resolution cascade in resolve_call_edges filters out via the symbol DB; the bench intentionally evaluates the raw extractor surface so changes to the tree-sitter queries are caught earlier than the resolver's downstream filter.

Future bumps to f1_threshold happen as the extractor improves — the bench fixture set itself is stable, so ratchet-up edits to the threshold alone are reviewable in isolation.