CodeLens MCP Product Readiness

This document is the product contract for moving CodeLens MCP from a strong local developer tool into a production-grade host-adaptive semantic context router.

It is deliberately evidence-first. Marketing claims belong in README.md only after the corresponding row here has a repeatable gate.

Product Definition

CodeLens MCP is a Rust MCP server that gives coding agents a bounded, auditable code-intelligence layer. The product-grade target is not "semantic search exists." The target is:

Codex, Claude Code, and generic MCP clients can attach to the same repository, discover the current host/tool/skill/memory/harness constraints, diagnose semantic-index readiness, retrieve the smallest useful code context, and recover from stale or mismatched indexes without guessing.

Execution Meta-Prompt

Use this prompt when starting a product-grade implementation or review pass:

You are the senior engineer responsible for making CodeLens MCP production-grade.

Mission:
Turn CodeLens MCP into a Rust host-adaptive semantic context router for Codex,
Claude Code, and generic MCP clients. The product must reduce wasted agent
tokens by detecting the host environment, skill roots, memory roots, MCP tool
surface, semantic-index readiness, and safe retrieval path before asking the
agent to read broad files.

Operating rules:
1. Start from the real local repo, current git state, host config, daemon
   status, and command output. Do not trust docs over runtime evidence.
2. Preserve unrelated dirty WIP. Modify only the smallest file set needed for
   the current product gate.
3. Prefer product evidence over architecture prose: install, doctor, index,
   retrieve, adapt, recover, benchmark, release.
4. Treat semantic visibility as P0. A user must be able to see compiled,
   model_assets, indexed_symbols, readiness_percent, stale_files, stale file
   reasons, model_mismatch, query_cache, and last_index_sha without guessing.
5. Treat retrieval quality as a measured product claim. Split benchmark results
   by identifier, short_phrase, natural_language, and issue_to_edit before
   claiming improvement.
6. Treat host adaptation as compatibility work, not prompt decoration. Codex,
   Claude Code, and generic MCP fixtures must prove graceful behavior with
   different tools, skills, memory, permissions, and harness settings.
7. Remove AI-shaped bloat. Avoid single-implementation wrappers, redundant
   verification, oversized files, and abstractions that do not reduce real
   complexity.
8. Write transient benchmark/model/cache artifacts only to /tmp or ignored
   runtime paths.

Loop:
1. Inspect the current product contract and runtime state.
2. Pick the highest-risk failing gate.
3. Implement the smallest root-cause change.
4. Run the narrowest relevant tests first, then the documented product gates.
5. Report evidence, benchmark numbers, token-cost implications, remaining gaps,
   and dirty-worktree classification.

Done means:
No product-grade claim is accepted unless the matching command, test, benchmark,
or release smoke gate passed in this repo during the current run.

Token Economics Bar

This product is only worth the extra indexing and doctor cost if it lowers session waste. Evaluate retrieval work against these questions:

Source Of Truth

Runtime resources and command output override prose. If a doc says semantic is ready but embedding_coverage_report says the model is missing, the report wins.

Boundary Alignment

Product Contract

Current Readiness Matrix

Host-Adaptive Flow

flowchart TD
  A["Host client"] --> B["prepare_harness_session(project, host_context, task_overlay)"]
  B --> C["Host profile snapshot"]
  C --> C1["MCP/tool surface"]
  C --> C2["Skill roots"]
  C --> C3["Memory roots"]
  C --> C4["Harness/profile policy"]
  B --> D["Semantic/index health"]
  D --> D1["compiled/model assets"]
  D --> D2["indexed symbols/stale files"]
  D --> D3["model/git/schema mismatch"]
  B --> E["Profile-scoped tool surface"]
  E --> F["Bounded retrieval or gated mutation"]
  F --> G["Benchmark/doctor/release evidence"]

Non-Negotiable Gates

Local development can run a smaller slice first, but a product-grade claim needs the full evidence set below:

cargo fmt --all -- --check
cargo check -p codelens-mcp --features semantic --quiet
cargo clippy -p codelens-mcp --features semantic -- -D warnings
cargo test -p codelens-engine --features semantic --quiet
cargo test -p codelens-mcp --features semantic --quiet
python3 scripts/test/test-smoke-embedding-coverage.py
python3 scripts/test/test-embedding-index-lifecycle.py
python3 scripts/test/test-regen-tool-defs-drift.py
python3 scripts/test/test-surface-manifest-contracts.py
python3 scripts/surface-manifest.py --check
python3 -m py_compile benchmarks/embedding-quality.py benchmarks/external-project-smoke.py benchmarks/embedding-index-lifecycle.py benchmarks/embedding_index_lifecycle_lib.py scripts/smoke-embedding-coverage.py scripts/test/test-smoke-embedding-coverage.py scripts/test/test-embedding-index-lifecycle.py
python3 scripts/smoke-embedding-coverage.py --binary target/debug/codelens-mcp --project .
python3 benchmarks/embedding-index-lifecycle.py . --binary target/debug/codelens-mcp --output /tmp/codelens-index-lifecycle.json
python3 benchmarks/external-project-smoke.py --binary target/debug/codelens-mcp --matrix benchmarks/external-project-smoke-matrix.json --output /tmp/codelens-external-project-smoke.json --check
git diff --check

Retrieval/ranker promotion additionally requires:

Fast ranker iteration should first run the hybrid lane only. This keeps root cause work observable without paying for every comparator subprocess on each loop:

python3 benchmarks/embedding-quality.py . \
  --binary target/debug/codelens-mcp \
  --methods get_ranked_context \
  --workers 4 \
  --batch-size 16 \
  --query-cache-probe off \
  --output /tmp/codelens-embedding-quality-hybrid-only.json \
  --stdout summary \
  --markdown-output /tmp/codelens-embedding-quality-hybrid-only.md \
  --triage-output /tmp/codelens-embedding-quality-hybrid-only-triage.json \
  --check \
  --min-hybrid-mrr 0.70 \
  --max-hybrid-candidate-missing-rate 0.10 \
  --max-hybrid-p95-response-tokens 20000

Promotion still requires the full default method set with the cache probe enabled so BM25/lexical, semantic, hybrid, and symbol-search comparators are measured together. Keep batching enabled so the promotion gate measures ranker quality without paying per-row process startup cost, and use method workers for the independent comparator lanes while preserving deterministic output order. Batch latency is reported separately as amortized batch latency; per-query latency ceilings such as --max-hybrid-avg-ms must be run with --batch-size 1:

python3 benchmarks/embedding-quality.py . \
  --binary target/debug/codelens-mcp \
  --method-workers 4 \
  --batch-size 16 \
  --query-cache-probe on \
  --output /tmp/codelens-embedding-quality-results.json \
  --stdout summary \
  --markdown-output /tmp/codelens-embedding-quality-summary.md \
  --triage-output /tmp/codelens-embedding-quality-triage.json \
  --check \
  --min-hybrid-mrr 0.70 \
  --min-lexical-mrr 0.50 \
  --min-hybrid-mrr-by-query-type natural_language=0.55 \
  --min-hybrid-recall-by-query-type issue_to_edit=0.80 \
  --max-hybrid-candidate-missing-rate 0.10 \
  --max-hybrid-p95-response-tokens 20000

Known Readiness Gaps

1. The current local ranker gate passes, but public benchmark prose still contains historical promoted baselines. Treat historical benchmark files as release-history claims, not proof for future ranker changes.
2. The current self dataset has 112 rows. Any doc or release note that claims a live 96-row or 104-row self dataset must be updated or explicitly labeled as a historical snapshot.
3. Built-in doctor/status --strict now carries HTTP-daemon semantic coverage and remediation and exits non-zero when attached semantic coverage is not ready or cannot be verified. It intentionally allows unconfigured hosts in --all reports so unsupported clients do not fail the whole aggregate check.
4. Host adaptation now has Codex, Claude Code, generic MCP fixtures, Codex/Claude/generic HTTP initialize snapshot coverage for MCP server/tool inventories, managed settings, and memory roots, plus representative doctor/status fixtures for unconfigured, malformed config, unreachable HTTP, stale HTTP, stdio-only hosts, and Cline/Windsurf-specific config paths. Memory-root entrypoint selection now returns bounded summary/registry/policy/skill-directory hints instead of forcing broad memory scans, and malformed file/missing roots degrade to zero entrypoints.
5. Token cost is measured mostly as response size. The product claim should add repeated-query/cache-hit and retry-avoidance evidence before claiming net session-token savings.
6. Productivity evidence now has a repeatable local loop, but the current local sample size is still small. Treat early runs as instrumentation proof until multiple real task sessions accumulate.

Overfitting Guardrails

The 112-row self dataset is a regression detector for this repository, not a generalization proof. The latest expansion deliberately added host-adaptation, coordination, report-job, and coverage/remediation queries that lowered hybrid MRR from the prior 96-row run (0.807) to 0.758; that drop is acceptable because it makes the self gate harder instead of inflating the headline.

Use the self gate this way:

• Passing self retrieval proves that current CodeLens concepts remain findable in this repository.
• candidate_missing, semantic_hit_dropped_by_hybrid, and hybrid_demoted_semantic_hit rows are diagnostic backlog, not reasons to tune the ranker to this dataset alone.
• Public retrieval claims require external project smoke or release-quality matrix evidence in addition to the self gate.
• The fast external-project smoke must assert expected search hits, not only successful tool exit codes.
• External smoke labels must validate the semantic target for the query; they must not be blindly copied from the rename dry-run target when the two workflows are testing different behavior.
• Default-on ranker changes must not be justified by self MRR alone, especially when the added rows use repository-specific vocabulary such as host profiles, coordination claims, or CodeLens tool names.

Overfitting analysis for the current 112-row gate:

Latest Local Operator Evidence

On 2026-07-06, python3 scripts/smoke-embedding-coverage.py --binary target/debug/codelens-mcp --project . passed with:

historical clean-index snapshot:
status=ready, compiled=True, model_assets.available=True,
model_assets.sha256=ef1d1e9cfa72,
indexed_symbols=11630, readiness_percent=100%, stale_files=0,
model_mismatch=False, remediation.action=none, query_cache.entries=164,
last_index_sha=aa49a9f4c79a88aca18497c29dee4161caa59862

After the current WIP changed benchmark/retrieval files, the same smoke first reported stale semantic coverage instead of pretending the old index was current:

status=stale, compiled=True, model_assets.available=True,
model_assets.sha256=ef1d1e9cfa72,
indexed_symbols=11672, readiness_percent=98%, stale_files=3,
stale_reason=benchmarks/external-project-smoke.py:embedding_keys_changed,
model_mismatch=False, remediation.action=refresh_embedding_index,
query_cache.entries=188,
last_index_sha=aa49a9f4c79a88aca18497c29dee4161caa59862

Running the documented recovery command refreshed the index for the current WIP:

index_embeddings: indexed_symbols=11687, bridges_generated=783,
query_cache.entries=188, status=ok

embedding_coverage_report:
status=ready, compiled=True, model_assets.available=True,
model_assets.sha256=ef1d1e9cfa72,
indexed_symbols=11687, readiness_percent=100%, stale_files=0,
stale_reason=none, model_mismatch=False, remediation.action=none,
query_cache.entries=188,
last_index_sha=aa49a9f4c79a88aca18497c29dee4161caa59862

The same day, bash scripts/mcp-doctor.sh . --strict passed for attached claude-code, codex, and cursor HTTP daemon configurations:

Summary: attached=3, unconfigured=2, issues=0
Strict verdict: PASS

Current reports include model_assets.sha256, readiness_percent, ready_files, bounded stale_file_reasons entries such as {"file_path":"src/main.rs","reason":"embedding_keys_changed"} so the operator can see the first stale-file causes without dumping every indexed file into the agent context, plus index.freshness.{schema,model,git,files} and remediation.action so model mismatch, schema drift, git SHA drift, and file-level index stale conditions do not collapse into one generic recovery hint.

When coverage is stale, the intended recovery command remains:

CODELENS_MODEL_DIR=crates/codelens-engine/models \
  /Users/bagjaeseog/.local/bin/codelens-mcp . --cmd index_embeddings

Built-in target/debug/codelens-mcp status --json --strict codex now exposes the same semantic coverage summary in hosts[].semantic_coverage and fails closed when that coverage is stale, unreachable, malformed, or unverified for an attached host. This is the desired operator loop: detect stale semantic coverage, run the single recovery command, and prove readiness with a doctor gate.

Next Implementation Order

1. Collect cross-machine lifecycle baselines and set conservative cold/warm threshold guidance without blocking release on noisy runner latency.
2. Expand ranker benchmarks to 300-500 labeled rows and reduce p95 response tokens; the self-retrieval quality and triage artifacts are now attached to upstream/release smoke evidence.
3. Run scripts/public_release_channel_smoke.py --version X.Y.Z --mode installer and --mode homebrew-info against the next published tag, then attach the generated transcript to release evidence.