Benchmarks
The question that matters: does an agent standing on Canopy answer more correctly, and what does it cost? Measured cold, per question, graded against ground truth that exists independently of Canopy — never against Canopy's own output. Numbers marked "coming soon" have not been measured on the reference hardware yet; we will not publish guesses.
Correctness & cost — canopy vs shell tooling
Same agent, same model, same questions. One arm answers with its shell tools
(grep, cat, git); the other stands on a prebuilt Canopy index and makes a single
routed canopy_answer call per question. Every cell is
graded against repo / git / graph ground truth computed independently of Canopy.
| Dataset | Correctness (canopy vs shell) | Tokens / question | Cost per correct answer |
|---|---|---|---|
| Library suite 3 OSS repos · 270 questions · cold, per-question | 0.932 vs 0.530 (+76%) | 16,173 vs 51,495 (−69%) | ~5× cheaper per correct answer |
| App corpus Cal.com · 6,118 files · 39 cross-layer questions | 0.923 vs 0.872 | 16,253 vs 80,915 (−80%) | 81% cheaper per correct answer |
The lift is the single routed call, not tool access: a control arm that handed the agent Canopy's full v1 tool catalog and let it orchestrate spent 2.3×–5.8× more than shell. Routing is a deterministic, corpus-validated function of the question — 360/360 on its labelled regression corpus, no LLM call, no network.
Full method, per-question artifacts, and every number's provenance are in the
benchmark report shipped in the Canopy repository
(docs/benchmarks/canopy-benchmark-report.md).
Indexing
Wall-clock time from canopy index . invocation to "index complete" on a cold cache.
Includes tree-sitter AST parsing, Tantivy search index build, and SQLite write.
Reproduction
git clone <repo> /tmp/bench-repo
cd /tmp/bench-repo
time canopy index --full # cold: wipes and rebuilds this repo's index| Repository | Files | Time | Index size | Status |
|---|---|---|---|---|
| gulf-shield-technologies/platform (Pith monorepo) Internal reference — TypeScript/React Native monorepo | 622 | 2.6s (AST + search) | ~11 MB | validated |
| torvalds/linux (v6.6) Linux kernel — ~76K C source files | ~76,000 | coming soon | coming soon | coming soon |
| facebook/react (main) React source — TypeScript monorepo | ~6,800 | coming soon | coming soon | coming soon |
| microsoft/TypeScript (main) TypeScript compiler — TypeScript on TypeScript | ~3,200 | coming soon | coming soon | coming soon |
gulf-shield-technologies/platform (Pith monorepo): Validated 2026-04-15 on build host, measured on v1.3.0 (the indexing pipeline; v2 changed the answer surface, not the indexer core). 98.4% parse success rate (10 config files excluded by JS grammar version mismatch).
Search latency
Measured end-to-end from tool call dispatch to result return within a running
canopy serve process.
Does not include MCP transport overhead or LLM inference time.
Reproduction (keyword)
cd /tmp/bench-repo && canopy index
# then time the answer path end-to-end:
time canopy answer "where is the function \`main\` defined?"| Query type | p50 | p95 | p99 | Status |
|---|---|---|---|---|
| Keyword search | 29 ms | — | — | validated |
| Semantic (vector) search | — | — | — | coming soon |
| Hybrid (keyword + semantic) | — | — | — | coming soon |
Keyword p50 of 29 ms includes cold process start (Rust binary init + SQLite open + Tantivy load).
In a persistent canopy serve process, hot-path query time is sub-millisecond.
canopy_answer latency by recipe
End-to-end latency for one routed canopy_answer call,
by the recipe the router dispatches to. A recipe runs its whole chain server-side
(graph traversal, coverage, git history) — each call replaces work the agent would
otherwise do in 3–8 separate tool round-trips. The v1-era per-tool measurements
were retired with the tool catalog; we don't relabel old numbers as new ones.
| Tool | What it does | p50 | p95 | Status |
|---|---|---|---|---|
| canopy_answer — symbol locate | 'where is the function `X` defined?' — AST symbol lookup | — | — | coming soon |
| canopy_answer — dependents | 'what depends on `path`?' — import-graph blast radius | — | — | coming soon |
| canopy_answer — pre-edit dossier | 'what should I know before editing `path`?' — dependents + coverage + history | — | — | coming soon |
| canopy_answer — architecture | 'architecture overview' — directory + module map | — | — | coming soon |
| canopy_answer — concept (semantic) | 'how does X work?' — two-tier semantic retrieval + cross-encoder rerank | — | — | coming soon |
Contribute a benchmark
Have Canopy running on a large or interesting codebase? We would love to include your numbers here. Send results to support@gulfshieldtech.com with:
- Repo name and approximate file count (you can anonymize the repo name)
- Hardware spec (processor, RAM)
-
Output of
time canopy index --full -
Canopy version (
canopy --version)
We include all submitted benchmarks that are reproducible, regardless of whether they reflect well or poorly on Canopy. Honest data builds trust.
Run these benchmarks yourself on your own codebase.
Start 14-Day Free Trial