编程语言接入_add-lang

以下为本文档的中文说明

Add Lang 是一个专门用于为 CodeGraph 代码分析系统添加新编程语言支持的端到端自动化工具。它的工作流程完整覆盖了从语法接入到质量验证的全过程：首先将 tree-sitter 语法接入 CodeGraph 的提取管道，然后编写相应的测试用例，最后通过基准测试评估提取质量和检索价值。使用场景非常明确，每当需要为 CodeGraph 增加新的语言支持时触发，典型目标语言包括 Lua、Elixir、Zig、OCaml 等。用户可以通过运行 /add-lang 命令来启动整个过程。该技能的核心特点在于高度自动化和质量导向。整个流程完全自主运行，自动选择测试仓库、运行基准测试、更新文档并生成报告，但严格遵守不提交、不推送、不发布的原则，所有变更留待用户审阅。设计原则强调数据驱动的验证，必须证明新语言支持在提取真实符号方面优于不使用 CodeGraph 的基准线，确保每次添加都有明确的价值提升。对于维护代码分析工具的团队来说，这显著降低了语言扩展的技术门槛，使非专业人员也能为系统添加新的语言支持。该技能的价值不仅体现在直接的功能实现上，还在于它与现有技术生态系统的良好兼容性和集成能力。无论是作为独立工具使用，还是嵌入到更大的工作流中，它都能发

Add a language to CodeGraph

Wire a new tree-sitter language into codegraph’s extraction pipeline, prove it
extracts real symbols on popular repos, and prove it beats no-codegraph for an
agent. Runsfully autonomously— pick repos, benchmark, update docs, then
report.Never commit, push, publish, or tag(house rule); leave all changes
for the user to review.

The argument is the language token used throughout theLanguageunion, e.g.
lua,elixir,zig. If none was given, ask which language. Use the lowercase
single-token form everywhere (csharp, notc#).

Prerequisites

• Run from the codegraph repo root.node,git,gh, and a logged-in
  claudeCLI (the benchmark spawns realclaude -pruns).
• The benchmark uses the local dev build — Step 8 builds + links it on PATH.

Workflow

Copy this checklist and work through it in order:

- [ ] 1. Resolve language; bail early if already supported (just benchmark) - [ ] 2. Find a grammar + health-check it (ABI / heap corruption) - [ ] 3. Discover the grammar's AST node types (dump-ast.mjs) - [ ] 4. Wire the language (4 files; sometimes a 5th core touch) - [ ] 5. Build + verify-extraction loop until PASS - [ ] 6. Add extraction tests; make them green - [ ] 7. Auto-pick 3 popular repos by size tier; add to corpus.json - [ ] 8. Benchmark all 3: extraction + with/without A/B - [ ] 9. Update README + CHANGELOG - [ ] 10. Report; do NOT commit

Step 1 — Resolve + short-circuit

Check whether the language is already wired: look for the token in the
LANGUAGESconst (src/types.ts) and theEXTRACTORSmap
(src/extraction/languages/index.ts). If it is already supported (e.g.
typescript,rust),skip Steps 2–6and go straight to benchmarking
(Steps 7–8) to validate/measure it — note in the report that no code changed.

Step 2 — Find a grammar, then health-check it

lsnode_modules/tree-sitter-wasms/out/|grep-i<lang># csharp -> c_sharp

• Present→ likely off-the-shelf;grammars.tsresolves it from
  tree-sitter-wasmsautomatically. (Many languages: elixir, zig, ocaml,
  solidity, toml, yaml, …)
• Absent→ vendor a.wasmintosrc/extraction/wasm/(likepascal/
  scala/lua) and add the token to the vendored branch in Step 4.

Always health-check before writing an extractor — apresentgrammar can
still be unusable:

nodescripts/add-lang/check-grammar.mjs<lang>path/to/valid-sample.<ext>

It prints the grammar’s ABI version and parses a valid sample many times in a
multi-grammar runtime. If itFAILs(ERROR trees on valid code — an old ABI
corrupting the shared WASM heap, which silently drops nested calls/imports on
every file after the first; e.g. the tree-sitter-wasmsLuagrammar is ABI 13
and fails), do NOT use that wasm.Vendor a newer (ABI 14/15) build instead:

npmpack @tree-sitter-grammars/tree-sitter-<lang># often ships a prebuilt *.wasm# or build one: npx tree-sitter build --wasm (needs Docker/emscripten)cp<the>.wasm src/extraction/wasm/tree-sitter-<lang>.wasm

then add the token to the vendored branch in Step 4 and re-run check-grammar on
the vendored path until it PASSes.If you cannot obtain a healthy wasm, STOP
and tell the user.

Step 3 — Discover AST node types

Get a representative source file (write a small sample covering functions,
classes/structs, imports, enums; orcurla raw file from a known repo), then:

nodescripts/add-lang/dump-ast.mjs<lang>path/to/sample.<ext># vendored grammar: pass the wasm path instead of the tokennodescripts/add-lang/dump-ast.mjs src/extraction/wasm/tree-sitter-<lang>.wasm sample.<ext>

The frequency table + field names (name:,parameters:,body:,
return_type:) tell you what to map. Open the existing extractor closest to the
language’s paradigm as a model:rust.ts/scala.ts(functional, traits),
java.ts/csharp.ts(OO),python.ts/ruby.ts(scripting),go.ts
(top-level methods + receivers).

Step 4 — Wire the language (4 f

iles)

These are exact, fragile wiring — match the existing style precisely:

1. src/types.ts— TWO edits:
   • add'<lang>',to theLANGUAGESconst (before'unknown');
   • add'**/*.<ext>',toDEFAULT_CONFIG.include.Don’t skip this— it’s
     the file-scan allowlist; without the glob,codegraph initfinds0
     fileseven though detection/extraction are wired.
2. src/extraction/grammars.ts— three maps:
   • WASM_GRAMMAR_FILES:<lang>: 'tree-sitter-<lang>.wasm',
   • EXTENSION_MAP: each file extension →'<lang>'(e.g.'.lua': 'lua',)
   • getLanguageDisplayName:<lang>: '<Display Name>',
   • vendored only: add<lang>to the
     (lang === 'pascal' || lang === 'scala' || …)wasm-path branch.
3. src/extraction/languages/<lang>.ts— new file exporting
   export const <lang>Extractor: LanguageExtractor = { … }. Map the node types
   from Step 3. Required fields:functionTypes,classTypes,methodTypes,
   interfaceTypes,structTypes,enumTypes,typeAliasTypes,
   importTypes,callTypes,variableTypes,nameField,bodyField,
   paramsField. Add hooks as the grammar needs them (getSignature,
   getVisibility,isExported,extractImport,visitNode,getReceiverType,
   interfaceKind,enumMemberTypes, etc. — see
   src/extraction/tree-sitter-types.ts).
4. src/extraction/languages/index.ts—import { <lang>Extractor } from './<lang>';and add<lang>: <lang>Extractor,toEXTRACTORS.

Sometimes a 5th, core touch insrc/extraction/tree-sitter.ts— variable
extraction has per-language branches inextractVariable(the generic fallback
only finds directidentifier/variable_declaratorchildren). If the grammar
nests declared names (e.g. Lua’svariable_declaration → variable_list), add a
} else if (this.language === '<lang>')branch there, mirroring the existing
ts/python/go ones. Import forms that aren’t a distinct node (Lua/Rubyrequire
is acall) are handled in the extractor’svisitNodehook instead.

Step 5 — Build + verify loop

npmrun build# tsc + copy-assets (copies any vendored *.wasm into dist/)

Index a small sample repo and check extraction:

(cd<sample-repo>&&codegraph init-i)nodescripts/add-lang/verify-extraction.mjs<sample-repo><lang>

verify-extraction.mjsfails (exit 1) if the language isn’t detected or only
file/importnodes were produced — the classic symptom of wrong node-type
names. On FAIL or a thin WARN: re-rundump-ast.mjson a richer file, fix the
mappings in<lang>.ts,npm run build, re-index, re-verify.Repeat until
PASS.

Step 6 — Tests

Add to__tests__/extraction.test.ts, modeled on theRust Extractionblock:

• adetectLanguageassertion indescribe('Language Detection')
• adescribe('<Lang> Extraction')block asserting functions/classes/imports
  are extracted from an inline source string.

npx vitest run __tests__/extraction.test.ts

Green before continuing.

Step 7 — Auto-pick 3 repos + corpus

Pickwithout asking. Find candidates, then curate 3 that are genuinely
<lang>-dominant, one per size tier:

gh search repos--language=<lang>--sort=stars--limit40\\--jsonfullName,stargazerCount,description

Tiers (matchcorpus.json):Small<~150 files ·Medium~150–1500 ·
Large>~1500. Skip repos that are tagged<lang>but mostly another
language. Write one cross-file architecturequestionper repo (the kind that
needs tracing across files). Add a"<Language>"block to
.claude/skills/agent-eval/corpus.json(fields:name,repo,size,
files,question) so/agent-evalcan reuse them.

Step 8 — Benchmark all 3 (extraction + A/B)

Make the dev build the codegraph on PATHonce, then loop:

npmrun build&&./scripts/local-install.sh scripts/add-lang/bench.sh<lang><name><url>"<question>"headless# ×3

bench.shclones (shared/tmp/codegraph-corpus), wipes + indexes, r
uns
verify-extraction.mjs, then the with/without retrieval A/B via
scripts/agent-eval/run-all.sh(skips the paid A/B if extraction is broken).
Read eachparse-run.mjssummary printed byrun-all.sh: tool calls, file
Reads, Grep/Bash, codegraph-tool calls, duration, andcost— for both the
withandwithoutarms. After the loop, restore the dev link if needed:
./scripts/local-install.sh.

Step 9 — Docs + CHANGELOG

• README.md: add<Lang>to the “19+ Languages” feature bullet, and add a
  row to theSupported Languagestable:
  | <Lang> | \\.ext\| Full support (classes, methods, …) |.
• CHANGELOG.md: add an## [Unreleased]section at the top (above the
  latest version) with### Added→ a user-perspective bullet, e.g.
  “CodeGraph now indexes (.ext) — functions, classes, imports, and
  call edges.”If## [Unreleased]already exists, append under it. (It’s
  folded into the next versioned block at release time.)

Step 10 — Report (do NOT commit)

Summarize for review:

• Files changed: the 4 wiring edits + new extractor + tests + README +
  CHANGELOG + corpus.json (+ any vendored.wasm).
• Extractionper repo: files / nodes / edges /verify-extractionresult.
• A/Bper repo:withvswithout(tool calls, file Reads, cost) and a
  one-line verdict — did codegraph reduce effort, and did both arms reach a
  correct answer?
• Gaps / follow-ups(node types not yet mapped, resolution edges missing,
  framework routes, etc.).

Hand the changes to the user.Do notrungit commit/pushor publish —
releases go through the GitHub Actions Release workflow.

Notes

• The A/B spawns realpaidclaude -pruns (opus,--max-budget-usd),
  2 arms × 3 repos. The corpus dir/tmp/codegraph-corpusis shared with
  /agent-eval, so clones are reused across runs.
• Any new*.wasmmust live insrc/extraction/wasm/—copy-assets(run by
  npm run build) ships it; otherwise it won’t be indist/.
• An index must be served by thesamebinary that built it. Step 8 builds +
  links the dev build first, so this holds.
• If a grammar can’t be obtained, or extraction can’t reach PASS,STOP and
  report— don’t ship a half-wired language.