Tree-sitter

Tree-sitter is a parser generator + incremental parser runtime, originally built by Max Brunsfeld at GitHub to power semantic highlighting and code navigation. It's not a single parser — it's a framework that compiles language-specific grammar files into fast, error-tolerant parsers that produce concrete syntax trees (CSTs).

Each language gets its own grammar file (typically 1-5k lines of declarative grammar rules) which compiles to a C parser shared library. Tree-sitter ships with grammars for 40+ languages out of the box, with community contributions adding more.

The key innovations: incremental re-parsing (only re-parse the changed region of a file, not the whole thing) and error recovery (parser keeps going past syntax errors instead of bailing). Both matter for editors and code-analysis tools that operate on in-progress code.

Regex on source code is brittle. A regex for 'function declarations' catches some, misses others, and fires on string literals and comments that happen to look like function signatures. The maintenance burden grows with every edge case.

Language-specific parsers (using each compiler's own AST library) work but require integrating N different parsers for N languages. A multi-language code-review tool would need to embed and update parsers for TypeScript, Python, Go, Rust, Java, etc. — each with different update cadences and APIs.

Tree-sitter solves both: one library, one API, every language. The CSTs are uniform structurally, so you can write queries (using tree-sitter's S-expression query syntax) that work across languages with small adjustments.

Tree-sitter ships a Lisp-like query language for matching patterns in syntax trees. A query like `(function_declaration name: (identifier) @function-name)` matches every named function declaration in a file and binds the name to a `@function-name` capture group.

Multi-language code analysis tools write language-specific queries that map to language-agnostic concepts (functions, classes, imports, calls). Each language gets ~20-50 lines of query definitions. Adding a new language to a tree-sitter-based tool is therefore a day's work, not a week's.

The query engine is fast — capable of running thousands of queries per second on a large file. That's why editors use tree-sitter for live highlighting and code-folding.

Diff-only AI code review (sending the diff and only the diff to an LLM) misses context. The LLM doesn't see the function that's called from a changed line, the type definition that's referenced, the convention used elsewhere in the file.

Tree-sitter lets a review tool extract symbols (functions, classes, types) from the entire repo, build a symbol graph, and at review-time pull in just the relevant symbol bodies for the changed code. This 'context indexing' is what separates context-aware AI code review from naive diff-only review.

The performance characteristics work: parsing a 5,000-file monorepo takes about 50 seconds with tree-sitter; incremental re-parsing on a push of 10 changed files takes under a second.

LGTM's context indexer uses tree-sitter for 12 languages: TypeScript, JavaScript, Python, Go, Rust, Java, Ruby, C, C++, C#, Swift, Kotlin. When you connect a repo and click 'Index Codebase', we walk the repo with the appropriate grammar per file, extract symbols (names + signatures + line ranges, NOT source bodies), and build a graph of who-calls-whom and who-imports-what.

Incremental updates run on every push to default branch — only the changed files get re-parsed, only the affected graph edges get re-ranked. Index stays fresh without rebuilding.

The 'never store source bodies' guarantee comes directly from tree-sitter's separation of structure from content: we keep what tree-sitter captures (symbol names, signatures), not the function bodies themselves. Bodies get fetched from GitHub fresh at review time, held in worker memory for the LLM call, then discarded.