How AST-grep Rewrote Tree-sitter in Rust and Made It 30% Faster

Made a tree-sitter powered text editor. Implementing incremental parsing improved performance from 5ms to 400us for small edits in a ~100LOC file. For many applications tree-sitter without incremental parsing makes no sense, but I guess it's not a problem for ast-grep.

The article explicitly says incremental parsing was not needed for this application (ingesting whole files for AI analysis).

> isn’t the entire point of tree-sitter that you can reuse old trees to make edit updates faster?

I've used tree-sitter for a few projects, but I never touched the tree-reuse stuff. The value of tree-sitter came from the fact that grammars compile to a .so and can be easily distributed through package managers. Adding support for a new grammar in ast-grep might just mean pointing to the .so at runtime (idk, I haven't checked). So this optimization might make sense for this project.

> isn’t the entire point of tree-sitter that you can reuse old trees to make edit updates faster?

No, not for ast-grep.

I've seen "load bearing" so many times in AI now I can't remember whether we always used "load bearing wall" to refer to a structural interior wall, or whether it's a recent AI-ism.

Pretty disappointing when you hit ⌘A, right-click, "Check for AI Content" and Pangram comes back with 100% AI... and on every section.

I like ast-grep and I think the changes are interesting, but it isn't fun reading an ai-generated article.

It seems like a lot of the speedup comes from using an arena, using realloc with offsets rather than raw pointers and some change to the graph that enabled this (though I don't want to re-read it to confirm).