Improve performance of finding indexables

[natematykiewicz]

Currently, all folders and files in the current tree are turned into IndexablePath, and then excluded files are filtered out after.

When there are large file trees that are meant to be excluded, this results in a lot of unnecessary work.

ActiveStorage stores files in the tmp directory in many many small folders. So does Bootsnap. Ruby LSP has to traverse all of these files, even though the entire directory should just be ignored.

Rubocop has solved this by breaking the includes patterns up into many patterns, applying the exclusions before the Dir.glob, so I followed in their footsteps. This works great for exclusions that end in "**/*". We still need to loop through all IndexablePath objects and see if they're excluded, in the case that an extension was provided on the excluded path, but this can cut down load time dramatically.

Before this PR in my Rails app, indexables took 76 seconds to run. Now it takes, 0.19 seconds. Before and after code both return the same exact file list.

Additionally, I added node_modules to the list of excluded trees, since that can be very large and never includes Ruby files.

I also removed the *.rb from the bundler path. Having a file extension on that means we need to scan all files. But we simply want to ignore the entire bundler path tree. I kind of wonder if we should always replace *.rb with *, to help people improve performance. Excluding an actual file name or partial file name makes sense. But excluding the only file extension we scan means we can do it faster by excluding the whole folder.

Motivation

Opening a ruby file caused my LSP server to print "Ruby LSP: indexing files" for 76 seconds at 0% before the progress bar starts moving.

Implementation

I knew that Rubocop has solved this problem before, so I looked at this file and followed what they did.

Automated Tests

I added tests for the new pattern exclude_pattern that gets used with fnmatch, while ensuring I didn't break any existing tests.

Manual Tests

I made a file that has both implementations of indexables on my computer. Then ran this in the Rails Console:

Benchmark.measure { RubyIndexer::Configuration.new.indexables }
=>
#<Benchmark::Tms:0x000000012a2dead0
 @cstime=0.0,
 @cutime=0.0,
 @label="",
 @real=0.19083199999295175,
 @stime=0.0784180000000001,
 @total=0.18055299999999974,
 @utime=0.10213499999999964>


Benchmark.measure { RubyIndexer::Configuration.new.old_indexables }
=>
#<Benchmark::Tms:0x000000012a374b20
 @cstime=0.0,
 @cutime=0.0,
 @label="",
 @real=76.83584400010295,
 @stime=13.666566,
 @total=14.943883,
 @utime=1.277317>

old_indexables = RubyIndexer::Configuration.new.old_indexables
indexables = RubyIndexer::Configuration.new.indexables

indexables.size == old_indexables.size
=> true

indexables.map(&:full_path).sort == old_indexables.map(&:full_path).sort
=> true

indexables.size
=> 13933

So here you can see that it finds the same ~14k files in 0.25% of the time.

[natematykiewicz]

I just resolved the merge conflict

[vinistock]

Great improvements 🚀

[natematykiewicz]

@vinistock I believe I've addressed all of your comments

[natematykiewicz]

@vinistock do you need anything else from me on this?

[vinistock]

This code is assuming that every pattern provided by the user is inside Dir.pwd, but that may not be true and I don't think we should tie ourselves to that yet.

I think we would need to change the check here to verify that the pattern both ends with wildcards and starts with Dir.pwd or **.

Suggested change

	relative_patterns = @excluded_patterns
	.select { |p| p.end_with?("/**/*") }
	.map { |p| p.delete_prefix("#{Dir.pwd}/") }
	relative_patterns = @excluded_patterns
	.select { |p| p.end_with?("/**/*") && (p.start_with?(Dir.pwd) || p.start_with?("**")) }
	.each { |p| p.delete_prefix!("#{Dir.pwd}/") }

[natematykiewicz]

That makes sense, though I don't think we want the ! on delete_prefix. We're allocating a new array, but the strings are still the ones in @excluded_patterns, which means we'd be mutating the content of @excluded_patterns.

[natematykiewicz]

I added that change to the select line, but kept what I had on the map line, since I don't think we want to mutate @excluded_patterns' strings. Let me know if you think otherwise.
3a8c7fe

[natematykiewicz]

Just to clarify, included_patterns should be relative to Dir.pwd, right? apply_config doesn't automatically prefix it with Dir.pwd, and I don't see how users would realistically set Dir.pwd in the YAML, considering they'd all have the repo checked out to different folders. Feels like we need to be prepending Dir.pwd to every entry in included_patterns. It also feels like excluded_patterns should similarly be expected to be within Dir.pwd. I'm struggling to understand what files someone would be including / excluding that aren't in a repo or a gem. But from what I can tell, excluded_patterns refers solely to application code, not gems. You exclude entire gems, not paths within a gem.

Right now I'm feeling like include_patterns doesn't even work, unless the user provides an absolute path to the folder on disk, which isn't very portable. What's the use-case for this? Understanding will help me figure out how to best optimize this, because the goal is to avoid calling Dir.glob on folders that are excluded, which means we need to take the included patterns, apply the exclude patterns that end with wildcards against them (essentially making include_patterns a larger list of folders), to cut down on which folders Dir.glob scans.

[vinistock]

That's good point. Now that I think of it, all patterns have to be relative to something, since you can clone a project in any directory.

I'll bring this up with the team, but I think we should indeed make everything relative to Dir.pwd.

[natematykiewicz]

Okay great, thanks!

[natematykiewicz]

If we're including File.join(Dir.pwd, "**", "*.rb") by default, all I can really think that people might be including is non-.rb ruby files, like a Gemfile or Guardfile. So that leaves me really wondering what people even use include_patterns for. When people specify include_patterns, it doesn't seem to remove the default pattern. So anything they're including is likely already in the existing pattern, unless it doesn't end with .rb or is outside of Dir.pwd.

[natematykiewicz]

Rubocop's Include config option is relative to Dir.pwd. And it just is meant for adding things like Gemfile or config.ru.

https://docs.rubocop.org/rubocop/configuration.html#unusual-files-that-would-not-be-included-by-default

This says that Rubocop by default:

Finds all Ruby source files under the current or other supplied directory. A Ruby source file is defined as a file with the .rb extension or a file with no extension that has a ruby shebang line as its first line.

https://github.com/rubocop/rubocop/blob/aee64e2d5d6615ef64803af52c1b82bf95b566ef/lib/rubocop/target_finder.rb#L33-L35

It accepts a directory because you can run the Rubocop CLI against a path relative to Dir.pwd (for example, just linting your models).

[natematykiewicz]

Hey @vinistock, where'd you guys land on this? Should I assume that all paths are relative to Dir.pwd? If so, should I update apply_config, so that include_patterns are prefixed with Dir.pwd?

[vinistock]

This method is quite complex. Can we add some comments explaining the steps that this takes and include an example of a pattern before and after?

[natematykiewicz]

I'm realizing I've totally forgotten about @included_patterns and just treated it as hardcoded.

[vinistock]

Indeed. We accept users defining extra glob patterns to index.