async search and index

  1. anonymous

    I am working with a really big project, with about 70 000+ files and CommandT feels really slow.

    Does command-t do async file search and indexing?

    Thanks!

  2. Greg Hurrell

    It's not doing any indexing, as such, beyond reading the file hierarchy the first time it is displayed and caching it in memory. I've spent quite a bit of time thinking about data structures that would allow me to trade some space for time, doing some up-front processing in order to deliver faster searches subsequently. To date, I haven't found anything viable.

    Back in ancient history, Command-T used to use a linear algorithm for scoring and matching. In mid-2010 it switched to an algorithm which is quadratic in the worst case, but tends to give much more intuitive results (see issue #1598).

    This still yields pretty good performance at around 20k files, but I am not surprised that it is starting to suffer at 70k+.

    We could add code that allowed you to fall back to the faster-but-less-accurate scoring method. That might make things more usable, although ironically, it's precisely on larger projects that you want the more intuitive matching to help you get to the files you want without having to perform too many keystrokes.

  3. anonymous

    Thanks for your quick reply. I understand it's reading the file hierarchy only - what I meant is: is that done in a separate thread? Would it be possible for the ruby script to have two threads - one that works with vim, updates UI and so on, and the other thread that receives requests for file lists, matching, sorting/scoring, etc. This way the UI would not block at all, we would have results "instantly". You could send smaller sets of results to the UI, in batches, as the file hierarchy reading goes further.

    I really like how fast Sublime Text 2 is - it works very well with 70k files.

    I did an investigation on my Linux machine: even a finely tuned |find| command that skips .hg/.git folders and .bak files and other binary file types... I still have to wait a couple of seconds before it's done. I don't see how Command-T could ever be "instant" without some kind of threading / async behavior.

    I have also tried Fuzzy Finder and CtrlP. Command-T is the nicest and most intuitive of all. Still, I'm forced to use FuF because it doesn't do deep folder hierarchy searching, and that makes it fast - with the cost of having to type more (folder names, not only file names).

    Keep up the good work!

  4. Greg Hurrell

    Right. Vim itself is single-threaded, so while it is possible to do threading in a plug-in (ie. something like Command-T, written in Ruby, or something in Python) it does feel a bit risky, as it may interact poorly with code in Vim which is operating under the assumption that it is the only thing running.

    I actually think the way to get Command-T to be responsive with that number of files isn't threading; it's just going back to the much-faster-but-not-quite-as-good linear scoring algorithm of old. A configuration option, perhaps.

Reply

This topic is now closed.

topic index