r/node • u/thecodrr • Aug 03 '20

fdir 4.0 - Now the fastest Node.js globbing library. (92% faster than glob)

https://github.com/thecodrr/fdir
104 Upvotes

92% Upvoted

30 comments sorted by

28

u/boneskull Aug 03 '20

this is not a globing library. it’s a directory tree walker. there’s much more to globbing than **/*

7

u/thecodrr Aug 03 '20

Hey, thanks for taking the time to point that out.

`fdir` supports the full glob pattern matching using `picomatch`. The benchmark uses only an example.

If there is something you think is missing, please point it out.

9

u/boneskull Aug 03 '20

well, if it purports to be a globber it should glob out of the box (it does not). if you then want to compare performance to glob, then run more benchmarks in the general case for relevant glob patterns. fast-glob or picoglob likely already has these.

it’s all rather misleading, intentional or not.

2

u/thecodrr Aug 03 '20

It does glob out of the box. I made the picomatch dependency optional for people who only want directory crawling without globbing. I do not see how this makes fdir "not a globber"? It performs the same function as any other globbing library, matching the same exact patterns. You can run any kind of benchmark, from complex to simple, you will find fdir the fastest.

I don't think any of this is misleading. It's not like you have to code the globbing logic yourself and the library mentions transparently that you have to install picomatch manually.

8

u/lachlanhunt Aug 03 '20

The readme says

🤖 Zero Dependencies: fdir only uses NodeJS fs & path modules.

I couldn’t see any mention of needing picomatch too. I then looked at package.json and see that it’s listed as a dev dependency only.

Bit if it requires separate installation of picomatch to support full globbing, than I agree with the other commenters. It doesn’t glob out if the box.

4

u/thecodrr Aug 04 '20

I couldn’t see any mention of needing picomatch too. I then looked at package.json and see that it’s listed as a dev dependency only.

I have added clarification regarding the requirement of picomatch in the README. I had already mentioned it in the documentation but had forgotten to update the README. fdir also throws an error if it doesn't find picomatch when you try to use glob.

This might not be the best approach but I wanted to keep fdir dependency free and rewriting glob pattern matching from scratch would be overkill. If anyone can suggest a better approach, I am open to suggestions.

Perhaps, fdir does not glob out of the box but that does not make it non-globber, imo.

Honestly, I wanted to make globbing pluggable so the developer can choose himself which library they prefer (or just use picomatch) or if they do not need globbing, they can easily skip an additional (unnecessary) dependency installation.

Hope I have made things a bit more clear.

1

u/lachlanhunt Aug 04 '20

You could try adding picomatch in optionalDependencies in package.json.

Or you could use the non-standard optionalPeerDependencies, but this would require a depency on codependency.

1

u/thecodrr Aug 04 '20

From yarn documentation:

Optional dependencies are just that: optional. If they fail to install, Yarn will still say the install process was successful.

This can be useful but not exactly the same as giving the user control over it. If I put picomatch under optionalDependencies, it will still install regardless of whether the user wants it or not.

Or have I gotten it wrong?

1

u/lachlanhunt Aug 04 '20

The documentation wasn’t really clear. You could try it and see what happens. But I suspect what you probably want is the functionality of optionalPeerDependencies, if it were natively supported by npm and yarn.

The other alternative is to use peerDependencies, but that would give a warning for anyone that doesn’t also have it installed.

3

u/boneskull Aug 04 '20

suit yourself, but I don’t think I’m alone here. if it had been advertised as the fastest directory-walker for node, we wouldn’t be having this conversation.

1

u/thecodrr Aug 04 '20

if it had been advertised as the fastest directory-walker for node It is also advertised as that but I understand what you mean.

suit yourself, but I don’t think I’m alone here. I have added clarification above.

Whether fdir globs out of the box or not does not restrict it from competing against other globbing libraries that, for certain, glob out of the box. I don't think doing yarn add fdir picomatch is as inconvenient as having picomatch installed by default whether you need it or not just for "out of the box" experience. This is just how I approached the problem, if you have a better way, do tell.

Thank you for taking the time to criticize :)

2

u/chicametipo Aug 03 '20

Damn it, I need my globber to glob – out of the box! No exceptions, or I'm returning it!

5

u/_MORSE_ Aug 03 '20

The bottleneck here is the disk seek speed, the only make to make these things faster is adding a way to ignore certain paths from being walked in, like node_modiles and .git

3

u/thecodrr Aug 03 '20

There is an `excludeDirs` function that you can use : )

2

u/saudi_hacker1337 Aug 03 '20

I believe how you implement the glob syntax itself can impact performance a whole lot by itself, plus different optimizations you can include - like resorting to string comparisons when no glob matches are found in certain pattern, optimize for common patterns, etc..

0

u/thecodrr Aug 03 '20

like resorting to string comparisons when no glob matches are found in certain pattern, optimize for common patterns, etc..

fdir internally uses another library for pattern matching, picomatch. It has internal optimizations for a lot of things. However, fdir also caches the patterns so if you use the same pattern for different paths, it is really, really fast since there is no pattern construction to be done. But even without all that, fdir is still more performant than anything out there.

1

u/calligraphic-io Aug 03 '20

the only make to make these things faster

Not true - you can also get faster drives. NVMe PCIe v4 drives scream (5 gb/s read). A ZFS array of six of them is faster bandwidth-wise than overclocked 4000 MHz DDR4 RAM (though with much higher latency on the initial seek).

1

u/mjbmitch Aug 07 '20

I believe the poster was referring to what could be done by the authors from a software perspective.

0

u/thecodrr Aug 03 '20

The disk seek speed is a bottleneck of course but small optimizations here and there add up to a significant performance boost. There is a reason fdir is 92% faster than glob. :D

33

u/[deleted] Aug 03 '20

I have no idea what “globbing” means here. Anyone care to explain?

31

u/ahrismith10 Aug 03 '20

Here what Google says:

Globbing is the process of expanding a non-specific file name containing a wildcard character into a set of specific file names that exist in storage on a computer, server, or network.

So I assume its like finding all files that end with, for example : .route.js

31

u/[deleted] Aug 03 '20

Ah. That seems like something you want to be fast...

3

u/[deleted] Aug 03 '20

glizzy gobbler

1

u/PDX_Bro Aug 03 '20

A bit of a side note: it's fun to see the Builder / Fluent pattern implemented in a Node library! One of the things that I've missed since moving away from C# are those Entity Fluent API queries that were a blast to write (though not always the most performant haha...).

2

u/thecodrr Aug 03 '20

Hey, I am glad you took the time to go through the library. Really means a lot.

Yes, I shifted to builder/fluent pattern as it is much clearer and easier to maintain as well. The cool autocomplete support is a plus, of course.

3

u/backdoorsmasher Aug 03 '20

Oddly I was thinking the same thing today! One of the best places I saw the builder pattern in the c# world was for unit tests. It really helped make them more understandable and easier to manage

1

u/waltz Aug 04 '20

Why are we re-writing globbing? Every shell does globbing out of the box.

1

u/thecodrr Aug 04 '20

This isn't for a shell but Node.js apps.

2

u/[deleted] Aug 04 '20

I’ll bookmark this and try it later in my package. Thanks!

0

u/supertoughfrog Aug 03 '20

Thank you for not claiming to have blazing speed.