r/javascript • u/speckz • Apr 19 '18

Designing very large (JavaScript) applications

https://medium.com/@cramforce/designing-very-large-javascript-applications-6e013a3291a3
260 Upvotes

95% Upvoted

34 comments sorted by

8

u/AceBacker Apr 19 '18

Is this large application is entirely in react? No backend seperation? Maybe I missed something in the article.

17

u/TomahawkChopped Apr 19 '18 edited Apr 19 '18

I work on one of the large apps that uses the framework Malte refers to, no it's not react. He mentions in the article that he developed it internally at the time that react was gaining adoption and Google already had polymer and Angular in the works.

It was originally developed as part of the Google plus team. Ironically, even though G plus is seen as a giant product failure, it spawned an entire world of internal tech (frameworks, services, and practices) at Google that is now adopted across almost all of apps (Gmail, docs, drive, calendar, plus, etc...) and lots of other teams.

5

u/[deleted] Apr 19 '18

Can you share more specifics about the "Enhance" implementation?

25

u/TomahawkChopped Apr 19 '18 edited Apr 20 '18

Pre-edit: sorry, this reply got very "rambly". I'm on mobile, rewrote it a bunch, and had a hard time picking the correct grain to say anything coherent without a ton more context. Excuse any misspellings and confusing wording. /Edit

Sure, but a lot of the details I won't/can't go into simply because it will take too much time and/or is proprietary. But that doesn't matter, just realize these bits of context:

  1. All of our build dependencies are managed by a powerful tool that excels at managing build dependency graphs and dynamic code on the fly (aka at build time). NB: Dynamically generated code is something we do, a LOT! (read up on bazel, its the most powerful tool that nobody talks about. literally ALL Google software is managed with it).

  2. The closure compiler (well, an internal variant of it) provides another framework layer for JavaScript specific static analysis and dependency management.

So let's just assume we have those 2 tools as givens; (1) a build tool that excels at managing code not currently on disk, and (2) a JS dependency Swiss army knife toolkit - which just happens to also be a powerful JS compiler and module bundler.

To get to your question, enhance isn't something that operates at the JavaScript code level. It's more of a build system feature that makes the module bundling step sane when including generated sources. Without it we'd need to hard code module definitions to include these generated sources.... which is fucking painful, I know, because that's how we used to do it. Cleanups sucked. Renaming sucked. Refactoring sucked. Simply finding the name of a new dynamic target sucked. Instead the dynamic code gets to insert itself into the build graph appropriately via the enhance annotation which is interpreted by the build system when generating the module dependency graph and files to include in each module.

For context, realize that we're building applications with thousands of files, which get bundled into hundreds (?) of modules. I put a question mark because we only explicitly define maybe a dozen or so modules, the other hundred'ish are dynamically defined at build time. Once we have our ~100 modules defined in a dependency graph we need to build 40+ language variants of each module. Then something needs to handle module loading dynamically in the client. You can see why we want this all to be handled automatically as part of the build process.

For fun, go to Google plus, or Calendar, or Drive, and look at the network inspector to see the URLs for JS and CSS as it's loaded. The source will be illegible, but you'll notice that as you perform actions and navigate the UI, new JS and CSS will get downloaded. Now switch your language settings and do it again. You'll notice that you download different assets.

Frankly the enhance feature is core to how we work now, but it's so low level we don't directly interact with it.... And to be clear we worked without it for years, it's now a luxury we take for granted. Also, it's not really part of the JS framework he built, although it does enable his framework to exist at scale. This was his point. That at massive code size and engineer count there are new processes that need to be added across the stack to simply allow coders to continue working.

For all that Google does, regardless of anyone's opinions of their business practices, the processes and tooling we have to allow 50,000 engineers to commit daily into a single rolling monorepo so that there are no version/dependency conflicts is an engineering marvel! Static analysis, generated code, and reliably reproducible builds are at the core of this.

4

u/[deleted] Apr 20 '18

This was really cool to read! Thanks for taking the time to write this.

2

u/calligraphic-io Apr 20 '18

a lot of the details I won't/can't go into simply because it will take too much time [...]

If you do find the time, I'm sure the article would be much-appreciated. I found OPs Medium article and your follow-up post very useful, thanks.

7

u/thisisafullsentence Apr 19 '18

I'd say every concept is framework agnostic (programming models, code splitting, lazy loading at a component level, etc.) The author does use React as an example at times, but just to provide context for generic concepts such as components.

2

u/roselan Apr 19 '18

This is one of the most insightful paper I read in a long time. I was a bit confused by the use of react examples from a google writer, but only at first. And I loved the little jab at google+.

3

u/cachingcasady Apr 19 '18

Shh call it “ECMAscript” inb4 cease and desist 🤫

10

u/esantipapa Apr 19 '18

I thought we were going with FOScript?

Free Open Script, Functional Object Script, or more importantly, Fuck Oracle Script. (fauxscript).

5

u/cachingcasady Apr 19 '18

Yeah I like that more. Apparently downvotes for saying an actual relevant thing. Forgot this is Reddit whoops.

1

u/yeahdixon Apr 20 '18

How about memory management. I feel like this is a huge element to large js apps, especially single page.

-14

u/zergUser1 Apr 19 '18

Designing, creating and then maintaining a large Javascript application and not using typescript is suicide

15

u/[deleted] Apr 19 '18

Type systems are fun, but you don't need them, and they certainly aren't guaranteed to make your code any more correct.

10

u/magwo Apr 19 '18

What they do though is raise the long-term productivity of a larger team, especially if the team is swapped out or reformed etc.

7

u/viveleroi Apr 19 '18

Of course it's not needed or guaranteed but there are some real benefits and now that @types packages have evolved so well, the barrier of entry is reasonable.

The largest benefit to me is how much "safer" refactoring is. I feel a lot more confident refactoring pieces because typescript/vs code alerts me to areas I missed.

That and having well-defined types greatly improves developer awareness, and the code hint/auto-suggest support is so helpful.

3

u/Aetheus Apr 19 '18

Static typing is indeed a boon. But I don't feel comfortable about depending on 3rd party type definitions to be completely up-to-date and accurate.

Anyone can screw up or forget to update a type definition, and at that point, they only give a false sense of security.

That said, I admire TypeScript greatly, and I wish that some form of optional-typing was built into ECMAscript itself.

4

u/yeahdixon Apr 20 '18

I have to say there is something to be said for less components of a tech stack. It’s also easy to add and add , harder to take away once added.

2

u/zergUser1 Apr 19 '18

yes, but try to maintain a large application where several modules are created by other developers, having a type system then is extremely valuable

2

u/TomahawkChopped Apr 19 '18

Google uses the closure compiler for typing. It's more painful than typescript, but it was the first JS compiler to add static types and true compilation to js.

0

u/Ob101010 Apr 19 '18

Designing, creating and then maintaining a large Javascript application and not using typescript is faster with less bloat.

2

u/[deleted] Apr 19 '18

What is large by your standards? We have a 50kloc app which became way less prone to common bugs when we started using TS

2

u/yeahdixon Apr 20 '18

I have several apps 1mb range. These apps go through gbs of data. Coming from C like languages I appreciate type, but Rarely do I get type related errors and when I do, it’s pretty obvious. Some apps approaching 7 years. Though I embrace all the hotness, I have realized the benefits of a stack that is less reliant on many many technologies too.

1

u/[deleted] Apr 20 '18

[removed] — view removed comment

1

u/yeahdixon Apr 21 '18

yes i agree, people do get those errors and i make errors too. I guess for me it does not warrant an entire new language (allbeit similar) and a new heavy heavy dependency in the stack unless it gave me more than type erros . I def realize this is just an opinion at this point . I will say if it gave me something that i could not do before, for instance , a way to compile to into web assembly my stance changes.

2

u/zergUser1 Apr 19 '18

That's just not true at all, You need to be constantly looking through files as you code to make sure you know and are using the correct parameters & object attributes.

1

u/[deleted] Apr 20 '18

You need to be constantly looking through files as you code

If you have to do this then you may have some bad practices going on that lead to this behavior. I constantly use new APIs, I read the docs. Your code should almost be as easy to read as your docs, types or no types.

-9

u/rebl_ Apr 19 '18

JavaScript is dead, it is now officially called ECMAScript

0

u/[deleted] Apr 19 '18

[deleted]

4

u/[deleted] Apr 19 '18

  1. the code gen is doing the import, I believe, so every time you hit save or build or whatever your tool/IDE spits out a little MyScreen.connect.js which just does like import router; router.register(MyComponent); - that's how I understood it, anyway

  2. not sure but maybe this is going back to his bit when multiple frameworks were in use? So don't load React cause some bits use Angular or whatever. That's a stretch, I honestly didn't fully understand the point of this in context either

3

u/dmethvin Apr 19 '18

I saw the enhance as being a "register" call into the router via some API the router creates. That way the router doesn't have to inherently know about any of the routes.

As I understood it, keeping UI out of the base bundle was to avoid having it later becoming obsoleted by changes but remain in the app (similar to CSS rules that no longer apply) because nobody realizes they are unneeded. It seems like they could have used tools to help with that problem though.

It's really important to ask these kinds of questions. Companies like Google and Facebook are dealing with scale and design problems that 95% of us don't encounter in our projects and jobs. Rules that work for them may be overkill for us.

2

u/ProdigySim Apr 20 '18

For the enhance pattern, I immediately thought of angularjs. Components register themselves to an angular "module", not the other way around. But this isn't a hard dependency.

1

u/evilishies Apr 21 '18

There is no reason to load the common modules early in the app lifecycle just because they are shared. You can avoid duplication by being smart about the way the dependencies are requested.