Lazy Loading of Client Resources

[IstoraMandiri]

One limitation of Meteor is that you can't choose which resources are bundled to the client based on their session state. As far as I can tell, the client always receives 'all or nothing' of the application (apart from /public, or translation strings if you're using tap:i18n).

For example, a common pattern is to have an /admin/ route, which most users don't visit, but all of the resources are shipped to all clients anyway. A minor redundancy issue, but for much larger apps at scale, I imagine this could have a substantial effect.

I believe the only way to achieve the separation of what is shipped right now is to have multiple meteor instances running on different ports, but attached to the same database -- e.g. at admin.myapp.com and www.myapp.com. I think it would be cleaner to avoid this.

For me the ideal folder structure of such an app would look something like:

/modules/admin/content/ 
/modules/frontend/posts/
/modules/frontend/comments/
[Perhaps with a special config file that is used during the build process.]

Ideally, this solution may would integrate with iron-router in some way, to simplify which routes trigger the loading of which resources.

Additionally, there could be a security benefit - don't ship certain code to certain clients based on allow/deny rules, just like DDP? This way we can ensure that only users logged in to an admin account can even download the admin UI logic.

I haven't thought too much about the API to achieve this, or even if it's feasible, but I didn't see this being discussed already (apologies if it was), and it'd be great to hear if anyone has thoughts and/or suggestions on how to achieve it.

[theosp]

+1

[delgermurun]

there is a trello card about it https://trello.com/c/24s6vyxo/55-incremental-loading

[mitar]

+1

[mpowaga]

+1

[IstoraMandiri]

Just saw some interesting work by @numtel that seems to relate!

https://github.com/numtel/meteor-component-example

[numtel]

Over the afternoon, I think I have come up with a much better solution.

Instead of using the manuelschoebel:wait-on-lib package and the numtel:es6-proxy package, and probably could even skip using the numtel:template-from-string package, it seems that a single resource-subscription package would be most useful.

Resource subscriptions could be published and contain a selection of HTML/CSS/JS files (or any other filetype Meteor has handlers for, e.g. ES6, COFFEE, LESS, JADE). The user's authentication status could be verified in the publish function.

I'm going to start looking at what it will take to implement this... stay tuned 🐇 🐢

[arunoda]

@numtel this is an awesome solution.
If you are going to return .ready() reactive method from publish call. That'll be great :)

[mquandalle]

So basically you will evaling data coming from a DDP connection, right? That doesn't feel better than doing a simple HTTP get query to load the javascript.

[mitar]

cc @MiroHibler

[arunoda]

I think there is no need to eval. We can load a script dynamically. From the server it wraps all the template compiling part. This will be just like JSONP.

[numtel]

I was hoping to transfer the data over DDP then try adding the script and link tags to the DOM instead of eval. The wait-on-lib package uses jQuery.ajax with eval.

Hot code pushes could refresh the css or refresh like a normal client side refresh when a DDP changed message comes in.

[arunoda]

@numtel that's the same as eval. It's blocked when you enabled browser policy package. But that's okay at first. Just publish the first version :) We can improve later.

[numtel]

@arunoda Ah too bad. Good to know.

[numtel]

@mquandalle I think the main advantage to using a DDP subscription over AJAX is that the client will receive notification when the code changes (and new data for CSS changes).

[mquandalle]

You can publish a URL using DDP, and then fetch the code using HTTP. This is how css “injection” is implemented for instance. It's good to keep the two protocols separate, HTTP for source distribution, DDP for data communication.

[MiroHibler]

Hey guys, check out miro:preloader ;)

Thnx @mitar :)

[matteodem]

Looks like preloader does not provide the functionality to lazy load templates, looks cool though.

[numtel]

😸 Ok, I've updated the example app to use miro:preloader and my new package, numtel:publicsources.

With this package, you can place a publicsources.json file in your application directory with a description of bundles of resources to build for client-side lazy-loading.
Source files will be transpiled and collected into a single .js and .css file for each bundle.

Big thanks to @mquandalle and @arunoda for their Bower and NPM packages, the patterns used in those saved me a ton of research and time.

This solution seems to work pretty well:

• Templates do not have be put into strings, leave them in .html files.
• Hot code push works with templates, CSS, and JS
• Use any file handler packages with lazy-loaded components like coffeescript, jade, less...

[jadsonlourenco]

Hi @numtel, just a thought on this: Meteor uses Node, we all know, and Node is not very good to work with "external" files, I say this because time ago I put Polymer inside the folder "public", but after 30 minutes Meteor work so slow for the auto-reload. Then after Polymer removed from the "public" and added as external(cdn) Meteor back to work fast.

In this use case will it happen, if you put too many files for Meteor handle these requests, will it become very slow after a few minutes? I will try it now!

Another issue, this method works with your other packages that create "WebComponents"? I will try it now too.

But for both issues if you or someone here has an answer that's great, perhaps explain why you have many files in the "public" folder Meteor is slow, etc.

Thank you.

[MiroHibler]

It really depends on what external libraries you load. I haven't been working with Polymer so far, but AFAIK it's a polyfill and that may be the root cause.

In other words, it really depends on the browser you're using, the type and size of the library as well as the task it's doing.

Another cause may be the fact that loaded libraries stay in browser's memory until you refresh the page but do not unload on template (re)rendering, so if you have a lot of templates within your page, it may load a lot of external libraries and slow your app down.

[jadsonlourenco]

Hi @MiroHibler, yes I understand and think the same. Also thanks for you package, after last update works fine again.

About this topic I do a test here with a more complex app, so I'm not sure how use the @numtel package to load the template that not need a router, for example: I have a template "main", thats load the "sidebar" and "yeld" of Iron Router, but how load this templates with this method?

I do performance test looking for load, and so great the result, because just load the "js" for the current template, and with cache no make new requests for loaded files. Feel the same Meteor way.

[matteodem]

I added a ticket for some enhancement ideas numtel/meteor-publicsources#1

[numtel]

@jadsonlourenco If that slow auto-reload issue with lots of files in public is a concern, it should be dealt with by patching Meteor. Placing files inside of the public directory should not inherently slow down an application.

That said, it's not the only way to use this package. Using the public directory was necessary to ensure that Meteor doesn't automatically include the source files in the app but there is no reason why you can't use a publicsource.json bundle like this to pull sources from the private directory:

{
  "priv/test": [ "../private/test.js" ]
}

[numtel]

To compliment the publicsources package, I have created the numtel:privatesources package which allows for authentication. 🌟 👍

[mitar]

Does anyone know of any way to load dynamically packages on the server side, on production, when Meteor app is bundled already?

[afareed007]

Running micro services for different sections of app on different ports could be a better solution as of now.

[dandv]

@numtel's two packages have become https://github.com/numtel/meteor-lazy-bundles.

@ahmadthd: how would that help with loading client files, e.g. templates and their JavaScript, on demand?