Webpacker compilation takes more than 1.5 hours

[karimelghazouly]

Currently we have tens or maybe hundreds of react components served from our rails app and we using capistrano for deployments.
we use for sure webpacker for assets compilation and right now this process is really painful as it it takes more than 1.5 hours on a high end machine.

I understand that this is because of the nature of our product, but webpacker right now compiles ALL the assets when there is a small change.
For example if we just add a new file, it recompiles everything from the start.

We looked into caching and found this issue a while ago
#1439
it means caching and checking for cached compiled files as a default behavior, put looking into the code here and the result of cache
https://github.com/rails/webpacker/pull/1743/files
I can see it only checks the digest of all files, which means if one file is added or one file is changed the whole digest changes and this trigger a complete compilation.

This is my understanding so far if there is something I don't get right please guide me.

[rossta]

What version of Webpacker and Rails?

What is your JS directory structure?

[karimelghazouly]

webpacker (4.3.0)
rails (6.0.3.5)

app/javascript
    react-configs/
    main_react_apps/

webpacker.yaml

source_path: app/javascript
source_entry_path: packs
public_root_path: public
public_output_path: assets
cache_path: tmp/cache/webpacker

inside packs we have this piece of code

import xxxconfig from '../react-configs/xxxconfig';
ReactOnRails.register({
    ...xxxconfig,
});

xxxconfig.js

const lessons = {
    "main_react_apps_1": require('main_react_apps/smthing/smthing/smthing/smthng').default,
    "main_react_apps_2": require('main_react_apps/smthing/smthing/smthing/smthng').default,
    ...
    ..
  }

Is that clear ?
@rossta

[ahmedAdel202090]

+1 Same Problem

[rossta]

Hard to say what’s going on with limited info.
If your build is taking 1.5hrs then some is either wrong or ill-matched with the setup or you have have thousands of source files (do you?)

Generally for slow builds, I’d add the following:

One suggestion is to upgrade Webpacker as it does look like you’re behind a few versions with v6 about to release.

You didn’t show what’s in your packs directory but if you have many files in there, that will definitely cause a slow build.

This article provides some debugging tips including how to use the speed-measure-webpack-plugin to identify a slow step—could be that disabling certain plugins in development could help with the local dev ex.

[karimelghazouly]

@rossta
Our packs folder has 5 js files, I included in my last comment the main file that has include most of the files.
I tried upgrading my webpacker before and didn't work.
and yes we have huge amount of assets, I would say maybe almost a thousand.

My main question is about how caching work, does it check if all assets are the same or it checks only modified/deleted/new files ?
as far as I see it only checks if the all the files are the same it skips compile step, but if only one file is edited or added it re compiles everything.

[justin808]

@karimelghazouly, first, can you try updating to the latest RC release?

Next, is your issue only with production or development as well.

You should be using hot-reloading with webpack for development work, per this example: https://github.com/shakacode/react_on_rails_tutorial_with_ssr_and_hmr_fast_refresh.

For production, very old versions of webpack and Node would clearly hurt.

My main question is about how caching work, does it check if all assets are the same or it checks only modified/deleted/new files ?

Caching? There is none. You're probably referring to the saved digest value that determines if recompilation should run during development. In general, don't use that as it's crazy slow. Use the webpack-dev-server or webpack --watch.

[justin808]

Please open this as a discussion here: https://discuss.rubyonrails.org/c/webpacker/10

[justin808]

We're saving issues for bug reports and features.

[karimelghazouly]

@justin808 we are using hot reloading for development, my main issue is in production where we need a precompilation.
I think webpack support such behavior using caching as mentioned here
https://webpack.js.org/configuration/cache/

webpack-dev-server or webpack --watch is for development only

[karimelghazouly]

@justin808 I believe this is an issue, as this behavior is not really efficient, recompilation of everything is a total waste of resources.

[justin808]

We have caching turned on. Once you identify this issue still occurs in using the current RC version, I'll reopen the issue.