Prevent webpack compilers from running on every page compile during development

[colbyfayock]

Currently our webpack compilers run on every page load which slows down performance a bit during development.

This doesn't impact production as it's statically compiled.

These plugins work by running at compile time to provide features like an RSS feed, OG images, and a sitemap

https://github.com/colbyfayock/next-wordpress-starter/tree/main/plugins

Ideally we can only run this when it will actulaly make a difference to avoid the performance hit during development

https://github.com/colbyfayock/next-wordpress-starter/blob/main/plugins/plugin-compiler.js#L56-L64

[petercr]

Hey Colby 👋 is this still an issue that needs to be worked on?
It seems like it's from a while ago 😅

Anyways let me know 👌

[colbyfayock]

yup still a problem!

[petercr]

Okay thanks. I just wanted to check in first and make sure before I spent any time on the issue.
But I guess now I'm free to jump in and see what the heck is going on 🤪

I'll let you know what I come up with 👍

[colbyfayock]

aweosme sounds great. im also pretty open to new solutions around the entire plugin chain here. @GuilleAngulo and i worked on it a while back as something that we were hoping would be easy to fit in new capabilities without having to rely on package.json-based scripts which wouldn't actually fall into the build process and may potentially get skipped? (not sure if that's a valid concern)

i'd also like to figure out a way to enhance code reuse. we have the util.js with a bunch of request capabilities in there. we did that because we weren't able to easily import the existing functions with the require vs import

[petercr]

So I finally got to give this code a good look over 👀 and think about it.
After looking through some Webpack docs, I came across this method: recordsPath

Here's the info on it:

Use this option to generate a JSON file containing webpack "records" – pieces of data used to store module identifiers across multiple builds. You can use this file to track how modules change between builds.

Hopefully that could give us a reference point to check the builds against, allowing for the build to be skipped if needed.

I'm going to play around with it and see where that gets me 👍🏻

[colbyfayock]

ohhhh thats super interesting. only thought, would that override other's recordsPath or is it per-plugin?

[petercr]

Hmm I don't know the answer to that, but I'll look into it 👀

Also I haven't quite figured out how to get it working yet 😂
It seems to run the command, but no records file is being generated.

[colbyfayock]

this might be worth looking into as well https://nextjs.org/blog/next-12-2#swc-plugins-experimental

haven't played with it, so not sure if it'll actually be useful for this use case, but worth thinking about and looking into

[petercr]

Hey Colby 👋 thanks for the link 👍
I checked it out, and found a list of the official swc plugins.
Link to the list of SWC plugins

Here's the list:

• emotion
• jest
• relay
• styled-components
• styled-jsx
• transform-imports

I'm not sure if any of those will be of help with this particular issue.
I was thinking maybe if we tried to implement some kind of caching for the webpack build, then it would only run if changes were made?

Perhaps even something like next-bundle-analyzer. And then we could query that data instead of displaying it to the user, of course 😉
Or even just some kind of regular cache plugin for Webpack.
Then hopefully it wouldn't run the build again if the input had not changed.

Let me know what you think 😄

[colbyfayock]

ah okay so you can't make a custom one? i guess that's where my mind was going, a custom plugin, kinda like we were doing for webpack, but maybe that's not the right route

caching sounds like a great strategy! especially if that's a common approach

totally open to seeing what you come up with, just hoping to avoid slowing down dev with the processing each time

if going the caching route, wonder if there's an apollo caching mechanism to take advantage of too 🤷 but idk how the requests work whether they're all sahred in the same process or not

looking forward to seeing what you come up with

[colbyfayock]

thought just come to mind, should we by default disable these from running on development? that's common isnt it?

a caching solution would totally be nice, but wondering if that would make more sense

we could add an npm script to manually trigger them to test and an environment variable / flag to enable / disable

only other thought is not all of them should be disabled though. Search for instance should always be available, so the problem persists there, as it's used clientside to make search requests

[petercr]

Nice well I think that would be an easy way to fix the problem! 😂
I mean sure I think it's common to disable running extra stuff in development.
You know how us devs love our fast refresh! 🚀

We could disable the ones we don't want running on the dev server and keep the ones we do want.
On that note, we could also declare some kind of variable like hasRun to hold the states of the plugins.
And that way we could only run each plugin 1 time on the dev server. So they're available.
Or only run the ones we want to run.

Maybe that's a good solution?
Let me know what you think 👌

[colbyfayock]

that sounds good to me! let's give it a shot

[petercr]

Hey Colby, sorry it took me so long to get back to you on this issue.
I had some personal issues come up that needed taking care of 😅

Anyways, so I tried coming up with 2 different solutions to fix the issue.
The first solution was what we had discussed using a boolean to hold the state of whether the plugins have run.
Here's the branch on my fork if you want to check it out: fix-webpack-compile-every-time.

The second solution I tried was adding a Webpack plugin called HardSourceWebpackPlugin.
Here's their intro blurb:

HardSourceWebpackPlugin is a plugin for webpack to provide an intermediate caching step for modules. In order to see results, you'll need to run webpack twice with this plugin: the first build will take the normal amount of time. The second build will be significantly faster.

It sounds exactly like what would fix this problem, if we could cache the modules then maybe we could reduce the build time.
Here's the branch on my fork if you want to check it out: hard-source-webpack-plugin.

From running the both of them locally it seems like the 1st solution is the only one that seems to reduce the build time for the plugins.
I don't have a production WP site that I can run this on to test it... 🤷🏻‍♂️ I didn't know if you wanted to give it a try?

Let me know what you think, and if you want me to push up either of those solutions as a PR.
Or if you have any questions.

Thanks 👍🏻

[colbyfayock]

ah i hope everything is okay, no worries, no rush, and at least if you get in a PR it'll be good for Hacktoberfest :)

i'll try to take a look sometime between tomorrow and early next week, will be a busy weekend for me

thanks for looking into it

[petercr]

Hey Colby, I totally thought I replied to this earlier whoops 😅
Yes everyone is fine, however, right after I finished working on these tasks I went and dropped my parents off for their flight to SW Florida.

Which was 2 days before hurricane Ian just smashed into SW Florida! [For the records I said they should wait to go 🙄 ]
Then they got stuck on the barrier island where they live for 6 days! With no power or water or cell service or anything.
On the sixth day they got a ride off the island with a good Samaritan in a small personal helicopter.
But for those 6 days there was only very limited communication with them.
So, you can see why I was a bit busy before 😂

Anyways hope all is going well with you, let me know if you have any thoughts on those PRs.

[colbyfayock]

i could only imagine 😓

let's start off with the first one where it'll keep track of if it has ran, want to submit a PR for that?

totally open to the caching one too, honestly didnt expect that we'd have to get deeper into the webpack plugin world haha

DM me on Twitter or email me or really wherever you want to find me and i'd happily let you test with my WP instance

[petercr]

Thanks yeah it was an ordeal for for but just us, but the family's of the 1,000s of other stranded people too 🙏🏻

I will go ahead and push up that first PR, and you can look it over.

And I'll send over a message on Twitter or discord or something 👍🏻