Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow Plugin Install time (504 Gateway Timeout errors) #68

Open
dave-gil opened this issue Jun 14, 2019 · 18 comments
Open

Slow Plugin Install time (504 Gateway Timeout errors) #68

dave-gil opened this issue Jun 14, 2019 · 18 comments

Comments

@dave-gil
Copy link

After successful clean install of this Wordpress reference architecture template, I experienced huge time lags in the admin interface.

Installing plugins regularly caused 504 timeout errors and even bypassing the cloudfront install didnt help. The only solution to get smooth installs that I have found so far is to edit the attributes of the Load Balancer and increase the Idle Timeout to 180 seconds or more. This permits most plugins to install successfully but they take forever!

Any similar experiences, thoughts?
@dave-gil dave-gil changed the title Extreme Plugin Install time (504 Gateway Timeout errors likely) Slow Plugin Install time (504 Gateway Timeout errors) Jun 14, 2019
@michael-newman
Copy link

In the past, I also ran into the 504 errors when installing or upgrading plugins; however, simply changing the Route53 domain A record from CloudFront to the ALB while installing/upgrading plugins resolved the issue and we have been doing this ever since. Installing/upgrading does take a little longer than a single instance setup; but, I would not characterize mine as taking forever. Note, I have my ALB Idle Timeout set to 120 seconds. Otherwise, any latency we experienced in the past was largely due to number of plugins and/or conflicts.

Sidenote re plugins... If you run into frontend page latency issues, you might want to consider a plugin management plugin--I've tried a few; but, ultimately, found Freesoul Deactive Plugins (https://wordpress.org/plugins/freesoul-deactivate-plugins/) to be the best; super nice interface, intuitive, and it simply works. Additionally, the plugin author, Jose, is a great to work with and he takes pride in what he does, and it shows.

@dave-gil
Copy link
Author

dave-gil commented Jun 15, 2019 via email

@michael-newman
Copy link

For a point of clarity, I was just sharing my Idle timeout setting as a point of comparison for you and did not mean to infer installing plugins takes 120 seconds--Sometimes a large plugin or update might take a minute; but, many times updates are a few seconds to 10 seconds or so, depending on size.

If you are running into admin latency issues when installing plugins on a fresh install, then it sounds like there is a configuration issue. Your highlighting EFS, and the only item I would suggest you double check there is the Burst Credit Balance and ensure you have enough dummy data saved in EFS so you have enough data throughput--I recall this was an issue for me when I first stood up this infrastructure (fresh install with only a couple of plugins). As far as admin interface speed of this reference architecture vs that of a single EC2 instance, my single EC2 instance is 2-3 seconds faster than the RefArch (same full blown site w/many plugins) --I believe this is just a result of all the extra jumps the RefArch has with all the subnets. Perhaps, I've gotten used to it; but, I don't find issue with it.

On the front end, my page loads are pretty quick, noting I leverage Freesoul deactivate plugins, W3TC, memcached, and CloudFront w/a secondary S3 origin for static content. My GTmetrix opportunities are focused on deferring parsing of javascript and optimizing images, otherwise very good.

I would also suggest reaching out to the AWS tech support as they have always done good by me. Good luck and stick with it--hopefully others chime in to provide you additional perspectives.

@dave-gil
Copy link
Author

dave-gil commented Jun 15, 2019 via email

@michael-newman
Copy link

Hey Dave, from your exampled installed times, it sure does look like there is an issue here... guess I've gotten used to the plugin load latency. Unfortunately, I can't use my environment for testing install times; we have it locked down.

As far as reaching out to AWS Tech, we utilize the AWS Business Support Plan... best $100/month we could ever spend--we have attained great support on a variety of topics. I believe you could subscribe and unsubscribe as needed. As far as AWS monitoring Github, I do not believe they are that active; but, then again, I'm only on from time to time.

Definitely a topic worth pursuing and believe we all would benefit from any changes...

Others, please chime in...

@tdondich
Copy link

For those with these problems, in order to help better troubleshoot, share the following:

  • File System Tier Configuration
  • Web Tier Configuration

Some of the things to note, EFS is not very performant with writes of small files. Plugin installs fall under this category. Unless a good amount of IOPS are allocated, then your EFS performance is going to be sad in this category. EFS is built for redundancy, not performance. However, once the files are there, performance can be snappy due to PHP file caching/etc.

Web Tier. The instance type matters here. If you are utilizing T* instance types, you may be running into CPU credit exhaustion which results in heavy throttling. So this is something to be very aware of when installing plugins.

Share your config info and let's see if we can find similarities.

@dave-gil
Copy link
Author

dave-gil commented Jul 1, 2019

@tdondich your comment makes sense to me.

I have tried EFS with and without dummy data (500GB) (for file system tier)
I have tried only instance types t2 small and medium (for web tier)

@tdondich
Copy link

tdondich commented Jul 1, 2019

@fippy yep, I would move away from the t2* instance types. Certain plugins may constantly be doing filesystem touches, which is not great. Anything that reads from media will also be incredibly slow. So I'm also looking into how to simply store the entire media library in s3 to avoid that read/write.

@badcrocodile
Copy link

badcrocodile commented Jul 28, 2019 •
edited
Loading

@fippy did you have any luck getting this issue resolved? I've just run my first AWS WP setup following the RefArch almost exactly and have the same problems and it's just not sitting right with me. Core updates fail completely. The only modifications from RefArch being to run on t2 mediums. Have you found a better configuration that solves these plugin installs / WP core update issues?

@michael-newman
Copy link

@tdondich, re Media Library and S3... how are you setting this up different than what is contained in the AWS White-paper (https://d1.awsstatic.com/whitepapers/wordpress-best-practices-on-aws.pdf)? Would welcome hearing your insights...

@tdondich
Copy link

The extremely harsh reality of this deployment is that utilizing EFS is simply a no-go. You can't get enough performance out of it to make wp-admin reasonably usable when you are using a handful of plugins, especially any plugins that require doing file scans. EFS was built for durability and large scaling requirements, not necessarily for performance when dealing with small files.

So the best thing to do is to have a localized NFS cluster or using high availability appliances such as SoftNAS.

I'm considering building a pull request that would offer the choice of EFS or deploying a localized NFS high availability cluster. Obviously EFS will most likely be more cost effective but will certainly hurt performance. NFS will cost more but you get dramatically faster performance when not caching on the front end or when using wp-admin.

@dave-gil
Copy link
Author

dave-gil commented Aug 12, 2019 via email

@dave-gil
Copy link
Author

dave-gil commented Aug 12, 2019 via email

@saluminati
Copy link

@fippy Are you a Messiah ? In my case you are.... I have wasted so much time fixing this issue until I found your solution suggesting idle timeout on ALB. Thanks for the help mate.

@HarishKM7
Copy link

In my case, increasing the ALB idle timeout from 60 to 300 did not help, but changing the "Origin Response Timeout" & "Origin Keep-alive Timeout" to 60 in CloudFront fixed the issue.

@angelovescio
Copy link

A relatively quick, but more costly solution, is to switch from "Burst Throughput" to "Provisioned Throughput" in AWS for the EFS volume. For my install off of this template, and after running for about a year, the size of the volume was ~400MB and 10M/s throughput was an additional $60/month. You can switch to this temporarily if it effects production, until you are able to get the cache stuff running. Then buy some burst credits, just in case, and switch it back after you make the cache changes.

@wesleywh
Copy link

wesleywh commented May 5, 2022 •
edited
Loading

I know this is really old but is a top result for this particular search. Have not tried the following yet but will soon...

The solution I'm building makes use of unison if anyone is interested. You can also just mount EFS async on an EC2 instance. If you're using something like fargate then you will have to use my unison method.

Found it via this blog: https://www.baeldung.com/linux/synchronize-linux-directories

Download unison: https://github.com/bcpierce00/unison

mkdir -p /unison
cd /unison
curl -LJO https://github.com/bcpierce00/unison/releases/download/v2.52.0/unison-v2.52.0+ocaml-4.01.0+x86_64.linux.tar.gz
tar -xvf unison-v2.52.0+ocaml-4.01.0+x86_64.linux.tar.gz

Then you have have it watch for directory changes and sync between two directories like the following:

./unison -batch=true -repeat watch /efs/ /usr/src/wordpress/

Then just make sure your efs volume is mounted to /efs. This mimics what mounting it asynchronously would do. That way file reads and writes return immediately but are still synced to EFS which in turn syncs it to your other instances/containers of wordpress.

Of course you need to wrap that unison repeat command in something like duminit to make sure the process is already running. Not sure the best approach there yet but I'm sure I'll find something. If I remember to post back here I will to let you know how it goes.

@wesleywh
Copy link

Okay I tried it out and it works really really well! Now I set this up with docker and fargate but this whole process can be setup with EC2 easy.

In my Dockerfile:

WORKDIR /unison
ADD https://github.com/bcpierce00/unison/releases/download/${UNISON_VERSION}/unison-${UNISON_VERSION}+ocaml-4.01.0+x86_64.linux.tar.gz /unison
RUN tar -xvf unison-${UNISON_VERSION}+ocaml-4.01.0+x86_64.linux.tar.gz && \
    rm /unison/unison-${UNISON_VERSION}+ocaml-4.01.0+x86_64.linux.tar.gz && \
    mv /unison/bin/* /bin/

Then in my startup script I run:

$(while "true"; do unison  -batch=true -repeat=watch -silent -group -owner -auto -times ${SYNC_SOURCE_DIR} ${SYNC_TARGET_DIR} >/dev/null 2>&1; done) &

This will make sure ${SYNC_SOURCE_DIR} and ${SYNC_TARGET_DIR} are always in sync. As a bonus if uninon dies for whatever reason it will restart. Could also put a sleep in the while incase of a crash loop, but this is good enough for now.

Then mount your EFS volume to your ${SYNC_SOURCE_DIR} and the path to the root of your wordpress installation to ${SYNC_TARGET_DIR}. Doing that it will write to the local filesystem and sync to EFS. Also if something get's written to EFS it will sync to the wordpress directory.

Setting it up like this I tried it and I no longer got timeouts installing or removing plugins because it is installing directly to the filesystem THEN copying to EFS. The same goes for reads as it will read directly from the filesystem and NOT from EFS so it will be really fast.

Hopefully this helps someone! It definitely solved my problem. Im hosting on fargate/load balancer/cloudfront combo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@badcrocodile @tdondich @dave-gil @saluminati @angelovescio @wesleywh @michael-newman @HarishKM7