[Deployment] Fix Backend Deployment Bug #386
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the courses backend bug from the previous transition to Kittyhawk, as well as making some nit improvements on the kittyhawk deployment code.
Cause for problem
After switching from Icarus to Kittyhawk in PR #358, our courses products began experiencing problems at any of the subroutes (e.g.
/adminor/accounts) with aBad Gatewayerror. The products are still viewable and frontend available, but anything that required logging in or searching for course review were down.Because all of the products were still accessible yet unfunctional, this must be caused by something shared by all courses product, which narrows the problem to the courses backend. After reinstalling the products using icarus (helm) last night and taking a closer comparison at the yaml today, I found out that the results were due to an extra line here.
Prevention in the future
This is a DevOops. I was trying to transition a lot of products over and should have not only taken a closer look at whether the yaml itself made sense, but also at the original configurations. Earlier, I also accidentally misspelled some of our products domains (e.g. penncourse.org vs .com) :'D
In the future, it would be beneficial to keep an improved checklist for deployment transitions, as well as have another person review non-time-sensitive migration PRs like these in order to catch any mistakes.
Also, with argo-cd, we can compare changes in deployment config more easily.