Redesign Build Deployment Process (External)

[MukuFlash03]

This PR serves as the implementation for this issue for redesigning our build and deployment processes across e-mission-server, join, admin-dash, public-dash primarily.

Collaborating on this along with @nataliejschultz .

All four redesign PRs for the external repos:
E-mission-server: e-mission/e-mission-server#961
Join: #29
Admin-dash: e-mission/op-admin-dashboard#113
Public-dash: e-mission/em-public-dashboard#125

---

A rough plan can be:

1st phase

• Consolidating Differences
  • Minimize the differences in internal and external versions, cleanup unnecessary files, so on.
• Image_Build_Push.yml:
  • CI / CD via GitHub actions to build and push images to Dockerhub for all four repos based on the current process for e-mission-server.
  • Image build push only builds for that specific repo when merge is made to a specific branch.
  • Additionally, for e-mission-server, image-build-push will now also need to trigger the GitHub action in join, admin-dash, public-dash.

2nd phase:

• Multi-tier uniform repo structure in nrelopenpath
  • Need to change the internal repos to pull from the externally built images for each repo without duplicating the entire external code repository internally.

[MukuFlash03]

Checking in the first initial set of changes for consolidating the differences in the external and internal versions.
Join repository did not have any differences between the two.

So, the major changes only involve the image_build_push GitHub action automation to build and push images to Dockerhub whenever changes are merged to the specified branch.
I have used a local test branch (image-push-merge) which can be changed to master once changes are finalized.

[nataliejschultz]

Mostly discussing comments that should be removed or un-commented.

[nataliejschultz]

Can be removed; replaced by environment variables

[MukuFlash03]

We would still need these.
The values are currently set in my forked repo's GitHub repository secrets.

For the three external repos (admin-dash, public-dash, join), these values would need to be set similar to how e-mission-server has these values set as it is the only one currently that uses a GitHub action workflow to automate the build and push for the docker image.

[nataliejschultz]

There is not necessarily testing to do for the changes to the external join page outside of running GitHub Actions, which we have proven to work. However, for a sanity check:

I pulled the code by downloading a zip of the forked join page directory on @MukuFlash03 image-push branch. I unzipped the file, cd'ed into the main directory, and ran docker compose -f docker-compose.yml up.

The containers spun up, and terminal gave the message:
Available on: mukul-forked-nrel-openpath-join-page-joinpage-1 | http://127.0.0.1:5050
I changed the port from 5050 to 3274 per the docker-compose.yml and went to http://127.0.0.1:3274, which looked like this:

The internal repository builds off of the image created in our image_build_push.yml workflow. Testing the internal page and building off of the new image:

1. Downloaded Mukul's nrelopenpath-multi-tier branch as a zip, opened and cd'ed into it.
2. Created the emission network: docker network create emission
3. Built database container: docker run --name db1 -d -p 27017:27017 --network emission mongo:4.4.0
4. Ran join page docker run --name int-join-1 -d -p 3274:5050 --network emission int-join
5. Changed local url to use correct port, and got the exact same result as before:

This should be similar to how the image is used internally, since docker compose is not supported.

[nataliejschultz]

Wanted to add that I tried adding different config file names as get requests per the join page readme, and it worked for both the internal and external tests.

[shankari]

@nataliejschultz

I am not sure what this means

This should be similar to how the image is used internally, since docker compose is not supported.

Given that this repo has both docker-compose.yml and docker-compose.dev.yml files
Also, how is using images generated by Mukul the same as using internal images?

[nataliejschultz]

@nataliejschultz

I am not sure what this means

This should be similar to how the image is used internally, since docker compose is not supported.
Given that this repo has both docker-compose.yml and docker-compose.dev.yml files Also, how is using images generated by Mukul the same as using internal images?

Per the docker usage policy:
"While docker-compose is a popular tool for defining and running multi-container Docker applications, it cannot be used for deploying Docker containers on Amazon Elastic Container Service (ECS). "

Per Jianli:
"Cloud team is not using docker-compose for deploy. The base image is pulled from EXTERNAL, i.e. your DockerHub, we wrapped the image to fit into AWS ECS deployment"

So using compose to test internally doesn't make sense to me, since that's not what's going to happen to the image in the jenkins pipeline anyway. Using images with our changes for testing (whether built locally or pushed to docker hub via our image building actions) makes the most sense to me.

[nataliejschultz]

Decided that, after everything I learned testing the public dash, I would re-test this page by building a local image and using that to run the docker-compose.dev.yml.
I spun up the db:

$ docker run --name db1 -d -p 27017:27017 --network emission mongo:4.4.0

Built a local image of the join page and ran container + checked localhost:

docker build -t joinpage-local frontend/
docker run --name join-ext-local -d -p 3274:5050 --network emission joinpage-local

Now using docker compose for the .dev files:

I changed the docker-compose.dev.yml to build off of the image that I created locally:

And ran:

$ docker compose -f docker-compose.dev.yml build
$ docker compose -f docker-compose.dev.yml up

Checked localhost to see if I could see both the plain and study config pages:

[shankari]

@nataliejschultz I agree that we will not test against the images that are built using jenkins as part of our dev testing. I think that they will be tested on staging. However, We can build the internal images using docker-compose build on the internal repos and run them locally. This is particularly important for images where the internal repo is very different from the external repo. Although we will be testing the images that are build that way in a dockerized (versus AWS) environment, we will at least know that the images were created correctly. I would say that this is the most important for server and then for public dashboard (until the internal and external repos are unified).

[MukuFlash03]

Reverted the hardcode run_id and commented out build-push job steps.
Can be merged now.

[nataliejschultz]

Removed extraneous comments from actions yml and set it to run on push to main.

[shankari]

Fairly straightforward