Setup local development environment using Nginx

DEVELOPMENT.md

@@ -0,0 +1,111 @@
+# Developer Guide
+
+The sections below explains how to build and run the website locally using Docker & Nginx. If you want to extend the style, content or functionality of this site, please see our [Customizing the ATT&CK Website](/CUSTOMIZING.md) document for tips and tricks.
+
+* Use **Workflow 1** if you need a comprehensive solution that handles both building and serving the website
+* Use **Workflow 2** if you're developing and need to quickly test changes without the overhead of the full Docker build process
+
+Use our [Github Issue Tracker](https://github.com/mitre-attack/attack-website/issues) to let us know of any bugs or other issues you encounter. We also encourage pull requests if you've extended the site in a cool way and want to share back to the community!
+
+If you find errors or typos in the site content, please let us know by sending an email to <[email protected]> with the subject **Website Content Error**, and make sure to include both a description of the error and the URL at which it can be found.
+
+_See [CONTRIBUTING.md](/CONTRIBUTING.md) for more information on making contributions to the ATT&CK website._
+
+## Prerequisites
+
+* Docker
+* Node.js and npm (for local build)
+* Python 3 and pip (for local build)
+
+## Workflow 1: Build and Run Using Docker
+
+This workflow is designed to be a comprehensive, all-in-one solution to building and serving the website. It utilizes a multi-stage Docker build process to handle the entire sequence of operations starting from installing dependencies to generating static files, and ultimately running the website.
+
+The first stage in this Dockerfile leverages a Node.js base image (node:16) to handle JavaScript and Webpack operations required for generating the search bundle of the site.
+
+The second stage uses a Python base image (python:3.10-slim-bullseye) to run the update-attack.py script, which generates static files for the entire website.
+
+The final stage is based on a lightweight Nginx image (nginx:stable-alpine) which is configured to serve the static files generated in the previous stage. In other words, once the Docker image is built, you can run a container from this image and have a fully functional version of the website served by an Nginx server.
+
+This approach is especially suitable if you need to have an isolated and reproducible build process, as each run starts from a clean state and goes through all the steps necessary to produce a running website.
+
+### Steps
+
+1. Build the Docker image:
+
+    ```shell
+    docker build -t attack_website .
+    ```
+
+2. Run the Docker container:
+
+    ```shell
+    docker run -p 80:80 attack_website
+    ```
+
+   This will start a Docker container with the image you built and forward port 80 from the container to your host machine.
+
+3. Now, you should be able to view the website by opening a web browser and navigating to `http://localhost`.
+
+## Workflow 2: Build Locally and Serve Using Docker
+
+This workflow, on the other hand, is optimized for developers who need a faster iteration cycle for testing changes to the website. It allows for the website to be built manually on your local system, which can offer more control and faster feedback while making changes to the codebase.
+
+In this workflow, the Dockerfile is much simpler and serves a single purpose: to run an Nginx server that hosts the website. However, instead of embedding the website's static files within the Docker image (as in Workflow 1), these files are provided to the Docker container at runtime through a Docker volume. This volume points to the output/ directory on your local file system, where the build artifacts are located.
+
+The main advantage of this approach is that you can modify the website's source files, run the build process locally, and then refresh your browser to see the changes without having to rebuild the Docker image. This can greatly accelerate the feedback loop when you're making frequent changes to the site.
+
+### Steps
+
+1. Ensure you have Node.js, npm, Python 3, and pip installed on your local machine.
+
+2. Build the static web content locally. The web application is composed of two modules: the Pelican content, and the ATT&CK search module.
+
+    * Build the Pelican content by running the following command from the root of the project:
+
+        ```shell
+        python3 update-attack.py --attack-brand --extras --no-test-exitstatus
+        ```
+
+      The static web content will be written to a folder called "output".
+
+    * Build the search module by running the following commands:
+
+        ```shell
+        cd attack-search
+        npm ci
+        npm run build
+        cp dist/search_bundle.js ../output/theme/scripts/
+        cd ..
+        ```
+
+3. Build the Docker image for the test environment:
+
+    ```shell
+    cd test
+    docker build -t attack-website-test .
+    ```
+
+4. Run the Docker container for the test environment:
+
+    ```shell
+    docker run -p 80:80 -v $(pwd)/../output:/workspace attack-website-test
+    ```
+
+   This will start a Docker container with the test environment image, forward port 80 from the container to your host machine, and mount the "output" directory from your local workspace to the "/workspace" directory inside the container. This allows Nginx to serve the static web content you built.
+
+   Please see [test/README.md](./test/README.md) for further usage details on the test environment image.
+
+5. Now, you should be able to view the website by opening a web browser and navigating to `http://localhost`.
+
+Please note that Workflow 1 is the preferred method as it closely emulates our production environment.
+Workflow 2 is recommended for those who prefer or need to build the website locally before testing it.
+
+## Disclaimer re: Pelican's Built-in Web Server
+
+We advise that you avoid using Pelican's built-in web server for serving the website.
+Pelican uses a different set of rules for path matching compared to Nginx, which is used in our production environment.
+As a result, the behavior of the built-in server may differ from the production environment, potentially leading to discrepancies and overlooked issues.
+
+To ensure that your testing environment is as close as possible to the production environment, we recommend using the workflows outlined in this guide.
+Both workflows leverage Nginx, which more closely emulate the behavior of the production environment, improving your ability to catch potential issues before they reach production.

Dockerfile

@@ -16,7 +16,7 @@ RUN cd attack-search && \
     npm run build
 
 # Use the official Python image as the base image
-FROM python:3.10-slim-bullseye
+FROM python:3.10-slim-bullseye as python-build
 
 ENV DEBIAN_FRONTEND noninteractive
 ENV LANG en_US.UTF-8
@@ -45,6 +45,12 @@ RUN python3 update-attack.py --no-test-exitstatus
 # Copy the search service webpack bundle from the node-build stage
 COPY --from=node-build /app/attack-search/dist/search_bundle.js output/theme/scripts/search_bundle.js
 
+# Nginx stage
+FROM nginx:stable-alpine as production-stage
+
+COPY --from=python-build /home/attackuser/attack-website/output /var/www/html
+COPY nginx.conf /etc/nginx/conf.d/default.conf
+
 # Label metadata
 LABEL name="attack-website" \
     description="Dockerfile for the ATT&CK Website" \
@@ -54,6 +60,6 @@ LABEL name="attack-website" \
     vendor="MITRE ATT&CK" \
     maintainer="[email protected]"
 
-WORKDIR /home/attackuser/attack-website
+EXPOSE 80
 
-CMD ["pelican", "--listen", "--bind", "0.0.0.0", "--port", "80"]
+CMD ["nginx", "-g", "daemon off;"]

README.md

@@ -1,80 +1,54 @@
 # MITRE ATT&CK® Website
 
-### See the live site at [attack.mitre.org](https://attack.mitre.org)!
-
 This repository contains the source code used to generate the MITRE ATT&CK® website as seen at `attack.mitre.org`. The source code is flexible to allow users to generate the site with custom content.
 
-## Usage
-The [Install and Run](#Install-and-Build) section below explains how to set up a local version of the site. You can also visit the live site at [attack.mitre.org](https://attack.mitre.org). If you want to extend the style, content or functionality of this site, please see our [Customizing the ATT&CK Website](/CUSTOMIZING.md) document for tips and tricks.
+## Visit the Site
 
-Use our [Github Issue Tracker](https://github.com/mitre-attack/attack-website/issues) to let us know of any bugs or other issues you encounter. We also encourage pull requests if you've extended the site in a cool way and want to share back to the community!
+You can view the live site at [attack.mitre.org](https://attack.mitre.org)!
 
-If you find errors or typos in the site content, please let us know by sending an email to [email protected] with the subject **Website Content Error**, and make sure to include both a description of the error and the URL at which it can be found. 
+## Reporting Issues
 
-_See [CONTRIBUTING.md](/CONTRIBUTING.md) for more information on making contributions to the ATT&CK website._
+If you encounter any bugs or other issues, please use our [Github Issue Tracker](https://github.com/mitre-attack/attack-website/issues).
 
-## Requirements
+If you find errors or typos in the site content, let us know by sending an email to <[email protected]> with the subject **Website Content Error**. Include a description of the error and the URL at which it can be found.
 
-- [python](https://www.python.org/) 3.6 or greater
+## Development
 
-## Install and Build
+Check out our [developer guide](DEVELOPMENT.md) if you are interested in extending the style, content, or functionality of this site.
+It includes instructions on setting up a local version of the site, and workflows for building and running the site using Docker or locally.
 
-### Install requirements
+We also have the additional following guides:
 
-1. Create a virtual environment: 
-    - macOS and Linux: `python3 -m venv env`
-    - Windows: `py -m venv env`
-2. Activate the virtual environment: 
-    - macOS and Linux: `source env/bin/activate`
-    - Windows: `env/Scripts/activate.bat`
-3. Install requirement packages: `pip3 install -r requirements.txt`
+* A [deployment guide](./test/README.md) for setting up our testing environment
+* A [release guide](./docs/RELEASE.md) for the release process
 
-### Build and serve the local site
+## Related MITRE Work
 
-1. Update ATT&CK markdown from the STIX content, and generate the output HTML from the markdown: `python3 update-attack.py`. _Note: `update-attack.py`, has many optional command line arguments which affect the behavior of the build. Run `python3 update-attack.py -h` for a list of arguments and an explanation of their functionality._
-2. Serve the HTML to `localhost:8000`: 
-    1. Ensure you are in the root of the repository, e.g. `path/to/attack-website`
-    2. `pelican -l`
+### ATT&CK STIX Data
 
-### (Optional) Build the search module
-1. Install Node.js. This is required in order to compile the search service webpack bundle.
-2. Generate the search service webpack bundle to enable search functionality:
-   ```shell
-   cd attack-search/
-   npm install # installs all third-party dependencies
-   npm run build # generates the webpack bundle
-   npm run copy # copies the resultant bundle to the Pelican server output directory
-   ```
+Data representing the ATT&CK Catalog can be found on the following repositories:
 
-### Installing, building, and serving the site via Docker 
+* [Cyber Threat Intelligence repository](https://github.com/mitre/cti) contains the ATT&CK and [CAPEC](https://capec.mitre.org/) datasets expressed in STIX 2.0 JSON.
+* [attack-stix-data](https://github.com/mitre-attack/attack-stix-data) contains the ATT&CK dataset expressed in STIX 2.1 Collections.
 
-1. Build the docker image:
-    - `docker build -t <your_preferred_image_name> .`
-2. Run a docker container:
-    - `docker run --name <your_preferred_container_name> -d -p <your_preferred_port>:80 <image_name_from_build_command>`
-3. View the site on your preferred localhost port
+### ATT&CK Navigator
 
-## Related MITRE Work
-#### ATT&CK STIX Data
-Data representing the ATT&CK Catalog can be found on the following repositories:
-- [Cyber Threat Intelligence repository](https://github.com/mitre/cti) contains the ATT&CK and [CAPEC](https://capec.mitre.org/) datasets expressed in STIX 2.0 JSON.
-- [attack-stix-data](https://github.com/mitre-attack/attack-stix-data) contains the ATT&CK dataset expressed in STIX 2.1 Collections.
+The ATT&CK Navigator is an open-source tool providing basic navigation and annotation of ATT&CK matrices, something that people are already doing today in tools like Excel. It is designed to be simple and generic - you can use the Navigator to visualize your defensive coverage, your red/blue team planning, the frequency of detected techniques, and more.
 
-#### ATT&CK Navigator
-The ATT&CK Navigator is an open-source tool providing basic navigation and annotation of ATT&CK matrices, something that people are already doing today in tools like Excel. It is designed to be simple and generic - you can use the Navigator to visualize your defensive coverage, your red/blue team planning, the frequency of detected techniques, and more. 
+<https://github.com/mitre-attack/attack-navigator>
 
-https://github.com/mitre-attack/attack-navigator
+### STIX
 
-#### STIX
 Structured Threat Information Expression (STIX<sup>&trade;</sup>) is a language and serialization format used to exchange cyber threat intelligence (CTI).
 
 STIX enables organizations to share CTI with one another in a consistent and machine readable manner, allowing security communities to better understand what computer-based attacks they are most likely to see and to anticipate and/or respond to those attacks faster and more effectively.
 
 STIX is designed to improve many different capabilities, such as collaborative threat analysis, automated threat exchange, automated detection and response, and more.
 
-https://oasis-open.github.io/cti-documentation/
+<https://oasis-open.github.io/cti-documentation/>
 
 ## Notice
+
 Copyright 2015-2023 The MITRE Corporation
 
 Approved for Public Release; Distribution Unlimited. Case Number 19-3504.
@@ -83,7 +57,7 @@ Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
 
-   http://www.apache.org/licenses/LICENSE-2.0
+   <http://www.apache.org/licenses/LICENSE-2.0>
 
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,

nginx.conf

@@ -0,0 +1,18 @@
+server {
+    listen 80;
+    server_name localhost;
+
+    location / {
+        root /var/www/html;
+        index index.html;
+        try_files $uri $uri/ =404;
+    }
+
+    error_page 404 /404.html;
+    location = /40x.html {
+    }
+
+    error_page 500 502 503 504 /50x.html;
+    location = /50x.html {
+    }
+}

test/Dockerfile

@@ -0,0 +1,18 @@
+# Dockerfile
+FROM ubuntu:latest
+
+# Install necessary packages
+RUN apt-get update && apt-get install -y \
+    nginx \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set working directory
+WORKDIR /workspace
+
+# Copy Nginx configuration file
+COPY nginx.conf /etc/nginx/sites-available/default
+
+# Expose port for Nginx
+EXPOSE 80
+
+CMD ["nginx", "-g", "daemon off;"]

test/README.md

@@ -0,0 +1,70 @@
+# Test Environment
+
+This directory contains a Docker-based testing environment for the ATT&CK website.
+The purpose of this environment is to closely emulate the production GitHub Pages environment, using Nginx to serve the static web content.
+This allows developers to catch and fix issues before pushing to GitHub Pages, thereby increasing development efficiency and reducing potential errors in the production environment.
+
+## Prerequisites
+
+Ensure you have the following installed on your local system:
+
+- Docker
+- Node.js and npm
+- Python 3 and pip
+
+## Building the Web Application
+
+Before starting the Docker container, you need to build the static web content locally.
+The web application is composed of two modules: the Pelican content, and the ATT&CK search module.
+
+1. Generate the static web pages (i.e., the Pelican content) by running the following command from the root of the project:
+
+    ```shell
+    python3 update-attack.py --attack-brand --extras --no-test-exitstatus
+    ```
+
+    The static web content will be written to a folder called "output".
+
+2. Build the search module by running the following commands:
+
+    ```shell
+    cd attack-search
+    npm ci
+    npm run build
+    cp dist/search_bundle.js ../output/theme/scripts/
+    cd ..
+    ```
+
+## Using the Docker Test Environment
+
+1. From the `/test` directory, build the Docker image:
+
+    ```shell
+    docker build -t attack-website-test .
+    ```
+
+2. Run the Docker container:
+
+    ```shell
+    docker run -p 80:80 -v $(pwd)/../output:/workspace attack-website-test
+    ```
+
+    This will start a Docker container with the image you built, forward port 80 from the container to your host machine, and mount the "output" directory from your local workspace to the "/workspace" directory inside the container. This allows Nginx to serve the static web content you built.
+
+3. Now, you should be able to view the website by opening a web browser and navigating to `http://localhost`.
+
+To stop the Docker container, press `Ctrl+C` in the terminal where the container is running.
+
+## Helper Script
+
+Alternatively, you can use the `run_test.sh` script to build the Docker image and start the container. Simply run the script from the `/test` directory:
+
+```shell
+./run_test.sh
+```
+
+Ensure that the script has execute permissions. If needed, you can add execute permissions with the following command:
+
+```shell
+chmod +x run_test.sh
+```

test/nginx.conf

@@ -0,0 +1,20 @@
+server {
+    listen 80;
+    listen [::]:80;
+
+    server_name localhost;
+
+    location / {
+        root /workspace;
+        index index.html;
+        try_files $uri $uri/ =404;
+    }
+
+    error_page 404 /404.html;
+    location = /40x.html {
+    }
+
+    error_page 500 502 503 504 /50x.html;
+    location = /50x.html {
+    }
+}

test/run_test.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+docker build -t attack-website-test .
+docker run -p 80:80 -v $(pwd)/../output:/workspace attack-website-test