CSS-10401 cleanup repo (#1331)

* improve readme and supporting docs

* PR tweaks

* Add a line explaining building/publishing section

* put jimm history in a separate doc

* add project links

.vscode/extensions.json

@@ -0,0 +1,6 @@
+{
+    "recommendations": [
+        "golang.go",
+        "babakks.vscode-go-test-suite"
+    ]
+}

CONTRIBUTING.md

@@ -0,0 +1,104 @@
+## Filing Bugs
+File bugs at https://github.com/canonical/jimm/issues.
+
+## Testing
+Many tests in JIMM require real services to be reachable i.e. Postgres, Vault, OpenFGA 
+and an IdP (Identity Provider).
+
+JIMM's docker compose file provides a convenient way of starting these services.
+
+### TLDR
+Run:
+```
+$ make test-env
+$ go test ./...
+```
+
+### Pre-requisite
+To check if your system has all the prequisites installed simply run `make sys-deps`.
+This will check for all test prequisites and inform you how to install them if not installed. 
+You will need to install `make` first with `sudo apt install make`
+
+### Understanding the test suite
+In order to enable testing with Juju's internal suites, it is required to have juju-db 
+(mongod) service installed.
+This can be installed via: `sudo snap install juju-db --channel=4.4/stable`.
+
+Tests inside of `cmd/` and `internal/jujuapi/` are integration based, spinning up JIMM and 
+a Juju controller for testing. To spin up a Juju controller we use the `JujuConnSuite` which 
+in turn uses the [gocheck](http://labix.org/gocheck) test library.
+
+Because of the `JujuConnSuite` and its use in JIMM's test suites, there are 2 test libraries in JIMM:
+- GoCheck based tests, identified in the function signature with `func Test(c *gc.C)`.
+  - These tests normally interact with a Juju controller.
+  - GoCheck should only be used when using the suites in `internal/jimmtest`.
+- Stdlib `testing.T` tests, identified in the function signature with `func Test(t *testing.T)`.
+  - These tests vary in their scope but do not require a Juju controller.
+  - To provide assertions, the project uses [quicktest](https://github.com/frankban/quicktest), 
+    a lean testing library.
+
+Because many tests rely on PostgreSQL, OpenFGA and Vault which are dockerised 
+you may simply run `make test-env` to be integration test ready.
+
+The above command won't start a dockerised instance of JIMM as tests are normally run locally. 
+Instead, to start a dockerised JIMM that will auto-reload on code changes, follow the instructions 
+in `local/README.md`.
+
+### Manual commands
+If using VSCode, we recommend installing the 
+[go-test-suite](https://marketplace.visualstudio.com/items?itemName=babakks.vscode-go-test-suite) 
+extension to enable running these tests from the GUI as you would with normal Go tests and the Go 
+VSCode extension.
+
+Because [gocheck](http://labix.org/gocheck) does not parse the `go test -run` flags, the examples 
+below show how to run individual tests in a suite: 
+```bash
+$ go test -check.f dialSuite.TestDialWithCredentialsStoredInVault`
+$ go test -check.f MyTestSuite
+$ go test -check.f "Test.*Works"
+$ go test -check.f "MyTestSuite.Test.*Works"
+```
+
+For more verbose output, add `check.v` and `check.vv`.
+
+**Note:** The `check.f` command only applies to Go Check tests, any package with both Go Check tests
+and normal `testing.T` tests will result in both sets of tests running. To avoid this look for where 
+Go Check registers its test suite into the Go test runner, normally in a file called `package_test.go`
+and only run that test function.  
+E.g. in `internal/jujuapi` an example command to only run a single suite test would be:
+```
+$ go test ./internal/jujuapi -check.f modelManagerSuite.TestListModelSummaries -run TestPackage ./internal/jujuapi
+```
+
+## Building/Publishing
+Below are instructions on building the various binaries that are part of the project as well as
+some information on how they are published.
+
+### jimmsrv
+To build the JIMM server run `go build ./cmd/jimmsrv`
+
+The JIMM server is published as an OCI image using 
+[Rockcraft](https://documentation.ubuntu.com/rockcraft/en/latest/) 
+(a tool to create OCI images based on Ubuntu).
+
+Run `make rock` to pack the rock. The images are published to the Github repo's container registry
+for later use by the JIMM-k8s charm.
+
+The JIMM server is also available as a snap and can be built with `make jimm-snap`. This snap is 
+not published to the snap store as it is intended to be used as part of a machine charm deployment.
+
+### jimmctl
+To build jimmctl run `go build ./cmd/jimmctl`
+
+The jimmctl tool is published as a [Snap](https://snapcraft.io/jimmctl).
+
+Run `make jimmctl-snap` to build the snap. The snaps are published to the Snap Store 
+from where they can be conveniently installed.
+
+### jaas plugin
+To build the jaas plugin run `go build ./cmd/jaas`
+
+The jaas plugin is published as a [Snap](https://snapcraft.io/jaas).
+
+Run `make jaas-snap` to build the snap. The snaps are published to the Snap Store 
+from where they can be conveniently installed.

Makefile

@@ -152,7 +152,6 @@ help:
 	@echo 'make sys-deps - Install the development environment system packages.'
 	@echo 'make format - Format the source files.'
 	@echo 'make simplify - Format and simplify the source files.'
-	@echo 'make get-local-auth - Get local auth to the API WSS endpoint locally.'
 	@echo 'make rock - Build the JIMM rock.'
 	@echo 'make load-rock - Load the most recently built rock into your local docker daemon.'
 

README.md

@@ -1,111 +1,73 @@
-# Juju Intelligent Model Manager
+# JIMM - Juju Intelligent Model Manager
 
-This service provides the ability to manage multiple juju models. It is
-considered a work in progress.
+[comment]: <> (Update the chat link below with a JIMM specific room)
+<h4 align="center">
+    <a href="https://app.element.io/#/room/#charmhub-juju:ubuntu.com">Chat</a> |
+    <a href="https://canonical-jaas-documentation.readthedocs-hosted.com/en/latest/">Docs</a> |
+    <a href="https://github.com/canonical/jimm-k8s-operator/">Charm</a>
+</h4>
 
-## Installation
+JIMM is a Go based webserver used to provide extra functionality on top of Juju controllers. 
+If you are unfamiliar with Juju, we suggest visiting the [Juju docs](https://juju.is/) - 
+the open source orchestration engine for software operators.
 
-To start using JIMM, first ensure you have a valid Go environment,
-then run the following:
+JIMM provides the ability to manage multiple Juju controllers from a single location with 
+enhanced enterprise functionality.
 
-    go get github.com/canonical/jimm
+JIMM is the central component of JAAS (Juju As A Service), where JAAS is a set of services 
+acting together to enable storing state, storing secrets and auth.
 
-## Go dependencies
+## Features
 
-The project uses Go modules (https://golang.org/cmd/go/#hdr-Module_maintenance) to manage Go
-dependencies. **Note: Go 1.11 or greater needed.**
+JIMM/JAAS provides enterprise level functionality layered on top of your Juju controller like:
+- Federated login via an external identity provider using OAuth2.0 and OIDC.
+- Fine grained access control with the ability to create user groups.
+- A single gateway into your Juju estate.
+- The ability to query for information across all your Juju controllers.
 
-## JIMM versioning
-
-JIMM v0 and v1 follow a different versioning strategy than future releases. JIMM v0 was the initial release and used MongoDB to store state.
-JIMM v1 was an upgrade that switched to using PostgreSQL for storing state but still retained similar functionality to v0.
-These versions worked with Juju v2 and v3.
-
-Since a refresh of the project, there was an addition of delegated authorization in JIMM. This means that users are authenticated and authorized in JIMM before requests are forwarded to Juju. This work encompassed a breaking change and required changes in Juju (requiring a Juju controller of at least version 3.3). To better align JIMM with Juju it was decided to switch our versioning strategy to align with Juju. As a result of this, there is no JIMM v2 and instead from JIMM v3, the versioning strategy we follow is to match JIMM's to the Juju major versions we support. As an example, JIMM v3 can speak to Juju v3 controllers AND the last minor version of the previous major (Juju v2.9) for migration purposes.
-
-## Development environment
+For a full overview of the capabilties, check out 
+[the docs](https://canonical-jaas-documentation.readthedocs-hosted.com/en/latest/explanation/jaas_overview/).
 
-### Local:
 
-A couple of system packages are required in order to set up a development
-environment. To install them, run the following:
-`make sysdeps`
+## Dependencies
 
-At this point, from the root of this branch, run the command:
-`make install`
+The project uses [Go modules](https://golang.org/cmd/go/#hdr-Module_maintenance) to manage 
+Go dependencies. **Note: Go 1.11 or greater needed.**
 
-The command above builds and installs the JIMM binaries, and places
-them in `$GOPATH/bin`. This is the list of the installed commands:
-
-- jemd: start the JIMM server;
-- jaas-admin: perform admin commands on JIMM;
-
-### Docker compose:
-See [here](./local/README.md) on how to get started.
-
-## Testing
+A brief explanation of the various services that JIMM depends on is below:
+- [Vault](https://www.vaultproject.io/): User cloud-credentials and private keys are stored in Vault. Cloud-credentials are API keys that 
+enable Juju to communicate with a cloud's API.
+- [PostgreSQL](https://www.postgresql.org/): All non-sensitive state is stored in Postgres.
+- [OpenFGA](https://openfga.dev/): A distributed authorisation server where authorisation rules are stored and queried 
+using relation based access control.
+- IdP: An identity provider which supports OAuth2.0 and OIDC.
 
-## TLDR
-Run:
-```
-$ make test-env
-$ go test ./...
-```
-### Pre-requisite
-To check if your system has all the prequisites installed simply run `make sysdeps`.
-This will check for all test prequisites and inform you how to install them if not installed. 
-You will need to install `make` first with `sudo apt install make`
-
-### Understanding the test suite
-As the juju controller internal suites start their our mongod instances, it is required to have juju-db (mongod).
-This can be installed via: `sudo snap install juju-db`.
-The latest JIMM has an upgraded dependency on Juju which requires in turn requires juju-db from channel `4.4/stable`,
- this can be installed with `sudo snap install juju-db --channel=4.4/stable`
-
-Tests inside of `cmd/` create a JIMM server and test the jimmctl and jaas CLI packages. The Juju CLI requires that it connects to
-an HTTPS server, but these tests also start a Juju controller which expects to be able to fetch a JWKS and macaroon publickey
-from JIMM (which is running as an HTTPS server). This would normally result in a TLS certificate error, however JIMM will
-attempt to use a custom self-signed cert from the certificate generated in `local/traefik/certs`. The make command `make certs` will generate these certs and place the CA in your system's cert pool which will be picked up by the Go HTTP client.
-
-The rest of the suite relies on PostgreSQL, OpenFGA and Hashicorp Vault which are dockerised
-and as such you may simple run `make test-env` to be integration test ready.
-The above command won't start a dockerised instance of JIMM as tests are normally run locally. Instead, to start a 
-dockerised JIMM that will auto-reload on code changes, follow the instructions in `local/README.md`.
-
-### Manual commands
-The tests utilise [go.check](http://labix.org/gocheck) for suites and you may run tests individually like so: 
-```bash
-$ go test -check.f dialSuite.TestDialWithCredentialsStoredInVault`
-$ go test -check.f MyTestSuite
-$ go test -check.f "Test.*Works"
-$ go test -check.f "MyTestSuite.Test.*Works"
-```
-
-For more verbose output, use `-check.v` and `-check.vv`
+## JIMM versioning
 
+The versioning strategy we follow is to match JIMM's major version to the corresponding
+Juju major version we support.
 
-### Make
-Run `make check` to test the application.
-Run `make help` to display help about all the available make targets.
+Additionally JIMM will also support Juju's last minor version of the previous major to 
+support model migrations.
 
-## Local QA
+E.g. JIMM v3 supports Juju v3 controllers AND the last minor version 
+of the previous major, v2.9.
 
-To start a local server for QA purposes do the following:
+For more information on JIMM's history and previous version strategy see [here](./doc/versioning.md).
 
-    sudo cp tools/jimmqa.crt /usr/local/share/ca-certificates
-    sudo update-ca-certificates
-    make server
+## Binaries
 
-This will start JIMM server running on localhost:8082 which is configured
-to use https://api.staging.jujucharms.com/identity as its identity
-provider.
+This repository contains 3 binaries:
+- jimmsrv: The JIMM server.
+- jimmctl: A CLI tool for administrators of JIMM to view audit logs, manage permissions, etc. 
+Available as a snap.
+- jaas: A plugin for the Juju CLI, extend the base set of command with extra functionality when
+communicating with a JAAS environment. 
 
-To add the new JIMM to your juju environment use the command:
+## Development environment
 
-   juju login localhost:8082 -c local-jaas
+See [here](./local/README.md) on how to get started.
 
-To bootstrap a new controller and add it to the local JIMM use the
-following commands:
+## Testing
 
-    juju bootstrap --config identity-url=https://api.staging.jujucharms.com/identity --config allow-model-access=true <cloud>/<region> <controller-name>
-    jaas-admin --jimm-url https://localhost:8082 add-controller <owner>/<controller-name>
+See [here](./CONTRIBUTING.md) on how to get started.

doc/versioning.md

@@ -0,0 +1,21 @@
+## Version History
+
+JIMM v0 and v1 follow a different versioning strategy than future releases. JIMM v0 was the initial
+release and used MongoDB to store state. JIMM v1 was an upgrade that switched to using PostgreSQL 
+for storing state but still retained similar functionality to v0.  
+These versions worked with Juju v2 and v3.
+
+Subsequently JIMM introduced a large shift in how the service worked:
+- JIMM now acts as a proxy between all client and Juju controller interactions. Previously 
+users were redirected to a Juju controller.
+- Juju controllers support JWT login where secure tokens are issued by JIMM.
+- JIMM acts as an authorisation gateway creating trusted short-lived JWT tokens to authorize 
+user actions against Juju controllers.
+
+The above work encompassed a breaking change and required changes in Juju (requiring a 
+Juju controller of at least version 3.3). 
+
+Further, to better align the two projects, JIMM's versioning now aligns with Juju.
+
+As a result of this, there is no JIMM v2 and instead from JIMM v3, the versioning strategy 
+we follow is to match JIMM's major version to the corresponding Juju major version we support.

internal/jujuapi/package_test.go

@@ -8,6 +8,7 @@ import (
 	jujutesting "github.com/juju/juju/testing"
 )
 
+// Registers Go Check tests into the Go test runner.
 func TestPackage(t *testing.T) {
 	jujutesting.MgoTestPackage(t)
 }

local/README.md

@@ -13,30 +13,20 @@ used for integration testing within the JIMM test suite.
 
 The service is started using Docker Compose, the following services should be started:
 - JIMM (only started in the dev profile)
+- Traefik (only started in the dev profile)
 - Vault
 - Postgres
 - OpenFGA
-- Traefik
 
-> Any changes made inside the repo will automatically restart the JIMM server via a volume mount. So there's no need
-to re-run the compose continuously, but note, if you do bring the compose down, remove the volumes otherwise
-vault will not behave correctly, this can be done via `docker compose down -v`
-
-Now please checkout the [Authentication Steps](#authentication-steps) to authenticate postman for local testing & Q/A.
-
-# Q/A Using Postman
-#### Setup
-1. Run `make get-local-auth`
-2. Head to postman and follow the instructions given by get-local-auth.
-#### Facades in Postman
-You will see JIMM's controller WS API broken up into separate WS requests.
-This is intentional.
-Inside of each WS request will be a set of `saved messages` (on the right-hand side), these are the calls to facades for the given API under that request.
-
-The `request name` represents the literal WS endpoint, i.e., `API = /api`.
-
-> Remember to run the `Login` message when spinning up a new WS connection, otherwise you will not be able to send subsequent calls to this WS.
+Some notes on the setup:
+- Local images are created in the repo's `/local/<service>` folder where any init scripts are defined for each service using the service's upstream docker image.
+- The docker compose has a base at `compose-common.yaml` for common elements to reduce duplication.
+- The compose has 2 additional profiles (dev and test). 
+  - Starting the compose with no profile will spin up the necessary components for testing.
+  - The dev profile will start JIMM in a container using [air](https://github.com/air-verse/air), a tool for auto-reloading Go code when the source changes.
+  - The test profile will start JIMM by pulling a version of the JIMM image from a container registry, useful in integration tests.
 
+> Any changes made inside the repo will automatically restart the JIMM server via a volume mount + air. So there's no need to re-run the compose continuously.
 
 # Q/A Using jimmctl
 
@@ -47,7 +37,7 @@ The `request name` represents the literal WS endpoint, i.e., `API = /api`.
 1. The following commands might need to be run to work around an [LXC networking
    issue](https://github.com/docker/for-linux/issues/103#issuecomment-383607773):
    `sudo iptables -F FORWARD && sudo iptables -P FORWARD ACCEPT`.
-2. Install Juju: `sudo snap install juju --channel=3.5/stable` (minimum Juju version is `3.5`).
+2. Install Juju: `sudo snap install juju --channel=3.5/stable` (minimum required Juju version is `3.5`).
 3. Install JQ: `sudo snap install jq`.
 
 ## All-In-One scripts