Merge branch 'v3' into use-cache-on-linter

.github/actions/test-server/action.yaml

@@ -16,7 +16,7 @@ inputs:
       The PAT token can be left empty when building the development version of JIMM.
     required: true
 
-output:
+outputs:
   url:
     description: 'URL where JIMM can be reached.'
     value: "https://jimm.localhost"
@@ -26,6 +26,9 @@ output:
   client-secret:
     description: 'Test client Secret to login to JIMM with a service account.'
     value: "2M2blFbO4GX4zfggQpivQSxwWX1XGgNf"
+  ca-cert:
+    description: 'The CA certificate used to genereate the JIMM server cert.'
+    value: ${{ steps.fetch-cert.outputs.jimm-ca }}
 
 runs:
   using: "composite"
@@ -50,6 +53,14 @@ runs:
       run: make dev-env
       shell: bash
 
+    - name: Retrieve server CA cert.
+      id: fetch-cert
+      run: |
+        echo 'jimm-ca<<EOF' >> $GITHUB_OUTPUT
+        cat ./local/traefik/certs/ca.crt >> $GITHUB_OUTPUT
+        echo 'EOF' >> $GITHUB_OUTPUT
+      shell: bash
+
     - name: Initialise LXD
       run: |
         sudo lxd waitready && \

.vscode/extensions.json

@@ -0,0 +1,6 @@
+{
+    "recommendations": [
+        "golang.go",
+        "babakks.vscode-go-test-suite"
+    ]
+}

CONTRIBUTING.md

@@ -0,0 +1,104 @@
+## Filing Bugs
+File bugs at https://github.com/canonical/jimm/issues.
+
+## Testing
+Many tests in JIMM require real services to be reachable i.e. Postgres, Vault, OpenFGA 
+and an IdP (Identity Provider).
+
+JIMM's docker compose file provides a convenient way of starting these services.
+
+### TLDR
+Run:
+```
+$ make test-env
+$ go test ./...
+```
+
+### Pre-requisite
+To check if your system has all the prequisites installed simply run `make sys-deps`.
+This will check for all test prequisites and inform you how to install them if not installed. 
+You will need to install `make` first with `sudo apt install make`
+
+### Understanding the test suite
+In order to enable testing with Juju's internal suites, it is required to have juju-db 
+(mongod) service installed.
+This can be installed via: `sudo snap install juju-db --channel=4.4/stable`.
+
+Tests inside of `cmd/` and `internal/jujuapi/` are integration based, spinning up JIMM and 
+a Juju controller for testing. To spin up a Juju controller we use the `JujuConnSuite` which 
+in turn uses the [gocheck](http://labix.org/gocheck) test library.
+
+Because of the `JujuConnSuite` and its use in JIMM's test suites, there are 2 test libraries in JIMM:
+- GoCheck based tests, identified in the function signature with `func Test(c *gc.C)`.
+  - These tests normally interact with a Juju controller.
+  - GoCheck should only be used when using the suites in `internal/jimmtest`.
+- Stdlib `testing.T` tests, identified in the function signature with `func Test(t *testing.T)`.
+  - These tests vary in their scope but do not require a Juju controller.
+  - To provide assertions, the project uses [quicktest](https://github.com/frankban/quicktest), 
+    a lean testing library.
+
+Because many tests rely on PostgreSQL, OpenFGA and Vault which are dockerised 
+you may simply run `make test-env` to be integration test ready.
+
+The above command won't start a dockerised instance of JIMM as tests are normally run locally. 
+Instead, to start a dockerised JIMM that will auto-reload on code changes, follow the instructions 
+in `local/README.md`.
+
+### Manual commands
+If using VSCode, we recommend installing the 
+[go-test-suite](https://marketplace.visualstudio.com/items?itemName=babakks.vscode-go-test-suite) 
+extension to enable running these tests from the GUI as you would with normal Go tests and the Go 
+VSCode extension.
+
+Because [gocheck](http://labix.org/gocheck) does not parse the `go test -run` flags, the examples 
+below show how to run individual tests in a suite: 
+```bash
+$ go test -check.f dialSuite.TestDialWithCredentialsStoredInVault`
+$ go test -check.f MyTestSuite
+$ go test -check.f "Test.*Works"
+$ go test -check.f "MyTestSuite.Test.*Works"
+```
+
+For more verbose output, add `check.v` and `check.vv`.
+
+**Note:** The `check.f` command only applies to Go Check tests, any package with both Go Check tests
+and normal `testing.T` tests will result in both sets of tests running. To avoid this look for where 
+Go Check registers its test suite into the Go test runner, normally in a file called `package_test.go`
+and only run that test function.  
+E.g. in `internal/jujuapi` an example command to only run a single suite test would be:
+```
+$ go test ./internal/jujuapi -check.f modelManagerSuite.TestListModelSummaries -run TestPackage ./internal/jujuapi
+```
+
+## Building/Publishing
+Below are instructions on building the various binaries that are part of the project as well as
+some information on how they are published.
+
+### jimmsrv
+To build the JIMM server run `go build ./cmd/jimmsrv`
+
+The JIMM server is published as an OCI image using 
+[Rockcraft](https://documentation.ubuntu.com/rockcraft/en/latest/) 
+(a tool to create OCI images based on Ubuntu).
+
+Run `make rock` to pack the rock. The images are published to the Github repo's container registry
+for later use by the JIMM-k8s charm.
+
+The JIMM server is also available as a snap and can be built with `make jimm-snap`. This snap is 
+not published to the snap store as it is intended to be used as part of a machine charm deployment.
+
+### jimmctl
+To build jimmctl run `go build ./cmd/jimmctl`
+
+The jimmctl tool is published as a [Snap](https://snapcraft.io/jimmctl).
+
+Run `make jimmctl-snap` to build the snap. The snaps are published to the Snap Store 
+from where they can be conveniently installed.
+
+### jaas plugin
+To build the jaas plugin run `go build ./cmd/jaas`
+
+The jaas plugin is published as a [Snap](https://snapcraft.io/jaas).
+
+Run `make jaas-snap` to build the snap. The snaps are published to the Snap Store 
+from where they can be conveniently installed.

Makefile

@@ -152,7 +152,6 @@ help:
 	@echo 'make sys-deps - Install the development environment system packages.'
 	@echo 'make format - Format the source files.'
 	@echo 'make simplify - Format and simplify the source files.'
-	@echo 'make get-local-auth - Get local auth to the API WSS endpoint locally.'
 	@echo 'make rock - Build the JIMM rock.'
 	@echo 'make load-rock - Load the most recently built rock into your local docker daemon.'
 

README.md

@@ -1,111 +1,73 @@
-# Juju Intelligent Model Manager
+# JIMM - Juju Intelligent Model Manager
 
-This service provides the ability to manage multiple juju models. It is
-considered a work in progress.
+[comment]: <> (Update the chat link below with a JIMM specific room)
+<h4 align="center">
+    <a href="https://app.element.io/#/room/#charmhub-juju:ubuntu.com">Chat</a> |
+    <a href="https://canonical-jaas-documentation.readthedocs-hosted.com/en/latest/">Docs</a> |
+    <a href="https://github.com/canonical/jimm-k8s-operator/">Charm</a>
+</h4>
 
-## Installation
+JIMM is a Go based webserver used to provide extra functionality on top of Juju controllers. 
+If you are unfamiliar with Juju, we suggest visiting the [Juju docs](https://juju.is/) - 
+the open source orchestration engine for software operators.
 
-To start using JIMM, first ensure you have a valid Go environment,
-then run the following:
+JIMM provides the ability to manage multiple Juju controllers from a single location with 
+enhanced enterprise functionality.
 
-    go get github.com/canonical/jimm
+JIMM is the central component of JAAS (Juju As A Service), where JAAS is a set of services 
+acting together to enable storing state, storing secrets and auth.
 
-## Go dependencies
+## Features
 
-The project uses Go modules (https://golang.org/cmd/go/#hdr-Module_maintenance) to manage Go
-dependencies. **Note: Go 1.11 or greater needed.**
+JIMM/JAAS provides enterprise level functionality layered on top of your Juju controller like:
+- Federated login via an external identity provider using OAuth2.0 and OIDC.
+- Fine grained access control with the ability to create user groups.
+- A single gateway into your Juju estate.
+- The ability to query for information across all your Juju controllers.
 
-## JIMM versioning
-
-JIMM v0 and v1 follow a different versioning strategy than future releases. JIMM v0 was the initial release and used MongoDB to store state.
-JIMM v1 was an upgrade that switched to using PostgreSQL for storing state but still retained similar functionality to v0.
-These versions worked with Juju v2 and v3.
-
-Since a refresh of the project, there was an addition of delegated authorization in JIMM. This means that users are authenticated and authorized in JIMM before requests are forwarded to Juju. This work encompassed a breaking change and required changes in Juju (requiring a Juju controller of at least version 3.3). To better align JIMM with Juju it was decided to switch our versioning strategy to align with Juju. As a result of this, there is no JIMM v2 and instead from JIMM v3, the versioning strategy we follow is to match JIMM's to the Juju major versions we support. As an example, JIMM v3 can speak to Juju v3 controllers AND the last minor version of the previous major (Juju v2.9) for migration purposes.
-
-## Development environment
+For a full overview of the capabilties, check out 
+[the docs](https://canonical-jaas-documentation.readthedocs-hosted.com/en/latest/explanation/jaas_overview/).
 
-### Local:
 
-A couple of system packages are required in order to set up a development
-environment. To install them, run the following:
-`make sysdeps`
+## Dependencies
 
-At this point, from the root of this branch, run the command:
-`make install`
+The project uses [Go modules](https://golang.org/cmd/go/#hdr-Module_maintenance) to manage 
+Go dependencies. **Note: Go 1.11 or greater needed.**
 
-The command above builds and installs the JIMM binaries, and places
-them in `$GOPATH/bin`. This is the list of the installed commands:
-
-- jemd: start the JIMM server;
-- jaas-admin: perform admin commands on JIMM;
-
-### Docker compose:
-See [here](./local/README.md) on how to get started.
-
-## Testing
+A brief explanation of the various services that JIMM depends on is below:
+- [Vault](https://www.vaultproject.io/): User cloud-credentials and private keys are stored in Vault. Cloud-credentials are API keys that 
+enable Juju to communicate with a cloud's API.
+- [PostgreSQL](https://www.postgresql.org/): All non-sensitive state is stored in Postgres.
+- [OpenFGA](https://openfga.dev/): A distributed authorisation server where authorisation rules are stored and queried 
+using relation based access control.
+- IdP: An identity provider which supports OAuth2.0 and OIDC.
 
-## TLDR
-Run:
-```
-$ make test-env
-$ go test ./...
-```
-### Pre-requisite
-To check if your system has all the prequisites installed simply run `make sysdeps`.
-This will check for all test prequisites and inform you how to install them if not installed. 
-You will need to install `make` first with `sudo apt install make`
-
-### Understanding the test suite
-As the juju controller internal suites start their our mongod instances, it is required to have juju-db (mongod).
-This can be installed via: `sudo snap install juju-db`.
-The latest JIMM has an upgraded dependency on Juju which requires in turn requires juju-db from channel `4.4/stable`,
- this can be installed with `sudo snap install juju-db --channel=4.4/stable`
-
-Tests inside of `cmd/` create a JIMM server and test the jimmctl and jaas CLI packages. The Juju CLI requires that it connects to
-an HTTPS server, but these tests also start a Juju controller which expects to be able to fetch a JWKS and macaroon publickey
-from JIMM (which is running as an HTTPS server). This would normally result in a TLS certificate error, however JIMM will
-attempt to use a custom self-signed cert from the certificate generated in `local/traefik/certs`. The make command `make certs` will generate these certs and place the CA in your system's cert pool which will be picked up by the Go HTTP client.
-
-The rest of the suite relies on PostgreSQL, OpenFGA and Hashicorp Vault which are dockerised
-and as such you may simple run `make test-env` to be integration test ready.
-The above command won't start a dockerised instance of JIMM as tests are normally run locally. Instead, to start a 
-dockerised JIMM that will auto-reload on code changes, follow the instructions in `local/README.md`.
-
-### Manual commands
-The tests utilise [go.check](http://labix.org/gocheck) for suites and you may run tests individually like so: 
-```bash
-$ go test -check.f dialSuite.TestDialWithCredentialsStoredInVault`
-$ go test -check.f MyTestSuite
-$ go test -check.f "Test.*Works"
-$ go test -check.f "MyTestSuite.Test.*Works"
-```
-
-For more verbose output, use `-check.v` and `-check.vv`
+## JIMM versioning
 
+The versioning strategy we follow is to match JIMM's major version to the corresponding
+Juju major version we support.
 
-### Make
-Run `make check` to test the application.
-Run `make help` to display help about all the available make targets.
+Additionally JIMM will also support Juju's last minor version of the previous major to 
+support model migrations.
 
-## Local QA
+E.g. JIMM v3 supports Juju v3 controllers AND the last minor version 
+of the previous major, v2.9.
 
-To start a local server for QA purposes do the following:
+For more information on JIMM's history and previous version strategy see [here](./doc/versioning.md).
 
-    sudo cp tools/jimmqa.crt /usr/local/share/ca-certificates
-    sudo update-ca-certificates
-    make server
+## Binaries
 
-This will start JIMM server running on localhost:8082 which is configured
-to use https://api.staging.jujucharms.com/identity as its identity
-provider.
+This repository contains 3 binaries:
+- jimmsrv: The JIMM server.
+- jimmctl: A CLI tool for administrators of JIMM to view audit logs, manage permissions, etc. 
+Available as a snap.
+- jaas: A plugin for the Juju CLI, extend the base set of command with extra functionality when
+communicating with a JAAS environment. 
 
-To add the new JIMM to your juju environment use the command:
+## Development environment
 
-   juju login localhost:8082 -c local-jaas
+See [here](./local/README.md) on how to get started.
 
-To bootstrap a new controller and add it to the local JIMM use the
-following commands:
+## Testing
 
-    juju bootstrap --config identity-url=https://api.staging.jujucharms.com/identity --config allow-model-access=true <cloud>/<region> <controller-name>
-    jaas-admin --jimm-url https://localhost:8082 add-controller <owner>/<controller-name>
+See [here](./CONTRIBUTING.md) on how to get started.

doc/versioning.md

@@ -0,0 +1,21 @@
+## Version History
+
+JIMM v0 and v1 follow a different versioning strategy than future releases. JIMM v0 was the initial
+release and used MongoDB to store state. JIMM v1 was an upgrade that switched to using PostgreSQL 
+for storing state but still retained similar functionality to v0.  
+These versions worked with Juju v2 and v3.
+
+Subsequently JIMM introduced a large shift in how the service worked:
+- JIMM now acts as a proxy between all client and Juju controller interactions. Previously 
+users were redirected to a Juju controller.
+- Juju controllers support JWT login where secure tokens are issued by JIMM.
+- JIMM acts as an authorisation gateway creating trusted short-lived JWT tokens to authorize 
+user actions against Juju controllers.
+
+The above work encompassed a breaking change and required changes in Juju (requiring a 
+Juju controller of at least version 3.3). 
+
+Further, to better align the two projects, JIMM's versioning now aligns with Juju.
+
+As a result of this, there is no JIMM v2 and instead from JIMM v3, the versioning strategy 
+we follow is to match JIMM's major version to the corresponding Juju major version we support.

internal/jimm/cloudcredential.go

@@ -342,6 +342,9 @@ func (j *JIMM) GetCloudCredentialAttributes(ctx context.Context, user *openfga.U
 		err = errors.E(op, err)
 		return
 	}
+	if len(attrs) == 0 {
+		return map[string]string{}, nil, nil
+	}
 
 	if hidden {
 		return
@@ -377,8 +380,5 @@ func (j *JIMM) getCloudCredentialAttributes(ctx context.Context, cred *dbmodel.C
 	if err != nil {
 		return nil, errors.E(op, err)
 	}
-	if len(attr) == 0 && cred.AuthType != "empty" {
-		return nil, errors.E(op, errors.CodeNotFound, "cloud-credential attributes not found")
-	}
 	return attr, nil
 }