Update requirements and documentation

CONTRIBUTING.md

@@ -0,0 +1,82 @@
+# Contributing to cldr-runner
+
+Thank you for considering contributions to the `cldr-runner` project!
+
+# Submitting a pull request
+
+You can start work on issues that are not yet part of a [Milestone](https://github.com/cloudera-labs/cldr-runner/milestones) -- anything in our issue tracker that isn't assigned to a Milestone is considered the [backlog](https://github.com/cloudera-labs/cldr-runner/issues?q=is%3Aopen+is%3Aissue+no%3Amilestone). 
+
+Before you start working, please announce that you want to do so by commenting on the issue. _([Create an issue](https://github.com/cloudera-labs/cldr-runner/issues/new?labels=enhancement) if there isn't one yet, and you can also check out our [Discussions](https://github.com/cloudera-labs/cldr-runner/discussions) for ideas.)_ We try to ensure that all active work is assigned to a Milestone in order to keep our backlog accurate.
+
+**When your work is ready for review, create a branch in your own forked repository from the `devel` branch and submit a pull request against `devel`, referencing your the issue.**
+
+As a _best practice_, you can prefix your branches with:
+
+|prefix|Description|Example|
+|------|-----------|-------|
+|`feature/`|A new feature or changes existing to existing code or documentation|`feature/update-some-params`|
+|`fix/`|A non-urgent bug fix|`fix/refactor-some-params`|
+|`hotfix/`|An urgent bug fix|`hotfix/patch-insecure-params`|
+
+> [!NOTE]
+> :fire_extinguisher: A **hotfix** should branch from `main`. It will then be committed to both the `main` and `devel` branches.
+
+# Signing your commits
+
+Note that we require signed commits inline with [Developer Certificate of Origin](https://developercertificate.org/) best-practices for open source collaboration.
+
+A signed commit is a simple one-liner at the end of your commit message that states that you wrote the patch or otherwise have the right to pass the change into open source.  Signing your commits means you agree to:
+
+```
+Developer Certificate of Origin
+Version 1.1
+
+Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+660 York Street, Suite 102,
+San Francisco, CA 94110 USA
+
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+
+Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I
+    have the right to submit it under the open source license
+    indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best
+    of my knowledge, is covered under an appropriate open source
+    license and I have the right under that license to submit that
+    work with modifications, whether created in whole or in part
+    by me, under the same open source license (unless I am
+    permitted to submit under a different license), as indicated
+    in the file; or
+
+(c) The contribution was provided directly to me by some other
+    person who certified (a), (b) or (c) and I have not modified
+    it.
+
+(d) I understand and agree that this project and the contribution
+    are public and that a record of the contribution (including all
+    personal information I submit with it, including my sign-off) is
+    maintained indefinitely and may be redistributed consistent with
+    this project or the open source license(s) involved.
+```
+
+(See [developercertificate.org](https://developercertificate.org/))
+
+To agree, make sure to add line at the end of every git commit message, like this:
+
+```
+Signed-off-by: John Doe <[email protected]>
+```
+
+> [!NOTE]
+> :rocket: TIP! Add the sign-off automatically when creating the commit via the `-s` flag, e.g. `git commit -s`.
+
+# Have questions? Opinions? Comments?
+
+Come find us on our [Discussions](https://github.com/cloudera-labs/cldr-runner/discussions)!

FAQ.md

@@ -0,0 +1,74 @@
+# Frequently Asked Questions
+
+Be sure to check out the [Discussions > Help](https://github.com/cloudera-labs/cldr-runner/discussions/categories/help) category for the latest answers. 
+
+# `ansible-navigator` FAQ
+
+## How do I add _extra variables_ and tags to `ansible-navigator`?
+
+If you want to run a playbook with a given tag, e.g. `-t infra`, then simply add it as a parameter to the `ansible-navigator` commandline. For example, `ansible-navigator run playbook.yml -t infra`. 
+
+Like tags, so you can pass _extra variables_ to `ansible-navigator` and the underlying Ansible command. For example, `ansible-navigator run playbook.yml -e @some_config.yml -e some_var=yes`.
+
+## How do I tell `ansible-navigator` where to find collections and roles?
+
+By default, `cloudera-deploy` expects to use the collections, roles, and libraries within the _execution environment_ container, that is, the `cldr-runner` image. Make sure you do _not_ have `ANSIBLE_COLLECTIONS_PATH` or `ANSIBLE_ROLES_PATH` set or `ansible-navigator` will pick up these environment variables and pass them to the running container. The underlying `ansible` application, like `ansible-playbook` will then pick up these environment variables and attempt to use them if set! 
+
+This behavior is great if you want to use host-based collections, e.g. local development, but you need to ensure that you update the `ansible-navigator.yml` configuration file to mount the host collection and/or role directories into the execution environment container. See [Advanced Usage: Execution Modes](NAVIGATOR.md#advanced-usage-execution-modes) to learn more about these execution modes.
+
+## `ansible-navigator` hangs when I run my playbook. What is going on?
+
+`ansible-navigator` does not handle user prompts when running in the `curses`, text-based UI , so actions in your playbook like:
+
+* Vault passwords
+* SSH passphrases
+* Debugger statements
+
+will not work out-of-the-box. You can enable `ansible-navigator` to run with prompts, but doing so will also disable the TUI and instead run its operations using `stdout`.
+
+Try adding:
+
+```bash
+ansible-navigator run --enable-prompts ...
+```
+
+to your execution to allow `ansible-navigator` to receive your prompt input.
+
+## How can I view a previous `ansible-navigator` run to debug an issue?
+
+`ansible-navigator` can be configured to save execution runs to a directory. You can reload a run by using the `replay` command:
+
+```bash
+ansible-navigator replay <playbook execution run file>.json
+```
+
+Then you can use the TUI to review the plays, tasks, and inventory for the previous run!
+
+You can learn more about [replays](https://ansible.readthedocs.io/projects/navigator/subcommands/#ansible-navigator-subcommands) and their [configuration](https://ansible.readthedocs.io/projects/navigator/settings/#subcommand-replay) in the `ansible-navigator` documentation.
+
+## How can I enable the playbook debugger?
+
+The [playbook debugger](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_debugger.html) is enabled in `ansible-navigator` by setting the debugger and then enabling prompts. For example,
+
+```bash
+ANSIBLE_ENABLE_TASK_DEBUGGER=True ansible-navigator run --enable-prompts main.yml
+```
+
+## How to I configure SSH to avoid a "Failed to connect to new control master" error?
+
+When running connecting to a host via SSH while running `ansible-navigator`, in particular when you are working with Terraform inventory managed by the `cloud.terraform` inventory plugin, you might encounter the following error:
+
+```bash
+Failed to connect to the host via ssh: Control socket connect(/runner/.ansible/cp/b44b170fff): Connection refused
+Failed to connect to new control master
+```
+
+To resolve, be sure to add the following variable to your `ansible-navigator.yml` configuration file:
+
+```yaml
+ansible-navigator:
+  execution-environment:
+    environment-variables:
+      set:
+        ANSIBLE_SSH_CONTROL_PATH: "/dev/shm/cp%%h-%%p-%%r"
+```

MANUAL_SETUP.md

@@ -0,0 +1,66 @@
+# Local Development Environment (Manual Setup)
+
+If you do not wish to run the `local_development.yml` playbook to set up your local development environment, the steps below can guide you through the manual process of getting set to run `ansible-navigator` without a `cldr-runner` _execution environment_.
+
+## Install an Execution Environment
+
+`ansible-navigator` typically runs within an _execution environment_ -- an `ansible-runner`-enabled container. To this end, you will need to have `docker` or an equivalent or `podman` running on your host.
+
+### Docker
+
+* [Windows](https://docs.docker.com/docker-for-windows/install/)
+* [Mac](https://docs.docker.com/docker-for-mac/install/)
+* Linux users, use your package manager
+
+> [!WARNING]
+> Be sure you uninstall any earlier versions of Docker, i.e. `docker`, and install the latest version, i.e. `docker-ce`. See [Install Docker Engine](https://docs.docker.com/engine/install/) for further details.
+
+> [!NOTE]
+> If you have not used Docker before, consider following their quick [Tutorial](https://docs.docker.com/get-started/#start-the-tutorial) to validate it is working and familiarize yourself with the interface.
+
+## Install Git
+
+> [!NOTE]
+> Git is required if you intend to clone the software for local editing. If you just intend to run the automation tools, you may skip this step.
+
+There are excellent instructions for installing Git on all Operating Systems on the [Git website](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
+
+## Install AWS CLI
+
+If you are going to be working with AWS, you will want the latest version of the **AWS CLI**.
+
+* [Windows](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html)
+* [Mac](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-mac.html)
+* Linux users, use your package manager
+
+If this is the first time you are installing the AWS CLI, configure the program by providing your credentials, and test that your credentials work
+
+```bash
+aws configure
+aws iam get-user
+```
+
+Visit the [AWS CLI User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) for further details regarding credential management.
+
+## Install CDP CLI
+
+Get the latest version of the **CDP CLI**.
+
+* [Install CDP CLI](https://docs.cloudera.com/cdp/latest/cli/topics/mc-installing-cdp-client.html)
+
+If this is the first time you are installing the CDP CLI, you will need to configure the program by providing the right credentials, and should then test that your credentials work.
+
+```bash
+cdp configure
+cdp iam get-user
+```
+
+Visit the [CDP CLI User Guide](https://docs.cloudera.com/cdp/latest/cli/topics/mc-configuring-cdp-client-with-the-api-access-key.html) for further details regarding credential management.
+
+## Confirm your SSH Keypair
+
+Ensure that you have a generated SSH keypair for your local profile. Visit the [SSH Keygen How-To](https://www.ssh.com/academy/ssh/keygen) for details.
+
+## Confirm your SSH Agent
+
+Ensure that you have a properly configured SSH Agent. Visit the [SSH Agent How-To](https://www.ssh.com/academy/ssh/keygen#adding-the-key-to-ssh-agent) for details.

NAVIGATOR.md

@@ -0,0 +1,113 @@
+# Installing, Running, Managing, and Troubleshooting `ansible-navigator`
+
+> `ansible-navigator` is a command-line tool and a text-based user interface (TUI) for creating, reviewing, running and troubleshooting Ansible content, including inventories, playbooks, collections, documentation and container images (execution environments).
+
+# Installation
+
+Setting up `ansible-navigator` is easy; you can spin up a new setup in **TWO** steps (plus your OS requirements, see below for important details)!
+
+> [!IMPORTANT]
+> Please note each OS has slightly different requirements for installing `ansible-navigator`. :woozy_face: Read more about [installing `ansible-navigator`](https://ansible.readthedocs.io/projects/navigator/installation/#install-ansible-navigator).
+
+1. Create and activate a new Python `virtualenv`.
+
+   You can name your virtual environment anything you want; by convention, we like to call it `cdp-navigator`.
+
+   ```bash
+   python -m venv ~/cdp-navigator; source ~/cdp-navigator/bin/activate;
+   ```
+
+2. Install the latest `ansible-core` and `ansible-navigator`.
+
+   These tools can be the latest versions, as the actual execution versions are encapsulated in the _execution environment_ container.
+
+   ```bash
+    pip install ansible-core ansible-navigator
+   ```
+
+# Usage
+
+`ansible-navigator` can be viewed as a wrapper around the core Ansible application, like `ansible-playbook`, `ansible-galaxy`, `ansible-doc`, etc. Read more about how to [configure](https://ansible.readthedocs.io/projects/navigator/settings/) your setup and your project execution as well as the [mapping](https://ansible.readthedocs.io/projects/navigator/subcommands/#mapping-ansible-navigator-commands-to-ansible-commands) of and [running](https://ansible.readthedocs.io/projects/navigator/subcommands/) of these subcommands.
+
+## Common commands
+
+| Command | Description |
+|---------|-------------|
+| `ansible-navigator run main.yml -e @config.yml` | Run the `main.yml` playbook with the contents of `config.yml` loaded as _extra variables_. |
+| `ansible-navigator doc cloudera.cloud.env_info` | View the `ansible-docs` of the [`cloudera.cloud.env_info` module](https://cloudera-labs.github.io/cloudera.cloud/env_info_module.html). |
+| `ansible-navigator doc cloudera.cloud.datahub_service -t lookup` | View the `ansible-docs` of the [`cloudera.cloud.datahub_service` lookup](https://wmudge.github.io/cloudera.cloud/datahub_service_lookup.html) plugin. |
+| `ansible-navigator exec -- ansible localhost -m cloudera.cloud.env_info -a 'name=my_env'` | Query the Cloudera Data Platform (CDP) Public Cloud for details on the `my_env` Environment. |
+
+# Advanced Usage: Execution Modes
+
+`ansible-navigator` typically executes Ansible using the Ansible runtime, collections, and roles built into the _execution environment_, but this is not the only way you can use the tool. In fact, there are **four** modes of execution with `ansible-navigator`, each providing a growing degree of control and customization as well as complexity. Yet, `ansible-navigator` provides a common interface to all modes, allowing you to switch seamlessly from one mode to another.
+
+Each mode is enabled by the presence of certain paths (e.g. `ANSIBLE_COLLECTIONS_PATHS`, `./collections`) and configuration parameters (e.g. `--execution-environment`).
+
+## User Mode
+
+| Collections path | Ansible runtime |
+|------------------|-----------------|
+| container | container |
+
+This is the default mode for using _execution environments_ like `cldr-runner`. All executable assets and dependencies are bundled into the image. All that is needed to run is `ansible-navigator` itself; the tool will collect and inject everything into the running container.
+
+## Power User Mode
+
+| Collections path | Ansible runtime |
+|------------------|-----------------|
+| project | container |
+
+If `ansible-navigator` discovers a `./collections` directory in the project, it will mount that directory into the running container and set the in-container Ansible configuration to use it. This allows you to develop and work with collections locally to a project. You can install local collections by using the `-p` flag with `ansible-galaxy`, for example, `ansible-galaxy collections install community.crypto -p ./collections`. Typically, you might keep a local `./collection/requirements.yml` configuration and install via `ansible-galaxy`.  `ansible-navigator` still uses the container for execution.
+
+See the [Placement of Ansible collections](https://ansible.readthedocs.io/projects/navigator/faq/#placement-of-ansible-collections) section for further details.
+
+## Developer Mode
+
+| Collections path | Ansible runtime |
+|------------------|-----------------|
+| host | container |
+
+In this mode, you set the `ANSIBLE_COLLECTIONS_PATH` environment variable in your host's shell, pointing at a host-installed, Ansible collections directory and specify the mount path in the `ansible-navigator.yml` configuration. `ansible-navigator` automatically sets this environment variable in the container, which now points to the custom mounted directory. For example:
+
+```yaml
+ansible-navigator:
+  execution-environment:
+    volume-mounts:
+      - src: "${ANSIBLE_COLLECTIONS_PATH}"
+        dest: "${ANSIBLE_COLLECTIONS_PATH}"
+        options: "Z"
+```
+
+Typically, this collections directory is a mix of Git-cloned and downloaded collections; the former grants you full SCM control. Meanwhile, `ansible-navigator` still uses the container for execution.
+
+See the [Placement of Ansible collections](https://ansible.readthedocs.io/projects/navigator/faq/#placement-of-ansible-collections) section for further details.
+
+The [local development](README.md#local-development) instructions are designed to support this mode. 
+
+## Platform Mode
+
+| Collections path | Ansible runtime |
+|------------------|-----------------|
+| host (or project) | host |
+
+In this final mode, you set both the `ANSIBLE_COLLECTIONS_PATH` to a host-installed collections directory (or use `./collections`) **and** disable the _execution engine_ parameter in the `ansible-navigator.yml` configuration. `ansible-navigator` automatically sets the environment variable yet runs the commands using the host system, not the container runtimes.
+
+```yaml
+ansible-navigator:
+  execution-environment:
+      enabled: False
+```
+
+> [!NOTE]
+> You can set this parameter via the CLI flags, `--ee` or `--execution-environment`.
+
+Arguably, this mode is the most complex, as it requires you to have installed all the Python and system dependencies on your host system; `ansible-navigator` only does the organization and coordination of assets and command, ignoring the container completely.
+
+See the [Execution Environment configuration](https://ansible.readthedocs.io/projects/navigator/settings/#execution-environment) section for further details.
+
+This is a fairly common task then [running CI jobs](https://github.com/cloudera-labs/cloudera.cluster/blob/main/.github/workflows/validate_pr.yml) for individual collections. The [local development](README.md#local-development) instructions are designed to support this mode.
+
+# Troubleshooting
+
+The [Frequently Asked Questions](FAQ.md) guide has a collection of relevant troubleshooting topics. You can also stop by the [Discussion > Help](https://github.com/cloudera-labs/cldr-runner/discussions/categories/help) category for the latest answers.