Skip to content

receive github hook, notify agent, receive task results, notify github

License

Notifications You must be signed in to change notification settings

zotoio/github-task-manager

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

github-task-manager

npm version Build Status Code Climate Test Coverage Greenkeeper badge Commitizen friendly semantic-release Docker Build Status

receive github hook, notify agent, receive task results, notify github (Unofficial)

Aims

Create an asynchronous CI agnostic mechanism for running custom test stage gates for github pull requests.

  • allow team leads to configure task sequences without leaving github
  • allow developers to see output from tasks without leaving github
  • trigger multiple jobs in parallel and indicate pending status on pr checks
  • then add results for each back to pull request check/comments as they complete
  • make extensible for other github event handling
  • stateless and deployable to Kubernetes at scale

Design

  • Deploy two functions to lambda via serverless framework ('gtmGithubHook', 'gtmGithubResults')
  • github PR open hook pointed at 'gtmGithubHook' lambda which adds event to a 'PendingQueue' SQS queue
  • agent(s) watch SQS for new test execution jobs
  • agent notifies 'ResultsQueue' SQS queue that a task has started (lambda updates github PR state)
  • agent triggers CI PR build, deploy and tests (plugin for each CI type)
  • agent formats and adds test results to 'ResultsQueue' SQS queue
  • agent notifies SNS topic which triggers 'gtmGithubResults' lambda
  • 'gtmGithubResults' lambda posts results to github pull request.

flow

Install

Environment variable description
GTM_AWS_KMS_KEY_ID aws kms key id
GTM_CRYPT_GITHUB_TOKEN encrypted access token for accessing github
GTM_CRYPT_GITHUB_WEBHOOK_SECRET encrypted shared secret from github webook config
GTM_CRYPT_AWS_ACCESS_KEY_ID encrypted aws key id - for agent only
GTM_CRYPT_AWS_SECRET_ACCESS_KEY encrypted aws secret - for agent only
GTM_CRYPT_AGENT_AWS_SECRET_ACCESS_KEY secret key for agent
GTM_CRYPT_AGENT_AWS_ACCESS_KEY_ID access key for agent
GTM_CRYPT_JENKINS_TOKEN encrypted token
GTM_CRYPT_TEAMCITY_PASSCODE encrypted teamcity executor passcode
GTM_CRYPT_SONAR_LOGIN encrypted sonar access token
GTM_CRYPT_SONAR_GITHUB_OAUTH encrypted github token for sonar to post comments and status
GTM_CRYPT_DOCKER_REG_PASSWORD encrypted docker private registry password
GTM_AWS_REGION awsregion to create resources in
GTM_SQS_PENDING_QUEUE name of SQS queue for new event
GTM_SQS_RESULTS_QUEUE name of SQS queue for results
GTM_SNS_RESULTS_TOPIC name of SNS topic for result ping
GTM_GITHUB_HOST api hostname can be updated for github enterprise
GTM_GITHUB_DEBUG debug mode for api calls
GTM_GITHUB_TIMEOUT github api timeout
GTM_GITHUB_PATH_PREFIX path prefix for github enterprise
GTM_GITHUB_PROXY github api client proxy
GTM_TASK_CONFIG_FILENAME filename in repo to look for for task config - default is .githubTaskManager
AWS_PROXY URL of proxy to use for network requests. Optional
GTM_AGENT_PORT defaults to 9091
GTM_JENKINS_USER login for jenkins executor
GTM_JENKINS_URL url executor uses to talk to jenkins
GTM_JENKINS_CSRF is csrf enabled? true or false
GTM_TEAMCITY_USER teamcity executor user
GTM_TEAMCITY_URL teamcity api url
GTM_DOCKER_IMAGE_WHITELIST comma separated list of regex of allows docker images eg. alpine:*,bash:latest
GTM_DOCKER_IMAGE_WHITELIST_FILE use an optional docker whitelist file .dockerImageWhitelistExample
GTM_DOCKER_COMMANDS_ALLOWED default is false, set to true to enable docker executor
GTM_DOCKER_ALLOW_PULL allow agent to pull from registry
GTM_DOCKER_DEFAULT_WORKER_IMAGE for running ci tasks, default is zotoio/gtm-worker:latest
GTM_DOCKER_REG_USERNAME username for docker private registry
GTM_DOCKER_REG_SERVER hostname for docker private registry
IAM_ENABLED agent host uses IAM ?
LAUNCHDARKLY_API_TOKEN token for launchdarkly sass executor
GTM_LOGSTASH_HOST optional logstash host for elasticsearch analysis
GTM_LOGSTASH_PORT optional logstash port
GTM_SONAR_HOST_URL sonar host url to connect to
GTM_SONAR_PROJECTNAME_PREFIX prefix if reporting to sonarqube
GTM_SONAR_ANALYSIS_MODE mode for sonar runner, default preview for PRs
GTM_SONAR_SOURCES default source dir is src
GTM_SONAR_JAVA_BINARIES default is target
GTM_SONAR_MODULES comma separated modules
GTM_SONAR_GITHUB_ENDPOINT optional enterprise github api url
GTM_TASK_CONFIG_DEFAULT_URL url to default sample config used when repo is missing .githubTaskManager.json
GTM_TASK_CONFIG_DEFAULT_MESSAGE_PATH path to markdown comment file added to PRs when repo is missing .githubTaskManager.json
GTM_DYNAMO_TABLE_EVENTS DynamoDB table to store event summaries
GTM_DYNAMO_TABLE_AGENTS DynamoDB table to store agent summaries
GTM_AWS_VPC_ID vpc id - only required for ddb endpoints
GTM_BASE_URL Base url used to render links to agent ui - eg elb cname
GTM_WELCOME_MESSAGE_ENABLED If not 'false', send a warning message on unconfigured repository pull requests
GTM_S3_DEPENDENCY_BUCKET aws s3 storage of build dependencies
GTM_AWS_S3_PROXY https_proxy for aws s3
GTM_REPO_BLACKLIST comma separated list of regex to blackist repo names from triggering events
GTM_SLS_EXECUTOR_AWS_STAGE stage override from default calculation of dev/test
GTM_SLS_EXECUTOR_AWS_REGION aws region for lambdas default ap-southeast-2
GTM_SLS_EXECUTOR_AWS_EXECUTION_ROLE docker serverless lambda execution role
GTM_SLS_EXECUTOR_DEPLOY_MODE deploy multiple lambdas 'parallel' (default) or 'sequential'
GTM_SLS_EXECUTOR_SNS_ERROR_TOPIC_ARN sns topic to notify on serverless error
GTM_SLS_EXECUTOR_APIGW_ENDPOINT_TYPE EDGE or REGIONAL or PRIVATE
GTM_SLS_EXECUTOR_APIGW_DOMAIN_SUFFIX reverse proxy domain name that will have apiId added as subdomain. eg. lambda.mysuffix.com will result in [apiId].lambda.mysuffix.com
GTM_SLS_EXECUTOR_HTTP_PROXY proxy passed to serverless executor
GTM_SLS_EXECUTOR_NO_PROXY no_proxy passed to serverless executor
GTM_SLS_EXECUTOR_VPC_ID vpc id for private apigw endpoints
GTM_SLS_EXECUTOR_VPC_SECURITY_GROUP_ID vpc security group id
GTM_SLS_EXECUTOR_VPC_SUBNET_A vpc az subnet
GTM_SLS_EXECUTOR_VPC_SUBNET_B vpc az subnet
GTM_SLS_EXECUTOR_VPC_SUBNET_C vpc az subnet
GTM_SLS_EXECUTOR_AWS_KMS_KEY_ID kms key id for sls env var encryption
GTM_SLS_EXECUTOR_CONFIG_TYPE 'ssm' (aws parameter store) or 'spring' or 'dotenv'
GTM_SLS_EXECUTOR_SPRING_CONFIG_ENDPOINT endpoint url of spring config server eg. http://spring:8888 if GTM_SLS_EXECUTOR_CONFIG_TYPE is 'spring'
GTM_WORKER_SCRIPTS_CLONE for docker executors using https://github.com/zotoio/gtm-worker based image - url of git repo to overlay on workspace eg. https://github.com/zotoio/gtm-worker-scripts.git
GTM_WORKER_SCRIPTS_PATH directory within scripts clone repo to overlay

important: values of env vars prefixed with GTM_CRYPT_* must be created via npm run sls-encrypt [name] [value]

Configure and deploy

  • run: npm run sls-deploy - note that this will create aws re$ources..
  • capture the hook url output in console and add to github repo pull request conf
  • run: npm run sls-logs-hook or npm run sls-logs-results to tail the logs
  • create a .githubTaskManager.json in your repo per https://github.com/zotoio/github-task-manager/wiki/Creating-a-Task-Configuration
  • start an agent locally using npm run build && npm start agent (or use docker/k8s)
  • create a pull request and confirm the hook is being hit and agent processes event

Docker and Kubernetes agents

You can run the latest image from docker hub: https://hub.docker.com/r/zotoio/github-task-manager

npm run docker-hub-run

..or run using the local checkout and tail logs:

npm run docker-local-bounce

..or if you have a k8s cluster and kubectl configured:

npm run k8s-apply
npm run k8s-delete

note that these k8s npm script inject vars from .env into the manifest

k8s

a starting point k8s manifest is in ./k8s/k8s-gtm-agent.yml

Agent Configuration

Running the Agent

The agent uses environment variables to configure itself as well as any executors running within it. The below environment variables are required for basic operation of the GTM Agent.

TODO: Provide Variables

Agent Homepage

The GTM Agent provides an information page summarising the ongoing operation of the agent. The page is available on port 9091 by default.

Plugins

Task executors for Jenkins, Teamcity, Travis, Http, Docker are in progress. Custom task executors can be added by adding this project as a dependency, and registering new Executors and EventHandlers. Please see https://github.com/zotoio/gtm-agent for an example that you can fork and modify as required while still using this project as the core.

  • Executors contain the logic to run tasks against a given system type, and format the results.
  • EventHandlers are used to map Github events to specific functionality such as pull requests.

Contributing

Fork this repository and work on your enhancements, then send a pull request. If you build custom plugins that may be useful to others in your forked gtm-agent, please let us know and may be able to assist wwith backporting to this project.

Use commitizen for conventional commit messages via git cz instead of git commit.
To setup if not already installed:

npm install -g commitizen
npm install -g cz-conventional-changelog
echo '{ "path": "cz-conventional-changelog" }' > ~/.czrc

...or you can just use npm run commit which will use local commitizen install.