Home

COAL installation & setup

Post setup

Run ~/dev/sdc/coal-post-setup.sh

Add your SSH public key to the list of authorized keys for user root in the global zone

In the global zone:

$ /usbkey/scripts/mount-usb.sh
$ cd /mnt/usbkey/config.inc
$ vi root.authorized_keys
# Paste your SSH public key

SDC-Docker

Switching TLS on/off in sdc-docker

Use sapiadm to update the docker service metadata like following:

[root@headnode (coal) ~]# sapiadm update $(sdc-sapi /services?name=docker | json -Ha uuid) metadata.USE_TLS=false

(this particular example disables use of TLS).

Then update the port used by your docker client so that it connects to the right TCP port: 2375 if TLS is disabled, 2376 if TLS is enabled. For instance:

export DOCKER_HOST=tcp://10.88.88.6:2375

if TLS is disabled.

Do not forget to check any alias you may have setup in your current shell for docker with alias | grep docker, and make sure the --tls option is passed or not depending on the desired setup.

Sniffing HTTP traffic on the wire

There are many ways to inspect HTTP requests/responses between the docker client and the docker server, but the method that I find easiest to work with is:

1. Disable SSL for the docker service as mentioned above.

2. Use sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-sniffer -w ~/capture.pcap vmnet8, then open ~/capture.pcap output with WireShark.

Useful commands

Listing servers from the headnode

sdc-server list

CloudAPI

Sending requests to CloudAPI with curl

First, setup that bash-specific function (it won't work in e.g zsh):

function cloudapi() {
  local now=`date -u "+%a, %d %h %Y %H:%M:%S GMT"`;
  local signature=`echo ${now} | tr -d '\n' | openssl dgst -sha256 -sign ~/.ssh/id_rsa | openssl enc -e -a | tr -d '\n'`;
  local url=$1
  shift

  curl -ks -i -H "Content-type: application/json" -H "Accept: application/json" -H "accept-version: ~8" -H "Date: ${now}" -H "Authorization: Signature keyId=\"/$SDC_ACCOUNT/keys/$SDC_KEY_ID\",algorithm=\"rsa-sha256\" ${signature}" --url $SDC_URL$url "$@";
  echo "";
}

then run e.g:

cloudapi /my/machines

Clearing tickets in CNAPI

Sometimes, workflow tasks can be stuck waiting on tickets that have been acquired, but won't be released before they expire. In this case, it can be useful to delete all tickets:

sdc-cnapi /servers/$(sysinfo | json UUID)/tickets?force=true -X DELETE

Troubleshooting

Accessing CNs serial console

$ ipmitool -I lanplus -U $username -P $password -H $BMC_IP -C 3 sol activate 
$ ipmitool -I lanplus -U $username -P $password -H $BMC_IP -C 3 power {status,on,off,cycle,reset}

General healthcheck

sdc-healthcheck

Global zone

In the global zone: sdc-healthcheck will show the state of each service. svcs -Zx will do the same.

Docker zone

sdc-dockeradm will display a list of available commands. For instance sdc-dockeradm tail tails the logs of the docker service.

Finding out why a provision failed

This will show DAPI's reasoning for the latest provision:

/opt/smartdc/cnapi/bin/alloc-reasons.sh -l

Updates

sdcadm up $zonename, e.g sdcadm up vmapi

Building your own image for a given zone

Build with Jenkins (choose the appropriate job from the sdc view) from a branch, then lookup the new image's uuid with sdcadm avail -C experimental, then reprovision a given zone with echo '{"image_uuid": "<some-uuid>"}' | vmadm reprovision <vm-uuid>.

When something goes wrong

How to retrieve the maintenance password?

$ cd sdc-headnode
$ tar xfvz cache/platform-master*.tgz platform-date/root.password
$ cat root.password

Using thoth

Make sure both MANTA_USER and THOTH_USER environment variables are set. MANTA_USER needs to be set to your actual manta user (e.g jgilli), and TOTH_USER must be thoth.

Make sure to use the latest versions of manta and manta-thoth too, as many known issues have been fixed over time.

Be careful when installing manta-thoth to npm install manta-thoth, not npm install toth. toth is a different thing in npm, and not what you want.