Configuration Management and Service Discovery

Lochness has several services. These services are ran on the Hypervisors directly. While each Hypervisor can run each service, an operator may want to select Hypervisors for each service. "Config management" is used to set up these services on the appropriate Hypervisors.

Some Assumptions:

• Every Hypervisor runs the Agent.
• All the needed binaries are part of the Mistify OS build

Each Hypervisor in the config store (etcd, currently) has a "services" tree that simply has the service name as the key and a "bool" as the value. Services that should run on the hypervisor are "true."

Each Hypervisor runs a local copy of Ansible. A "wrapper" queries etcd and generates a simple local inventory file with host vars for simple config options and service status. In general, Ansible never talks to etcd. Ansible is ran out of cron and can also watch a key in etcd.

Each service has a role in Ansible and each role has an "enable" and "disable" tasks file. Enable tasks set up any configs, including systemd unit files, and enables the service. Disable tasks disable the service. Each service includes a "side-car" service that registers it in etcd with a TTL.

Each service (except the agent) has a well known name. A simple etcd based DNS server allows both A and SRV lookups of each service. All DHCPD servers, for example are registered at dhcpd.services.lochness.local. (Note: the "local" domain is configurable.) Each Hypervisor is also available at <machine_id>.nodes.lochness.local. This allows "legacy" applications, such as tftp, to use DNS while still being able to discover nodes for a service. Each service should also have a well known port as legacy applications cannot use SRV records to discover ports.