diff --git a/README.md b/README.md index 80f31cd8b..d16f54725 100644 --- a/README.md +++ b/README.md @@ -5,133 +5,176 @@ neco logo -**Neco** is a collection of tools to build and maintain large data center -infrastructure. It is used in [cybozu.com](https://www.cybozu.com/), the -top B2B groupware service in Japan. +**Neco** is a project to build and maintain highly automated on-premise data centers using [Kubernetes][] at the center of its system. +The project is led by a [CNCF][] Silver member [Cybozu][] which is also known as [Kintone][] in the United States. + +Neco is also the name of the architecture for systems built on the deliverables of the project. + +Table of Contents: + +- [Repositories](#repositories) +- [System overview](#system-overview) + - [Boot servers](#boot-servers) + - [Neco Continuous Delivery System (Neco CD)](#neco-continuous-delivery-system-neco-cd) + - [Kubernetes](#kubernetes) +- [Network](#network) + - [Design](#design) + - [Node BGP and redundancy](#node-bgp-and-redundancy) + - [DHCP and location of boot servers](#dhcp-and-location-of-boot-servers) +- [Tests](#tests) + - [Virtual data center](#virtual-data-center) + - [Large tests](#large-tests) + - [Middle/Small tests](#middlesmall-tests) +- [Directories](#directories) + +Repositories +------------ -**Project Status**: Perpetual beta. +The project consists of a lot of repositories including: -Architecture ------------- +- [cybozu-go/neco][neco]: The main repository. +- [cybozu-go/neco-apps][neco-apps]: Kubernetes application manifests for Argo CD. +- [cybozu/neco-containers][neco-containers]: Dockerfiles to build container images. +- [cybozu-go/sabakan][Sabakan]: Versatile network boot server. +- [cybozu-go/cke][CKE]: Cybozu Kubernetes Engine. +- [cybozu-go/coil][Coil]: [CNI][] plugin in favor of UNIX philosophy. +- [cybozu-go/topolvm][TopoLVM]: Capacity-aware [CSI][] plugin using LVM. +- [cybozu-go/placemat][placemat]: Virtual data center construction tool. + +System overview +--------------- + +![necosystem](https://www.plantuml.com/plantuml/svg/ZPHDJzj048RFtLynWajxe1Akk4GL0Wm2HFYeGkaXuh2nXyJAPiUQtQMD8FtjdJN-SEKKDGV7ddTcvzbtl3voNhCFAwCCpj7cuAa4BkOBhq1-hv2I8PR_rxVdhxlZA9wOW8UlSByeE8TILbZ3xMJI9TqLTYdw3MO49LCUCkz4bGPlRPHR5bRgz0eOCw_jt22yg0zCmUOfiKM6v41jbCeCXHpE2tyHPh1xfmsUCw9LqsEU4wIdZw0TQBdFSkZOr5HQjxB5h98ZbQfpzmm--Rt3PkKH_D0-MwWgLTHG3ZNLeQOcE8yCBxtOLPYXW7jqi7lvJBmMDTenJSbw2lpOvwk4i_Hg6Bn6hytuP2YasXGP3UDtJ2lq2mnkgDsZmPBORr6efF73jGYLAp5MSyBUT57aETh7d7vQiQ5EbMbuciTE34z4iZjBESBEJRn-bA7XxUJJJkMtHFRhZkUiMoCkrYNoKcogfh9edlQ7MwN2L3cPtEntYBsQZf9ZhM1MkMhpBbNo_rJ9-rJ9DgfaErKYL9kMx8J8mmJv6Hb-mPVV1nNTB4Q7SEdLMNz4qucUwQMsVyKUT3332PzgBxbE6gBFyfs-l57rI0UZZXJ_5AknlgicIkdC4yDtXwpY1KHfNa8OmPZcXLNj6MnCwXsT4PoYAQ8tnsNfslFOu-kbzArCQLMIHUkTgasgr__SeCEeAcZbvXCG_H5j8k2enF3wz3Tf8_WMaDTHsLvI-GyIbGm9NZeNsc0ejpHem0xxxtEtgyuDTOGs3olp1m00) -Neco adopts [network booting][netboot] to manage thousands of servers -efficiently. The center of Neco is therefore a few netboot servers. +A Neco data center consists of a few (from 3 to 5) management servers called _boot sever_ and thousands of servers for a Kubernetes cluster; pretty big, huh? In fact, Neco is designed to manage such large clusters. -[neco-ubuntu][] creates a custom Ubuntu installer to setup netboot servers. +### Boot servers -[Sabakan][sabakan], a product from Neco, implements functions necessary -for netboot including [DHCP][], [UEFI HTTP Boot][], [iPXE][], [ignition][] -and file server service to download assets. +Boot servers are symmetrical with each other except for host names and IP addresses. -![sabakan_architecture](http://www.plantuml.com/plantuml/svg/ZOv1IyGm58Jl-HN3BdZBOTkRY2ohu1uyBBX_uBLvhMAQIFAYYFZVBIbHIoruoOFvPZApZq91qc1Lu5R8zPQnOMaDMfkYSDZWGm66X1gAZ8mevhjR0zKQg4UWC8MXZNzpUWfWUnVe_IzKpr05hIrtekVmK_sUV_1UyC3XjQp_OP4QUYQ7xVrJ_oW7crXzbrvDFnTFQLpHwuK-Zd3UCF93CT_TKgfK1j3fHL-Ny2LkZpSdNE1uFf_G-O36Ur7P_o_ddfr9W_q2) - +All persistent data are stored in an [etcd][] cluster. Thanks to etcd, adding or removing a boot server can be done easily. -To share and keep data between netboot servers, [etcd][] is used widely. -To protect data and authorize users, [Vault][] is used too. +[Sabakan][] is a network boot server that supports DHCP and HTTP. It can also manage an inventory of Node servers and assign IP addresses automatically. Node servers should use [UEFI HTTP boot][HTTPBoot] to load [CoreOS][] from sabakan. -![etcd_vault_architecture](http://www.plantuml.com/plantuml/svg/TOwzQWCn48HxFSLmgKM8hmiXc3HvW90kpKPQcmDPEdPN3YRatUEpyC-1hv5WFhwTMQkHMDqb9noCyZOnEhOG4L9LO-dmwu18Hj-aZ1CYFVrFIs2r1FeZS6WoV2m_sJS13-z2Xtkedw4Ll4-yCJ-7V-vs_bifXW-M_NdzbUsf9fiaFhXBsqixsU2bw5xQpzEfbu8LGVUfB2W26iSq1BAXv0wagCVSJS_PV6tgCyPgZrisA0TXqwyyg5P6OB5XCvrWdOjjxLMCPEJMd6FTfNy0) - +[CKE][], or Cybozu Kubernetes Engines, is a CNCF certified Kubernetes installer. It queries Node server status to Sabakan and chooses Nodes to construct Kubernetes control plane and worker nodes. TLS certificates required for Kubernetes are issued by [Vault][]. -As to network, Neco uses layer-3 technologies only. Specifically, -[BGP][] is used to advertise/receive routes together with [BFD][] to -reduce route convergence and [ECMP][] for high availability. +### Neco Continuous Delivery System (Neco CD) -![network_architecture](http://www.plantuml.com/plantuml/svg/ZLB1Ri8m3BtdAopk7E8ZcYO6aoPkKxexYjeCAjgu2kdGDF7l2mviHR9eBr2_z_py77bvZ3R4lctKyL3xpWRRGbDx5xyx1nJYdbJPK5_1naSN4gw2AwFrkyR1R4t1GN6gOxcVWJq2rp_gFDGKNN8RYXZGqsJ8ijjeU9fNTFBpPnwaUDeVb6qb45Nc_c5ZD2p0hTxUCuKI9NIXt7L7gSwM1xjBAxqKinGVm5ELAfDWdG60mU8VPAvhW-R54w2px5veg8yEZFji4aQ1jMOWFIl-VU2FDt-SxezZ_YkY28KBNowtS2tOhpRcDGlIn_QYaltMr7QN8DaIT3wiGdoIZYgcqx_UwX4UHpBfINdmcWT7yk187XpDWuCynWpkaCF20kfqRZAB3rb-_7i5okuoYp8hkQlB0cUrTBxgs-ON) - +To bootstrap and maintain boot servers, a simple continuous delivery system called Neco CD is implemented in this repository. -Repository contents -------------------- +Neco CD consists of these programs and artifacts: -### neco +- [`neco-updater`](./docs/neco-updater.md): a daemon to watch GitHub and notify `neco-worker` of new artifacts. +- [`neco-worker`](./docs/neco-worker.md): a daemon to install/update programs on a boot server. +- [`neco`](./docs/neco.md): CLI tool to bootstrap and configure boot servers. +- Neco debian package: archive of program binaries to be installed on boot servers. -`neco` is a deploy automation tool written in Go. -It installs and updates miscellaneous utilities in boot servers. +Debian packages are built by CircleCI and released on [GitHub releases][releases]. -See [docs/neco.md](docs/neco.md) +Read [docs/cicd.md](docs/cicd.md) for details. -### neco-updater +### Kubernetes -`neco-updater` is a background service to detect new releases of `neco`. -When detected, it sends the update information to `neco-worker` through etcd. +Kubernetes cluster installed with CKE has nothing other than essential functions. +To make it useful, many applications such as [MetalLB][] or [Teleport][] are installed. -Neco itself is released as a Debian package in [releases][]. +Manifests of these applications are in [neco-apps][] repository and continuously delivered by [Argo CD][ArgoCD]. -See [docs/neco-updater.md](docs/neco-updater.md) +Network +------- -### neco-worker +### Design -`neco-worker` is an automate maintenance service. It installs/updates `neco` package, applications, -and sabakan contents when receives information from `neco-updater` through etcd. +Network of Neco is designed with these goals: -See [docs/neco-worker.md](docs/neco-worker.md) +- Bare-metal performance, i.e., no overlays like [VXLAN][] +- Scalability, especially for [east-west traffic](https://searchnetworking.techtarget.com/definition/east-west-traffic) +- Vendor neutrality -### generate-artifacts +The data center network for Neco should employ [leaf-spine topology](https://blog.westmonroepartners.com/a-beginners-guide-to-understanding-the-leaf-spine-network-topology/). More specifically, each data center rack works as an autonomous system (AS) and exchanges routes with [BGP][]. -`generate-artifacts` is a command-line tool to generate `artifacts.go` which is a collection of latest components. +Since BGP is a layer-3 routing protocol, use of layer-2 can be limited within a single rack. -See [docs/generate-artifacts.md](docs/generate-artifacts.md) +![leaf-spine](https://www.plantuml.com/plantuml/svg/VPB1IiGm48RlUOfXSz53OJjJFShgnPl8xb4y13DkosQJagQ8YE-kROKKdCrb2FD_Cny9sGt9n3IUdHfEdUzDD6TmdUVqsJE4cDgWKZXoQZd2b_B1CXpwtnpC0CDqgAKcIarIt-hViitUl9rgkBe_FC7jpNPRNs-c0SVclLuHc0dC1Au8oWHbWbP48-verEo2wgNjh6N_6hjZcwgxFy5SvP3KNEMmcIwxF13eoeOFNsbOJFh1ZInc2FenCdin9D3FO-oTFDg0thDT8YmOiMJ4eX4l6wbWf9AHYaOIZJlsTlgmFm00) -### Test suite +### Node BGP and redundancy -[dctest](dctest/) directory contains test suites to run integration -tests in a virtual data center environment. +Kubernetes nodes need to route packets for Pods. Without overlay networking, this means that each node needs to advertise routing information. In Neco, each node runs [BIRD][] to speak BGP and advertise routes for Pods. -[Placemat][placemat] is a tool to create arbitrarily complex network topology -and virtual servers using Linux networking stacks, [rkt][], and [QEMU][]. +For high availability, network links must be redundant. Therefore, Neco requires that each rack has two Top-of-Rack (ToR) switches and each server has two ethernet links. To reduce convergence time of BGP, [BFD][] is used together. -See [docs/dctest.md](docs/dctest.md) +This means that ToR switches and node servers in a rack are all BGP speakers and need to connect each other as iBGP peers. To make configurations simple and flexible, ToR switches are configured as [route reflectors](https://en.wikipedia.org/wiki/Route_reflector). -### CI/CD +The two ToR switches are independent; each has its own managing subnet. As a result, each server has two IP addresses for each ToR switch. To tolerate single link or ToR switch failure, the server has another IP address called **the node address** with `/32` netmask and advertises it via BGP. -CI/CD in **Neco** is running by CircleCI. -Then `neco-updater` service is updated automatically. +![iBGP](https://www.plantuml.com/plantuml/svg/ZPAnJiCm48RtUugJEYVOLs38L5ZO40AsgaCadsghgOsSGmWXthsnhQGPF3H3TFhrvVzoYNUZz56-pmEC9-U_-jYVSN2UH_aT64CK6q2SJFFJn-E3KXf0mf75SiG_yC4mIeWQzu3JzHdUKUT9grQrkjLtT0-7XQCLHrUEPYvNDNlhZ65_YEvefSdFmJ-KadIRRQyfq9Zcd7UfZLIxLLFVjlIpMDKjRvCqSwoxHLxiAmEj3LGrKDr0TGDTlyDgGHD-V5E4PVLDawDDMNF2XJ7uz4DE7RhdbxU4qGrOfOvkggjZE_OcdP4B) -See [docs/cicd.md](docs/cicd.md) +### DHCP and location of boot servers -### Run unit tests at local machine +[UEFI HTTP Boot][HTTPBoot] requires a DHCP server which requires layer-2 broadcast domain. Without DHCP relay servers, a dedicated server must be run as a boot server for each rack because layer-2 domain does not span racks. -First, start up etcd server manually. +If ToR switches can provide DHCP relay, this requirement can be relaxed. -```console -$ make start-etcd -``` +Tests +----- -Then, run `go test` on another terminal. +### Virtual data center -```console -$ go test -v -count=1 -race -mod=vendor ./... -``` +To make continuous delivery reliable enough, thorough and comprehensive system tests need to be performed. +Since Neco is the system to bootstrap and maintain an on-premise data center, this means that the tests should do the same thing, that is, to bootstrap and maintain a data center! -### GCP suite +To simulate an on-premise data center, we have created [placemat][], a tool to construct a virtual data center with containers, virtual machines and Linux network stacks. The virtual data center implements the aforementioned leaf-spine networks with BGP routers. -[gcp](gcp/) directory contains utilities for provisioning the Google Compute Engine services for Neco project. -Test suite, and other Neco github projects deploy GCE instance based on an image created by this tool. +### Large tests -See [docs/gcp](docs/gcp/) +[dctest](dctest/) directory contains test suites to run large tests using the virtual data center. +Details are described in [docs/dctest.md](docs/dctest.md). -### `git-neco` +### Middle/Small tests -`git-neco` is a git extension utility to help Neco developers. +Other than large tests, this repository contains smaller tests that can be run as follows: -Documentation -------------- +1. Setup the environment by `make setup`. This is a one shot job. +2. Run etcd by `make start-etcd` +3. Run tests by `go test -v -count 1 -race -mod=vendor ./...` +4. Stop etcd by `make stop-etcd` -[docs](docs/) directory contains documents about designs and specifications. +Directories +----------- + +Other than Go packages, this repository has the following directories: + +- [`dctest/`](dctest/): Large tests using a virtual data center. +- [`debian/`](debian/): Ingredients for Debian package. +- [`docs/`](docs/): miscellaneous documentation. +- [`etc/`](etc/): Kubernetes manifests for networking and other configuration files. +- [`ignitions/`](ignitions/): [CoreOS Ignition][Ignition] template files. [releases]: https://github.com/cybozu-go/neco/releases [godoc]: https://godoc.org/github.com/cybozu-go/neco -[netboot]: https://en.wikipedia.org/wiki/Network_booting -[neco-ubuntu]: https://github.com/cybozu/neco-ubuntu -[sabakan]: https://github.com/cybozu-go/sabakan -[DHCP]: https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol -[UEFI HTTP Boot]: https://github.com/tianocore/tianocore.github.io/wiki/HTTP-Boot -[iPXE]: https://ipxe.org/ -[ignition]: https://github.com/coreos/ignition -[etcd]: http://etcd.io/ -[Vault]: http://vaultproject.io/ -[BGP]: https://en.wikipedia.org/wiki/Border_Gateway_Protocol +[ArgoCD]: https://argoproj.github.io/argo-cd/ [BFD]: https://en.wikipedia.org/wiki/Bidirectional_Forwarding_Detection -[ECMP]: https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing +[BGP]: https://en.wikipedia.org/wiki/Border_Gateway_Protocol +[BIRD]: https://bird.network.cz/ +[CKE]: https://github.com/cybozu-go/cke +[CNCF]: https://www.cncf.io +[CNI]: https://github.com/containernetworking/cni +[Coil]: https://github.com/cybozu-go/coil +[CoreOS]: https://coreos.com/os/docs/latest/ +[CSI]: https://github.com/container-storage-interface/spec +[Cybozu]: https://cybozu-global.com +[etcd]: https://etcd.io +[HTTPBoot]: https://github.com/tianocore/tianocore.github.io/wiki/HTTP-Boot +[Ignition]: https://coreos.com/ignition/docs/latest/ +[Kintone]: https://www.kintone.com +[Kubernetes]: https://kubernetes.io +[MetalLB]: https://metallb.universe.tf +[neco]: https://github.com/cybozu-go/neco +[neco-apps]: https://github.com/cybozu-go/neco-apps +[neco-containers]: https://github.com/cybozu/neco-containers [placemat]: https://github.com/cybozu-go/placemat -[rkt]: https://coreos.com/rkt/ -[QEMU]: https://www.qemu.org/ +[Sabakan]: https://github.com/cybozu-go/sabakan +[Teleport]: https://gravitational.com/teleport/ +[TopoLVM]: https://github.com/cybozu-go/topolvm +[Vault]: https://www.vaultproject.io +[VXLAN]: https://en.wikipedia.org/wiki/Virtual_Extensible_LAN diff --git a/docs/dctest.md b/docs/dctest.md index 6a809a283..bd78b6a1f 100644 --- a/docs/dctest.md +++ b/docs/dctest.md @@ -9,7 +9,7 @@ Generating deb Package Dctest uses a generated deb package to install Neco. Details of artifacts definition used in generating the deb package are -described in [Artifacts](artifacts). +described in [Artifacts](artifacts.md). Type of Test Suites ------------------- diff --git a/docs/placemat-menu.md b/docs/placemat-menu.md index 42d85e475..8801b0215 100644 --- a/docs/placemat-menu.md +++ b/docs/placemat-menu.md @@ -9,7 +9,6 @@ The cluster of the overview is in the following figure. ![Network Design](http://www.plantuml.com/plantuml/png/hPF1IiGm48RlUOgX9pq4d3p1uYBePGNhJKH26tVBran9P-a-lcsIjIarwyLBwNp__y_03Ddqh1sVlbhXJCNQxbi3nIFr33TFbespXcyBq3qqiH9LIwSQYeVpLEiMHZQGEtgJEV_epvrncXkoi4iCr74wQ4lEG3aqN1syN8rrgfTTOnU6VWBujqMbbbTwyOhJrV7kWydXLJMRnQjP35bXgJQX6Ro9UoA6tKY4b59iI_-FQQ5yKQPAUL7Uasxu7zqkHmHPqs1bsFVqYI3kTurKHAxP7rY6EtlGca-M_gmX6bF9hdE2-aN0N0AJXChDM0gv1EOISSRSTT5kvchDSUN7cQibtnXRJ-_j6m00) - The network is designed as a spine-and-leaf architecture. There is a *core switches* at the top of the cluster. The *spine switches* connects core switches and racks. It is a L3 switch to routing between inter cluster,