Developer On-boarding

Overview

Functionality:

  • Airship Bootstrapping

  • Core Kubernetes Management

  • Misc.

Code structure:

  1. Jinja templates (promenade/templates/**).

  2. Helm charts for Kubernetes components, etcd, CoreDNS, Promenade charts/**.

  3. Python support code.

    • API

    • Config access object

    • CLI

    • Certificate generation code

Since Promenade is largely templates + charts, unit testing is not enough to provide confidence for code changes. Significant functional testing is required to test small changes to, e.g. the etcd or calico charts, which can completely break deployment (or break reboot recovery). Developers can run the functional tests locally:

./tools/setup_gate.sh  # Run once per machine you're on, DO NOT USE SUDO.
./tools/gate.sh        # Runs a 4 node resiliency test.

This runs the test defined in tools/g2/manifests/resiliency.json. There are a few additional test scenarios defined in adjacent files.

There are helpful tools for troubleshooting these gates in tools/g2/bin/*, including tools/g2/bin/ssh.sh, which will let you ssh directly to a node to debug it, e.g.:

./tools/g2/bin/ssh.sh n0

Running Resilency Tests Behind Corporate Proxy

If your development environment is behind a corporate proxy, you will need to update following files to add your envrionment’s proxy information, dns, or possibly your internal ntp servers, in order to deploy airship:

  • charts/coredns/values.yaml: Update the upstream coredns nameserver IPs to your internal DNS addresses.

  • examples/basic/KubernetesNetwork.yaml: Since resilency manifest uses the examples/basic environment configuration, you will need to Update the kubernetes network configuration in this folder. Update the upstream nameserver IPs to your internal DNS addresses. Add the http(s) proxy URL and additional_no_proxy list. Also, if your enviornment requires that, update the ntp server list to your internal ntp server addresses for more reliable time sync.

  • tools/g2/templates/network-config.sub: Update the upstream nameserver IPs to your internal DNS addresses.

Bootstrapping

Promenade is responsible for converting a vanilla Ubuntu 16.04 VM into a proper Airship.

How You Run It

Assuming you have a valid set of configuration. Generate genesis.sh, which is a self-contained script for bootstrapping the genesis node.

promenade build-all -o output-dir config/*.yaml

What genesis.sh does:

  1. Basic host validation (always room for more).

  2. Drops pre-templated files in place:

    • Manifests to run initial Kubernetes components /etc/kubernetes/manifests

      • Basic components (apiserver, scheduler, controller-manager)

      • Etcd

      • Auxiliary Etcd

    • Docker configuration

    • Kubelet configuration

    • Apt configuration (proxy)

    • Bootstrapping Armada configuration

      • Dedicated Tiller

      • Dedicated Kubernetes API server

      • API server points at auxiliary etcd.

  3. Installs some apt packages (docker + user-defined)

  4. Starts Docker and Kubernetes.

  5. Waits for bootstrapping services to be up (healthy Kubernetes API).

  6. Applies configured labels to node.

  7. Waits for Armada to finish bootstrapping deployment.

  8. Final host validation.

When it’s done, you should have a working Airship deployed as defined by your configuration (e.g. with or without LMA, keystone, etc) with no configuration loaded into Deckhand (via Shipyard).

How It Works

The templates that get dropped in place generally live in promenade/templates/**. The genesis node gets everything under roles/genesis/** and roles/common/** directly in place. Note that the templates under roles/join/** are used instead of the files under genesis for joining nodes to the existing cluster.

The “real” work happens inside kubelet managed “static” pods (defined by flat files in /etc/kubernetes/manifests), primarily via Armada.

Charts do a bunch of work to take control of essentially everything behind the scenes. Trickiest is etcd, for which we run multiple server processes to keep the cluster happy throughout bootstrapping + initial node join.

Note that we deploy two separate etcd clusters: one for Kubernetes itself, and one for Calico. The Calico one is a bit less sensitive.

Anchor Pattern

To provide increased resiliency, we do something a bit unusual with the core components. We run a DaemonSet for them which simply copy static Pod definitions into the /etc/kubernetes/manifests directory on the hosts (along with any supporting files/configuration). This ensures that these workloads are present even when the Kubernetes API server is unreachable. We call this pattern the Anchor pattern.

The following components follow this pattern:

  • Kubernetes core components

    • API server

    • Scheduler

    • Controller Manager

  • Kubernetes etcd

  • Calico etcd

  • HAProxy (used for API server discovery)

The HAProxy DaemonSet runs on every machine in the cluster, but the others only run on “master” nodes.

Kubernetes Cluster Management

Promenade is responsible for managing the Kubernetes lifecycle of nodes. That primarily consists of “joining” them to the cluster and adding labels, but also includes label updates and node removal.

Node Join

This is done via a self-contained script that is obtained by Drydock querying the Promenade API GET /api/v1.0/join-scripts (and providing a configuration link to Deckhand originally specified by Shipyard).

The join script is delivered to the node by Drydock and executed via a systemd unit. When it runs, it follows a similar pattern to genesis.sh, but naturally does not use any Kubernetes bootstrapping components or run Armada:

  1. Basic host validation (always room for more).

  2. Drops pre-templated files in place:

    • Docker configuration

    • Kubelet configuration

    • Apt configuration (proxy)

  3. Installs some apt packages (docker + user-defined)

  4. Starts Docker and Kubernetes.

  5. Waits for node to be recognized by Kubernetes.

  6. Applies configured labels to node.

  7. Final host validation.

After the node has successfully joined, the systemd unit disables itself so that it is not run again on reboot (though it would be safe to do so).

Other Management Features

Re-labeling and node removal API development has been delayed for other priorities, but is recently underway. While changing labels is generally easy, there are a few trickier bits around Kubelet and etcd management.

It is currently possible to fully de-label and remove a node from the cluster using a script that gets placed on each node (it requires kubectl so that must be in place), but that work is not exposed via API yet. The resiliency gate exercises this to reprovision the genesis node as a normal node.

Miscellaneous

Promenade does a few bits of additional work that’s hard to classify, and probably don’t belong in scope long term. Most notably is certificate generation.

Certificate generation is configured by the PKICatalog configuration document, which specifies the details for each certificate (CN, groups, hosts). Promenade then translates those requirements into calls to cfssl. The following will create a certificates.yaml file in output-dir containing all the generated certs:

promenade generate-certs -o output-dir config/*.yaml

If there are existing certs in config/*.yaml, then they will be used if applicable.

Troubleshooting

The context for this section is the functional gates described above. You can run them with:

./tools/gate.sh <gate_name>

When something goes wrong with this, you can ssh into individual nodes for testing (the nodes are named n0 through n3):

./tools/g2/bin/ssh.sh <node_name>

When you get into a node and see various failures, or have an Armada error message saying a particular chart deployment failed, it is important to assess the overall cluster rather than just digging into the first thing you see. For example, if there is a problem with etcd, it could manifest as the Kubernetes API server pods failing.

Here is an approximate priority list of what to check for health (i.e. things higher up in the list break things lower down):

  1. Kubernetes etcd

  2. Kubernetes API Server

  3. Other Kubernetes components (scheduler, controller-manager, kubelet).

  4. Kubernetes proxy

  5. Calico etcd

  6. Calico node

  7. DNS (CoreDNS)

For almost any other application, all of the above must be healthy before they will function properly.