etcd

In this lesson, you'll explore etcd, the distributed key-value store that serves as the primary data store for all cluster state in Kubernetes.

Objectives:

• Understand etcd's role within a Kubernetes cluster
• Install and configure etcd from scratch
• Learn how to interact with etcd using the etcdctl command-line tool
• Secure etcd with TLS certificates for encrypted communication

By the end of this lesson, you'll understand how etcd stores and manages the critical state data that powers a Kubernetes cluster.

etcd is a distributed key-value store that serves as the primary data store for all cluster state in Kubernetes. It stores configuration data, state information, and metadata for the entire cluster.

Originally developed by CoreOS, etcd is now maintained by the CNCF. It is designed to be highly available, consistent, and reliable, all essential qualities for the system that holds the "source of truth" for your entire Kubernetes cluster.

etcd implements the Raft consensus algorithm to ensure consistency across multiple nodes. The cluster can continue operating even if some nodes fail, as long as a majority of nodes (a quorum) remain healthy.

etcd cluster

etcd's strong consistency guarantee requires that all consistency-sensitive operations (for example, any write operations) flow through a leader node, which gets elected through the Raft consensus algorithm.

etcd cluster electing a leader

Strong consistency is essential for Kubernetes to ensure that all components see the same cluster state.

Without this guarantee, Kubernetes could schedule pods on nodes lacking capacity, or controllers could make conflicting decisions based on stale data.

Only the Kubernetes API server communicates directly with etcd. All other components (like kubelet, scheduler, and controllers) interact with cluster state through the API server, which acts as a gateway to etcd.

kube-apiserver interacting with etcd

This design keeps etcd safe from direct access by cluster components and centralizes control over authentication, authorization, admission, and validation to ensure the cluster state stays valid and consistent.

etcd in a Kubernetes cluster

Follow these steps to install etcd:

Set the version to install:

ETCD_VERSION=v3.6.4

Download and install etcd from GitHub:

curl -fsSLO https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION?}/etcd-${ETCD_VERSION?}-linux-amd64.tar.gz
tar xzvof etcd-${ETCD_VERSION?}-linux-amd64.tar.gz

sudo install -m 755 etcd-${ETCD_VERSION?}-linux-amd64/{etcd,etcdctl,etcdutl} /usr/local/bin

etcdctl completion bash | sudo tee /etc/bash_completion.d/etcdctl

Create a dedicated user for the etcd service:

sudo adduser \
    --system \
    --group \
    --disabled-login \
    --disabled-password \
    --home /var/lib/etcd \
    etcd

Download the systemd unit file to run etcd as a systemd service:

sudo wget -O /etc/systemd/system/etcd.service https://labs.iximiuz.com/content/files/courses/kubernetes-the-very-hard-way-0cbfd997/03-control-plane/01-etcd/__static__/etcd.service?v=1770643331

Start the etcd service:

sudo systemctl daemon-reload
sudo systemctl enable --now etcd

Congratulations! You have successfully installed etcd. 🎉

With etcd installed and running, you can now explore its capabilities.

etcd exposes both HTTP and gRPC APIs. You can test the HTTP API using curl:

curl http://127.0.0.1:2379/health

However, etcdctl provides a more convenient way to interact with etcd:

etcdctl endpoint health

As a key-value store, etcd allows you to store and retrieve data in the form of key-value pairs:

etcdctl put foo bar

Retrieve data from the store:

etcdctl get foo

By default, etcdctl returns both the key and the value. To retrieve only the value, use the --print-value-only flag:

etcdctl get foo --print-value-only

etcd supports prefix-based key retrieval using the --prefix flag, allowing you to organize data in a hierarchical structure (like Kubernetes does).

Create some keys with a path-like structure:

etcdctl put /foo bar
etcdctl put /foo/bar baz
etcdctl put /foo/bar/baz bat

You can retrieve all keys that start with /foo:

etcdctl get --prefix /foo --keys-only

Securing etcd with TLS is critical in any production Kubernetes cluster: without proper TLS encryption, sensitive information travels in plaintext over the wire.

This exposes critical data to anyone who can intercept network traffic:

• ServiceAccount tokens
• TLS certificates
• Cluster configuration
• Secrets and ConfigMaps

TLS provides more than just encryption, though. It also enables authenticating clients via mTLS (mutual TLS) authentication, ensuring that only authorized components can access or modify cluster state data.

Setting up TLS for etcd requires generating the necessary certificates and keys.

Create a directory to store the certificates and keys:

sudo mkdir -p /etc/etcd/pki
cd /etc/etcd/pki

Create a Certificate Authority (CA) to sign certificates:

sudo openssl genrsa -out ca.key 4096
sudo openssl req -x509 -new -nodes -key ca.key -out ca.crt -subj "/CN=etcd" -sha256 -days 3650

Create a config file for the server certificate:

cat <<EOF | sudo tee server.cnf
[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext
prompt             = no

[ req_distinguished_name ]
CN = server

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = localhost
DNS.2 = $(hostname)
IP.1  = 127.0.0.1
IP.2  = ::1
IP.3 = $(ip -o -4 addr show | grep 'eth' | awk '{split($4,a,"/"); print a[1]}' | paste -sd,)
EOF

Generate the server certificate:

sudo openssl genrsa -out server.key 2048
sudo openssl req -new -key server.key -out server.csr -config server.cnf
sudo openssl x509 -req -in server.csr -out server.crt \
  -CA ca.crt -CAkey ca.key \
  -days 365 -extfile server.cnf -extensions req_ext

Generate a client certificate for the Kubernetes API server and other etcd clients:

sudo openssl genrsa -out client.key 2048
sudo openssl req -new -key client.key -out client.csr -subj "/CN=etcd/O=etcd"
sudo openssl x509 -req -in client.csr -out client.crt \
  -CA ca.crt -CAkey ca.key \
  -days 365

Set ownership of all generated certificates and keys to the etcd user:

sudo chown -R etcd:etcd .

Make the client key accessible to all users:

sudo chmod 644 client.key

Go back to the home directory:

cd

Configure etcd to use TLS for server connections and mutual TLS for client authentication:

cat <<EOF | sudo tee -a /etc/default/etcd

ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379

ETCD_CLIENT_CERT_AUTH=true
ETCD_CERT_FILE=/etc/etcd/pki/server.crt
ETCD_KEY_FILE=/etc/etcd/pki/server.key
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

ETCD_NAME=$(hostname)
ETCD_ADVERTISE_CLIENT_URLS=https://$(hostname):2379

EOF

Restart the etcd service to apply the changes:

sudo systemctl restart etcd

Configure etcdctl to communicate with etcd using TLS:

cat <<EOF | tee -a $HOME/.bashrc $HOME/.profile

export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/pki/client.crt
export ETCDCTL_KEY=/etc/etcd/pki/client.key
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
EOF

Verify that you can communicate with etcd using TLS:

bash --login -c "etcdctl endpoint health"

With these certificates in place, etcd now provides secure, encrypted communication with authenticated clients, adding an essential layer of protection to your cluster's data store.

In this lesson, you learned about etcd, the distributed key-value store that serves as the backbone of every Kubernetes cluster.

Key takeaways:

• Cluster state storage: etcd serves as the single source of truth for all Kubernetes cluster data, storing configuration, state information, and metadata with strong consistency guarantees
• Raft consensus algorithm: Ensures data consistency and high availability across multiple nodes, with automatic leader election and the ability to survive node failures as long as a majority remains healthy
• API server gateway: Only the Kubernetes API server communicates directly with etcd, providing centralized access control and keeping etcd protected from direct component access
• Key-value operations: Supports hierarchical data organization through prefix-based queries, enabling efficient storage and retrieval of Kubernetes resource information
• TLS security: Provides encrypted communication and mutual authentication between etcd servers and clients, protecting sensitive cluster data during transmission

With etcd running and secured with TLS, you have established the foundational data layer that will store all your cluster's state.

Next, you'll install kube-apiserver, the control plane component that will serve as the gateway to etcd and expose the Kubernetes API.

🧪 Playgrounds

• etcd
• etcd Cluster

• etcd
  • Quickstart
  • Learning Resources
• Raft Consensus Algorithm
• Operating etcd clusters for Kubernetes