Add an HAProxy Sidecar to Serve an MQTT Broker over TLS

Scenario

Temperature sensors in a factory send their readings over MQTT to a broker. A sensor publishes a reading to a topic, such as factory/line1/temperature, and every client subscribed to that topic receives it. The broker is Eclipse Mosquitto, running as the mqtt-broker Deployment in the iot-platform namespace.

In this lab, the Pod sensor-client plays the role of a sensor. It has the Mosquitto client tools and the CA certificate at /certs/ca.crt. The broker account for clients is user sensor with the password Sens0r-Pass-2026.

Mosquitto speaks plain MQTT only and listens on 127.0.0.1:1883. The security team requires every client to connect with MQTT over TLS on port 8883, and the platform team chose HAProxy to handle the TLS. Today, sensor-client fails when it connects to mqtt-service:8883, because nothing in the broker Pod listens on that port.

The manifests of the broker are already applied. They are in /home/laborant/mqtt-broker/ on dev-machine:

/home/laborant/mqtt-broker/
├── deployment.yaml                 # Deployment mqtt-broker (Mosquitto only)
├── haproxy-configmap.yaml          # ConfigMap mqtt-haproxy-config
├── mosquitto-configmap.yaml        # ConfigMap mosquitto-config
├── mosquitto-passwd-secret.yaml    # Secret mosquitto-passwd
└── service.yaml                    # Service mqtt-service on port 8883

One more object is not in the folder: the Secret mqtt-tls in iot-platform, created by the security team. It holds mqtt.pem, the server certificate and its private key for mqtt-service.

Task

Update /home/laborant/mqtt-broker/deployment.yaml so that HAProxy runs in the broker Pod and terminates TLS for Mosquitto, then apply it. HAProxy must meet these requirements:

After the rollout, the broker Pod must show both containers ready, and sensor-client must be able to publish and receive messages through mqtt-service:8883 over TLS.

kubectl exec -n iot-platform sensor-client -- mosquitto_sub -h mqtt-service -p 8883 --cafile /certs/ca.crt -u sensor -P Sens0r-Pass-2026 -t factory/line1/temperature -C 1 -v

kubectl exec -n iot-platform sensor-client -- mosquitto_pub -h mqtt-service -p 8883 --cafile /certs/ca.crt -u sensor -P Sens0r-Pass-2026 -t factory/line1/temperature -m 21.5

kubectl logs -n iot-platform deployment/mqtt-broker -c mosquitto

kubectl get configmap mqtt-ca -n iot-platform -o jsonpath='{.data.ca\.crt}' > /home/laborant/ca.crt

kubectl port-forward -n iot-platform service/mqtt-service 8883:8883

openssl s_client -connect localhost:8883 -servername mqtt-service -verify_hostname mqtt-service -CAfile /home/laborant/ca.crt -brief < /dev/null

kubectl explain deployment.spec.template.spec.containers
kubectl explain deployment.spec.template.spec.containers.ports

cat <folder>/<haproxy-configmap-file>
cat <folder>/<mosquitto-configmap-file>

kubectl explain deployment.spec.template.spec.volumes.configMap
kubectl explain deployment.spec.template.spec.volumes.secret
kubectl explain deployment.spec.template.spec.containers.volumeMounts

kubectl apply -f <file> --dry-run=server
kubectl apply -f <file>
kubectl rollout status deployment/<name> -n <namespace>
kubectl get pods -n <namespace>

kubectl logs deployment/<name> -n <namespace> -c <container>

kubectl get pods -n <namespace>
kubectl logs deployment/<name> -n <namespace> -c <container>

⚒ Test Cases

Understanding the challenge

Clients must reach the broker with MQTT over TLS on port 8883, but Mosquitto speaks only plain MQTT and listens only on 127.0.0.1:1883 inside its Pod. The missing piece is a process in the same Pod that accepts TLS connections on port 8883, decrypts them with the certificate from the Secret mqtt-tls, and forwards plain MQTT to 127.0.0.1:1883. That is the job of the HAProxy sidecar.

The broker is managed from the manifests in /home/laborant/mqtt-broker/, so the change goes into deployment.yaml, and kubectl apply brings it to the cluster. The other files stay as they are.

The HAProxy sidecar ends TLS on port 8883 and forwards plain MQTT to Mosquitto on 127.0.0.1:1883 inside the same Pod.

Sidecar containers

A Pod can run several containers. A sidecar is a helper container that runs next to the main application container and adds a feature without changing the application. Here, HAProxy adds TLS to Mosquitto.

Containers in one Pod share:

• The network namespace. They have the same IP address and the same localhost. That is why HAProxy can forward to 127.0.0.1:1883, and why nothing outside the Pod can reach Mosquitto directly.
• The Pod's volumes. Each container mounts only the volumes it needs, at its own paths.
• The Pod's life cycle. The containers run on the same node and are created and removed together with the Pod. The Pod is Ready only when all of its containers are ready, so READY shows 2/2.

ConfigMaps and Secrets as files

HAProxy needs two files: its configuration and the certificate. Both exist as Kubernetes objects, and a container sees them as files in two steps:

• A volume in the Pod template points to the ConfigMap or the Secret.
• A volume mount in the container places that volume at a directory. Every key becomes a file with the same name.

So the ConfigMap key haproxy.cfg mounted at /usr/local/etc/haproxy becomes /usr/local/etc/haproxy/haproxy.cfg, the file the HAProxy image reads at start. The Secret key mqtt.pem mounted at /etc/haproxy/certs becomes /etc/haproxy/certs/mqtt.pem, the file named in haproxy.cfg.

Step 1: Look at the broker

Run all commands on dev-machine, where the manifests are. The Terminal 1 and Terminal 2 links in Steps 5 and 6 open new terminals on dev-machine. The output in these steps comes from a real run. Your Pod names, IP addresses and times will be different.

The manifests of the broker:

ls /home/laborant/mqtt-broker

deployment.yaml
haproxy-configmap.yaml
mosquitto-configmap.yaml
mosquitto-passwd-secret.yaml
service.yaml

What runs in the cluster:

kubectl get deployment,pods,service -n iot-platform

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mqtt-broker   1/1     1            1           8s

NAME                               READY   STATUS    RESTARTS   AGE
pod/mqtt-broker-7bb4689989-pjk9h   1/1     Running   0          8s
pod/sensor-client                  1/1     Running   0          8s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/mqtt-service   ClusterIP   10.43.158.194   <none>        8883/TCP   8s

The broker Pod shows 1/1: it runs only the mosquitto container.

Try to publish over TLS from sensor-client:

kubectl exec -n iot-platform sensor-client -- mosquitto_pub -h mqtt-service -p 8883 --cafile /certs/ca.crt -u sensor -P Sens0r-Pass-2026 -t factory/line1/temperature -m 21.5

Error: Bad file descriptor
command terminated with exit code 1

The Service forwards the connection to port 8883 of the broker Pod, but no process in that Pod listens on 8883, so the connection fails.

Step 2: Read the configuration the sidecar uses

cat /home/laborant/mqtt-broker/haproxy-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mqtt-haproxy-config
  namespace: iot-platform
data:
  haproxy.cfg: |
    global
      log stdout format raw local0

    defaults
      mode tcp
      log global
      timeout connect 5s
      timeout client 1h
      timeout server 1h

    frontend mqtt_tls
      bind *:8883 ssl crt /etc/haproxy/certs/mqtt.pem
      default_backend mqtt_broker

    backend mqtt_broker
      server mosquitto 127.0.0.1:1883 check

• mode tcp: HAProxy passes the bytes through without reading MQTT.
• bind *:8883 ssl crt /etc/haproxy/certs/mqtt.pem: HAProxy listens for TLS on port 8883 and loads the certificate and key from /etc/haproxy/certs/mqtt.pem.
• server mosquitto 127.0.0.1:1883: HAProxy forwards plain MQTT to 127.0.0.1:1883. This works only if HAProxy runs in the same Pod as Mosquitto.

Where Mosquitto listens:

cat /home/laborant/mqtt-broker/mosquitto-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mosquitto-config
  namespace: iot-platform
data:
  mosquitto.conf: |
    listener 1883 127.0.0.1
    allow_anonymous false
    password_file /mosquitto/auth/passwordfile
    persistence true
    persistence_location /mosquitto/data/
    log_timestamp_format %H:%M:%S
    log_type all

• listener 1883 127.0.0.1: Mosquitto accepts plain MQTT only from inside its own Pod.
• log_type all: Mosquitto logs every MQTT packet, so its logs show each message it receives and each message it sends to a subscriber.

The Secret mqtt-tls is not in the folder. It holds one key, mqtt.pem, which becomes the file HAProxy loads.

These files explain the two mount paths from the task:

• ConfigMap mqtt-haproxy-config at /usr/local/etc/haproxy: the HAProxy image reads /usr/local/etc/haproxy/haproxy.cfg at start (see the haproxy image page).
• Secret mqtt-tls at /etc/haproxy/certs: the bind line in haproxy.cfg loads /etc/haproxy/certs/mqtt.pem.

Step 3: Add the HAProxy sidecar to deployment.yaml

Open the manifest:

vi /home/laborant/mqtt-broker/deployment.yaml

Make two additions to the Pod template:

• A second entry under containers, after mosquitto, for the haproxy container.
• Two entries at the end of volumes, for the ConfigMap and the Secret the new container mounts.

The mosquitto container and the rest of the file stay as they are. The updated file, with the new lines highlighted:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mqtt-broker
  namespace: iot-platform
  labels:
    app: mqtt
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mqtt
  template:
    metadata:
      labels:
        app: mqtt
    spec:
      initContainers:
        - name: prepare-passwordfile
          image: public.ecr.aws/docker/library/eclipse-mosquitto:2.1.2-alpine
          command:
            - sh
            - -c
            - cp /secret/passwordfile /mosquitto/auth/passwordfile && chown mosquitto:mosquitto /mosquitto/auth/passwordfile && chmod 0600 /mosquitto/auth/passwordfile
          volumeMounts:
            - name: mosquitto-passwd
              mountPath: /secret
              readOnly: true
            - name: mosquitto-auth
              mountPath: /mosquitto/auth
      containers:
        - name: mosquitto
          image: public.ecr.aws/docker/library/eclipse-mosquitto:2.1.2-alpine
          ports:
            - containerPort: 1883
              name: mqtt
          volumeMounts:
            - name: mosquitto-config
              mountPath: /mosquitto/config/mosquitto.conf
              subPath: mosquitto.conf
              readOnly: true
            - name: mosquitto-auth
              mountPath: /mosquitto/auth
            - name: mqtt-data
              mountPath: /mosquitto/data
        - name: haproxy                            # the sidecar container
          image: public.ecr.aws/docker/library/haproxy:3.2.26-alpine
          ports:
            - containerPort: 8883                  # the port the Service sends traffic to
              name: mqtt-tls
          volumeMounts:
            - name: haproxy-config
              mountPath: /usr/local/etc/haproxy    # the image reads haproxy.cfg from here
              readOnly: true
            - name: tls-certs
              mountPath: /etc/haproxy/certs        # haproxy.cfg loads mqtt.pem from here
              readOnly: true
      volumes:
        - name: mosquitto-config
          configMap:
            name: mosquitto-config
        - name: mosquitto-passwd
          secret:
            secretName: mosquitto-passwd
        - name: mosquitto-auth
          emptyDir: {}
        - name: mqtt-data
          emptyDir: {}
        - name: haproxy-config                     # files from the ConfigMap
          configMap:
            name: mqtt-haproxy-config
        - name: tls-certs                          # files from the Secret
          secret:
            secretName: mqtt-tls

Check the file against the API server before you apply it:

kubectl apply -f /home/laborant/mqtt-broker/deployment.yaml --dry-run=server

deployment.apps/mqtt-broker configured (server dry run)

Apply it:

kubectl apply -f /home/laborant/mqtt-broker/deployment.yaml

deployment.apps/mqtt-broker configured

Step 4: Watch the rollout

The Pod template changed, so the Deployment creates a new Pod and removes the old one:

kubectl rollout status deployment/mqtt-broker -n iot-platform

Waiting for deployment "mqtt-broker" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "mqtt-broker" rollout to finish: 1 old replicas are pending termination...
deployment "mqtt-broker" successfully rolled out

kubectl get pods -n iot-platform

NAME                           READY   STATUS        RESTARTS   AGE
mqtt-broker-744b5d645f-dwch8   2/2     Running       0          9s
mqtt-broker-7bb4689989-pjk9h   1/1     Terminating   0          29s
sensor-client                  1/1     Running       0          29s

The new Pod shows 2/2: both containers are ready. The old Pod stays Terminating for a few seconds. Run kubectl get pods -n iot-platform again until only the new Pod is listed, because kubectl logs deployment/mqtt-broker can pick the old Pod, which has no haproxy container:

NAME                           READY   STATUS    RESTARTS   AGE
mqtt-broker-744b5d645f-dwch8   2/2     Running   0          10s
sensor-client                  1/1     Running   0          30s

HAProxy loaded its configuration and the certificate:

kubectl logs -n iot-platform deployment/mqtt-broker -c haproxy

[NOTICE]   (1) : Initializing new worker (9)
[NOTICE]   (1) : Loading success.

The challenge checks also connect to the broker, so a few Connect from ... lines can already follow these two.

Step 5: Publish and subscribe over TLS

In Terminal 1, subscribe and wait for one message:

kubectl exec -n iot-platform sensor-client -- mosquitto_sub -h mqtt-service -p 8883 --cafile /certs/ca.crt -u sensor -P Sens0r-Pass-2026 -t factory/line1/temperature -C 1 -v

In Terminal 2, publish a reading:

kubectl exec -n iot-platform sensor-client -- mosquitto_pub -h mqtt-service -p 8883 --cafile /certs/ca.crt -u sensor -P Sens0r-Pass-2026 -t factory/line1/temperature -m 21.5

mosquitto_pub prints nothing when it succeeds. The subscriber in Terminal 1 prints the topic and the message, then exits:

factory/line1/temperature 21.5

This line means the subscriber received the message. If Terminal 1 keeps waiting after the publish, the message did not reach it.

• -h mqtt-service is the DNS name of the Service. sensor-client runs in the same namespace, so the short name is enough. mqtt-service.iot-platform and mqtt-service.iot-platform.svc.cluster.local work too.
• --cafile /certs/ca.crt tells the client to trust the CA that signed the broker certificate.
• The client checks that the name in -h is in the certificate. The certificate holds only the DNS names of the Service (mqtt-service, mqtt-service.iot-platform, mqtt-service.iot-platform.svc and mqtt-service.iot-platform.svc.cluster.local), not an IP address. A connection to the Service IP or the Pod IP fails with Error: Protocol error.

Step 6: Confirm that the connection uses TLS

The subscriber received the message, but that alone does not show that the traffic was encrypted. openssl s_client opens a TLS connection and prints what the client and the server agreed on.

sensor-client has no openssl, but dev-machine has. dev-machine is outside the cluster network, so kubectl port-forward opens a tunnel from a local port to the Service.

Save the CA certificate from the ConfigMap mqtt-ca to a file:

kubectl get configmap mqtt-ca -n iot-platform -o jsonpath='{.data.ca\.crt}' > /home/laborant/ca.crt

In Terminal 1, forward local port 8883 to the Service. The command keeps running:

kubectl port-forward -n iot-platform service/mqtt-service 8883:8883

Forwarding from 127.0.0.1:8883 -> 8883
Forwarding from [::1]:8883 -> 8883

In Terminal 2, open a TLS connection through the tunnel:

openssl s_client -connect localhost:8883 -servername mqtt-service -verify_hostname mqtt-service -CAfile /home/laborant/ca.crt -brief < /dev/null

• -connect localhost:8883: the local end of the tunnel.
• -servername mqtt-service and -verify_hostname mqtt-service: the name to check in the certificate. The connection goes to localhost, so the name is given separately.
• -CAfile /home/laborant/ca.crt: trust the CA that signed the broker certificate, like --cafile for mosquitto_pub.
• -brief: print only a short summary of the connection.
• < /dev/null: close the connection right after the handshake, instead of waiting for input.

CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN = mqtt-service.iot-platform.svc
Hash used: SHA256
Signature type: RSA-PSS
Verification: OK
Verified peername: mqtt-service
Server Temp Key: X25519, 253 bits
DONE

• Protocol version: TLSv1.3 and Ciphersuite: TLS_AES_256_GCM_SHA384: the connection is encrypted with TLS 1.3.
• Peer certificate: CN = mqtt-service.iot-platform.svc: HAProxy presented the certificate from the Secret mqtt-tls.
• Verification: OK and Verified peername: mqtt-service: the certificate is signed by the trusted CA and is valid for the name mqtt-service.

Stop the port-forward in Terminal 1 with Ctrl+C. Mosquitto itself listens only on 127.0.0.1:1883, so every client that reaches it came through this TLS port.

Summary

• Problem: Mosquitto listens only on 127.0.0.1:1883 with plain MQTT, and nothing in the Pod listened on 8883.
• Fix: a second container, haproxy, in the Pod template of deployment.yaml, listening on 8883, applied with kubectl apply.
• Configuration: ConfigMap mqtt-haproxy-config mounted read-only at /usr/local/etc/haproxy, where the image reads haproxy.cfg.
• Certificate: Secret mqtt-tls mounted read-only at /etc/haproxy/certs, the path named in haproxy.cfg.
• Why 127.0.0.1 works: containers in one Pod share the network namespace.
• Proof: the Pod shows 2/2, the subscriber receives 21.5 over TLS, and openssl s_client shows a verified TLS 1.3 connection.