Lesson  in  Workload Identity from Scratch: Re-Inventing SPIFFE with OpenSSL, MariaDB, SoftHSM2, and a TPM

mTLS Identity: Why a Valid Certificate Is Not Enough

In mutual TLS, "the certificate chains to our CA" only proves the holder has some cert we issued, not who they are. Build a CA, hand the same trust to two workloads, and watch one impersonate the other until you bind identity to the certificate subject. Done end to end against MariaDB.

One CA, Two Identities

In Lab 1 the browser checked the server and that was the whole story: one-directional trust. Mutual TLS adds the other direction. Now the server checks the client's certificate too, and that is where a subtle, dangerous gap opens up.

A server doing mTLS verifies two things, and it is easy to do only the first:

  1. Chain validity. Does this client cert chain up to a CA I trust? This proves only that the holder has a certificate we issued.
  2. Subject match. Is this the specific identity I expect? This proves it is this workload, not merely anyone holding a cert from our CA.

Skip the second check and you get the flatness trap: every workload that owns any cert from your CA can impersonate every other one. In this lab you will build that trap on purpose with MariaDB, watch one identity log in as another, then close it by binding the account to the certificate subject.

The mental model, a cluster or a fleet is a private PKI

On the public web you trust Mozilla's root list. In a private fleet you are Mozilla: you mint one root, distribute its public cert to every verifier, and that distribution is how "the crowd knows who you trust." Each workload holds its own key and proves its own identity; the CA only signs. The server then checks chain validity (is this one of ours) and subject (is this specifically billing). Chain validity without subject binding is the flat network where one stolen cert is every cert.

Step 1: Build one CA and two identities

Install cfssl (Cloudflare's PKI CLI: JSON in, key plus signed cert out in one pipeline).

sudo apt-get update && sudo apt-get install -y golang-cfssl

Create a working directory and the CA definitions:

mkdir -p ~/pki && cd ~/pki

cat > ca-csr.json <<'JSON'
{ "CN": "Demo Root CA", "key": { "algo": "ecdsa", "size": 256 }, "names": [{ "O": "Demo" }] }
JSON

cat > ca-config.json <<'JSON'
{ "signing": {
    "default": { "expiry": "8760h" },
    "profiles": {
      "server": { "expiry": "8760h", "usages": ["signing","key encipherment","server auth"] },
      "client": { "expiry": "8760h", "usages": ["signing","key encipherment","client auth"] }
    } } }
JSON

Generate the root, then a server cert for the database and two client certs from the very same CA:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

echo '{"CN":"db.local","hosts":["db.local","127.0.0.1"],"key":{"algo":"ecdsa","size":256}}' \
  | cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile server - \
  | cfssljson -bare server

echo '{"CN":"billing.internal","key":{"algo":"ecdsa","size":256}}' \
  | cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile client - \
  | cfssljson -bare billing

echo '{"CN":"intruder.internal","key":{"algo":"ecdsa","size":256}}' \
  | cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile client - \
  | cfssljson -bare intruder

The [WARNING] ... lacks a "hosts" field cfssl prints for the client certs is expected: client certs are never addressed as servers, so they need no SAN.

Check it. Both client certs are different identities that both chain to the one CA. That dual fact is the entire lab:

openssl verify -CAfile ca.pem billing.pem intruder.pem   # expect: both OK
openssl x509 -in billing.pem  -noout -subject            # subject=CN = billing.internal
openssl x509 -in intruder.pem -noout -subject            # subject=CN = intruder.internal

Step 2: Stand up MariaDB with TLS

Install the server and give it the CA plus its own cert and key:

sudo apt-get install -y mariadb-server
sudo mkdir -p /etc/mysql/ssl
sudo cp ~/pki/ca.pem ~/pki/server.pem /etc/mysql/ssl/
sudo cp ~/pki/server-key.pem /etc/mysql/ssl/
sudo chown -R mysql:mysql /etc/mysql/ssl && sudo chmod 640 /etc/mysql/ssl/server-key.pem

Point the daemon at them:

sudo tee /etc/mysql/mariadb.conf.d/99-ssl.cnf > /dev/null <<'CNF'
[mariadbd]
ssl-ca   = /etc/mysql/ssl/ca.pem
ssl-cert = /etc/mysql/ssl/server.pem
ssl-key  = /etc/mysql/ssl/server-key.pem
CNF
sudo systemctl restart mariadb

Map db.local so the client can verify the server against its SAN (this is the Lab 1 direction, still in force). Note that the MariaDB client does not check the server's name unless you ask it to, which is why every client command below passes --ssl-verify-server-cert:

echo "127.0.0.1 db.local" | sudo tee -a /etc/hosts > /dev/null

Check it. The server should now offer TLS:

sudo mariadb -N -e "SHOW STATUS LIKE 'Ssl_cipher'; SELECT @@have_ssl;"

The Flatness Trap

Step 3: The flatness trap, demonstrated

Create an account that requires only that the client present some cert from our CA:

sudo mariadb <<'SQL'
CREATE USER 'billing'@'%' REQUIRE X509;
GRANT ALL PRIVILEGES ON *.* TO 'billing'@'%';
FLUSH PRIVILEGES;
SQL

Log in as billing, presenting billing's cert. This is the legitimate path:

cd ~/pki
mariadb --protocol=tcp -h db.local --ssl-ca=ca.pem --ssl-verify-server-cert \
  --ssl-cert=billing.pem --ssl-key=billing-key.pem \
  -u billing -e "SELECT CURRENT_USER();"

Now log in as billing again, but present the intruder cert:

mariadb --protocol=tcp -h db.local --ssl-ca=ca.pem --ssl-verify-server-cert \
  --ssl-cert=intruder.pem --ssl-key=intruder-key.pem \
  -u billing -e "SELECT CURRENT_USER();"

Check it. Both succeeded. The intruder cert, a completely different identity, logged in as billing, because REQUIRE X509 only asked "is this cert from our CA," and it was. This is the flatness trap: chain validity is not identity.

Binding Identity to the Subject

Step 4: Bind identity to the subject

Recreate the account, this time pinning it to the exact certificate subject:

sudo mariadb <<'SQL'
DROP USER 'billing'@'%';
CREATE USER 'billing'@'%' REQUIRE SUBJECT '/CN=billing.internal';
GRANT ALL PRIVILEGES ON *.* TO 'billing'@'%';
FLUSH PRIVILEGES;
SQL

billing's own cert still works:

mariadb --protocol=tcp -h db.local --ssl-ca=ca.pem --ssl-verify-server-cert \
  --ssl-cert=billing.pem --ssl-key=billing-key.pem \
  -u billing -e "SELECT CURRENT_USER();"

The intruder cert is now rejected, even though it is still perfectly valid and still chains to the trusted CA:

mariadb --protocol=tcp -h db.local --ssl-ca=ca.pem --ssl-verify-server-cert \
  --ssl-cert=intruder.pem --ssl-key=intruder-key.pem \
  -u billing -e "SELECT 1;"

Check it. The second command fails with ERROR 1045 (28000): Access denied. The cert is valid; it is simply not billing. That is the subject binding doing the job REQUIRE X509 could not.

Important

MariaDB matches the subject in the slash-separated form, /CN=billing.internal. The comma form CN=billing.internal will silently fail to match and you will get Access denied with a valid cert. Read the exact string with openssl x509 -in billing.pem -noout -subject -nameopt compat (it prints the slash form) and use that verbatim. MySQL 8 differs here, so never copy a REQUIRE SUBJECT string between the two without checking.

Recap

You issued two valid identities from one CA and proved the two distinct checks a mTLS server must make:

  • Chain validity answers "is this one of ours." With REQUIRE X509 alone, any cert from your CA impersonated billing.
  • Subject match answers "is this billing specifically." REQUIRE SUBJECT rejected a valid-but-wrong identity.

The lesson generalizes far past databases: every mTLS system (service meshes, Kubernetes components, internal APIs) makes exactly these two checks, and the flatness trap is what you get whenever the second one is missing. In Lab 3 we make these certs short-lived and watch what that buys you.