Skip to main content

EMQX with CRL and OCSP stapling

This how-to walks through configuring an EMQX broker deployed as an Avassa application to use Avassa's strongbox CA for two complementary revocation paths:

  • CRL — EMQX periodically fetches the CRL from supd and rejects client certificates that have been revoked. This is the right default: clients need do nothing, the server enforces revocation.
  • OCSP stapling — EMQX periodically fetches an OCSP response for its own server certificate from supd, caches it, and staples it into every TLS handshake. Clients can verify in one round trip that the server certificate is still valid, without contacting the responder themselves.

The two paths are independent and can be enabled together. CRL revokes clients; OCSP stapling proves the server cert is still good.

Prerequisites​

  • An Avassa environment with at least one edge site available (udc1, udc2, …).
  • A tenant configured in Control Tower with permission to create strongbox CAs, vaults, and applications.
  • The supctl client and access to topdc.

The examples below assume the site is named udc2 and the tenant has a deployment named emqx. Adjust to match your environment.

Look up the tenant UUID​

OCSP URLs identify the tenant by UUID (CRL URLs accept either form). Get the UUID once and reuse it when constructing URLs:

supctl do strongbox get-tenant-uuid

This returns a UUID such as 5a3e…-ab12. The CRL endpoints are then:

http://api.internal:4664/crl/<tenant-uuid>/<ca-name>/<version>
http://api.internal:4664/crl-pem/<tenant-uuid>/<ca-name>/<version>

and the OCSP responder is at:

http://api.internal:4664/ocsp/<tenant-uuid>/<ca-name>

The api.internal hostname is reachable from inside any container deployed on a site. The host port 4664 is supd's CRL/OCSP listener.

note

/crl/... returns DER (application/pkix-crl, RFC 5280 mandated encoding); /crl-pem/... returns the same CRL re-encoded as PEM. EMQX 5's CRL fetcher tries public_key:pem_decode/1 first and only falls back to DER, so pointing the CA's crl-dist-urls at the PEM endpoint is the most reliable choice.

Step 1: create the CA​

Create a CA dedicated to MQTT certificates. Pin the CRL and OCSP URLs that should be baked into every certificate the CA issues, and distribute the CA cert to the emqx deployment so the trust bundle follows the application to the site:

supctl create strongbox tls ca <<EOF
name: mqtt-ca
ttl: 1y350d
cert-key-type: ecdsa
cert-key-curve: secp256r1
digest: sha256
crl-dist-urls:
  - http://api.internal:4664/crl-pem/\${SYS_TENANT_NAME}/\${SYS_CA_NAME}/\${SYS_CA_VERSION}
ocsp-responder-urls:
  - http://api.internal:4664/ocsp/\${SYS_TENANT_UUID}/\${SYS_CA_NAME}
distribute:
  deployments:
    - emqx
EOF

The crl-dist-urls ends up in each issued certificate's CRL Distribution Points extension, and ocsp-responder-urls ends up in each certificate's Authority Information Access (AIA) OCSP URI. Some TLS stacks discover both URLs from those extensions; others prefer to be told explicitly via configuration. EMQX is in the latter camp — see Step 4 — so the values are also plumbed straight into the EMQX listener configuration.

Due to a bug in some versions of EMQX (5.10 and earlier at least) it only accepts PEM encoded CRLs for updates. To allow for this we configure the .../crl-pem/... URL as crl-dist-urls above.

Step 2: vault and auto-rotated server certificate​

Create a vault that distributes to the same deployment, and an auto-cert secret that will issue and rotate the EMQX server certificate signed by mqtt-ca:

supctl create strongbox vaults <<EOF
name: mqtt-vault
distribute:
  deployments:
    - emqx
EOF
supctl create strongbox vaults mqtt-vault secrets <<EOF
name: server-cert
allow-image-access: [ "*" ]
auto-cert:
  issuing-ca: mqtt-ca
  host: mqtt.emqx.internal
  cert-type: server
  ttl: 30d
  refresh-threshold: 15d
EOF

The secret produces three files (cert.pem, cert.key, ca-cert.pem) that you mount into the EMQX container in Step 4.

Step 3: issue a client certificate​

Issue a client certificate manually (in production you would normally issue these to clients via your provisioning flow):

supctl do strongbox tls ca mqtt-ca issue-cert --input - <<EOF > client-cert.json
ttl: 1d
host: mqtt-client.example
cert-type: client
EOF

jq -r .cert            client-cert.json > client.pem
jq -r '."private-key"' client-cert.json > client.key
jq -r .serial          client-cert.json   # save for later revocation

Fetch the CA certificate so external clients can validate the chain:

supctl do strongbox tls ca mqtt-ca get-ca-cert | jq -r .cert > mqtt-ca.pem

Step 4: deploy EMQX​

EMQX 5 reads its listener configuration from environment variables. Each __ becomes a config-path component, so for example EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__CERTFILE corresponds to the HOCON path listeners.ssl.default.ssl_options.certfile.

The example below enables both CRL and OCSP stapling on a single TLS listener at port 8883. If you only want one of the two, simply leave out the corresponding block of variables.

supctl create applications <<EOF
name: emqx
version: "1.0"
services:
  - name: mqtt
    mode: replicated
    replicas: 1
    volumes:
      - name: emqx-certs
        vault-secret:
          vault: mqtt-vault
          secret: server-cert
          # EMQX runs as uid 1000 in the container; the vault-secret
          # default of 400 root:root is unreadable by the emqx user.
          file-ownership: 1000:1000
      - name: emqx-data
        ephemeral-volume: { size: 100MiB, file-mode: "777" }
      - name: emqx-log
        ephemeral-volume: { size: 100MiB, file-mode: "777" }
    containers:
      - name: emqx
        image: emqx/emqx:5.10.3
        env:
          EMQX_LISTENERS__SSL__DEFAULT__BIND: "0.0.0.0:8883"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__CERTFILE:    "/etc/emqx-certs/cert.pem"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__KEYFILE:     "/etc/emqx-certs/cert.key"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__CACERTFILE:  "/etc/emqx-certs/ca-cert.pem"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__VERIFY: "verify_peer"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__FAIL_IF_NO_PEER_CERT: "true"

          # --- CRL: EMQX checks client certs against the CRL ---
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__ENABLE_CRL_CHECK: "true"
          EMQX_CRL_CACHE__REFRESH_INTERVAL: "5m"
          EMQX_CRL_CACHE__HTTP_TIMEOUT:     "15s"

          # --- OCSP stapling: EMQX staples a response for its own cert ---
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__OCSP__ENABLE_OCSP_STAPLING: "true"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__OCSP__ISSUER_PEM:           "/etc/emqx-certs/ca-cert.pem"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__OCSP__RESPONDER_URL:        "http://api.internal:4664/ocsp/<tenant-uuid>/mqtt-ca"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__OCSP__REFRESH_INTERVAL:     "5m"
          EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__OCSP__REFRESH_HTTP_TIMEOUT: "15s"
        mounts:
          - volume-name: emqx-certs
            files:
              - { name: cert.pem,    mount-path: /etc/emqx-certs/cert.pem }
              - { name: cert.key,    mount-path: /etc/emqx-certs/cert.key }
              - { name: ca-cert.pem, mount-path: /etc/emqx-certs/ca-cert.pem }
          - volume-name: emqx-data
            mount-path: /opt/emqx/data
          - volume-name: emqx-log
            mount-path: /opt/emqx/log
    network:
      ingress-ip-per-instance:
        protocols:
          - { name: tcp, port-ranges: "8883" }
        access: { allow-all: true }
EOF

Then deploy to the chosen site:

supctl create application-deployments <<EOF
name: emqx
application: emqx
application-version: "1.0"
placement:
  match-site-labels: >
    system/name = udc2
EOF

Configuration notes​

  • enable_crl_check without a populated CRL cache fails closed — EMQX rejects every client until the first CRL refresh succeeds. Watch the EMQX log for crl_cache: refresh OK after startup.
  • ocsp.responder_url must be set explicitly. EMQX does not pull it out of the AIA extension on the cert.
  • ocsp.issuer_pem points at the CA cert that signed the server cert; EMQX needs it to construct the certID in the OCSP request. The same file mounted as cacertfile works.
  • refresh_interval governs how quickly revocations propagate. The defaults (5 minutes for both CRL and OCSP) are appropriate for production. For testing, drop them to 5s.

Step 5: verify the golden path​

Get the ingress IP for the EMQX service instance:

supctl show --site udc2 applications emqx | grep -A2 ingress

CRL: verify a valid client connects​

echo | openssl s_client -connect <emqx-ip>:8883 \
    -CAfile mqtt-ca.pem -cert client.pem -key client.key \
    -servername mqtt.emqx.internal -verify_return_error

The handshake completes and Verify return code: 0 (ok) is printed.

OCSP stapling: verify the server cert is stapled​

Add -status to ask openssl to print the stapled OCSP response:

echo | openssl s_client -connect <emqx-ip>:8883 -status \
    -CAfile mqtt-ca.pem -cert client.pem -key client.key \
    -servername mqtt.emqx.internal

The output now includes an OCSP block such as:

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Responses:
    Certificate ID:
      ...
    Cert Status: good
    This Update: ...
    Next Update: ...

If the response shows OCSP response: no response sent, EMQX has not yet fetched its first OCSP response. Wait one refresh_interval and try again.

Step 6: revoke a certificate​

Revoke the client certificate using the serial number captured in Step 3:

supctl do strongbox tls ca mqtt-ca \
    revoke-cert --reason keyCompromise <serial>

Inspect the new CRL to confirm the serial is listed:

supctl do strongbox tls ca mqtt-ca get-crl | jq -r .crl | openssl crl -text -noout

Within one CRL refresh interval EMQX picks up the new CRL. Re-running the openssl probe with the revoked client cert now ends the handshake with a TLS alert:

SSL alert number 44
... certificate revoked

Revoking the server certificate works the same way; the next OCSP refresh causes EMQX to staple a revoked response. Clients running openssl s_client -status will see Cert Status: revoked and can take the server out of rotation.

Troubleshooting​

CRL fetched but TLS handshake fails with bad_crls​

Ensure the CRL endpoint baked into the cert is crl-pem, not crl: EMQX 5's CRL fetcher prefers PEM and the DER fallback path is fragile. Re-issue the CA with crl-dist-urls pointing at /crl-pem/... and rotate the server cert.

OCSP stapling never activates​

Check the EMQX log (supctl do volga query-topics …) for the OCSP fetch outcome. Common causes:

  • responder_url points at a host name that doesn't resolve inside the container.
  • issuer_pem is missing or unreadable — same file-mode fix as above.
  • The OCSP responder returns 500 — check supctl show strongbox tls ca mqtt-ca to confirm the CA exists and is healthy on the Control Tower.