Cert-Manager

certificates have 90 days life time and rotated 30 days before expire, there is certmanager_certificate_expiration_timestamp_seconds prometheus metric to watch for this

Installation

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml

Wait for pods in Cert-manager namespace to be up and running

kubectl -n cert-manager get po

note: to upgrade just apply new version but before read upgrading notes if any

You need to wait few minutes after installing certmanager before proceeding, underneath it will create and sign self signed certificate for webhooks which will be used by cainjector and while it is not done trying to create cluster issues will error:

Error from server (InternalError): error when creating "issuer.yml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority

Cluster Issuer

Next, we need to create cluster wide certificate issuer, done once per cluster

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: fRVcLb3iZ0RHF9YL2Y4yGWDx11XvO_MMBt-jvOHh
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: svc@rabota.ua
    privateKeySecretRef:
      name: letsencrypt
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token

Notes:

if everything fine kubectl get clusterissuer should show our letstencrypt issuer in ready state

Wildcard Certificate

To not create certificates each time we may want to create them once and reuse them

---
apiVersion: v1
kind: Secret
metadata:
  name: wildcard-tls
  namespace: dev
type: kubernetes.io/tls
stringData:
  tls.key: ""
  tls.crt: ""
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-tls
  namespace: dev
spec:
  secretName: wildcard-tls
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt
  dnsNames:
    - "dev.rabota.ua"
    - "*.dev.rabota.ua"

Notes:

secret in this example is added just for kubectl delete -f will delete it also
secret will be created or updated and filled with tls keys
need to be done per each namespace
wait till kubectl -n dev get certificate wildcard-tls will become ready (may take up to few minutes, in my initial case it was almost 3min)
after that kubectl -n dev get secret wildcard-tls -o yaml should be filled by certificates

Ingress wildcard usage example

For ingress to use an pre-created wildcard certificate we need to add the following to its spec:

tls:
  - hosts:
      - demo.dev.rabota.ua
    secretName: wildcard-tls

here is a deployment example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: dev
  labels:
    app: demo
spec:
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: demo
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: demo
  namespace: dev
spec:
  type: ClusterIP
  selector:
    app: demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
  namespace: dev
  labels:
    app: demo
spec:
  # ADDED
  tls:
    - hosts:
        - demo.dev.rabota.ua
      secretName: wildcard-tls
  # -----

  ingressClassName: external
  rules:
    - host: demo.dev.rabota.ua
      http:
        paths:
          - backend:
              service:
                name: demo
                port:
                  number: 80
            path: /
            pathType: ImplementationSpecific

and to check:

curl -s -i -v --resolve demo.dev.rabota.ua:443:20.13.179.68  https://demo.dev.rabota.ua/

also kubectl get ing demo should state that it is listening not only port 80 but 443 as well

to cleanup demo run kubectl delete ing,svc,deployment demo -n dev

Ingress personal certificate per service

For this to work DNS record should be precreated and pointing to cluster, otherwise certificate wont be created. Do not forget that domain for this example must be demo.dev. Do not forget that in future access will be allowed only from cloudflare so proxy mode should be turned on.

In case we need to have a dedicated certificate per service we may do so by applying the following to ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
  namespace: dev
  labels:
    app: demo

  # ADDED
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
  # -----

spec:
  # ADDED
  tls:
    - hosts:
        - demo.dev.rabota.ua
      secretName: demo-tls
  # -----

  ingressClassName: external
  rules:
    - host: demo.dev.rabota.ua
      http:
        paths:
          - backend:
              service:
                name: demo
                port:
                  number: 80
            path: /
            pathType: ImplementationSpecific

So technically we are notifying cert manager about the need of new certificate by adding cert-manager.io/issuer: letsencrypt annotation to ingress.

After that kubectl get certificates will show our new demo-tls certificate and in few minutes when it will become ready kubectl get secrets demo-tls -o yaml will contain our certs (in my case I did not created DNS and realized that only after 7mins of looking around, after adding DNS it took

All this may be checked the same as before:

curl -s -i -v --resolve demo.dev.rabota.ua:443:20.13.179.68  https://demo.dev.rabota.ua/

And to cleanup:

kubectl -n dev delete ing,svc,deployment demo

# certificate will be deleted automaticaly by cert-manager
#kubectl -n dev delete certificate demo-tls

kubectl -n dev delete secret demo-tls

Uninstall

Delete all resources (except secrets) created by cert-manager

kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces

Delete cert-manager

kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml

Delete namespace

kubectl delete ns cert-manager

Cloudflare Token

https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/

cert-manager

xxxxxxxxxxxxxxxxxxxxxx

curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
     -H "Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxx" \
     -H "Content-Type:application/json"

Command line tool

https://cert-manager.io/docs/reference/cmctl/

brew install cmctl

Manual certificate renew

kubectl -n dev get certificates
cmctl status certificate wildcard-tls -n dev
cmctl renew wildcard-tls -n dev
# Manually triggered issuance of Certificate dev/wildcard-tls
``

## Troubleshooting

[https://cert-manager.io/docs/troubleshooting/webhook/#error-x509-certificate-signed-by-unknown-authority](https://cert-manager.io/docs/troubleshooting/webhook/#error-x509-certificate-signed-by-unknown-authority)

Certmanager website has very detailed troubleshooting guides

Example of troubleshooting dedicated certificate

```bash
kubectl get certificate demo-tls

Shows as not ready for more than 5 minutes

kubectl get cecrtificaterequest

Shows that our request is approved but not ready

kubectl describe certificaterequest demo-tls-rkn65

Explains what wen wrong, in my case initially I had wrong configuration of ingress so it was not able to find issuer

Normal  IssuerNotFound   9m40s  cert-manager-certificaterequests-issuer-acme        Referenced "Issuer" not found: issuer.cert-manager.io "letsencrypt" not found

https://cert-manager.io/docs/troubleshooting/

By mistake in ingress annotation I have asked for issuer rather than cluster issuer, and because there is no such issuer in dev namespace certificate creation process stuck