Kubernetes HTTPS ingress with ingress-nginx and cert-manager

2021-03-13 15:41:00 +00:00Freyja

In this post, I'll continue the process of setting up a Kubernetes cluster from my last post by setting up our own load balancer, ingress and certificate manager. This lets us easily deploy web services protected by HTTPS in our Kubernetes cluster.

This article is the second in a series:

  1. Setting up self-hosted Kubernetes
  2. Kubernetes HTTPS ingress with ingress-nginx and cert-manager

Installing the load balancer

To let us assign IP addresses to services running in the cluster, we can set up MetalLB and set up a subnet for the cluster. I've chosen the 172.31.0.0/16 subnet, since it doesn't conflict with anything else running on my home network, and set up my home network to route all local traffic for the subnet to the cluster nodes.

The installation is pretty simple, as MetalLB already provides the manifests we need in their installation guide:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

In addition, we need to deploy a config map for MetalLB. In ours we will just assign a range of IP addresses that can be assigned to services:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.31.0.1-172.31.255.254
EOF

Installing the ingress

The next step is to deploy ingress-nginx. The ingress serves multiple purposes:

  • It lets us serve all public HTTP-based services from a single port, routing traffic to the right Kubernetes service based on host and path.
  • It terminals SSL traffic, so that we do not need individual services to care about HTTPS. Additionally, when combined with cert-manager later on, it lets us trivially issue TLS certificates for our services.

In this case we also start out by just deploying the manifests for ingress-nginx found in the installation guide:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml

The default setup will use NodePort for the service, meaning that HTTP and HTTPS gets assigned to some arbitrary ports on each node. However, we may want to serve on port 80 and 443 on a static IP assigned by the load balancer we set up in the last step. To do this, we just need to change the service type to LoadBalancer, and optionally specify a static IP from the subnet that was set up for MetalLB:

# run this command to edit a deployed service manifest:
kubectl edit -n ingress-nginx svc/ingress-nginx-controller

 ...
 spec:
-  type: NodePort
+  type: LoadBalancer
+  loadBalancerIP: 172.31.0.1    # static IP for ingress-nginx
   ...

After this change has been applied, we can test out our ingress by connecting to it. Since we haven't configured any services yet, we'd expect it to just always give a "404 Not Found" error:

# check HTTP:
curl http://172.31.0.1
# check HTTPS (-k is necessary since we haven't set up cert-manager yet)
curl -k https://172.31.0.1

Deploying a web service using the ingress

Now, let's try to deploy a web service. For this, we'll deploy the nginxdemos/hello image, which serves a simple "hello world" page.

First, let's make manifests for the deployment and the service:

kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hello
  labels:
    app: nginx-hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-hello
  template:
    metadata:
      labels:
        app: nginx-hello
    spec:
      containers:
      - name: nginx-hello
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-hello
  labels:
    app: nginx-hello
spec:
  selector:
    app: nginx-hello
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
EOF

To use this service with the ingress, we need to create an ingress manifest as well:

kubectl apply -f - <<EOF
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-hello
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: hello.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-hello
          servicePort: 80
EOF

In the above manifest, we assume that hello.example.com is a DNS name that is mapped to the IP address of the ingress. (In my setup, it's mapped to my home network's public IP, which port forwards traffic to the MetalLB issued IP for the ingress.)

Once we have deployed the ingress, we can test it with curl:

curl 'http://hello.example.com/'

Server address: 10.42.1.7:80
Server name: nginx-hello-8bb945fb-5m2zs
Date: 13/Mar/2021:14:50:46 +0000
URI: /
Request ID: 159f091d524ed851ccf85ce13c0ce833

Adding SSL support with cert-manager

Finally, let's deploy cert-manager to let us issue TLS certificates and integrate that with our ingress setup.

As with the other steps, deploying cert-manager is as simple as just applying the manifest file found in the installation guide:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml

With cert-manager installed, we can set up a cluster-wide ACME issuer to let us issue certificates for our domains. While you can use HTTP to validate your ownership of a domain name, I prefer to use DNS challenges instead, so that's what I have used in my setup.

I'm using Cloudflare as my DNS provider, so I set up my ClusterIssuer to automatically set the needed TXT records for my domain names as I issue certificates for them.

kubectl -n cert-manager create secret generic cloudflare-api-token --from-literal=api-token=<SOME-API-TOKEN>

kubectl -n cert-manager apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: public-ca
spec:
  acme:
    email: user@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: public-ca-account-key
    solvers:
    - dns01:
        cloudflare:
          email: user@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
EOF

Now adding a TLS certificate for our cluster becomes trivial:

kubectl edit ingress/nginx-hello

 ---
 apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   name: nginx-hello
   annotations:
     kubernetes.io/ingress.class: nginx
+    cert-manager.io/cluster-issuer: public-ca
 spec:
   rules:
   - host: hello.example.com
     http:
       paths:
       - path: /
         backend:
           serviceName: nginx-hello
           servicePort: 80
+  tls:
+  - hosts:
+    - hello.example.com
+    secretName: nginx-hello-tls

We can see that a certificate is being requested:

kubectl get certificaterequests

NAME                    READY   AGE
nginx-hello-tls-pptqw   False   42s

After waiting a while, if all goes well it should become ready, and a certificate should be issued and automatically used by the ingress.

We can try to call our service again, but this time over HTTPS, to check that the certificate is working:

curl https://hello.example.com

Server address: 10.42.1.9:8080
Server name: nginx-hello-778774fc79-5b6kp
Date: 13/Mar/2021:15:32:01 +0000
URI: /
Request ID: 221c863e3613b7b431761b3785be5ce3

Success! 😄

Conclusion

For a personal Kubrenetes cluster, I've found that this trifecta of services (metallb, ingress-nginx and cert-manager) are all easy to install and configure, and makes it easy to deploy web services in the cluster and have them be protected by HTTPS, without having to do any custom configuration or HTTPS setup for each service: it's all handled automatically when we create the ingresses.