Prerequisites
Obviously, you need a Kubernetes cluster. Whether this is a full-fledged one running at a cloud provider of your choice or a small k3s single-node cluster on your Raspberry Pi does not matter (much), the setup is pretty generic.
You need to have access to your cluster using kubectl and helm. The instructions below assume that your kubeconfig file or environment variable are pointing to the right cluster, so using plain kubectl
or helm
is enough.
Adding the helm repository for cloudflared
We will be using a community-maintained helm chart from Pascal Iske that is available on the ArtifactHub. Feel free to peruse the instructions on the page.
For the impatient reader, here is the shortcut:
helm repo add pascaliske https://charts.pascaliske.dev
helm repo update
Creating the values.yaml
for the installation via helm
It is advisable to use a values.yaml
file for all installations via helm, rather than using the --set foo=bar
commandline flags. For one, the file can be kept locally, whereas your shell history might easily get lost. Second, you could check that file into your version control system (think git
).
To get an idea what values are supported by the helm chart, use the helm show values pascaliske/cloudflared
command, which gives you the complete defaults from the helm chart.
Create a new file values.yaml
and add the necessary options, maybe something like this:
env:
- name: TZ
value: Europe/Berlin
- name: TUNNEL_DNS_UPSTREAM
value: https://dns.digitale-gesellschaft.ch/dns-query
service:
dns:
enabled: true
type: LoadBalancer
port: 53
metrics:
enabled: true
type: ClusterIP
We specified two sections: environment and service. Let’s have a look at each of them.
The env
section configures the behaviour of the cloudflared application itself inside the pod by specifying environment variables. In our case, we use the Europe/Berlin
timezone and specify which upstream server we want to use. For this example I picked the server hosted by Digitale Gesellschaft in Switzerland, but you can also use the Mozilla server or any DoH server. You can (and should) use more than one server, but for the sake of simplicity let’s start with one.
The next section is for service
, which means it configures the Kubernetes resources related to Services. We enable the dns
service and specify that we want to use a service of type LoadBalancer
. This is the easiest to setup, but might be pricey if you are running it in a public cloud, where each and every LoadBalancer costs a lot of money. In this case you might want to switch to using a service of type ClusterIP
. We will have a short look at how to make your resolver available in this case.
The port
option for the dns
service is where you specify which ports the service should listen on. The cloudflared application running inside the container will always listen on port 5053. But this option allows us to make the service reachable from the outside on port 53 (on the LoadBalancer IP). To make our resolver usable with normal clients, using port 53 is the only sensible option. While you can use different ports when testing with e.g. dig
or drill
, most operation systems do not allow setting a port for DNS resolution in e.g. /etc/resolv.conf
.
We’ll skip the metrics
part for now and have a closer look at it later on.
Installing cloudflared
With the values.yaml
file at hand, we can install cloudflared into our cluster into its own namespace called cloudflared
:
helm install cloudflared pascaliske/cloudflared -n cloudflared --create-namespace -f values.yaml
After that, you should see one pod running and three services being created:
kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
pod/cloudflared-97874dd45-vx2w9 1/1 Running 0 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cloudflared-dns-udp LoadBalancer 10.82.242.51 192.168.99.166 53:30842/UDP 10m
service/cloudflared-dns-tcp LoadBalancer 10.82.150.97 192.168.99.166 53:30979/TCP 10m
service/cloudflared-metrics ClusterIP 10.82.8.11 <none> 49312/TCP 10m
Wait, three services? We only specified two services above (dns
and metrics
), where is the other one coming from? Apparently we have one each for DNS using UDP and DNS using TCP? Why can’t we combine them into one service?
We need to take a little detour into Kubernetes services of type LoadBalancer here. For historical reasons, you cannot run TCP and UDP services on Loadbalancers in cloud environments. So, as DNS unfortunately uses both UDP and TCP and switches between the two if needed (requests bigger than 512 bytes will be resent using TCP) we need different services.
If you are running your Kubernetes cluster on baremetal (and e.g. a Raspberry Pi counts as baremetal in this case) and you are using MetalLB, then you are lucky. For MetalLB, you can workaround this issue and tell MetalLB that those services are allowed to run on the same LoadBalancer IP (in the example 192.168.99.166
) by annotating the services. Modify the values.yaml
like this:
[...]
service:
dns:
enabled: true
type: LoadBalancer
port: 53
annotations:
metallb.universe.tf/allow-shared-ip: "allow cloudflared services on the same IP"
[...]
The value for the annotation does not matter, as long as it is identical on the services that should be allowed to share the IP (and only those services).
You need to check your cloud provider’s documentation, if sharing the LoadBalancer’s IP is possible. If not, using services of type ClusterIP
and exposing them via e.g. Traefik might be the better approach, see below.
An alternative would be to only force your clients to use TCP, and only expose the TCP service. Please note that this is not yet possible with the current version of the chart.
Checking the resolver
Now that we have a running resolver, let’s check if it is working. You can use all of the usual tools like dig
,drill
or nslookup
. For dig
it would look like this:
$ dig +short +tcp @192.168.99.166 b1-systems.de
95.216.238.34
$ dig +short +notcp @192.168.99.166 b1-systems.de
95.216.238.34
As you can see, we get a valid reply when trying to resolve b1-systems.de
using UDP or TCP.
What’s up with the metrics?
In the output showing our Kubernetes services we noticed that there is a third service called cloudflared-metrics
. We enabled this in our values.yaml
and set the type to ClusterIP
. But what does it do?
The cloudflared application can output metrics that can be scraped by e.g. Prometheus. Depending on the setup of your monitoring, you might need to adapt the settings we made in the values.yaml
.
Enabling the service in the values.yaml
already gives you a Kubernetes service resource. By using a type of ClusterIP
we do not waste another LoadBalancer IP for this service. But this means that by default it is not reachable from outside the cluster.
As the metrics are provided over a simple HTTP endpoint you can put an Ingress
in front of it if you want to scrape it from outside the cluster. In case you scrape from inside the cluster, e.g. using the kube-prometheus
stack (or something like the cluster monitoring if you manage your clusters with Rancher), you do not need to expose it to the outside, so the settings we used are fine.
How to expose your service without a LoadBalancer
It might be generally preferrable to not use a LoadBalancer for each and every Kubernetes Service. In this case you can use Traefik as an Ingress Controller to make your resolver available. Please note that this also has the same limitations regarding UDP and TCP, although in this case it would be the Traefik services for TCP and UDP that are running as Kubernetes services of type LoadBalancer
…
In a previous article on running Blocky I described how to configure Traefik in detail, here is the short version:
You need to add the entrypoints for DNS to your Traefik installation, using high ports (so the Traefik pod can still run as non-root). Let’s call them dns-udp
and dns-tcp
.
If you are using MetalLB, you can annotate your Traefik services to allow them to run on the same Loadbalancer IP, just like we did above with the cloudflared services.
You can then create resources of type IngressRouteTCP
and IngressRouteUDP
that forward all traffic they receive on port 53 to the cloudflared service on port 53 (which will forward it to the targetPort
5053 which is where the cloudflared application inside the pod is listening). As an example, a IngressRouteUDP
would look like this:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteUDP
metadata:
name: cloudflared-udp
namespace: cloudflared
spec:
entryPoints:
- dns-udp
routes:
- services:
- name: cloudflared-dns-udp
port: 53
Conclusion
As usual with Kubernetes, the actual application is easy to run if you have a well-built and well-maintained image. It is the surrounding bits and pieces that make it a little more challenging. But I hope I was able to show you how to start and set up your own cloudflared
resolver in Kubernetes. Have fun!