Requests need a way to get in. Initially this is where I was using MetalLB to solve my problem. I would create a service using the MetalLB LoadBalancer configuration specifying an IP address, and the service would map 1:1 to a pod where the application was running. Now I am using an ingress to point multiple applications to the same DNS entry with paths. This means any request to k8s.jmoore53.com
now gets routed to the ingress, which then points to the service, which then points to the pod.
An extra hop is being added, but the configuration for this is all done with DNS now.
Starting with What I Know, NGINX
Because I am already familiar with NGINX and the way it is configured in a standard installation, I thought it would be the best solution out of the box. The installation looked like the following (of course I pulled down the deploy.yaml):
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
kubectl apply -f deploy.yaml
For this installation, I actually made no modifications to the deploy.yaml
before deploying it.
After it was installed, I needed to create the ingress/service/pod yaml example which looked like the following:
# Ingress!
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: "k8stest.jmoore53.com"
http:
paths:
- path: "/testpath"
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
- path: "/"
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
---
# Service
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
ports:
- port: 80
targetPort: 80
selector:
app: nginx
---
# Pod Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
After this was created, the Ingress was assigned an IP and requests to “/” and “/testpath” were routing properly to the nginx service which routed to the nginx pod. This is all fine and great, but was a bit difficult to troubleshoot at times because everything was configured with NGINX.. Aka, any kind of nginx error was a bit difficult to troubleshoot because the ingress and the pods showed the same 404 error (oops, however this can be solved with a whoami
pod instead of an nginx
pod).
Moving To Traefik!
Nginx was a great solution, but Traefik is another solution available for Ingress, so I wanted to check it out. (It also provides support for API Gateway (L7/L4 support), and has a pretty cool dashboard for all the routes.)
As a quick install, I used the “Configuring Kubernetes Ingress Controller” from the Traefik documentation. I created the rbac.yaml
, ingress.yaml
, traefik.yaml
, and whoami.yaml
files.
In a short and somewhat unhelpful description, the files do the following:
rbac.yaml
: Installs the cluster roles and bindingstraefik.yaml
: configures a ServiceAccount, a traefik Deployment, and a Service, see below, also note there is a container port exposing 8080 for the admin portwhoami.yaml
: A simple whoami service and deployment, this is a really basic web page that deploys information on a get request basicallyingress.yaml
: This is similar to the nginx ingress.
# traefik.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.6
args:
- --api
- --api.insecure=true
- --api.dashboard=true
- --entrypoints.web.address=:80
- --providers.kubernetesingress
ports:
- name: web
containerPort: 80
- name: admin
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
type: LoadBalancer
selector:
app: traefik
ports:
- protocol: TCP
port: 80
name: web
targetPort: 80
- protocol: TCP
port: 8080
name: admin
targetPort: 8080
Note this is similar, if not for a few lines almost identical to the NGINX ingress.
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myingress
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: k8stest.jmoore53.com
http:
paths:
- path: /bar
pathType: Exact
backend:
service:
name: whoami
port:
number: 80
- path: /foo
pathType: Exact
backend:
service:
name: whoami
port:
number: 80
Gateway API with Traefik
Since the Gateway API is in Alpha, it needs to be installed with the following:
kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.4.0" \
| kubectl apply -f -
Then from here I used the Traefik Kubernetes Gateway Provider.
Essentially it’s similar to the traffic ingress. Below are the files:
rbac.yaml
: Access Controlwhoami.yaml
: Again, whoami service returning requesttraefik-service.yaml
: Service account, Deployment, and Service for Traefikgateway-api.yaml
: Has a GatewayClass, a Gateway, and an HTTPRoute which looks like the following:
# gateway-api.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
name: my-gateway-class
spec:
controllerName: traefik.io/gateway-controller
---
kind: Gateway
metadata:
name: my-gateway
spec:
gatewayClassName: my-gateway-class
listeners:
- name: http
protocol: HTTP
port: 80
#tls:
# certificateRefs:
# - kind: Secret
# name: mysecret
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: http-app
namespace: default
spec:
parentRefs:
- name: my-gateway
hostnames:
- apigw.k8stest.jmoore53.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
- path:
type: PathPrefix
value: /bar
backendRefs:
- name: whoami
port: 80
weight: 1
I did end up disabling all traffic over 443 and SSL for testing purposes. Also note, basically all of these are default configurations.
SSL?
Right now SSL Termination is done on the HAProxy end and then data is proxied back to the Kubernetes service.
Links
- Kubernetes Docs - Ingress
- DigitalOcean Kubernetes Ingress Tutorial
- Github Ingress NGINX Controller
- NGINX Ingress Controller
- Traefik as an Ingress Controller
- Traefik as an Ingress Controller
- Kubernetes Ingress
- Traefik Ingress
- Traefik Gateway API Configuration
- Installing Gateway API CRDs manually
- Kubernetes Gateway API - HTTP Route
- Kubernetes Gateway API - TCP