Kusk blog

Kusk-gateway OpenAPI-Driven K8s Gateway Alpha 2

Jan 28, 2022
5 min
read
Taras Yatsurak
Software Engineer
Kusk

Kusk-gateway v0.0.0.alpha.2 was released today with lots of enhancements, new features and bug fixes.

Kusk-gateway OpenAPI-Driven K8s Gateway Alpha 2
Share on Twitter
Share on LinkedIn
Share on Reddit
Share on HackerNews
Copy URL

Table of Contents

Get started today

Kusk-gateway v0.0.0.alpha.2 was released today with lots of enhancements, new features and bug fixes.

For the unfamiliar, Kusk-gateway is an OpenAPI driven open-source Kubernetes-based Gateway brought to you by Kubeshop, a K8s Accelerator-Incubator.

What makes Kusk-gateway unique is that it uses the ubiquitous OpenAPI specification file as a single source of truth for making an API available to consumers, which includes routing configuration, request validation, timeouts, etc. It will especially resonate with developers and teams like ours, who have adopted a design-first approach to API development. Thanks to Kusk-gateway and OpenAPI (Swagger), you can quickly publish your APIs deployed in Kubernetes without having to add any additional configuration resources, which plays nicely with both manual and automated/GitOps-based development workflows.

Under the hood, Kusk-gateway consists of an Operator (Kusk-gateway Manager) that configures the battle proven Envoy Proxy to expose your APIs or any deployed in K8s web service to outside consumers. Check out the architecture diagram for more information on how Kusk-gateway works.

New in this Release

This release was mostly focused on improving Envoy deployment and its configuration in various K8s related scenarios, making the Gateway capable of running in production environments.

The notable changes from the first alpha release are:

  • Websockets connection upgrade can be enabled per any path/method.
  • Dynamic request validation based on OpenAPI schema with automatic 400 error responses with meaningful error messages.
  • TLS support - we’re now able to serve traffic using HTTPS with the configurable certificates list and cipher suites.
  • Multiple gateways (read: Envoy Fleets) are now possible to be deployed and the API and Static Routes configurations can be bound to a specific fleet of Envoy pods. This enables a configuration of multiple websites on different Load Balancing IP addresses.
  • Envoy pods now can be flexibly configured with custom CPU/Memory resources limits, custom Envoy image tag and pod scheduling directives - we can start them on any node pool with different node taints tolerations and pod affinity/anti-affinity configurations. This makes it possible to set up a Highly Available Gateway.
  • Gateway K8s service configuration is much more flexible now, with the ability to specify the preconfigured Load Balancing IP address and the list of opened ports or even change the type of the service itself. E.g. you can disable HTTPS or even HTTP port or open it on a different port number than usual, try to use Envoy gateway as a completely internal K8s service, etc. One can also configure the custom K8s annotations for this Load Balancing service, enabling the different Load Balancer tunables that are cloud provider specific. E.g. if you want to create the gateway as a GKE internal load balancing service, you can specify the related annotations to setup the internal LB.
  • Observability improvements. Envoy access logs can be enabled to stream to pod stdout, with either text (unstructured) or JSON (structured) log format. JSON is more suitable for the injection into any log aggregation service, while text based is more human-readable. Log entries are configurable, i.e. we can log only the needed for the analysis fields. Custom annotations can be added to Envoy pods - with the specific annotations Prometheus will be able to scrape the traffic stats. This way it is possible to get requests metrics like the number of 5xx HTTP codes, indicating the percentage of failed requests.

Automatic request validation

This feature allows you to configure Kusk-gateway to automatically validate the incoming requests against OpenAPI schema before they are forwarded to the upstream services. Kusk-gateway will validate the request (including its body) and return a HTTP 400 error with a helpful message in case it’s appropriate.

This feature can immensely improve developer’s productivity with quick iteration cycles when working with new APIs and services.

Validation can be enabled on any level, be it the whole service — top-level x-kusk extension or a single endpoint. To enable validation, simply add the following lines to your OpenAPI schema before uploading it to Kusk Gateway:

x-kusk:
validation:
request:
enabled: true

So the final result would look like this:

openapi: 3.0.0
info:
title: Todo-Backend API
version: 1.0.0
x-kusk:
validation:
request:
enabled: true
cors:
origins:
- '*'
methods:
- POST
- PATCH
- DELETE
- PUT
- GET
- OPTIONS
headers:
- Content-Type
credentials: true
max_age: 86200
upstream:
service:
namespace: default
name: todo-backend
path:
prefix: /
paths:
/todos/{id}:
...

Or like this — in case you need validation enabled on a specific endpoint:

paths:
/todos/{id}:
patch:
x-kusk:
validation:
request:
enabled: true
responses:
'200':
description: 'Updates the to do item'
...

New EnvoyFleet configuration options

To give you a quick glance on how it looks, here’s EnvoyFleet Custom Resource with all tunables enabled or commented:

apiVersion: gateway.kusk.io/v1alpha1
kind: EnvoyFleet
metadata:
name: default
namespace: default
spec:
image: "envoyproxy/envoy-alpine:v1.20.0"
service:
# NodePort, ClusterIP, LoadBalancer
type: LoadBalancer
# Specify annotations to modify service behavior, e.g. for GCP to create internal load balancer:
annotations:
networking.gke.io/load-balancer-type: "Internal"
# Specify preallocated static load balancer IP address if present
loadBalancerIP: 10.10.10.10
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: http
# externalTrafficPolicy: Cluster|Local
resources:
# limits:
# cpu: 1
# memory: 100M
requests:
cpu: 10m
memory: 100M
# Put any additional annotations to the Envoy pod.
# Here we add the annotations for the Prometheus service discovery to scrape Envoy pods for the Prometheus metrics.
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '19000'
prometheus.io/path: /stats/prometheus
##### Scheduling directives
# Read https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ for the details.
# Optional - schedule Envoy pods to the node with the label "disktype=ssd".
# nodeSelector:
# disktype: "ssd"
# Optional - allow to be scheduled on the "tainted" node. Taint with "kubectl taint nodes node1 key1=value1:NoSchedule".
# Taints will repel the pods from the node unless the pods have the specific toleration.
# The line below will allow this specific Envoy pod to be scheduled there (but scheduler decides where to put it anyway).
# tolerations:
# - key: "key1"
# operator: "Exists"
# effect: "NoSchedule"
# Optional - provide pods affinity and anti-affinity settings.
# This is more flexible than nodeSelector scheme but they can be specified together.
# For the scalability and fault tolerance we prefer to put all Envoy pods onto different nodes - in a case one node fails we survive on others.
# The block below will search for all matching labels of THIS "default" envoy fleet pods and will try to schedule pods onto different nodes.
# Other fleets (if present) are not taken into consideration. You can specify nodeAffinity and podAffinity as well.
# affinity:
# podAntiAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# - labelSelector:
# matchExpressions:
# - key: app.kubernetes.io/name
# operator: In
# values:
# - kusk-gateway-envoy-fleet
# - key: fleet
# operator: In
# values:
# - default
# topologyKey: kubernetes.io/hostname
# optional, the number of Envoy Proxy pods to start
size: 1
# Access logging to stdout
# Optional, if this is missing no access logging to stdout will be done
accesslog:
# json|text
format: text
# Depending on format we can specify our own log template or if template is not specified - default Kusk Gateway will be used.
# See https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-strings for the details.
# Below are specified the examples of similar and minimalistic formats for both text and json format types.
# Text format field order is preserved.
# The output example:
# "[2021-12-15T16:50:50.217Z]" "GET" "/" "200" "1"
text_template: |
"[%START_TIME%]" "%REQ(:METHOD)%" "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" "%RESPONSE_CODE%" "%DURATION%"
# Json format fields order isn't preserved
# The output example:
# {"start_time":"2021-12-15T16:46:52.135Z","path":"/","response_code":200,"method":"GET","duration":1}
json_template:
start_time: "%START_TIME%"
method: "%REQ(:METHOD)%"
path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
response_code: "%RESPONSE_CODE%"
duration: "%DURATION%"
# TLS configuration
tls:
# cipherSuites:
# - ECDHE-ECDSA-AES128-SHA
# - ECDHE-RSA-AES128-SHA
# - AES128-GCM-SHA256
# use only TLS v1.2 - 1.3, disable older protocols.
tlsMinimumProtocolVersion: TLSv1_2
tlsMaximumProtocolVersion: TLSv1_3
# Where the certificates are deployed as k8s TLS secrets.
tlsSecrets:
- secretRef: my-cert
namespace: default
view raw envoyfleet.yaml hosted with ❤ by GitHub

Get Started

The new Kusk-gateway Manager version is available as a Docker image and can be deployed with our Helm charts. If you’d like to know more about the Kusk Gateway or want to take it for a spin- the documentation will guide you through the configuration and the TodoMVC example can be set up and destroyed within 10 minutes locally with Minikube.

In its current alpha state, Kusk-Gateway is already capable of serving your traffic, however there are improvements and features to add before we are production ready.

Some of the things we are working on:

  • Mocking of OpenAPI endpoints using the same OpenAPI definition and its examples.
  • Dashboard for the better visibility of the Gateway state and its traffic, calling the endpoints using the usual Swagger dashboard and more.
  • Requests Rate Limiting.

We are eager for your feedback! If you have questions or comments, please join us on Discord, create an issue on GitHub, or on Twitter at @thekubeshop.

If you’d like to stay updated to the future releases, blogs or events, please join a growing community of our users by subscribing here.

Related Content