Are you starting to sail with your application in Kubernetes? Check these best practices before you go to production

1. Create optimal Docker images

Using Buildpacks

Instead of manually creating a Dockerfile with the instructions to compile, expose and create an image of your app, you can use Buildpacks , a tool by CNCF designed to convert artifacts that run on the JVM into executable apps, ensuring that apps meet security and compliance requirements without developer intervention. In most cases, it works out of the box, without any configuration. Follow the installation instructions here.

Then, at your project root folder you can generate an optimized image for Java only by doing:

pack build your-app-name --builder cnbs/sample-builder:bionic

That’s it! you’ve got now a runnable app image called your-app-name available on your local daemon, without needing to install a JDK, or maven or configure the environment. It will analyze your app and download any required dependency and apply the best practices for building your image in the most optimal way.

With Spring Boot 2.3.x, you can generate a build image with a maven goal spring-boot:build-image

mvn spring-boot:build-image -Dspring-boot.build-image.imageName=your-app-name

Using Jib

Jib is a tool from Google Containers Tools that lets you build optimized Docker and OCI images for your Java applications without a Docker daemon – and without deep mastery of Docker best-practices. It’s available for Maven and Gradle. Whereas traditionally a Java application is built as a single image layer with the application JAR, both Jib and Buildpack strategy is to separate the Java application into multiple layers for more granular control of your builds. When you change your code, only your change’s layer is rebuilt, not your entire application.

Here’s how to add the Jib Maven Plugin in your pom.xml file

<build>
    <plugins>
        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>jib-maven-plugin</artifactId>
            <version>2.4.0</version>
        </plugin>
    </plugins>
</build>

With it, you can generate a build image with the following command

mvn compile jib:build -Dimage=gcr.io/my-project/your-app-name

2. Use Liveness and Readiness Probes

Kubernetes uses two probes to determine if the app is ready to accept traffic and whether the app is alive.

  • The Readiness state tells whether the application is ready to accept client requests. If the application is busy, your cluster will avoid sending traffic to your pod.
  • The Liveness state of an application tells whether the internal state is valid. If Liveness is broken, this means that the application itself is in a failed state and cannot recover from it. In those cases, the best solution is to restart the application.

With Spring-boot 2.3.x you can leverage the Actuator module to provide endpoints (available at /actuator/health) for exposing Kubernetes probes to check your application status. Add the following properties to your Deployment/ReplicationController YAML configuration

Readiness Probe

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
  name: kubernetes-demo
spec:
...
  template:
    ...
    spec:
      containers:
        ...
        readinessProbe:
          httpGet:
            port: 8080
            path: /actuator/health/readiness

Liveness Probe

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
  name: k8s-demo-app
spec:
...
  template:
    ...
    spec:
      containers:
        ...
        livenessProbe:
          httpGet:
            port: 8080
            path: /actuator/health/liveness

3. Shut down your app gracefully

It’s possible to configure a pre-stop step on your deployment, to impede that Kubernetes shut down your application while still receiving requests. That way, you can give enough time for requests to stop being routed to the application before it’s terminated. Use the deployment’s lifecycle attribute like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
  name: k8s-demo-app
spec:
...
  template:
    ...
    spec:
      containers:
        ...
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 10"]

You can also enable configure a “grace period” with spring boot to finish processing pending requests whenever a shutdown signal is received within your application. Add the following properties to your src/main/resources/application.properties file:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30

4. Enhance your developer experience (DX)

Developer Experience is the equivalent of User Experience when the primary user of the product is a developer. DX cares about the developer experience of using a product, its libs, SDKs, documentation, frameworks, open-source solutions, general tools, APIs, etc.

Skaffold

Skaffold is a command-line tool created by Google to handle the workflow for building, pushing, and deploying your application on Kubernetes for non-production environments. It mainly uses two commands:

  • skaffold dev lets you build and deploy your app every time your code changes.
  • skaffold run builds and deploys your app once, similar to a CI/CD pipeline.

Installing Skaffold

Depending on your operative system, you need to run the following script:

Linuxcurl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64sudo install skaffold /usr/local/bin/
Macbrew install skaffold
Windowschoco install skaffold

In order to implement it, create skaffold.yaml in the root of your project, we’ll be using the spring-boot plugin goal build-image in order to build a docker container of the application

apiVersion: skaffold/v2beta3
kind: Config
metadata:
  name: kubernetes-demo
build:
  artifacts:
  - image: localhost:5000/apps/demo
    custom:
      buildCommand: ./mvnw spring-boot:build-image -D spring-boot.build-image.imageName=$IMAGE && docker push $IMAGE
      dependencies:
        paths:
          - src
          - pom.xml
deploy:
  kubectl:
    manifests:
    - k8s/deployment.yaml
    - k8s/service.yaml

Development with Skaffold

In development mode, skaffold will do the following procedures to your workflow

  • Build the app
  • Create a container
  • Push the container to the registry
  • Apply the deployment and service configuration to Kubernetes.
  • Stream the logs from the deployed pod to your terminal
  • Port-forward your application automatically

Debugging with Skaffold

We can use the skaffold debug command to attach a debugger to the container running in Kubernetes, by forwarding the Http port 4503 and the remote debug port 5055, which you can then configure in your favorite IDE to attach the process and set breakpoints as if you were running it locally.

5. Simplify your configuration

As you start creating more and more resources and different environments when working with Kubernetes, tagging them and maintaining them is tedious, could turn into a demanding task, and become a mess really easily. Here is a tool to simplify that:

Kustomize

Kustomize is a tool that will ease our environment configuration for Kubernetes deployment. You can start creating a base set of resources and then apply customization on top of that base, tailored for different environments, letting you keep all common properties in one place. With it, you can declaratively manage workloads and applications by simplifying the authoring of resource configurations, automatically generating, composing, reusing, and composing them to reach your desired state.

Installing Kustomize

Run the following script, and it will try to detect and download the latest release and copy the kustomize binary to your working folder

curl -s "https://raw.githubusercontent.com/\
kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

Running Kustomize

Create a kustomize/base directory at the root of your project and put there your common deployment & service configuration files, then create a kustomize.yaml file with the following:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:	
- service.yaml
- deployment.yaml

Now, we can generate a specific resource configuration for your environment by creating a new folder with a YAML file and the desired state in a declarative way. Let’s say production environments need two replicas for our services, we create a kustomize/prod/update-replicas.yaml like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-demo
spec:
  replicas: 2

Also, we need to create a kustomization.yaml with the following:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../base

patchesStrategicMerge:
- update-replicas.yaml

Then, we tell kustomize by using kustomize build ./kustomize/prod we want to patch the base resources with the update-replicas by just adding the properties need to run our services in the production environment.

You can also pipe the output from kustomize into the Kubernetes controller by doing

kustomize build kustomize/prod | kubectl apply -f -

Also, it can be combined with skaffold by telling it to use kustomize for the deployment paths. Let’s review our skaffold YAML and change the deploy section to use kustomize instead of kubectl,  like this:

apiVersion: skaffold/v2beta3
kind: Config
metadata:
  name: kubernetes-demo
build:
  artifacts:
  - image: localhost:5000/apps/demo
    custom:
      buildCommand: ./mvnw spring-boot:build-image -D spring-boot.build-image.imageName=$IMAGE && docker push $IMAGE
      dependencies:
        paths:
          - src
          - pom.xml
deploy:
  kustomize:
    paths: ["kustomize/base"]
profiles:
  - name: qa
    deploy:
      kustomize:
        paths: ["kustomize/prod"]

We can now run our normal skaffold dev --port-forward to use the deployment configuration from the base folder, and we can activate the production profile by running the following: skaffold dev -p prod --port-forward

6. Use Service Discovery

If you need to call other services, with Kubernetes is very easy, as each container has a list of the other services in your cluster with their DNS entry, so you can call them by their name.

Let’s say we have another service called auxiliary-service with the following YAML:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: auxiliary-service
  name: auxiliary-service
spec:
  ports:
  - name: 80-8080
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: auxiliary-service
  type: ClusterIP
status:
  loadBalancer: {}

Within our application, we can call it via RestTemplate like this:

private RestTemplate restService = new RestTemplateBuilder().build();
String response = restService.getForObject("http://auxiliary-service/hello", String.class);
System.out.println(response);

7. Validate Kubernetes YAML Files for best practices

There are several static tools out there to validate and score your YAML files for best-practices and security compliance. They can even be integrated with your CI/CD environment. They are mostly divided into three main categories

  • API validators: validate a given YAML manifest against the Kubernetes API server.
  • Built-in checkers: they include opinionated checks for security, best practices, performance, etc.
  • Custom validators: these tools allow writing custom rules to check in your development lifecycle.

Kubeval

Kubeval falls into the API validators category and is used to verify your YAML manifest against the Kubernetes API schema. Installation instructions can be found on the project website.

Once installed, you can run it with the following command

kubeval my-app-replication-controller.yaml
kubeval my-service.yaml

When successful, Kubeval exits with code 0 and the following message

[32mPASS[0m - .\my-app-replication-controller.yaml contains a valid ReplicationController (my-app)
[32mPASS[0m - .\my-service.yaml contains a valid Service (my-service)

The good thing is, you don’t need a cluster to validate your YAML files, so you can any errors early in the development cycle.

Kube-score

Kube-score, as the name says, performs a static analysis against a set of recommendations and best practices for Kubernetes resources, and you can define a threshold to set the result of the analysis to either OK, Warning or Critical. It can be used online or installed locally.

Kube-score is ideal for continuous integration environments, as it displays all recommendations in human-friendly output, which can be made more concise by adding the flag --output-format ci. Here’s a sample output

kube-score score base-valid.yaml --output-format ci
[OK] http-echo apps/v1/Deployment
[OK] http-echo apps/v1/Deployment
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) CPU limit is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Memory limit is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) CPU request is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Memory request is not set
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Image with latest tag
[OK] http-echo apps/v1/Deployment
[CRITICAL] http-echo apps/v1/Deployment: The pod does not have a matching network policy
[CRITICAL] http-echo apps/v1/Deployment: Container is missing a readinessProbe
[CRITICAL] http-echo apps/v1/Deployment: (http-echo) Container has no configured security context
[CRITICAL] http-echo apps/v1/Deployment: No matching PodDisruptionBudget was found
[WARNING] http-echo apps/v1/Deployment: Deployment does not have a host podAntiAffinity set

Polaris

Polaris is a static analysis tool that you can use to validate your manifest YAML files either locally or inside a cluster. It comprehends several checks from security, performance, and best practices, as it can be seen from the list of checks on the documentation. You can also write your custom checks also. The installation instructions can be found here. Then, you can validate the manifest by running the following command

polaris audit --audit-path my-pod.yaml

Polaris will return a JSON displaying the checks performed, with all checks tagged by severity and category, similar to this:

{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "test-data/base-valid.yaml",
  "DisplayName": "test-data/base-valid.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 2,
    "Namespaces": 0,
    "Controllers": 2
  },
  "Results": [
    {
      "Name": "http-echo",
      "Namespace": "",
      "Kind": "Deployment",
      "Results": {},
      "PodResult": {
        "Name": "",
        "Results": {
          "hostIPCSet": {
            "ID": "hostIPCSet",
            "Message": "Host IPC is not configured",
            "Success": true,
            "Severity": "danger",
            "Category": "Security"
          },
    ]
}

A concise output can opt-in via the --format score flag. It will print an integer from the 0 – 100 range.

polaris audit --audit-path my-pod.yaml
68

Also, a threshold the exit the validation successfully is available with the --set-exit-code-below-score which will throw and exit code of 4 when the scores is below the desired level. The --set-exit-code-on-danger lets you define an exit code 3 when any of the danger tagged checks fail.


I hope that you can use and combine these recommendations and tools you can embark on your journey to production as safe and efficiently as possible, Happy sail!

Deja un comentario

Tu dirección de correo electrónico no será publicada. Los campos obligatorios están marcados con *