ksonnetでkubernetesのmanifestを環境ごとに生成/applyする

(2018-06-20)

ksonnetはJSONのテンプレートエンジンjsonnetからk8sのmanifestを環境ごとに生成してapplyするツール。kubeflowでも使われている。

$ brew install ksonnet/tap/ks
$ ks version
ksonnet version: 0.11.0
jsonnet version: v0.10.0
client-go version:

init

まずks initしてディレクトリを作成する。

$ kubectl config current-context
minikube

$ ks init kstest
$ cd kstest
$ ls
app.yaml	components	environments	lib		vendor

$ cat app.yaml
apiVersion: 0.1.0
environments:
  default:
    destination:
      namespace: default
      server: https://192.168.99.100:8443
    k8sVersion: v1.10.0
    path: default
kind: ksonnet.io/app
name: kstest
registries:
  incubator:
    gitVersion:
      commitSha: 40285d8a14f1ac5787e405e1023cf0c07f6aa28c
      refSpec: master
    protocol: github
    uri: github.com/ksonnet/parts/tree/master/incubator
version: 0.0.1

Prototypeのインストール

app.yamlにあるように最初はincubatorというregistryが登録されている。追加する場合はks registry addks pkg listでregistryのpackage一覧を見て、ks pkg installするとpackageに含まれるいくつかのprototypeを持ってくることができる。 prototypeというのはparameterを足してcomponentになるもと。

# ks registry add *** github.com/***
$ ks pkg list
REGISTRY  NAME      INSTALLED
========  ====      =========
incubator apache
incubator efk
incubator mariadb
incubator memcached
incubator mongodb
incubator mysql
incubator nginx
incubator node
incubator postgres
incubator redis
incubator tomcat

$ ks pkg install incubator/[email protected]
$ ks prototype list
NAME                                  DESCRIPTION
====                                  ===========
...
io.ksonnet.pkg.redis-all-features     A Redis deployment with metrics, ingress, and persistent storage.
io.ksonnet.pkg.redis-persistent       A simple Redis deployment, backed by persistent storage.
io.ksonnet.pkg.redis-stateless        A simple, stateless Redis deployment,

$ ls vendor/incubator/redis/
README.md       examples        parts.yaml      prototypes      redis.libsonnet

Componentの生成

ks generateでprototypeからcomponentを生成する。deployed-serviceは最初から入っているprototype。

$ ks generate deployed-service myapp --image gcr.io/google-samples/hello-app:2.0 --type ClusterIP
$ ks generate redis-stateless redis
$ ls components/
myapp.jsonnet           params.libsonnet        redis.jsonnet

jsonnetではparamsを参照しているが、environmentごとの設定がなければcomponentsのparams.libsonnetの値になる。

$ cat components/myapp.jsonnet
local env = std.extVar("__ksonnet/environments");
local params = std.extVar("__ksonnet/params").components.myapp;
[
   {
      "apiVersion": "v1",
      "kind": "Service",
      "metadata": {
         "name": params.name
      },
      "spec": {
         "ports": [
            {
               "port": params.servicePort,
               "targetPort": params.containerPort
            }
         ],
         "selector": {
            "app": params.name
         },
         "type": params.type
      }
   },
   {
      "apiVersion": "apps/v1beta2",
      "kind": "Deployment",
      "metadata": {
         "name": params.name
      },
      "spec": {
         "replicas": params.replicas,
         "selector": {
            "matchLabels": {
               "app": params.name
            },
         },
         "template": {
            "metadata": {
               "labels": {
                  "app": params.name
               }
            },
            "spec": {
               "containers": [
                  {
                     "image": params.image,
                     "name": params.name,
                     "ports": [
                     {
                        "containerPort": params.containerPort
                     }
                     ]
                  }
               ]
            }
         }
      }
   }
]

$ cat components/params.libsonnet
{
  global: {
    // User-defined global parameters; accessible to all component and environments, Ex:
    // replicas: 4,
  },
  components: {
    // Component-level parameters, defined initially from 'ks prototype use ...'
    // Each object below should correspond to a component in the components/ directory
    redis: {
      name: "redis",
      redisPassword: "null",
    },
    myapp: {
      containerPort: 80,
      image: "gcr.io/google-samples/hello-app:2.0",
      name: "myapp",
      replicas: 1,
      servicePort: 80,
      type: "ClusterIP",
    },
  },
}

Environmentの追加

Environmentごとに向き先を変えて、異なるパラメータでapplyできる。

$ ks env add prd -n prd
$ ks env list
NAME    OVERRIDE KUBERNETES-VERSION NAMESPACE SERVER
====    ======== ================== ========= ======
default          v1.10.0            default   https://192.168.99.100:8443
prd              v1.10.0            prd       https://192.168.99.100:8443

ks param setでパラメータを設定できる。これがcomponentsのparams.libsonnetのものより優先される。

$ ks param set myapp replicas 2 --env prd
$ cat environments/prd/params.libsonnet
local params = std.extVar('__ksonnet/params');
local globals = import 'globals.libsonnet';
local envParams = params + {
  components+: {
    myapp+: {
      replicas: 2,
    },
  },
};

{
  components: {
    [x]: envParams.components[x] + globals
    for x in std.objectFields(envParams.components)
  },
}

yamlマニフェストを表示

$ ks show prd
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: redis
  name: redis
spec:
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        image: bitnami/redis:3.2.9-r2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 30
          timeoutSeconds: 5
        name: redis
        ports:
        - containerPort: 6379
          name: redis
        readinessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 5
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
        volumeMounts:
        - mountPath: /bitnami/redis
          name: redis-data
      volumes:
      - name: redis-data
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: redis
  name: redis
spec:
  ports:
  - name: redis
    port: 6379
    targetPort: redis
  selector:
    app: redis
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: myapp
  type: ClusterIP
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:2.0
        name: myapp
        ports:
        - containerPort: 80

apply

$ ks apply default -c myapp