Prometheus metrics and time based scaling with KEDA (Kubernetes Event-driven Autoscaling)

2024-06-05 kubernetes prometheus

KEDA is a component that enables event-driven scaling in Kubernetes, and works with HorizontalPodAutoscaler. It provides scalers for various event sources such as CPU, Prometheus, and Cron, and can increase or decrease replicas with HPA or change them from 0 to 1 based on the event.

Update replicas using K8s HorizontalPodAutoscaler and set determination interval and increase/decrease limit - sambaiz-net

Install with Helm.

cluster.addHelmChart('KedaChart', {
  chart: 'keda',
  release: 'kedacore',
  repository: 'https://kedacore.github.io/charts',
  namespace: 'keda',
  version: '2.14.2',
})

Create ScaledObject.

apiVersion: 'keda.sh/v1alpha1'
kind: 'ScaledObject'
metadata:
  name: 'test-scaledobject'
spec:
  scaleTargetRef:
    kind: 'Deployment'
    name: 'testapp'
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: 'http://kube-prometheus-stack-prometheus.prometheus.svc.cluster.local:9090'
      query: 'prometheus_http_requests_total{code="200",handler="/graph"}'
      threshold: '50'
  - type: cron
    metadata:
      timezone: Asia/Tokyo
      start: 40 22 * * *
      end: 50 22 * * *
      desiredReplicas: '10'

When you apply this, metrics will be registered,

$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s0-prometheus?labelSelector=scaledobject.keda.sh/name=test-scaledobject" | jq
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "s0-prometheus",
      "metricLabels": null,
      "timestamp": "2024-06-05T21:54:04Z",
      "value": "2"
    }
  ]
}

HPA will be created that references these metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
...
spec:
  maxReplicas: 10
  metrics:
  - external:
      metric:
        name: s0-prometheus
        selector:
          matchLabels:
            scaledobject.keda.sh/name: test-scaledobject
      target:
        averageValue: "50"
        type: AverageValue
    type: External
  - external:
	    metric:
	      name: s1-cron-Asia-Tokyo-4022xxx-5022xxx
	      selector:
	        matchLabels:
	          scaledobject.keda.sh/name: test-scaledobject
	    target:
	      averageValue: "1"
	      type: AverageValue
	  type: External
  minReplicas: 1
  ...

If HPA already exists, an error will occur, but you can make it managed by ScaledObject by adding the scaledobject.keda.sh/transfer-hpa-ownership: true annotation.

metadata:
  annotations:
    scaledobject.keda.sh/transfer-hpa-ownership: "true"
spec:
  ...
  advanced:
    horizontalPodAutoscalerConfig:
      name: testapp-hpa

You can confirm that replicas will increase to desiredReplicas during the cron period.

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  15m   horizontal-pod-autoscaler  New size: 4; reason: external metric s1-cron-Asia-Tokyo-4022xxx-5022xxx(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: test-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  14m   horizontal-pod-autoscaler  New size: 8; reason: external metric s1-cron-Asia-Tokyo-4022xxx-5022xxx(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: test-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  14m   horizontal-pod-autoscaler  New size: 10; reason: external metric s1-cron-Asia-Tokyo-4022xxx-5022xxx(&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: test-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  39s   horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

References

KEDAによるKubernetesのイベントドリブンなスケーリングの実践｜hayajo

Prometheusメトリクスに基づいて Kubernetes Pod をオートスケーリングさせてみた - QG Tech Blog