Get communication in Kubernetes cluster with Pixie's PxL script

2024-04-07 kubernetes

Install Pixie CLI and login. If you use Pixie from New Relic, pass api key included in the install commmand.

Install newrelic-bundle to EKS cluster with CDK and monitor it - sambaiz-net

$ bash -c "$(curl -fsSL https://withpixie.ai/install.sh)"
$ px version
Pixie CLI
0.8.2+Distribution.401c92c.20230531033620.1.jenkins

$ px auth login --api_key='xxxxx'
Pixie CLI
Authentication Successful

Let’s write PxL Script to read and display the table of data collected by Pixie. Built-in scripts are helpful to write.

import px
df = px.DataFrame(table='http_events', start_time='-30s')
df['pod'] = df.ctx['pod']
px.display(df)

Run it with px run or live.

$ px run -f test_script.pxl -o json | jq
{
  "_tableName_": "output",
  "time_": "2024-04-07T19:03:31.857543719+09:00",
  "upid": "00000003-0000-13c1-0000-0000000067ac",
  "remote_addr": "10.18.36.193",
  "remote_port": 40818,
  "trace_role": 2,
  "major_version": 1,
  "minor_version": 1,
  "content_type": 0,
  "req_headers": "{\"Accept\":\"*/*\",.}",
  "req_method": "GET",
  "req_path": "/health",
  "req_body": "",
  "req_body_size": 0,
  "resp_headers": "{\"Connection\":\"close\",...}",
  "resp_status": 200,
  "resp_message": "OK",
  "resp_body": "OK",
  "resp_body_size": 2,
  "latency": 94640,
  "pod": "newrelic/newrelic-bundle-nri-metadata-injection-c8fdfdcc-zdlt6"
}
...

$ px live -f test_script.pxl

Events whose trace_role is 1 is a request from pod to remote_addr in outside of the cluster, and ones whose trace_role is 2 is a request from remote_addr to pod. px.ip_to_pod_id() and px.pod_id_to_pod_name() can covert remort_addr to pod’s name.

import px
df = px.DataFrame(table='http_events', start_time='-30s')
df['pod'] = df.ctx['pod']

df.ra_pod = px.pod_id_to_pod_name(px.ip_to_pod_id(df.remote_addr))
df.ra_name = px.select(df.ra_pod != '', df.ra_pod, df.remote_addr)

df.source = px.select(df.trace_role == 1, df.pod, df.ra_name)
df.destination = px.select(df.trace_role == 1, df.ra_name, df.pod)

df = df[df.ra_pod != '']
px.display(df[['source', 'destination', 'latency']])

It’s successful to obtain HTTP communication between pods.

{
  "_tableName_": "output",
  "source": "newrelic/vizier-query-broker-bb9d87cc9-lq7tw",
  "destination": "newrelic/vizier-metadata-69c585d799-pd2hl",
  "latency": 10760328
}
{
  "_tableName_": "output",
  "source": "newrelic/vizier-query-broker-bb9d87cc9-lq7tw",
  "destination": "newrelic/vizier-metadata-69c585d799-pd2hl",
  "latency": 11376994
}
{
  "_tableName_": "output",
  "source": "newrelic/vizier-query-broker-bb9d87cc9-lq7tw",
  "destination": "newrelic/vizier-metadata-69c585d799-pd2hl",
  "latency": 16669085
}