About newrelic-lambda-extension and how it works telemetry without CloudWatch Logs

newrelicinfraawslog

Theare are two methods to send Lambda logs to New Relic. First is a conventional method that uses a Lambda function aws-log-ingestion to subscribe and transfer CloudWatch Logs, and second is a method that uses a Lambda layer newrelic-lambda-extension. The latter send trace logs etc. without outputting to CloudWatch Logs so it can minimize the cost.

Install

Doing newrelic-lambda integrations install, Secret containing API Key Layer refers is deployed.

$ pip3 install newrelic-lambda-cli
$ newrelic-lambda integrations install \
    --nr-account-id <account id> \
    --nr-api-key <api key> \
    --linked-account-name <linked account name> \
    --enable-license-key-secret \
    --aws-profile <aws_profile_name>
    --aws-region <aws_region>
Validating New Relic credentials
Retrieving integration license key
Creating the AWS role for the New Relic AWS Lambda Integration
Waiting for stack creation to complete... ✔️ Done
✔️ Created role [NewRelicLambdaIntegrationRole_*****] in AWS account.
Linking New Relic account to AWS account
✔️ Cloud integrations account [*****] already exists in New Relic account [*****] with IAM role [arn:aws:iam::*****:role/newrelic].
Enabling Lambda integration on the link between New Relic and AWS
✔️ Cloud integrations account [*****] is using CloudWatch Metric Streams
Creating the managed secret for the New Relic License Key
Setting up NewRelicLicenseKeySecret stack in region: us-east-1
Creating change set: NewRelicLicenseKeySecret-CREATE-1648894216
Waiting for change set creation to complete, this may take a minute... Waiting for change set to finish execution. This may take a minute... ✔️ Done
Creating newrelic-log-ingestion Lambda function in AWS account
Setting up 'newrelic-log-ingestion' function in region: us-east-1
Fetching new CloudFormation template url
Creating change set: NewRelicLogIngestion-CREATE-1648894271
Waiting for change set creation to complete, this may take a minute... Waiting for change set to finish execution. This may take a minute... ✖️ Failed to create 'newrelic-log-ingestion' function: 'Status'
✖️ Install Incomplete. See messages above for details.

Grant the read policy for arn which is exported by the stack and set the layer and environment values. This time, output debug logs to easy to track the process.

$ cat template.yaml
...
Resources:
  TestNewRelicFunction:
    Type: AWS::Serverless::Function 
    Properties:
      Runtime: go1.x
      Handler: main
      CodeUri: hello-world/
      Environment:  
        Variables:
          NEW_RELIC_ACCOUNT_ID: *****
          NEW_RELIC_TRUSTED_ACCOUNT_KEY: *****
          NEW_RELIC_EXTENSION_SEND_FUNCTION_LOGS: true
          NEW_RELIC_EXTENSION_LOG_LEVEL: DEBUG
      Layers:
        - !Sub arn:${AWS::Partition}:lambda:${AWS::Region}:451483290750:layer:NewRelicLambdaExtension:12
      Policies:
        - AWSSecretsManagerGetSecretValuePolicy:
            SecretArn: !ImportValue NewRelicLicenseKeySecret-NewRelic-LicenseKeySecretARN

Replacing lambda.Start(handler) to nrlambda.Start(handler, app), Transaction is written to Context same as other integrations. Current implementation waits for telemetry data until lambda’s timeout so the replacing is required.

In following codes, logrus integration adds trace.id to logs.

Monitor infrastructure and applications with New Relic - sambaiz-net

package main

import (
	"context"
	"fmt"

	"github.com/newrelic/go-agent/v3/integrations/logcontext/nrlogrusplugin"
	"github.com/newrelic/go-agent/v3/integrations/nrlambda"
	"github.com/newrelic/go-agent/v3/newrelic"
	"github.com/sirupsen/logrus"
)

func handler(ctx context.Context) (interface{}, error) {
	logger := logrus.New()
	logger.SetFormatter(nrlogrusplugin.ContextFormatter{})
	logger.WithContext(ctx).Info("test")

	return "ok", nil
}

func main() {
	app, err := newrelic.NewApplication(
		nrlambda.ConfigOption(),
	)
	if nil != err {
		fmt.Println("error creating app (invalid config):", err)
	}
	nrlambda.Start(handler, app)
}

Invoking this function, following logs are outputted to CloudWatch Logs and you can see the traces and logs on New Relic.

START RequestId: 58fd7908-283f-4b7e-9b78-d23c0c41d586 Version: $LATEST
[NR_EXT] New Relic Lambda Extension starting up
[NR_EXT] Registration response: {"functionName":"newrelic-test-app-TestNewRelicFunction-BhEz8V8OMy0F","functionVersion":"$LATEST","handler":"main"}
[NR_EXT] Log registration with request {"buffering":{"maxBytes":1048576,"maxItems":10000,"timeoutMs":500},"destination":{"URI":"http://sandbox:39273","protocol":"HTTP"},"types":["platform","function"]}
[NR_EXT] Starting log server.
LOGS Name: newrelic-lambda-extension State: Subscribed Types: [platform,function]
[NR_EXT] Registered for logs. Got response code 200 "OK"
EXTENSION Name: newrelic-lambda-extension State: Ready Events: [SHUTDOWN,INVOKE]
{"span.id":"28eed3cf47348008","entity.type":"SERVICE","hostname":"169.254.217.61","message":"test","log.level":"info","trace.id":"578ff961a66d3bf45c3e1fc247bcb9d6","timestamp":1649377719029}
[NR_EXT] Agent telemetry bytes: ****
[NR_EXT] Aggressive harvest yielded 1 invocations
[NR_EXT] Sent 1/1 New Relic payload batches with 1 log events successfully in 164.061ms (163ms to transmit 1.1kB).
END RequestId: 58fd7908-283f-4b7e-9b78-d23c0c41d586
REPORT RequestId: 58fd7908-283f-4b7e-9b78-d23c0c41d586 Duration: 203.51 ms Billed Duration: 204 ms Memory S

How it works

newrelic-lambda-extension obtains and sends function logs and telemetry information differently.

SendFunctionLogs()

When environment variable NEW_RELIC_EXTENSION_SEND_FUNCTION_LOGS is true, function logs are sent. newrelic-lambda-extension starts a log server and listen sandbox.localdomain to recive logs from Runtime Log API of Lambda extension, and handle it. Then function logs are send and platform logs for starting etc. are added to telemetry information.

Environment variable AWS_LAMBDA_RUNTIME_API has the endpoint of Runtime Log API so an extension can subscribe logs with requesting PUT /logs.

SendTelemetry()

Send telemetry information including trace data. This method listen a pipe created with mkfifo() system call. If there is the pipe, agent writes telemetry information to it instead of stdout.

Unixのパイプをmkfifo()で作ってdup2()で標準出力にコピーして書き込む - sambaiz-net