Receive NewRelic alerts on Lambda through EventBridge

awsnewrelic

NewRelic’s alert destinations include Slack and PagerDuty, and you can specify these in Workflows and add additional data with NRQL. However, in order to customize the alert text or perform other processing, it is necessary to handle it with Lambda, etc. It can also receive the alert with Webhook & Custom header, but with EventBridge a public endpoint is not needed.

Creating an EventBridge destination,

and setting as Notify destination of Workflows, when you select event source, it it hasn’t existed yet, the partner event source in EventBridge will be created.

By default, event template has the following value.

{
	"id": {{ json issueId }},
	"issueUrl": {{ json issuePageUrl }},
	"title": {{ json annotations.title.[0] }},
	"priority": {{ json priority }},
	"impactedEntities": {{json entitiesData.names}},
	"totalIncidents": {{json totalIncidents}},
	"state": {{ json state }},
	"trigger": {{ json triggerEvent }},
	"isCorrelated": {{ json isCorrelated }},
	"createdAt": {{ createdAt }},
	"updatedAt": {{ updatedAt }},
	"sources": {{ json accumulations.source }},
	"alertPolicyNames": {{ json accumulations.policyName }},
	"alertConditionNames": {{ json accumulations.conditionName }},
	"workflowName": {{ json workflowName }}
}

When you associate the partner event source with an event bus, the event bus with the same name is created, and you can check the schema by performing Schema discovery.

Make the corresponding struct.

type Event struct {
	Account    string    `json:"account"`
	DetailType string    `json:"detail-type"`
	ID         string    `json:"id"`
	Region     string    `json:"region"`
	Resources  []string  `json:"resources"`
	Source     string    `json:"source"`
	Time       time.Time `json:"time"`
	Version    string    `json:"version"`
	Detail     struct {
		ID                  string   `json:"id"`
		IssueURL            string   `json:"issueURL"`
		Title               string   `json:"title"`
		Priority            string   `json:"priority"`
		ImpactedEntities    []string `json:"impactedEntities"`
		TotalIncidents      int      `json:"totalIncidents"`
		State               string   `json:"state"`
		Trigger             string   `json:"trigger"`
		IsCorrelated        bool     `json:"isCorrelated"`
		CreatedAt           int64    `json:"createdAt"`
		UpdatedAt           int64    `json:"updatedAt"`
		Sources             []string `json:"sources"`
		AlertPolicyNames    []string `json:"alertPolicyNames"`
		AlertConditionNames []string `json:"alertConditionNames"`
		WorkloadNames       []string `json:"workloadNames"`
	}
}

func handler(ctx context.Context, event Event) (interface{}, error) {
	fmt.Printf("Received: %+v\n", event)
	...
}

After that, enable EventBridge to invoke the Lambda function, and create a rule on EventBus so that it will be executed when an alert is fired.

import * as events from 'aws-cdk-lib/aws-events'
import * as targets from 'aws-cdk-lib/aws-events-targets'
    
fn.grantInvoke(new iam.ServicePrincipal('events.amazonaws.com'));

new events.Rule(this, 'NewRelicEventBusRule', {
  eventBus: events.EventBus.fromEventBusArn(
    this,
    'NewRelicEventBus',
    'arn:aws:events:ap-northeast-1:xxxxxx:event-bus/aws.partner/newrelic.com/xxxxxx',
  ),
   eventPattern: {
    source: events.Match.prefix('aws.partner/newrelic.com'),
  },
  targets: [new targets.LambdaFunction(fn, {
    retryAttempts: 1,
  })],
})

References

AWS EventBridgeとの連携によるインシデント対応の効率化(架電編)

[AWS CDK tips]EventBridge (L2) で Content based filtering を記述する