Sending Logs to Datadog via CloudWatch Logs Subscription Filter with Firehose as Destination
datadogawsOne way to send AWS service logs to Datadog is through Firehose. Creating a stack from the official template provisions a Firehose pointed at a specified endpoint, an IAM role, and an S3 Bucket for storing failed delivery logs.
$ export DD_API_KEY=*****
$ export DD_INTAKE_LOG_ENDPOINT=https://aws-kinesis-http-intake.logs.datadoghq.com/v1/input
$ aws cloudformation create-stack \
--stack-name datadog-kinesis-logs \
--template-body "$(curl -sL --compressed https://docs.datadoghq.com/resources/json/kinesis-logs-cloudformation-template.json)" \
--capabilities CAPABILITY_IAM \
--region ap-northeast-1 \
--parameters \
ParameterKey=DatadogAPIKey,ParameterValue=$DD_API_KEY \
ParameterKey=DatadogHttpEndpointUrl,ParameterValue=$DD_INTAKE_LOG_ENDPOINT
Creating a CloudWatch Logs Subscription Filter with this Firehose as the destination starts sending logs.
$ export DELIVERY_STREAM_ARN=$(aws cloudformation describe-stacks \
--stack-name datadog-kinesis-logs \
--query "Stacks[0].Outputs[?OutputKey=='DatadogDeliveryStreamARN'].OutputValue" \
--output text)
$ export ROLE_ARN=$(aws cloudformation describe-stacks \
--stack-name datadog-kinesis-logs \
--query "Stacks[0].Outputs[?OutputKey=='CloudWatchLogsRoleARN'].OutputValue" \
--output text)
$ aws logs put-subscription-filter \
--filter-name datadog-kinesis-logs \
--log-group-name "/aws/lambda/helloworld" \
--filter-pattern "\"REPORT RequestId:\"" \
--destination-arn $DELIVERY_STREAM_ARN \
--role-arn $ROLE_ARN
When you send Lambda metrics to Datadog via AWS integration or Forwarder, Serverless billing applies, but simply sending logs this way does not count.