Configure FireLens to set up Fluent Bit sidecar in ECS to forward logs

2025-05-05 aws fluentd

FireLens is an ECS feature that forwards stdout/stderr to the Fluent Bit/Fluentd sidecar container via /var/run/fluent.sock and automatically generates and places output configuration files for CloudWatch Logs, S3, etc., inside the sidecar container. What you need to do is just to specify the awsfirelens log driver to the application container and set firelensConfiguration to the sidecar container.

const taskRole = new iam.Role(this, 'TaskRole', {
  assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
})

const logBucket = new s3.Bucket(this, 'LogBucket')
logBucket.grantWrite(taskRole)

const executionRole = new iam.Role(this, 'ExecutionRole', {
  assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonECSTaskExecutionRolePolicy'),
  ]
})
    
const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', {
  cpu: 256,
  memoryLimitMiB: 512,
  taskRole,
  executionRole,
})

taskDef.addContainer('AppContainer', {
  containerName: 'app',
  image: ecs.ContainerImage.fromRegistry('nginx'),
  portMappings: [{ containerPort: 80 }],
  logging: ecs.LogDrivers.firelens({
    options: {
      Name: 's3',
      region: cdk.Stack.of(this).region,
      bucket: logBucket.bucketName,
    }
  }),
})

taskDef.addFirelensLogRouter('FluentBitContainer', {
  containerName: 'fluent-bit',
  image: ecs.ContainerImage.fromRegistry('amazon/aws-for-fluent-bit:latest'),
  firelensConfig: {
    type: ecs.FirelensLogRouterType.FLUENTBIT,
  },
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'firelens' }),
})

Enter the sidecar and check that the FireLens configuration file is placed.

Deploy a container to ECS on Fargate, execute commands by ECS Exec, and perform port forwarding by Session Manager - sambaiz-net

$ aws ecs execute-command \
  --cluster (cluster_name) \ 
  --task (task_name) \
  --container fluent-bit \
  --interactive \
  --command "/bin/sh"

# cat /fluent-bit/etc/fluent-bit.conf

[INPUT]
    Name tcp
    Listen 127.0.0.1
    Port 8877
    Tag firelens-healthcheck

[INPUT]
    Name forward
    Mem_Buf_Limit 25MB
    unix_path /var/run/fluent.sock

[INPUT]
    Name forward
    Listen 127.0.0.1
    Port 24224

[FILTER]
    Name record_modifier
    Match *
    Record ecs_cluster *****
    Record ecs_task_arn *****
    Record ecs_task_definition *****

[OUTPUT]
    Name null
    Match firelens-healthcheck

[OUTPUT]
    Name s3
    Match app-firelens*
    bucket ****
    region ap-northeast-1

You can also add custom configuration with configFileValue. On EC2, you can set FileType to S3, but for Fargate, only FILE is supported, so you need to include it in the image, etc.

/*
# assets/fluent-bit/Dockerfile 
FROM amazon/aws-for-fluent-bit:latest
COPY fluent-bit.conf /extra.conf
*/

taskDef.addFirelensLogRouter('FluentBitContainer', {
  containerName: 'fluent-bit',
  image: ecs.ContainerImage.fromAsset('assets/fluent-bit', {
    platform: ecrassets.Platform.LINUX_AMD64,
  }),
  firelensConfig: {
    type: ecs.FirelensLogRouterType.FLUENTBIT,
    options: {
      configFileType: ecs.FirelensConfigFileType.FILE,
      configFileValue: '/extra.conf',
    },
  },
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'firelens' }),
})

Then, the specified file will be loaded with @INCLUDE.

# cat /fluent-bit/etc/fluent-bit.conf
...

[FILTER]
    Name record_modifier
    Match *
    Record ecs_cluster *****
    Record ecs_task_arn *****
    Record ecs_task_definition *****

@INCLUDE /extra.conf

[OUTPUT]
    Name null
    Match firelens-healthcheck

...