fluentdでElasticsearchに送る

2017-02-01 elasticsearch fluentd

必要なものをいれていく。Amazon LinuxのAMIから。

Failed to build gem native extension.

$ yum install -y ruby-devel

serverengine requires Ruby version >= 2.1.0.

rbenvでバージョンを上げる。

$ git clone https://github.com/rbenv/rbenv.git ~/.rbenv
$ cd ~/.rbenv && src/configure && make -C src
$ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
$ ~/.rbenv/bin/rbenv init
$ echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
$ source ~/.bash_profile
$ git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build

$ rbenv -v
rbenv 1.1.0-2-g4f8925a

Ruby install aborted due to missing extensions

$ yum install -y openssl-devel readline-devel zlib-devel

$ rbenv install -l
1.8.5-p113
1.8.5-p114
1.8.5-p115
...

$ rbenv install 2.4.0
$ rbenv global 2.4.0
$ ruby -v
ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]

$ td-agent-gem install fluent-plugin-elasticsearch

td-agent.confはこんな感じ。

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match hoge.log>
  @type elasticsearch
  host *****
  port 9200
  index_name test_index
  type_name test_type
</match>

$ echo '{"a": "b"}' | /opt/td-agent/embedded/bin/fluent-cat hoge.log
$ curl *****:9200/test_index/test_type/_search?pretty
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test_index",
        "_type" : "test_type",
        "_id" : "AVn5puy79PEDL_x5e_u3",
        "_score" : 1.0,
        "_source" : {
          "a" : "b"
        }
      }
    ]
  }
}

logstash formatでも入れてみる。

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match hoge.log>
  @type elasticsearch
  host *****
  port 9200
  logstash_format true
  logstash_prefix aaaa
  type_name test_type
</match>

$ echo '{"a": "b"}' | /opt/td-agent/embedded/bin/fluent-cat hoge.log
$ curl *****:9200/aaaa-2017.02.02/_search?pretty
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "aaaa-2017.02.02",
        "_type" : "test_type",
        "_id" : "AVn_FyQP7q9Gyu5HC4Mq",
        "_score" : 1.0,
        "_source" : {
          "a" : "b",
          "@timestamp" : "2017-02-02T22:49:33+09:00"
        }
      }
    ]
  }
}

forwardと同じく Buffered Output pluginを継承しているので buffer_typeのデフォルトはmemory。必要ならfileにする。いずれにせよスパイクなどでbuffer_queue_limitを超えないように余裕をもっておく。また、buffer_chunk_limitがElasticsearchのhttp.max_content_length(デフォルト100mb)を超えないようにする。