actually planing use fluentd on nodes server fetch logs , forward logs our log collector(main fluentd server)and can transfer logs s3 buckets able write logs s3 bucket getting error when restart td-agent in debug mode.
fluent nodes configuration:-
root@ip-172-31-32-115:/var/log/td-agent# cat /etc/td-agent/td-agent.conf #apache access logs <source> @type tail format apache2 path /var/log/apache2/access.log pos_file /var/log/td-agent/apache2.access.pos tag apache.access </source> #log forwardin <match apache.access> @type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s #primary host <server> host xx.xx.xx.xx port 24224 </server> flush_interval 60s </match> log forwarder logs
root@ip-172-31-34-201:/home/ubuntu# td-agent -v 2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:457:read_config: reading config file path="/etc/td-agent/td-agent.conf" 2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:323:supervise: starting fluentd-0.12.20 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-config-placeholders' version '0.3.1' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mongo' version '0.7.12' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-s3' version '0.6.5' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-scribe' version '0.10.14' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td' version '0.10.28' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.1' 2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.12.20' 2016-06-06 18:21:44 +0000 [info]: fluent/agent.rb:125:add_match: adding match pattern="apache2.." type="forward" 2016-06-06 18:21:44 +0000 [info]: plugin/out_forward.rb:141:block in configure: adding forwarding server '172.31.33.196:24224' host="172.31.33.196" port=24224 weight=60 plugin_id="object:11e9668" 2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail" 2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail" plugin/out_forward.rb:232:rebuild_weight_array: rebuilding weight array lost_weight=0 2016-06-06 18:21:44 +0000 [info]: plugin/in_tail.rb:499:initialize: following tail of /var/log/apache2/access.log log aggregator config:-
#input <source> @type forward bind 0.0.0.0 port 24224 </source> # s3 config <match s3.*.*> @type s3 aws_key_id akierpe6jxxxkxsg3gwsa aws_sec_key 31kncojdqrxxxrxew0dyy5rqv9zegtedfrere25f1hsvw s3_bucket rana.sahu s3_region us-west-2 path logs/ buffer_type file buffer_path /var/log/td-agent/s3/buffer/td buffer_chunk_limit 2m buffer_queue_limit 4098 retry_wait 1s max_retry_wait 5s disable_retry_limit true time_slice_format %y%m%d%h time_slice_wait 10m utc </match> stuck on thing
when started td-agent -v getting error 2016-06-07 07:11:00 +0000 [info]: plugin/in_forward.rb:81:listen: listening fluent socket on 0.0.0.0:24224 2016-06-07 07:11:00 +0000 [error]: fluent/engine.rb:196:rescue in run: unexpected error error_class=errno::eaddrinuse error=# 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:innew' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:82:innew' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:82:in listen' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:51:instart' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:111:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:110:ineach' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:110:in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:225:instart' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:175:in run' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:597:inrun_engine' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:148:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:incall' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:in `main_process'
please let me know if have solution this.
i got answer , these configuration can forward apache2 logs log collector , store them s3 bucket.
thanks all.
