Install Elasticsearch, Kibana 4 , fluentd (Opensource splunk) with syslog clients

So used splunk some times but it has its limit (money) so now Im testing

1. Java

first install java on your server. Get java from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

 yum localinstall jdk-8u25-linux-x64.rpm

And install it on your server.

2. Elasticsearch

Get it from here http://www.elasticsearch.org/download I installed the rpm and run

https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.0.Beta1.noarch.rpm
yum localinstall elasticsearch-1.4.0.Beta1.noarch.rpm

I hade to make some settings in this file my vps only hade 512m

vi /etc/sysconfig/elasticsearch

/etc/init.d/elasticsearch start

So moving on

3. Kibana 4

Download kibana from here http://www.elasticsearch.org/overview/kibana/installation/

cd /var/www/html
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-BETA1.1.tar.gz
tar zxvf kibana-4.0.0-BETA1.1.tar.gz
mv kibana-4.0.0-BETA1.1 kibana
chown apache:apache -R kibana

4. Install fluentd

http://docs.fluentd.org/articles/install-by-rpm

curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh

Install gems needed

yum install libcurl-devel
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

open this file and have only this in the file

vi /etc/td-agent/td-agent.conf

<match td.*.*>
 type tdlog
 apikey YOUR_API_KEY
auto_create_table
buffer_type file
buffer_path /var/log/td-agent/buffer/td
</match>

<source>
 type syslog
 port 42185
 tag syslog
</source>
<source>
type forward
</source>
<match syslog.**>
type elasticsearch
logstash_format true
flush_interval 10s # for testing
</match>

Restart the agent

/etc/init.d/td-agent restart

Time for sending some logs to the server

5. Client

in rsyslog open the file /etc/rsyslog.conf and add at the buttom

*.* @127.0.0.1:42185