ELK Stack is the world’s most popular log management platform. It is a collection of open-source products including Elasticsearch, Logstash, and Kibana. All these 3 products are developed, managed and maintained by Elastic.
ELK Stack is a powerful and open source platform that can manage a massive amount of logged data. The inputs log are generally from a graphical web interface (GUI).
- Elasticsearch is a JSON-based search and analytics engine intended for horizontal scalability and easier management.
- Logstash is a server-side data processing interface that has the capability to collect data from several sources concurrently. It then transforms it, and then sends the data to your desired stash. It is an open source application.
- Kibana is used to visualize your data and navigate the Elastic Stack. It is an open source tool as well.
Install and configure ELK Stack on Ubuntu
In this tutorial, we are going to use filebeat to send log data to Logstash. Beats are lightweight data shippers and to begin with, we should have to install the agent on servers.
Step 1) Installing Java 8
ElasticSearch supports Java 8 and 9, but the trouble is Logstash is only compatible with Java 8. Java 9 is not supported yet. Therefore, we are going to install Oracle Java 8.
Launch the Terminal and add Oracle Java 8 repository, followed by a system update, and actual installation.
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt install oracle-java8-set-default
Pay attention to the Terminal. You will have to agree to license agreement windows and select “yes” to continue. After installation is complete, you can check the java version by using the following commands:
.sudo java -version
Java Version Check
sudo echo $JAVA_HOME
Check Java Home
Step 2) Installing and Configuring Elasticsearch
Let’s start with wget command to download Elasticsearch followed by public signing key:
sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Secondly, install the apt-transport-https package (Debian based distros needs this).
sudo apt-get install apt-transport-https
Add the repository:
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
Update the repo list and install the package:
sudo apt-get update
sudo apt-get install elasticsearch
Let’s modify “elasticsearch.yml” file:
sudo vim /etc/elasticsearch/elasticsearch.yml
Uncomment “network.host” and “http.port”. Following configuration should be added:
network.host: localhost http.port: 9200
Next, save and close the file.
To make sure ElasticSearch works seamlessly, enable it on boot and Start ElasticSearch.
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
Check installation:
sudo curl -XGET 'localhost:9200/?pretty'
Check ElasticSearch
Step 3) Installing Kibana
Let’s start installing Kibana now and modify Kibana settings:
sudo apt-get install kibana
sudo vim /etc/kibana/kibana.yml
Uncomment following lines:
server.port: 5601 server.host: "localhost" elasticsearch.url: "http://localhost:9200"
Save and exit the File.
Enable it on boot and start Kibana service:
sudo systemctl enable kibana.service
sudo systemctl start kibana.service
Step 4) Configuring Nginx as Reverse Proxy for Kibana
In the similar lines, let’s install Nginx, configure it, and start the service. Use the following commands one at a time:
sudo apt-get install nginx apache2-utils
Configure Virtual host:
sudo vim /etc/nginx/sites-available/elk
Add the following configuration to file:
server {
listen 80;
server_name elk.fosslinux.com;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.elkusersecret;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
Create user and password file for web browser authentication:
sudo htpasswd -c /etc/nginx/.elkusersecret elkusr
ELK Authentication
Enter password and repeat. Check Nginx Configurations:
sudo nginx -t
Enable Nginx on system boot and restart the service:
sudo systemctl enable nginx.service
sudo systemctl restart nginx.service
Step 5) Installing and Configuring Logstash
Install Logstash:
sudo apt-get install logstash
Here we are going to generate SSL certificate key to secure log transfer from file beat client. Modify the “hosts” file before creating the SSL certificate.
sudo vim /etc/hosts
Add the following line to file. Make sure to change IP and server name to yours.
172.31.31.158 elk-server elk-server
When done, save and exit the file.
Now change directory to Logstash.
sudo cd /etc/logstash/
Create a folder for SSL:
sudo mkdir ssl
Generate SSL certificate. Change elk-server to your server name in the below command.
sudo openssl req -subj '/CN=elk-server/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout ssl/logstash-forwarder.key -out ssl/logstash-forwarder.crt
Create following files inside “/etc/logstash/conf.d”.
sudo cd /etc/logstash/conf.d/
create a filebeat-input file using vim.
sudo vim filebeat-input.conf
Add the following lines to it.
input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/ssl/logstash-forwarder.crt"
ssl_key => "/etc/logstash/ssl/logstash-forwarder.key"
}
}
Save and close the file and create a new configuration file.
sudo vim syslog-filter.conf
Add the following contents to it.
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
Save and exit the file. Create elasticsearch output file.
sudo vim output-elasticsearch.conf
Add the following lines to it.
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Let’s enable Logstash on boot and start the service:
sudo systemctl enable logstash.service
sudo systemctl start logstash.service
Step 6) Installing and Configuring Filebeat on Client servers
Start with editing the hosts file to add elk host entries. Make sure to replace IP and name with yours.
sudo vim /etc/hosts
172.31.31.158 elk-server
Save and exit the file.
Download and install the Public Signing Key:
sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Install “apt-transport-https” and add repo.
sudo apt-get install apt-transport-https
sudo echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
Update repo and install Filebeat.
sudo apt-get update
sudo apt-get install filebeat
Modify Filebeat configurations.
sudo vim /etc/filebeat/filebeat.yml
Find the following line and modify the value to “true”.
enabled: true
Here we are not modifying log path and Filebeat will forward all logs inside “var/log” folder
paths:
- /var/log/*.log
Uncomment the following lines:
output.logstash: # The Logstash hosts hosts: ["elk-server:5443"] ssl.certificate_authorities: ["/etc/filebeat/logstash-forwarder.crt"]
Comment Elasticsearch:
#output.elasticsearch: # Array of hosts to connect to. # hosts: ["localhost:9200"]
Save and exit the file.
Now go to ELK server and get “logstash-forwarder.crt” contents
sudo cat /etc/logstash/ssl/logstash-forwarder.crt
SSL Crt
copy output and then go to Elk client-server.
Create a certificate file
sudo vim /etc/filebeat/logstash-forwarder.crt
insert copied output and save & exit.
Enable filebeat on system boot Start filebeat service.
sudo systemctl enable filebeat.service
sudo systemctl start filebeat.service
Step 7) Browsing the Kibana Dashboard
Launch your favorite web browser and enter the domain name followed by username and password.
http://elk.fosslinux.com
Browser Authentication
Enter the created user name and password. You should see the Kibana Welcome page. Click “Explore my Own” button.
Kibana Welcome Page
You should be directed to the Kibana Home Page.
Kibana Home Page
Click “Discover” on the left side. Click “Create index pattern”.
Create Index
Then define the index pattern “filebeat-*”.
Define Index
Click next and choose @timestamp’ and click ‘Create index pattern’.
Choose TimeStamp
Index pattern should get created.
Created Index Pattern
Click the “Discover” Menu to see the server logs.
Discover logs
Logs will be shown as per the time stamp. Click on any timestamp to expand it and see the log file contents and its details.
Check Log Files
If you reached here, it implies you have successfully installed and configured the ELK stack with filebeat. Got any issues? Feel free to let us know in the comments below.
12 comments
Hi Darshana. Thanks for the tutorial! It was very interesting and helpful, but I have some doubts about it.
1.- When configurating Nginx, you enabled the authentication and refered to an htpasswd file called “.elkusr”. After that, you created an htpasswd file called “.elkusrsecret”. In this case, when it tries to reach the authentication file, the one specified in the configuration doesn’t exist. How’s it going to find the correct file?
2.- When configurating Filebeat, in the logstash output hosts field, you specified “elk-master” but this name is not in the hosts list of the server and neither in the client configuration. The names added to the hosts lists are “elk-server”, does it work fine like that?
I followed your tutorial and everything was installed and configured ok, but Filebeat is not able to connect to the server and I don’t know the reason yet. I installed both client and server in separated Ubuntu 18.04 LTS servers and would like to make them talk to each other but I don’t know the reason it doesn’t work yet, maybe it’s something wrong with the configuration of the machine. If you have any suggestion or advice it would be very appreciated!
Thanks!
Hi Pablo,
Thank you for the comment.
1. It is my bad. Nginx configurations should change as follows
“auth_basic_user_file /etc/nginx/.elkusersecret;”
2. elk-master must change to elk-server
I will update the post.
Please run below command on elk-server to check logstash port.
netstat -tunlp | grep 5443
If the port is available to go to the filebeat server.
Then try to ping
ping elk-server (you should add elk-server IP to your hosts file)
if ping is working do telnet
telnet elk-server 5443
if you cant it may be the issue on the firewall.
check logstash log file and verify that it started without errors.
Just try to disable firewall and check
ufw disable
If you have any issue I am really happy to help you mate.
Regards
Darshana
I can’t find filebeat index like u ? I follow all of your instructions and test them but I can’t .why sir? thanks u
me too, I can’t find filebeat index.
elk-server
netstat -tunlp | grep 5443
filebeat server
ping elk-server
telnet elk-server 5443
all good
but I can’t find filebeat index on kibana
thank you
I followed all the directions, but when I go to my url at the end, I just get an nginx page.
I get nginx page as well. Trying http://elk-server:5601 but no Kibana…
He put the nginx config in /etc/nginx/sites-available/elk, but he should also put it in sites-enabled, so
ln -s /etc/nginx/sites-available/elk /etc/nginx/sites-enabled/elk
sudo nginx reload
@Mezoloth, @srslol: I also got the same nginx page. As I’m new to nginx, took me few Google-minutes to find. Create the symlink below
sudo ln -s /etc/nginx/sites-available/elk /etc/nginx/sites-enabled/elk
then restart nginx
sudo systemctl stop nginx.service
sudo systemctl start nginx.service
Worked for me.
Just a small correction to the instructions.
One of the steps says to do this:
sudo cd /etc/logstash/conf.d/
but since cd is a shell built-in, that doesn’t work. The command should be run without sudo, like this:
cd /etc/logstash/conf.d/
My logstash throws following exception
[2020-05-07T16:23:23,083][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-05-07T16:24:13,430][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:270) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:112) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.9.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.security.KeyException: could not find key file: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:114) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
… 20 more
[2020-05-07T16:24:13,445][ERROR][logstash.inputs.beats ] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.
[2020-05-07T16:24:13,455][WARN ][io.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x52a5b0e3, L:/172.25.0.31:5443 – R:/10.213.96.194:55724]
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:270) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:112) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.9.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.security.KeyException: could not find key file: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:114) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
… 20 more
[2020-05-07T16:24:47,140][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:270) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:112) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.9.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.security.KeyException: could not find key file: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:114) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
… 20 more
[2020-05-07T16:24:47,142][ERROR][logstash.inputs.beats ] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.
[2020-05-07T16:24:47,143][WARN ][io.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x9a8d96f0, L:/172.25.0.31:5443 – R:/10.213.96.194:55726]
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:270) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:112) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.9.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.security.KeyException: could not find key file: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:114) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
… 20 more
[2020-05-07T16:25:42,936][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:270) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:112) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.9.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.security.KeyException: could not find key file: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:114) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
… 20 more
[2020-05-07T16:25:42,941][ERROR][logstash.inputs.beats ] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.
[2020-05-07T16:25:42,942][WARN ][io.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x4389b220, L:/172.25.0.31:5443 – R:/10.213.96.194:55728]
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:270) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:112) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.9.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.9.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.security.KeyException: could not find key file: /etc/logstash/ssl/logstash-forwarder.key
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:114) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1015) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:268) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
… 20 more
if your logstash logs this error:
“Exception caught in channel initializer
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-forwarder.key”
it’s because the `/etc/logstash/ssl/logstash-forwarder.key` file is only readable by root, so logstash cant read it because of permission error.
I choose to change the owner to logstash and solve the problem: `sudo chown logstash logstash-forwarder.key`
reference
1. https://discuss.elastic.co/t/logstash-input-beats-file-does-not-contain-valid-private-key/228140/18
Hey , I am new to this logstash . i am configuring filebeat-> logstash -> elasticsearch -> kibana
when i do this command on logstash server , i am getting some error . please help me as soon as possible .
root@ip-10-0-1-187:~# sudo /usr/share/logstash/bin/logstash -e ‘input { stdin{ } } output { stdout{ } }’
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-08-15 22:45:55.590 [LogStash::Runner] multilocal – Ignoring the ‘pipelines.yml’ file because modules or command line options are specified
[FATAL] 2020-08-15 22:45:55.614 [LogStash::Runner] runner – Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the “path.data” setting.
[ERROR] 2020-08-15 22:45:55.616 [LogStash::Runner] Logstash – java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit