ElasticSearch Fails to Start on Ubuntu 16.04.3 - status=1 Failure

Question

The ElasticSearch 6.2.2 service on this Ubuntu 16.04.3 machine has been running fine for a few months. I rebooted the machine today and found the ElasticSearch service had not started. So I tried sudo systemctl start elasticsearch.service. I was not given any error. But I noticed it still hadn't started.

Upon running systemctl status elasticsearch I get this:

`● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: inactive (dead) (Result: exit-code) since Tue 2018-09-18 02:35:58 UTC; 19min ago
     Docs: http://www.elastic.co
  Process: 14147 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 14147 (code=exited, status=1/FAILURE)

Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Unit entered failed state.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Service hold-off time over, scheduling restart.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: Stopped Elasticsearch.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Start request repeated too quickly.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: Failed to start Elasticsearch.

The solution which seemed to help many people who couldn't start the ES service on Ubuntu found here (Can not start elasticsearch as a service in ubuntu 16.04), the one accepted as the answer, did not help me.

This is a production machine so I hope someone can help! Thanks!!

EDIT 1 - after much pain trying to track down the log file, it's now at the following url per suggestion from @ArnavRay https://file.io/ASnEG4

Answer 1

The only error on the logs are related to the x-pack security. Are you using any of the features of x-pack? Here is a link that describes what happens when the x-pack license expires. https://www.elastic.co/guide/en/x-pack/current/license-expiration.html

Try running elasticsearch-plugin remove x-pack

Default location for the elasticsearch-plugin is /usr/share/elasticsearch/bin/. You might need to use sudo. Remove all the x-pack related configuration from elasticsearch.yml and restart ElasticSearch.

https://github.com/deviantony/docker-elk/issues/154 - This has the same issue as yours.

If you are still facing issues you can try the following things.

1. Increase the heap memory on jvm.options under /etc/elasticsearch/
2. Use ls -a and check if any other elasticsearch.yml files has been created. If so remove them.

3. Run these commands

   echo -e "elasticsearch - memlock unlimited storm - nproc 257597" >> /etc/security/limits.conf ulimit -n 32768 ulimit -u 65536 echo -e "* - nofile 32768 * - nproc 65536" >> /etc/security/limits.conf

4. Try restarting the system. This has saved me a lot of headaches over the years :)