Elastic Agent: unexpected EOF
Elastic Agent fails to connect to Elasticsearch with unexpected EOF error message.
Recently, I installed a Fleet Managed Elastic Agent (v8.12.2) on a Linux server which refused to ingest any data into Elasticsearch despite starting up successfully.
Running the Elastic Agent status command showed that the Agent was running successfully but no data was being ingested into Elasticsearch.
After a scratching around we found the following error in the logs:
Failed to connect to backoff(elasticsearch(https://{{es-server}}:9200)): Get "https://{{es-server}}:9200": unexpected EOF
By the way, to view the Elastic Agent logs on the server, run the following command: /opt/Elastic/Agent/elastic-agent logs.
At first, it seemed like this error might be related to an SSL/TLS issue. However, this was not the case - we verified it with openssl s_client -debug -connect {{es-server}}:9200.
The next option was to ensure that the Linux server could connect to Elasticsearch by executing curl -vvv https://{{es-server}}:9200. It successfully connected.
Next, we executed the Elastic Agent interactively with /opt/Elastic/Agent/elastic-agent -e and it connected to Elasticsearch, without any issues.
Clearly, there was a difference running it interactively vs. running it as a service.
Diagnostics
The elastic-agent includes a diagnostics command that generates a diagnostics archive with all the configuration data. Of particular interest, was the variables.yaml file.
The variables.yaml file showed that a proxy was used to connect to Elasticsearch.
In this case, a proxy was not necessary to connect to Elasticsearch.
To resolve the issue, we had to override the proxy settings for the Elastic Agent Service in the unit file by adding the following lines:
Environment="http_proxy="
Environment="https_proxy="
Execute the following commands to reload the unit file and restart the Elastic Agent:
sudo systemctl daemon-reload
sudo systemctl restart Elastic-Agent.serviceConclusion
The error message was not helpful. Thankfully, the Elastic Agent can create a diagnostics bundle which pointed us in the right direction to override the proxy for the Elastic Agent.