Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle server closing connection after response body is sent #223

Merged
merged 7 commits into from
Feb 21, 2025

Conversation

JoshMock
Copy link
Member

@JoshMock JoshMock commented Feb 21, 2025

There are cases where Elasticsearch may do the following:

  1. decide to not read the entire request body
  2. return a response with an HTTP error code
  3. close the connection

One example of this is, if Elasticsearch receives a _bulk request where the content-length is larger than it is configured to handle, it will stop reading the request body after the first chunk is sent, respond with a 413 Content Too Large, and close the request.

In this case, the transport would see this as an EPIPE error. The problem is that, when using HttpConnection (which many customers still do, as well as Kibana), it would see the response body was finished being sent, via the request end event, and stop listening to all events on the request, including error events, before the connection was closed by the server and the EPIPE error would be raised. So, that unhandled error would bubble up the stack, and the transport would attempt to finish sending the request body over a now-closed connection.

HttpConnection was being too optimistic about what signals the end of a request/response cycle. When using the Node.js http library, as HttpConnection does, the following conditions are a better way to ensure that the cycle has finished:

  • request finish and response end events have both fired (success)
  • request error event has fired, signaling a connection-related issue (failure)
  • response close event has fired before either of the above conditions are met, signaling an early connection closure (failure)

This fixes elastic/elasticsearch-js#2605.

@JoshMock JoshMock merged commit b44dd13 into main Feb 21, 2025
21 checks passed
@JoshMock JoshMock deleted the server-closed-connection-2 branch February 21, 2025 19:03
Copy link

The backport to 8.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-8.x 8.x
# Navigate to the new working tree
cd .worktrees/backport-8.x
# Create a new branch
git switch --create backport-223-to-8.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b44dd134cc9bf1fbca5204851ec38030cdfd06e9
# Push it to GitHub
git push --set-upstream origin backport-223-to-8.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.x

Then, create a pull request where the base branch is 8.x and the compare/head branch is backport-223-to-8.x.

JoshMock added a commit that referenced this pull request Feb 21, 2025
* Handle EPIPE error when server unexpectedly closes the connection

* Add tests for EPIPE server disconnect edge case

* Drop irrelevant comment

* Cleanup of unneeded logic

* Explicitly handle EPIPE error

response 'close' event catches it implicitly on Linux, but apparently
not on Windows or Mac.

* Make error message check more permissive

* Make error message check even more permissive

(cherry picked from commit b44dd13)
JoshMock added a commit that referenced this pull request Feb 21, 2025
…s sent (#224)

* Handle server closing connection after response body is sent (#223)

* Handle EPIPE error when server unexpectedly closes the connection

* Add tests for EPIPE server disconnect edge case

* Drop irrelevant comment

* Cleanup of unneeded logic

* Explicitly handle EPIPE error

response 'close' event catches it implicitly on Linux, but apparently
not on Windows or Mac.

* Make error message check more permissive

* Make error message check even more permissive

(cherry picked from commit b44dd13)

* Use callback style of setTimeout for 8.x

* Test 8.x transport with 8.x client
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Client does not handle connections being closed by the server in all cases
1 participant