Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Reason: 'HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)'. #165

Closed
airplane03 opened this issue Nov 27, 2023 · 8 comments
Assignees

Comments

@airplane03
Copy link

airplane03 commented Nov 27, 2023

It seems like a libcurl Error or a server Error

To Reproduce

from curl_cffi import requests

url = 'https://booking.jetstar.com/cn/zh/booking/select-flights'
sess = requests.Session()
req = requests.get(url, impersonate="chrome110")
print(req.text)

Expected behavior

    566     rsp.request = req
--> 567     raise RequestsError(str(e), e.code, rsp) from e
    568 else:
    569     rsp = self._parse_response(c, buffer, header_buffer)

RequestsError: Failed to perform, ErrCode: 92, Reason: 'HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)'. This may be a libcurl error, See https://curl.se/libcurl/c/libcurl-errors.html first for more details.
@coletdjnz
Copy link
Contributor

I can also reproduce on 0.5.10 (stream, sync). What version are you running?

I was getting this error too the other day with stream=True, sync, 0.5.10b5 on an internal resource. I couldn't repro on any other site so assumed it was a server issue. 🤔

@perklet
Copy link
Collaborator

perklet commented Nov 29, 2023

The url does not work in my browser, too. So I guess it's a server error.

This error(http/2 stream 0) has been reported many times ever since curl_cffi was published, but I stil can not find a reproducible way to trigger it. Given that the majority users are behind proxies, the situation is even more difficult to deal with. I'm even not sure it's a bug introduced in libcurl, curl-impersonate or curl_cffi, or it's just a server error. Depending on your context, here are some general suggestions for future comers:

  1. Try to see if this error was caused by proxies, if so, use better proxies.
  2. If it stops working after a while, maybe you're just being blocked by, such as, Akamai.
  3. Force http/1.1 mode.
  4. See if the url works in your real browser.
  5. Find a stable way to reproduce it, so we can finally fix, or at least bypass it.

@coletdjnz if you can still reproduce this, please continue to comment.

@airplane03
Copy link
Author

@yifeikong TKS for suggestion

Yup, this websit is protected with Akamai.

It works fine on my browser(http2, response with a redirect) and postman(http1.1, response a 400 bad requests page). when I tried it with curl_cffi in http/1.1 mode, the requests time out, acting like the python requests module. So I guess the libcurl based on openssl is blocked by this website.

I see the google's boringssl is in the dependency of curl , so I wonderd if I replace the libcurl.dll with a new one based on boringssl is feasible.

@perklet
Copy link
Collaborator

perklet commented Nov 29, 2023

Akamai uses http2 fingerprints to detect browsers. This value was not changed for many chrome versions until 117, which curl_cffi does not support by now. It should be resolved by #163

BoringSSL was compiled staticlly into the .so file, and it's not related to http2 fingerprints.

@gospider007
Copy link

@yifeikong TKS for suggestion

Yup, this websit is protected with Akamai.

It works fine on my browser(http2, response with a redirect) and postman(http1.1, response a 400 bad requests page). when I tried it with curl_cffi in http/1.1 mode, the requests time out, acting like the python requests module. So I guess the libcurl based on openssl is blocked by this website.

I see the google's boringssl is in the dependency of curl , so I wonderd if I replace the libcurl.dll with a new one based on boringssl is feasible.

@airplane03 Your test method is wrong. The URL you mentioned requires cookies and headers to access. Of course, your browser can succeed if it has complete headers and cookies, but your test code does not have these.

@airplane03
Copy link
Author

@yifeikong TKS for suggestion
Yup, this websit is protected with Akamai.
It works fine on my browser(http2, response with a redirect) and postman(http1.1, response a 400 bad requests page). when I tried it with curl_cffi in http/1.1 mode, the requests time out, acting like the python requests module. So I guess the libcurl based on openssl is blocked by this website.
I see the google's boringssl is in the dependency of curl , so I wonderd if I replace the libcurl.dll with a new one based on boringssl is feasible.

@airplane03 Your test method is wrong. The URL you mentioned requires cookies and headers to access. Of course, your browser can succeed if it has complete headers and cookies, but your test code does not have these.

That dosn't mater, the curl_cffi with impersonate takes headers itself, For cookies, the anonymous browser and postman do not have cookies either

@gospider007
Copy link

@yifeikong TKS for suggestion
Yup, this websit is protected with Akamai.
It works fine on my browser(http2, response with a redirect) and postman(http1.1, response a 400 bad requests page). when I tried it with curl_cffi in http/1.1 mode, the requests time out, acting like the python requests module. So I guess the libcurl based on openssl is blocked by this website.
I see the google's boringssl is in the dependency of curl , so I wonderd if I replace the libcurl.dll with a new one based on boringssl is feasible.

@airplane03 Your test method is wrong. The URL you mentioned requires cookies and headers to access. Of course, your browser can succeed if it has complete headers and cookies, but your test code does not have these.

That dosn't mater, the curl_cffi with impersonate takes headers itself, For cookies, the anonymous browser and postman do not have cookies either

I don’t know how you tested it. If I open this address directly in an incognito browser, I will be unable to access it. However, if I visit the homepage first and then visit the address you provided, I can access it. When I complete the browser Copy the headers and send the request to the address you provided, and the request library will run normally.

@airplane03
Copy link
Author

@yifeikong TKS for suggestion
Yup, this websit is protected with Akamai.
It works fine on my browser(http2, response with a redirect) and postman(http1.1, response a 400 bad requests page). when I tried it with curl_cffi in http/1.1 mode, the requests time out, acting like the python requests module. So I guess the libcurl based on openssl is blocked by this website.
I see the google's boringssl is in the dependency of curl , so I wonderd if I replace the libcurl.dll with a new one based on boringssl is feasible.

@airplane03 Your test method is wrong. The URL you mentioned requires cookies and headers to access. Of course, your browser can succeed if it has complete headers and cookies, but your test code does not have these.

That dosn't mater, the curl_cffi with impersonate takes headers itself, For cookies, the anonymous browser and postman do not have cookies either

I don’t know how you tested it. If I open this address directly in an incognito browser, I will be unable to access it. However, if I visit the homepage first and then visit the address you provided, I can access it. When I complete the browser Copy the headers and send the request to the address you provided, and the request library will run normally.

Tks, what an idiot i am, I got the response after taking a header and changing my IP,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants