Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Proxy doesn't change when using different username #308

Open
MrDebugger opened this issue Apr 30, 2024 · 2 comments
Open

[BUG] Proxy doesn't change when using different username #308

MrDebugger opened this issue Apr 30, 2024 · 2 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@MrDebugger
Copy link

Describe the bug:
The curl_cffi library does not switch proxies correctly when only the username in the proxy URL changes, while other components (host, port, and password) remain the same. This issue is observed when multiple proxies are used sequentially in a single session, resulting in all requests incorrectly using the first proxy's credentials.

To Reproduce:
Steps to reproduce the behavior:

Set up a Session with proxies that differ only in the username.
Execute requests to a target website (e.g., httpbin.org/ip) that returns the requester's IP.
Notice that the session fails to update the proxy after the initial set, using the first proxy for all requests.

from curl_cffi.requests import Session

proxies = {
    'Proxy1': 'http://username1:password@host:port',
    'Proxy2': 'http://username2:password@host:port',
}

s = Session()

for proxy_key, proxy_url in proxies.items():
    s.proxies = {
        'http': proxy_url,
        'https': proxy_url,
    }
    r = s.get('https://httpbin.org/ip')
    print(proxy_key, r.json())

Expected behavior:
Each request should reflect the IP of the corresponding proxy defined by its username. However, the actual output shows the IP of the first proxy for both requests, indicating that the proxy change was not recognized.

Versions:
OS: Ubuntu 20.04.4 LTS
curl_cffi version: 0.6.3
Python version: 3.10.12

Additional context:
The session used is synchronous.
The proxy credentials issue suggests a possible mishandling in the proxy management within the session, particularly with caching or parsing mechanisms.

@MrDebugger MrDebugger added the bug Something isn't working label Apr 30, 2024
@perkfly
Copy link
Collaborator

perkfly commented Apr 30, 2024

I'm afraid this is how libcurl works. See my comments here.

#210 (comment)

You can try adding Connection: close header or downgrading to HTTP/1.1, but it's not guaranteed to work.

IMHO, this is the correct behavior actually. Establishing new connection to the same address is quite inefficient, and these auto-rotating proxies abuse the behavior.

@perkfly perkfly added this to the v0.8 milestone Jul 4, 2024
@a-n-i-m-e-z
Copy link

解决方法是对于你认为不同的代理单独创建一个会话,每个会话对应一个代理,否则单一的会话即便是不同IP也会很容易被检测到,这就导致你用这个库没有任何意义

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants