Python 3: Add key "Content-Type" check for python 3 in webdriver test

[ziransun]

In webdriver getheaders() API of http response returns lowercase key
"content-type" in python 2 and capitalized key "Content-Type" in python 3.
This probably is caused by different http libraries used. Python 2 uses
httplib and python 3 uses http.client for http connection.

[ziransun]

Tracing back code in wpt/tools/webdriver/webdriver/transport.py
send->response = self._request(method, uri, payload, headers, timeout=None) -> response = self.connection.getresponse()->self._conn = self._conn = HTTPConnection(self.host, self.port, **conn_kwargs) ->from six.moves.http_client import HTTPConnection

[andreastt]

Python 2 overrides the casing in headers returned from the server? This strikes me as odd, but it does not surprise me.

In any case, the fix looks straight forward to me. Thanks!

[andreastt]

Some of the stability tests on Chrome and Firefox are failing:
https://community-tc.services.mozilla.com/tasks/groups/FKWi0dL4T0y8Ct1_LvLV-g

Could you take a look?

[stephenmcgruer]

Stability checks fail because anything that touches webdriver tests takes too long for them to complete (webdriver tests are slow...).

There are some diffs on Firefox and Safari results, but they look like expected flakes (the webdriver tests - or the implementations - are also flaky).

Ziran; are you able to find any instances online of people reporting that headers have different cases between the two libs or between Python 2 and 3? I'd like to understand this better before making an if PY3 change.

[ziransun]

Stephen, I found a similar issue reported here -
mozilla/agithub#67

[stephenmcgruer]

Ok, so I spent some time digging into this, and I'm happy that this is a fundamental difference in Py2/Py3. This is repeating some of what Ziran said, but:

tools/webdriver/webdriver/transport.py
-> calls Response.from_http, which internally uses headers = dict(http_response.getheaders())
-> The http_response here is from response = self.connection.getresponse()
-> And self.connection is self._conn = HTTPConnection(self.host, self.port, **conn_kwargs)

So we're using HTTPConnection, which in both Python 2 and 3 returns a HTTPResponse. Now how is getheaders() implemented? (In CPython).

Python 2: Uses an HTTPMessage which is (via mimetools.Message) an rfc822.Message, which calls lower() on the header key.

Python 3: Uses an email.Parser, which uses an email.FeedParser. This has horribly obtuse code (uses generators, a message passing stack, ...), but ultimately calls self.policy.header_source_parse. There are different policies but all have the same logic, and the one used is Compat32 anyway, which doesn't modify the header key.

In terms of code samples, one can see this via:

smcgruer@penguin:~/github/wpt$ python2
Python 2.7.13 (default, Sep 26 2018, 18:42:22) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> conn = httplib.HTTPConnection("www.python.org")
>>> conn.request("GET", "/index.html")
>>> r1 = conn.getresponse()
>>> r1.getheaders()
[('content-length', '0'), ('via', '1.1 varnish'), ('x-cache', 'HIT'), ('accept-ranges', 'bytes'), ('x-timer', 'S1586714954.221388,VS0,VE0'), ('strict-transport-security', 'max-age=63072000; includeSubDomains'), ('server', 'Varnish'), ('retry-after', '0'), ('connection', 'close'), ('x-served-by', 'cache-yyz4545-YYZ'), ('x-cache-hits', '0'), ('location', 'https://www.python.org/index.html'), ('date', 'Sun, 12 Apr 2020 18:09:14 GMT')]
>>> 
smcgruer@penguin:~/github/wpt$ python3
Python 3.5.3 (default, Sep 27 2018, 17:25:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import http.client
>>> conn = http.client.HTTPConnection("www.python.org")
>>> conn.request("GET", "/index.html")
>>> r1 = conn.getresponse()
>>> r1.getheaders()
[('Server', 'Varnish'), ('Retry-After', '0'), ('Location', 'https://www.python.org/index.html'), ('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Date', 'Sun, 12 Apr 2020 18:09:36 GMT'), ('Via', '1.1 varnish'), ('Connection', 'close'), ('X-Served-By', 'cache-yyz4551-YYZ'), ('X-Cache', 'HIT'), ('X-Cache-Hits', '0'), ('X-Timer', 'S1586714977.600726,VS0,VE0'), ('Strict-Transport-Security', 'max-age=63072000; includeSubDomains')]

Now that we have that out of the way, I'll take a look at the PR :)

[ziransun]

Added comment lines as suggested. Thanks Stephen!

[dralley]

This must have changed sometime between Python 3.5 (what you used) and Python 3.8 (what I just ran the test on) because the new expectation actually causes the assertion to fail, for me.

AssertionError: assert 'Content-Type' in {'cache-control': 'no-cache', 'content-length': '14', 'content-type': 'application/json; charset=utf-8', 'date': 'Fri, 01 May 2020 03:54:48 GMT'}

edit: pinged the wrong person

[ziransun]

@dralley I'm trying to reproduce the bug you mentioned above. Any particular test you ran that gave this error?

[dralley]

It's possible I'm doing something wrong or there is something else "off" with my environment, but here's what I'm doing:

python3 ./mach test-wpt --timeout-multiplier=4 --headless /webdriver/tests/get_element_rect/get.py

(this is a Servo build environment)

Python details:

Python 3.8.2 (default, Feb 28 2020, 00:00:00) 
[GCC 10.0.1 20200216 (Red Hat 10.0.1-0.8)] on linux
Type "help", "copyright", "credits" or "license" for more information.

[ziransun]

@dralley I tried this using Python 3.8.2 with Chrome browser in Ubuntu environment and didn't notice this error.

[ziransun]

@dralley: I can confirm that this issue exists when running firefox browser in PY3. Sorry that I wasn't able to reproduce it at the time you raised the issue!

[ziransun]

Further change at PR #25039