Download speed up

[mrx23dot]

I managed to speed up downloading a new quarter by 0.7sec, only be internally replacing https with http in urls.
It bypasses the SSL which is slow. It's future proof, so even when SEC disables http it redirects to https.

http only:

565662 function calls (479577 primitive calls) in 5.358 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
86 1.651 0.019 1.651 0.019 {method 'recv_into' of '_socket.socket' objects}
14 1.423 0.102 1.423 0.102 {built-in method time.sleep}
43 0.433 0.010 0.433 0.010 {method 'read' of '_ssl._SSLSocket' objects} <- not sure what the remaining is!
21 0.318 0.015 0.318 0.015 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}

INFO:xbrl.helper.connection_manager:200 http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327.htm
INFO:xbrl.helper.connection_manager:200 http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/srt/2020/elts/srt-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/srt/2020/elts/srt-types-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-gaap-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-types-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-roles-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-gaap-eedm-def-2020-01-31.xml
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/srt/2020/elts/srt-eedm1-def-2020-01-31.xml
INFO:xbrl.helper.connection_manager:200 http://xbrl.sec.gov/dei/2020/dei-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_lab.xml
INFO:xbrl.helper.connection_manager:200 http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_cal.xml
INFO:xbrl.helper.connection_manager:200 http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_def.xml
INFO:xbrl.helper.connection_manager:200 http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_pre.xml

original mixed http+https:

537159 function calls (451090 primitive calls) in 6.054 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
45 1.557 0.035 1.557 0.035 {method 'read' of '_ssl._SSLSocket' objects}
14 1.455 0.104 1.455 0.104 {built-in method time.sleep}
88 1.360 0.015 1.360 0.015 {method 'recv_into' of '_socket.socket' objects}
581 0.245 0.000 0.245 0.000 {built-in method new of type object at 0x0000000060CC9C00}
21 0.207 0.010 0.207 0.010 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}

INFO:xbrl.helper.connection_manager:200 https://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327.htm
INFO:xbrl.helper.connection_manager:200 https://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/srt/2020/elts/srt-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/srt/2020/elts/srt-types-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-gaap-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-types-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-roles-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/us-gaap/2020/elts/us-gaap-eedm-def-2020-01-31.xml
INFO:xbrl.helper.connection_manager:200 http://xbrl.fasb.org/srt/2020/elts/srt-eedm1-def-2020-01-31.xml
INFO:xbrl.helper.connection_manager:200 https://xbrl.sec.gov/dei/2020/dei-2020-01-31.xsd
INFO:xbrl.helper.connection_manager:200 https://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_lab.xml
INFO:xbrl.helper.connection_manager:200 https://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_cal.xml
INFO:xbrl.helper.connection_manager:200 https://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_def.xml
INFO:xbrl.helper.connection_manager:200 https://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_pre.xml

I recommend adding this:
query_response = self.connection_manager.download(file_url.replace('https','http'), headers=self.headers)

[manusimidt]

I don't think that just replacing the protocol in the url will increase the download speed.
Most sites (sec.gov included) redirect all incoming traffic on port 80 (http) to port 443 (https). This means that the request will be SSL-encryped no matter if you use http or https.

Please correct me if i am wrong :)

[mrx23dot]

Yeah, you are right, http is redirected to https, not sure where the difference is coming from.

Also session.verify = False helps a few percent.

python -m cProfile -s tottime http_test.py > prof.txt

from requests import Session,packages
packages.urllib3.disable_warnings()

session = Session()
session.verify = False
session.headers.update({'From': '[email protected]', 'User-Agent': 'tester v1'})

url='http://www.sec.gov/Archives/edgar/data/0000320193/000032019321000056/aapl-20210327_lab.xml'
resp = session.get(url)
assert resp.ok
print(len(resp.text))

[mrx23dot]

into here to be more precise:

xbrl\helper\connection_manager.py

from requests import Session,packages
packages.urllib3.disable_warnings()
response = self._session.get(url, headers=headers, allow_redirects=True, **verify=False**)

I will also try out the zip download to speed things up.