fix build curl byte range error #1143

zerog2k · 2023-11-15T16:22:37Z

fixes #920

Downloading Buildroot sources to cache directory ...
curl --continue-at - --output /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz https://buildroot.org/downloads/buildroot-2023.02.1.tar.gz
** Resuming transfer from byte position 7181664
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.

seems like buildroot.org servers not supporting byte range requests

zerog2k · 2023-11-15T16:26:13Z

alternatively, maybe it should use mirrors, or host the file on github (as release binary)?

themactep · 2023-11-15T16:57:53Z

it works fine for me

$ curl --verbose --continue-at - --output /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz https://buildroot.org/downloads/buildroot-2023.02.1.tar.gz
** Resuming transfer from byte position 7181664
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.122:443...
* Connected to buildroot.org (140.211.167.122) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4067 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=buildroot.org
*  start date: Sep 30 03:25:10 2023 GMT
*  expire date: Dec 29 03:25:09 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
} [5 bytes data]
> GET /downloads/buildroot-2023.02.1.tar.gz HTTP/1.1
> Host: buildroot.org
> Range: bytes=7181664-
> User-Agent: curl/8.4.0
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/1.1 416 Requested Range Not Satisfiable
< Date: Wed, 15 Nov 2023 16:56:28 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000
< Strict-Transport-Security: max-age=63072000
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=iso-8859-1
< 
{ [390 bytes data]
100   378    0   378    0     0   1265      0 --:--:-- --:--:-- --:--:--  1272
* Connection #0 to host buildroot.org left intact

themactep · 2023-11-15T17:00:08Z

even with an incomplete file in cache

$ dd if=/tmp/buildroot_dl/buildroot-2023.02.1.tar.gz of=/tmp/buildroot_dl/buildroot-2023.02.1.tar.gz2 bs=1 count=30000
30000+0 records in
30000+0 records out
30000 bytes (30 kB, 29 KiB) copied, 0.0652167 s, 460 kB/s

$ mv /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz2 /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz 

$ curl --verbose --continue-at - --output /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz https://buildroot.org/downloads/buildroot-2023.02.1.tar.gz
** Resuming transfer from byte position 30000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.122:443...
* Connected to buildroot.org (140.211.167.122) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4067 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=buildroot.org
*  start date: Sep 30 03:25:10 2023 GMT
*  expire date: Dec 29 03:25:09 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
} [5 bytes data]
> GET /downloads/buildroot-2023.02.1.tar.gz HTTP/1.1
> Host: buildroot.org
> Range: bytes=30000-
> User-Agent: curl/8.4.0
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/1.1 206 Partial Content
< Date: Wed, 15 Nov 2023 16:58:45 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000
< Last-Modified: Tue, 09 May 2023 20:55:00 GMT
< ETag: "6d9560-5fb48f8070031"
< Accept-Ranges: bytes
< Content-Length: 7151664
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1
< Content-Range: bytes 30000-7181663/7181664
< Content-Type: application/x-gzip
< 
{ [7784 bytes data]
100 6984k  100 6984k    0     0  5753k      0  0:00:01  0:00:01 --:--:-- 5757k
* Connection #0 to host buildroot.org left intact

zerog2k · 2023-11-15T18:55:08Z

interesting... perhaps related to something on my curl client side?

from a mac big sur, working, but only on initial (cache miss) pull:

hackpro:~ jens$  curl --verbose --continue-at - --output /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz https://buildroot.org/downloads/buildroot-2023.02.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.122:443...
* Connected to buildroot.org (140.211.167.122) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [227 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4060 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=buildroot.org
*  start date: Sep 30 03:25:10 2023 GMT
*  expire date: Dec 29 03:25:09 2023 GMT
*  subjectAltName: host "buildroot.org" matched cert's "buildroot.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.1
> GET /downloads/buildroot-2023.02.1.tar.gz HTTP/1.1
> Host: buildroot.org
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 15 Nov 2023 18:38:59 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000
< Last-Modified: Tue, 09 May 2023 20:55:00 GMT
< ETag: "6d9560-5fb48f8070031"
< Accept-Ranges: bytes
< Content-Length: 7181664
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1
< Content-Type: application/x-gzip
< 
{ [7841 bytes data]
100 7013k  100 7013k    0     0  8994k      0 --:--:-- --:--:-- --:--:-- 9061k
* Connection #0 to host buildroot.org left intact

hackpro:~ jens$ curl --version
curl 8.1.2 (x86_64-apple-darwin20.0) libcurl/8.1.2 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.41.0
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

from my ubuntu-18.04 build box:

jens@mars:~$ curl --verbose --continue-at - --output /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz https://buildroot.org/downloads/buildroot-2023.02.1.tar.gz
** Resuming transfer from byte position 7181664
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.122...
* TCP_NODELAY set
* Connected to buildroot.org (140.211.167.122) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Unknown (8):
{ [25 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4067 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=buildroot.org
*  start date: Sep 30 03:25:10 2023 GMT
*  expire date: Dec 29 03:25:09 2023 GMT
*  subjectAltName: host "buildroot.org" matched cert's "buildroot.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
> GET /downloads/buildroot-2023.02.1.tar.gz HTTP/1.1
> Host: buildroot.org
> Range: bytes=7181664-
> User-Agent: curl/7.58.0
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
< HTTP/1.1 416 Requested Range Not Satisfiable
< Date: Wed, 15 Nov 2023 18:39:16 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000
< Strict-Transport-Security: max-age=63072000
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=iso-8859-1
< 
* HTTP server doesn't seem to support byte ranges. Cannot resume.
* stopped the pause stream!
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (21):
} [1 bytes data]
* TLSv1.3 (OUT), TLS alert, Client hello (1):
} [2 bytes data]
curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.

jens@mars:~$ curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

on the subsequent pulls on mac:

hackpro:~ jens$  curl --verbose --continue-at - --output /tmp/buildroot_dl/buildroot-2023.02.1.tar.gz https://buildroot.org/downloads/buildroot-2023.02.1.tar.gz
** Resuming transfer from byte position 7181664
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.122:443...
* Connected to buildroot.org (140.211.167.122) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [227 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4060 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=buildroot.org
*  start date: Sep 30 03:25:10 2023 GMT
*  expire date: Dec 29 03:25:09 2023 GMT
*  subjectAltName: host "buildroot.org" matched cert's "buildroot.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.1
> GET /downloads/buildroot-2023.02.1.tar.gz HTTP/1.1
> Host: buildroot.org
> Range: bytes=7181664-
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 416 Requested Range Not Satisfiable
< Date: Wed, 15 Nov 2023 18:42:42 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000
< Strict-Transport-Security: max-age=63072000
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=iso-8859-1
< 
{ [390 bytes data]
100   378    0   378    0     0   1510      0 --:--:-- --:--:-- --:--:--  1549
* Connection #0 to host buildroot.org left intact
hackpro:~ jens$

looks like server gives 416 error when sending Range request header, but the difference here is that older curl exits non-zero (33) whereas newer curl behavior seems to not consider this an error (exit code 0).

  33     HTTP range error. The range "command" didn't work.

not sure if perhaps it can be handled, i.e. permitting error code 33 to pass

looking for ideas on this. I do see the need for caching (to not hammer buildroot's servers unnecessarily).

flyrouter

We plan to stop using this script and remove it in the near future.
Please keep this in mind.

fix build curl byte range error

3fe7845

zerog2k requested review from widgetii, dimerr and flyrouter as code owners November 15, 2023 16:22

flyrouter approved these changes Nov 16, 2023

View reviewed changes

flyrouter merged commit 6998d1f into OpenIPC:master Nov 16, 2023
97 checks passed

zerog2k deleted the fix-buildroot-curl branch November 16, 2023 22:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix build curl byte range error #1143

fix build curl byte range error #1143

zerog2k commented Nov 15, 2023

zerog2k commented Nov 15, 2023

themactep commented Nov 15, 2023

themactep commented Nov 15, 2023

zerog2k commented Nov 15, 2023 •

edited

Loading

flyrouter left a comment

fix build curl byte range error #1143

fix build curl byte range error #1143

Conversation

zerog2k commented Nov 15, 2023

zerog2k commented Nov 15, 2023

themactep commented Nov 15, 2023

themactep commented Nov 15, 2023

zerog2k commented Nov 15, 2023 • edited Loading

flyrouter left a comment

Choose a reason for hiding this comment

zerog2k commented Nov 15, 2023 •

edited

Loading