Failure to build against nginx 1.23.2 on Ubuntu 20.04 (focal)

[ronilaukkarinen]

I'm opening a separate issue about this since other issues have been either closed or not resolved and getting a clear consensus on how to proceed is rather "messy" to say the least. Feel free to close and point out to the right main issue if necessary.

Issues: #1758 #1757 #1756 #1749 #1752

There is a probable solution here, but we're not ready to try this kind of experimenting in production environment even during maintenance window. Is it guaranteed it works for Ubuntu 20.04.5 LTS (focal)? We don't know.

Is there anyone working on this? Is there a bounty, would be glad to compensate for the effort as this affects for many servers for us.

Version information

Linux distribution:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"

nginx version:

$ sudo nginx -V

nginx version: nginx/1.23.2
built by gcc 9.3.0 (Ubuntu 9.3.0-10ubuntu2)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.23.2/debian/debuild-base/nginx-1.23.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

How to reproduce:

1. Use rather new Ubuntu, tested on Ubuntu 20.04

2. Install nginx 1.23.2 (released 19 Oct 2022, at the moment it's the latest version) via Official Ubuntu Intsallation instructions on Nginx.org

3. Use install script to fetch/install latest ngx-pagespeed:

   bash <(curl -f -L -sS https://ngxpagespeed.com/install) \
     --nginx-version latest

4. About to build nginx.  Do you have any additional ./configure
   arguments you would like to set?  For example, if you would like
   to build nginx with https support give --with-http_ssl_module
   If you do not have any, just press enter.

   Add following:

   --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module

5. Answer Y to following:

   Does this look right? [Y/n]
   Build nginx? [Y/n]
   Install nginx? [Y/n]
   

Build goes fine, but make install fails badly. See logs below.

This also happens on nginx 1.23.1 (released 19 Jul 2022).

Detailed information/log:

[...] truncated [...]
	/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_list_iterator.cc
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g   -D_GLIBCXX_USE_CXX11_ABI=0 --std=c++11  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules -I src/http/v2 -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/chromium/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/google-sparsehash/src/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/google-sparsehash/gen/arch/linux/x64/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/grpc/src/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/protobuf/src/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/re2/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/out/Release/obj/gen -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/out/Release/obj/gen/protoc_out/instaweb -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/apr/src/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/aprutil/src/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/apr/gen/arch/linux/x64/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/aprutil/gen/arch/linux/x64/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/url -I src/mail -I src/stream \
	-o objs/addon/src/ngx_message_handler.o \
	/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_message_handler.cc
cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g   -D_GLIBCXX_USE_CXX11_ABI=0 --std=c++11  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules -I src/http/v2 -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/chromium/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/google-sparsehash/src/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/google-sparsehash/gen/arch/linux/x64/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/grpc/src/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/protobuf/src/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/re2/src -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/out/Release/obj/gen -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/out/Release/obj/gen/protoc_out/instaweb -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/apr/src/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/aprutil/src/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/apr/gen/arch/linux/x64/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/third_party/aprutil/gen/arch/linux/x64/include -I /home/rolle/incubator-pagespeed-ngx-latest-stable/psol/include/url -I src/mail -I src/stream \
	-o objs/addon/src/ngx_pagespeed.o \
	/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc: In function ‘ngx_int_t net_instaweb::{anonymous}::ps_set_cache_control(ngx_http_request_t*, char*)’:
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:407:36: error: request for member ‘elts’ in ‘r->ngx_http_request_s::headers_out.ngx_http_headers_out_t::cache_control’, which is of pointer type ‘ngx_table_elt_t*’ {aka ‘ngx_table_elt_s*’} (maybe you meant to use ‘->’ ?)
  407 |   if (r->headers_out.cache_control.elts == NULL) {
      |                                    ^~~~
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:408:35: error: cannot convert ‘ngx_table_elt_t**’ {aka ‘ngx_table_elt_s**’} to ‘ngx_array_t*’
  408 |     ngx_int_t rc = ngx_array_init(&r->headers_out.cache_control, r->pool,
      |                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   ngx_table_elt_t** {aka ngx_table_elt_s**}
In file included from src/core/ngx_core.h:65,
                 from src/http/ngx_http.h:13,
                 from /home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.h:33,
                 from /home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:26:
src/core/ngx_array.h:32:29: note:   initializing argument 1 of ‘ngx_int_t ngx_array_init(ngx_array_t*, ngx_pool_t*, ngx_uint_t, size_t)’
   32 | ngx_array_init(ngx_array_t *array, ngx_pool_t *pool, ngx_uint_t n, size_t size)
      |                ~~~~~~~~~~~~~^~~~~
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:415:22: error: cannot convert ‘ngx_table_elt_t**’ {aka ‘ngx_table_elt_s**’} to ‘ngx_array_t*’
  415 |       ngx_array_push(&r->headers_out.cache_control));
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                      |
      |                      ngx_table_elt_t** {aka ngx_table_elt_s**}
In file included from src/core/ngx_core.h:65,
                 from src/http/ngx_http.h:13,
                 from /home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.h:33,
                 from /home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:26:
src/core/ngx_array.h:27:35: note:   initializing argument 1 of ‘void* ngx_array_push(ngx_array_t*)’
   27 | void *ngx_array_push(ngx_array_t *a);
      |                      ~~~~~~~~~~~~~^
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc: In function ‘bool net_instaweb::{anonymous}::ps_get_cache_control(ngx_http_request_t*, GoogleString*)’:
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:439:74: error: request for member ‘elts’ in ‘r->ngx_http_request_s::headers_out.ngx_http_headers_out_t::cache_control’, which is of pointer type ‘ngx_table_elt_t*’ {aka ‘ngx_table_elt_s*’} (maybe you meant to use ‘->’ ?)
  439 |   auto ccp = static_cast<ngx_table_elt_t**>(r->headers_out.cache_control.elts);
      |                                                                          ^~~~
/home/rolle/incubator-pagespeed-ngx-latest-stable/src/ngx_pagespeed.cc:444:59: error: request for member ‘nelts’ in ‘r->ngx_http_request_s::headers_out.ngx_http_headers_out_t::cache_control’, which is of pointer type ‘ngx_table_elt_t*’ {aka ‘ngx_table_elt_s*’} (maybe you meant to use ‘->’ ?)
  444 |   for (ngx_uint_t i = 0; i < r->headers_out.cache_control.nelts; i++) {
      |                                                           ^~~~~
make[1]: *** [objs/Makefile:1945: objs/addon/src/ngx_pagespeed.o] Error 1
make[1]: Leaving directory '/home/rolle/nginx-1.23.2'
make: *** [Makefile:10: build] Error 2
Error: Failure running 'make', exiting.

Full logs can be provided if needed.

Currently the only way around this is to revert andhold package:

sudo apt-cache showpkg nginx |grep 1.21
sudo apt install nginx=1.21.6-1~focal

Then rebuild nginx with defining the specific version:

bash <(curl -f -L -sS https://ngxpagespeed.com/install) \
     --nginx-version 1.21.6

Finally, hold nginx:

sudo apt-mark hold nginx

Latest working version:

$ sudo nginx -V
nginx version: nginx/1.21.6
built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --add-module=/home/rolle/incubator-pagespeed-ngx-latest-stable --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module