Load balancing to a backed websocket service using "least_conn" #1960

vkbodanki · 2021-09-13T17:42:03Z

vkbodanki
Sep 13, 2021

Hi, I am trying to set up the ingress controller to load balance to a websocket backed svc using the "least_conn" algorithm for load balancing.

To being with, I have a service that accepts incoming websocket connections for clients. Kubeproxy is currently responsible for load balancing the incoming websocket requests across different service pods. Everything works fine with round-robin algorithm that Kubeproxy supports.

Now I wanted to replace Kubeproxy with Nginx ingress controller for load balancing the websocket connections to my backend service pods. I configured all the moving parts necessary for Nginx and got websocket load balancing to work with Nginx ingress controller. However, I still see that the load balancing is still round-robin even though I configured the lb-method to "least_conn". I am not sure if I missed a configuration that enables using different load balancing options. Any help is greatly appreciated.

Here is the Ingress config.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: mytest-ingress-nginx
 annotations:
  nginx.org/websocket-services: "mytest-svc"
  nginx.org/ssl-services: "mytest-svc"
  nginx.org/lb-method: "least_conn"
spec:
 ingressClassName: nginx
 tls:
 - hosts:
   - "testApp.westus2.cloudapp.azure.com"
   secretName: nginx-ingress-tls
 rules:
 - host: "testApp.westus2.cloudapp.azure.com"
   http:
    paths:
    - path: /direct
      pathType: Prefix
      backend:
       service:
        name: mytest-svc
        port:
         number: 10443

Answered by jasonwilliams14

Oct 15, 2021

@vebodank I believe we know what is going on and why you are seeing the different behavior between NGINX Ingress OSS and NGINX In gress Plus.
What is happening is that it works fine for NGINX Ingress Plus, because NGINX Ingress Plus does not have to do a reload when upstreams are changed (brought down brought online) This can be done dynamically.

WIth NGINX Ingress Open source, there has to be a reload when upstreams are changed. In your case, when you bring down one, then bring it back online.. That is important because when the reload occurs, there is a new set of worker processes that is spawned and is not aware of any current existing connections to the backend. (those connections are…

View full answer

jasonwilliams14 · 2021-09-29T18:12:09Z

jasonwilliams14
Sep 29, 2021
Collaborator

@vebodank
Couple things.
Can you post the output of your nginx -T from the Ingress pod?
Second, you can try setting the load balancing method in a configmap settings as an alternate.
We will look for your updates.

1 reply

vkbodanki Sep 29, 2021
Author

@jasonwilliams14 Thank you for taking the time. Find below the output of nginx -T (I only have one pod running btw). I also tried setting the lbmethod in config map and got the same results.

Note: I was able to get the least_conn lbmethod to work when I have the NginxPlus ingress controller running. So not sure if this is just a bug with the community version.

PS D:\Repos\kubernetes-ingress\deployments> kubectl exec nginx-ingress-65c8d8cc7f-rlkpd -n nginx-ingress -- nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

worker_processes  auto;
daemon off;

error_log  stderr notice;
pid        /var/lib/nginx/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main
                     '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" rn="$resource_name" "$resource_type" "$resource_namespace" svc="$service" "$request_id" rt=“$request_time” ua=“$upstream_addr” uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time" cs=“$upstream_cache_status“'
                     ;


    access_log  /dev/stdout  main;




    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 65s;
    keepalive_requests 100;

    #gzip  on;

    server_names_hash_max_size 1024;
    server_names_hash_bucket_size 256;

    variables_hash_bucket_size 256;
    variables_hash_max_size 1024;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    map $http_upgrade $vs_connection_header {
        default upgrade;
        ''      $default_connection_header;
    }








    server {
        # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
        set $default_connection_header "";
        set $resource_type "";
        set $resource_name "";
        set $resource_namespace "";
        set $service "";

        listen 80 default_server;


        listen 443 ssl default_server;



        ssl_certificate /etc/nginx/secrets/default;
        ssl_certificate_key /etc/nginx/secrets/default;






        server_name _;
        server_tokens "on";






        location / {
            return 404;
        }
    }
    # stub_status
    server {
        listen 8080;

        allow 127.0.0.1;
        deny all;

        location /stub_status {
            stub_status;
        }
    }
    server {
        listen unix:/var/lib/nginx/nginx-status.sock;
        access_log off;



        location /stub_status {
            stub_status;
        }
    }

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;

    server {
        listen unix:/var/lib/nginx/nginx-502-server.sock;
        access_log off;



        return 502;
    }

    server {
        listen unix:/var/lib/nginx/nginx-418-server.sock;
        access_log off;



        return 418;
    }
}

stream {
    log_format  stream-main  '$remote_addr [$time_local] '
                      '$protocol $status $bytes_sent $bytes_received '
                      '$session_time "$ssl_preread_server_name"';

    access_log  /dev/stdout  stream-main;





    include /etc/nginx/stream-conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/config-version.conf:
server {
    listen unix:/var/lib/nginx/nginx-config-version.sock;
        access_log off;



    location /configVersion {
        return 200 94;
    }
}
map $http_x_expected_config_version $config_version_mismatch {
        "94" "";
        default "mismatch";
}
# configuration file /etc/nginx/conf.d/testapp-mytest-ingress-nginx.conf:
# configuration for testapp/mytest-ingress-nginx

upstream testapp-mytest-ingress-nginx-testApp.westus2.cloudapp.azure.com-mytest-svc-10443 {
        zone testapp-mytest-ingress-nginx-testApp.westus2.cloudapp.azure.com-mytest-svc-10443 256k;
        least_conn;

        server 172.21.1.242:10443 max_fails=1 fail_timeout=10s max_conns=30;
        server 172.21.2.17:10443 max_fails=1 fail_timeout=10s max_conns=30;

}


server {


        listen 80;




        listen 443 ssl;


        ssl_certificate /etc/nginx/secrets/mytest-ingress-nginx-tls;
        ssl_certificate_key /etc/nginx/secrets/mytest-ingress-nginx-tls;







        server_tokens on;

        server_name testApp.westus2.cloudapp.azure.com;

        set $resource_type "ingress";
        set $resource_name "mytest-ingress-nginx";
        set $resource_namespace "testapp";






        if ($scheme = http) {
                return 301 https://$host:443$request_uri;
        }



        location /v1/websocket {
                set $service "mytest-svc";


                proxy_http_version 1.1;

                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass https://testapp-mytest-ingress-nginx-testApp.westus2.cloudapp.azure.com-mytest-svc-10443;


        }


}

jasonwilliams14 · 2021-10-01T19:16:40Z

jasonwilliams14
Oct 1, 2021
Collaborator

@vebodank
Ok. The output looks correct. I can see the load balancing algorithm is defined as "least_conn" in the upstreams, which is what you are looking for.
Are you seeing some unexpected behavior in your traffic?
Additionally, how are you testing this out?

From a NGINX point of view the "least_conn" does not care about the protocol itself. NGINX will load balance/proxy accordingly.

1 reply

vkbodanki Oct 1, 2021
Author

Thanks for confirming the configuration is correct. Here is my test scenario.

Start with 3 backend pods for mytest-svc.
Send 90 requests to the ingress-controller.
The requests are load-balanced to all three backend pods. Each pod gets 30 requests(As expected).
Restart one of the backend pods. All 30 websockets connections to the pod are broken(client's don't attempt to reconnect when a websocket connection is broken).
Now we are left with 3 backend pods. 2 pods with 30 websocket connections each and one pod with 0 websocket connections.
Send another 30 websocket connection requests to the ingress-controller.
That left me with 2 pods having 40 connections each and 1 pod with 10 connections as opposed to 3 pods with 30 connections each.

jasonwilliams14 · 2021-10-04T20:52:14Z

jasonwilliams14
Oct 4, 2021
Collaborator

@vebodank thanks. That is helpful. A few of the engineers and I took a look at your config and have some suggestions.

We noticed that you are not using any keepalives in your upstreams, which we recommend. You can use the following annotation and specify your value:

nginx.org/keepalive

More information here:
https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

NGINX Ingress will automatically upgrade the connections to websockets, as you can see with these two settings:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

We are wondering if you are seeing the number of requests, instead of the number of connections.
If this is NGINX+, you can use the dashboard to watch the connections in realtime.
the other option we can do use extend NGINX Logging and grab some additional information and review the logs.

I would start with the keepalives and use a basic setup, like you have and proceed to test again.
Let us know how it goes.

1 reply

vkbodanki Oct 5, 2021
Author

Thank you @jasonwilliams14. I will repeat the test with your suggestions and report back. Is there a recommended number of connections for the keepalive ?

I would like to clarify a couple of things.

I have done the same test with NGINX+ ingress controller and everything works as expected with a least_conn lb algorithm. So I think this is only an issue with the community version of NGINX.
I have a metric on the service side that track's the number of websocket connections in each pod, so I don't think there is any confusion between requests and connections. Again, as I mentioned above, the same test gives me expected results with the NGINX+ controller.

jasonwilliams14 · 2021-10-05T16:52:00Z

jasonwilliams14
Oct 5, 2021
Collaborator

@vebodank
Ok. Good to hear about NGINX+ Ingress, which is this project. Sounds like ours is working correctly.
I cannot speak for community is that is a different product entirely than this project.
Thanks for the update. 👍

1 reply

vkbodanki Oct 9, 2021
Author

@jasonwilliams14
I apologize for the confusion from my earlier response. By community version , I meant the non-paid version of Nginx+ which is still this project as I understand. I guess what I was trying to say was that the least_conn works fine when I run the ingress controller from ~deployments\deployment\nginx-plus-ingress and does not work with the ingress controller from ~deployments\deployment\nginx-ingress.

I repeated the tests after configuring nginx.org/keepalive and there was no difference in behavior(it was still round robin). Please advise and let me know if there are log files that I can provide to help with the investigation. Find below the upstream config after configuring keepalive.

upstream testapp-mytest-ingress-nginx-testApp.westus2.cloudapp.azure.com-mytest-svc-10443 {
        zone testapp-mytest-ingress-nginx-testApp.westus2.cloudapp.azure.com-mytest-svc-10443 256k;
        least_conn;

        server 172.21.1.235:10443 max_fails=1 fail_timeout=10s max_conns=30;
        server 172.21.2.19:10443 max_fails=1 fail_timeout=10s max_conns=30;

        keepalive 30;

}

jasonwilliams14 · 2021-10-15T03:45:36Z

jasonwilliams14
Oct 15, 2021
Collaborator

@vebodank I believe we know what is going on and why you are seeing the different behavior between NGINX Ingress OSS and NGINX In gress Plus.
What is happening is that it works fine for NGINX Ingress Plus, because NGINX Ingress Plus does not have to do a reload when upstreams are changed (brought down brought online) This can be done dynamically.

WIth NGINX Ingress Open source, there has to be a reload when upstreams are changed. In your case, when you bring down one, then bring it back online.. That is important because when the reload occurs, there is a new set of worker processes that is spawned and is not aware of any current existing connections to the backend. (those connections are handled by the previous worker connections)

That said, when NGINX Ingress OSS is reloaded and the new 30 requests is sent in, NGINX Ingress OSS distributes those connections evenly across the three upstream pods.
That is why you are seeing this behavior in NGINX Ingress OSS and is different from NGINX Pus Ingress

Let me know if that is helpful and answers your question.

0 replies

rthamrin · 2022-03-19T01:57:43Z

rthamrin
Mar 19, 2022

if you don't mind can you check mine? I am also confused about what actually running on my ingress. i tought is rounud robin but seems is not, ewma instead. thank you

root# kubectl exec ingress-controller-nginx-ingress-controller-lqvt7 -n ingress -- nginx -T 

2022/03/18 12:00:17 [warn] 81#81: the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /opt/bitnami/nginx/conf/nginx.conf:143

nginx: [warn] the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /opt/bitnami/nginx/conf/nginx.conf:143

2022/03/18 12:00:17 [warn] 81#81: the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /opt/bitnami/nginx/conf/nginx.conf:144

nginx: [warn] the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /opt/bitnami/nginx/conf/nginx.conf:144

2022/03/18 12:00:17 [warn] 81#81: the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /opt/bitnami/nginx/conf/nginx.conf:145

nginx: [warn] the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /opt/bitnami/nginx/conf/nginx.conf:145

nginx: the configuration file /opt/bitnami/nginx/conf/nginx.conf syntax is ok

nginx: configuration file /opt/bitnami/nginx/conf/nginx.conf test is successful

# configuration file /opt/bitnami/nginx/conf/nginx.conf:



# Configuration checksum: 9211372373012630290



# setup custom paths that do not require root access

pid /tmp/nginx.pid;



daemon off;



worker_processes 1;



worker_rlimit_nofile 1047552;



worker_shutdown_timeout 240s ;



events {

	multi_accept        on;

	worker_connections  16384;

	use                 epoll;

}



http {

	lua_package_path "/etc/nginx/lua/?.lua;;";

	

	lua_shared_dict balancer_ewma 10M;

	lua_shared_dict balancer_ewma_last_touched_at 10M;

	lua_shared_dict balancer_ewma_locks 1M;

	lua_shared_dict certificate_data 20M;

	lua_shared_dict certificate_servers 5M;

	lua_shared_dict configuration_data 20M;

	lua_shared_dict global_throttle_cache 10M;

	lua_shared_dict ocsp_response_cache 5M;

	

	init_by_lua_block {

		collectgarbage("collect")

		

		-- init modules

		local ok, res

		

		ok, res = pcall(require, "lua_ingress")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		lua_ingress = res

		lua_ingress.set_config({

			use_forwarded_headers = false,

			use_proxy_protocol = false,

			is_ssl_passthrough_enabled = false,

			http_redirect_code = 308,

			listen_ports = { ssl_proxy = "442", https = "443" },

			

			hsts = true,

			hsts_max_age = 15724800,

			hsts_include_subdomains = true,

			hsts_preload = false,

			

			global_throttle = {

				memcached = {

					host = "", port = 11211, connect_timeout = 50, max_idle_timeout = 10000, pool_size = 50,

				},

				status_code = 429,

			}

		})

		end

		

		ok, res = pcall(require, "configuration")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		configuration = res

		configuration.prohibited_localhost_port = '10246'

		end

		

		ok, res = pcall(require, "balancer")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		balancer = res

		end

		

		ok, res = pcall(require, "monitor")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		monitor = res

		end

		

		ok, res = pcall(require, "certificate")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		certificate = res

		certificate.is_ocsp_stapling_enabled = false

		end

		

		ok, res = pcall(require, "plugins")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		plugins = res

		end

		-- load all plugins that'll be used here

		plugins.init({  })

	}

	

	init_worker_by_lua_block {

		lua_ingress.init_worker()

		balancer.init_worker()

		

		monitor.init_worker(10000)

		

		plugins.run()

	}

	

	geoip_country       /etc/nginx/geoip/GeoIP.dat;

	geoip_city          /etc/nginx/geoip/GeoLiteCity.dat;

	geoip_org           /etc/nginx/geoip/GeoIPASNum.dat;

	geoip_proxy_recursive on;

	

	aio                 threads;

	aio_write           on;

	

	tcp_nopush          on;

	tcp_nodelay         on;

	

	log_subrequest      on;

	

	reset_timedout_connection on;

	

	keepalive_timeout  75s;

	keepalive_requests 100;

	

	client_body_temp_path           /tmp/client-body;

	fastcgi_temp_path               /tmp/fastcgi-temp;

	proxy_temp_path                 /tmp/proxy-temp;

	ajp_temp_path                   /tmp/ajp-temp;

	

	client_header_buffer_size       1k;

	client_header_timeout           60s;

	large_client_header_buffers     4 8k;

	client_body_buffer_size         8k;

	client_body_timeout             60s;

	

	http2_max_field_size            4k;

	http2_max_header_size           16k;

	http2_max_requests              1000;

	http2_max_concurrent_streams    128;

	

	types_hash_max_size             2048;

	server_names_hash_max_size      1024;

	server_names_hash_bucket_size   64;

	map_hash_bucket_size            64;

	

	proxy_headers_hash_max_size     512;

	proxy_headers_hash_bucket_size  64;

	

	variables_hash_bucket_size      256;

	variables_hash_max_size         2048;

	

	underscores_in_headers          off;

	ignore_invalid_headers          on;

	

	limit_req_status                503;

	limit_conn_status               503;

	

	include /etc/nginx/mime.types;

	default_type text/html;

	

	# Custom headers for response

	

	server_tokens off;

	

	more_clear_headers Server;

	

	# disable warnings

	uninitialized_variable_warn off;

	

	# Additional available variables:

	# $namespace

	# $ingress_name

	# $service_name

	# $service_port

	log_format upstreaminfo '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';

	

	map $request_uri $loggable {

		

		default 1;

	}

	

	access_log /var/log/nginx/access.log upstreaminfo  if=$loggable;

	

	error_log  /var/log/nginx/error.log notice;

	

	resolver 10.43.0.10 valid=30s;

	

	# See https://www.nginx.com/blog/websocket-nginx

	map $http_upgrade $connection_upgrade {

		default          upgrade;

		

		# See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

		''               '';

		

	}

	

	# Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.

	# If no such header is provided, it can provide a random value.

	map $http_x_request_id $req_id {

		default   $http_x_request_id;

		

		""        $request_id;

		

	}

	

	# Create a variable that contains the literal $ character.

	# This works because the geo module will not resolve variables.

	geo $literal_dollar {

		default "$";

	}

	

	server_name_in_redirect off;

	port_in_redirect        off;

	

	ssl_protocols TLSv1.2 TLSv1.3;

	

	ssl_early_data off;

	

	# turn on session caching to drastically improve performance

	

	ssl_session_cache shared:SSL:10m;

	ssl_session_timeout 10m;

	

	# allow configuring ssl session tickets

	ssl_session_tickets off;

	

	# slightly reduce the time-to-first-byte

	ssl_buffer_size 4k;

	

	# allow configuring custom ssl ciphers

	ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';

	ssl_prefer_server_ciphers on;

	

	ssl_ecdh_curve auto;

	

	# PEM sha: 679b4302df1fcd1d8fb5bc5099ea16c4694c8f0e

	ssl_certificate     /etc/ingress-controller/ssl/default-fake-certificate.pem;

	ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;

	

	proxy_ssl_session_reuse on;

	

	upstream upstream_balancer {

		### Attention!!!

		#

		# We no longer create "upstream" section for every backend.

		# Backends are handled dynamically using Lua. If you would like to debug

		# and see what backends ingress-nginx has in its memory you can

		# install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.

		# Once you have the plugin you can use "kubectl ingress-nginx backends" command to

		# inspect current backends.

		#

		###

		

		server 0.0.0.1; # placeholder

		

		balancer_by_lua_block {

			balancer.balance()

		}

		

		keepalive 320;

		

		keepalive_timeout  60s;

		keepalive_requests 10000;

		

	}

	

	# Cache for internal auth checks

	proxy_cache_path /tmp/nginx-cache-auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=30m use_temp_path=off;

	

	# Global filters

	

	## start server _

	server {

		server_name _ ;

		

		listen 80 default_server reuseport backlog=4096 ;

		listen [::]:80 default_server reuseport backlog=4096 ;

		listen 443 default_server reuseport backlog=4096 ssl http2 ;

		listen [::]:443 default_server reuseport backlog=4096 ssl http2 ;

		

		set $proxy_upstream_name "-";

		

		ssl_reject_handshake off;

		

		ssl_certificate_by_lua_block {

			certificate.call()

		}

		

		location / {

			

			set $namespace      "";

			set $ingress_name   "";

			set $service_name   "";

			set $service_port   "";

			set $location_path  "";

			set $global_rate_limit_exceeding n;

			

			rewrite_by_lua_block {

				lua_ingress.rewrite({

					force_ssl_redirect = false,

					ssl_redirect = false,

					force_no_ssl_redirect = false,

					preserve_trailing_slash = false,

					use_port_in_redirects = false,

					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },

				})

				balancer.rewrite()

				plugins.run()

			}

			

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any

			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`

			# other authentication method such as basic auth or external auth useless - all requests will be allowed.

			#access_by_lua_block {

			#}

			

			header_filter_by_lua_block {

				lua_ingress.header()

				plugins.run()

			}

			

			body_filter_by_lua_block {

				plugins.run()

			}

			

			log_by_lua_block {

				balancer.log()

				

				monitor.call()

				

				plugins.run()

			}

			

			access_log off;

			

			port_in_redirect off;

			

			set $balancer_ewma_score -1;

			set $proxy_upstream_name "upstream-default-backend";

			set $proxy_host          $proxy_upstream_name;

			set $pass_access_scheme  $scheme;

			

			set $pass_server_port    $server_port;

			

			set $best_http_host      $http_host;

			set $pass_port           $pass_server_port;

			

			set $proxy_alternative_upstream_name "";

			

			client_max_body_size                    1m;

			

			proxy_set_header Host                   $best_http_host;

			

			# Pass the extracted client certificate to the backend

			

			# Allow websocket connections

			proxy_set_header                        Upgrade           $http_upgrade;

			

			proxy_set_header                        Connection        $connection_upgrade;

			

			proxy_set_header X-Request-ID           $req_id;

			proxy_set_header X-Real-IP              $remote_addr;

			

			proxy_set_header X-Forwarded-For        $remote_addr;

			

			proxy_set_header X-Forwarded-Host       $best_http_host;

			proxy_set_header X-Forwarded-Port       $pass_port;

			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			

			proxy_set_header X-Scheme               $pass_access_scheme;

			

			# Pass the original X-Forwarded-For

			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			

			# mitigate HTTPoxy Vulnerability

			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

			proxy_set_header Proxy                  "";

			

			# Custom headers to proxied server

			

			proxy_connect_timeout                   5s;

			proxy_send_timeout                      60s;

			proxy_read_timeout                      60s;

			

			proxy_buffering                         off;

			proxy_buffer_size                       4k;

			proxy_buffers                           4 4k;

			

			proxy_max_temp_file_size                1024m;

			

			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			

			proxy_cookie_domain                     off;

			proxy_cookie_path                       off;

			

			# In case of errors try the next upstream server before returning an error

			proxy_next_upstream                     error timeout;

			proxy_next_upstream_timeout             0;

			proxy_next_upstream_tries               3;

			

			proxy_pass http://upstream_balancer;

			

			proxy_redirect                          off;

			

		}

		

		# health checks in cloud providers require the use of port 80

		location /healthz {

			

			access_log off;

			return 200;

		}

		

		# this is required to avoid error if nginx is being monitored

		# with an external software (like sysdig)

		location /nginx_status {

			

			allow 127.0.0.1;

			

			allow ::1;

			

			deny all;

			

			access_log off;

			stub_status on;

		}

		

	}

	## end server _

	

	## start server hitlocalk3s.com

	server {

		server_name hitlocalk3s.com ;

		

		listen 80  ;

		listen [::]:80  ;

		listen 443  ssl http2 ;

		listen [::]:443  ssl http2 ;

		

		set $proxy_upstream_name "-";

		

		ssl_certificate_by_lua_block {

			certificate.call()

		}

		

		location /document/ {

			

			set $namespace      "ingress";

			set $ingress_name   "nginx-ingress";

			set $service_name   "nginx-document";

			set $service_port   "80";

			set $location_path  "/document";

			set $global_rate_limit_exceeding n;

			

			rewrite_by_lua_block {

				lua_ingress.rewrite({

					force_ssl_redirect = false,

					ssl_redirect = true,

					force_no_ssl_redirect = false,

					preserve_trailing_slash = false,

					use_port_in_redirects = false,

					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },

				})

				balancer.rewrite()

				plugins.run()

			}

			

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any

			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`

			# other authentication method such as basic auth or external auth useless - all requests will be allowed.

			#access_by_lua_block {

			#}

			

			header_filter_by_lua_block {

				lua_ingress.header()

				plugins.run()

			}

			

			body_filter_by_lua_block {

				plugins.run()

			}

			

			log_by_lua_block {

				balancer.log()

				

				monitor.call()

				

				plugins.run()

			}

			

			port_in_redirect off;

			

			set $balancer_ewma_score -1;

			set $proxy_upstream_name "ingress-nginx-document-80";

			set $proxy_host          $proxy_upstream_name;

			set $pass_access_scheme  $scheme;

			

			set $pass_server_port    $server_port;

			

			set $best_http_host      $http_host;

			set $pass_port           $pass_server_port;

			

			set $proxy_alternative_upstream_name "";

			

			client_max_body_size                    1m;

			

			proxy_set_header Host                   $best_http_host;

			

			# Pass the extracted client certificate to the backend

			

			# Allow websocket connections

			proxy_set_header                        Upgrade           $http_upgrade;

			

			proxy_set_header                        Connection        $connection_upgrade;

			

			proxy_set_header X-Request-ID           $req_id;

			proxy_set_header X-Real-IP              $remote_addr;

			

			proxy_set_header X-Forwarded-For        $remote_addr;

			

			proxy_set_header X-Forwarded-Host       $best_http_host;

			proxy_set_header X-Forwarded-Port       $pass_port;

			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			

			proxy_set_header X-Scheme               $pass_access_scheme;

			

			# Pass the original X-Forwarded-For

			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			

			# mitigate HTTPoxy Vulnerability

			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

			proxy_set_header Proxy                  "";

			

			# Custom headers to proxied server

			

			proxy_connect_timeout                   5s;

			proxy_send_timeout                      60s;

			proxy_read_timeout                      60s;

			

			proxy_buffering                         off;

			proxy_buffer_size                       4k;

			proxy_buffers                           4 4k;

			

			proxy_max_temp_file_size                1024m;

			

			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			

			proxy_cookie_domain                     off;

			proxy_cookie_path                       off;

			

			# In case of errors try the next upstream server before returning an error

			proxy_next_upstream                     error timeout;

			proxy_next_upstream_timeout             0;

			proxy_next_upstream_tries               3;

			

			proxy_pass http://upstream_balancer;

			

			proxy_redirect                          off;

			

		}

		

		location = /document {

			

			set $namespace      "ingress";

			set $ingress_name   "nginx-ingress";

			set $service_name   "nginx-document";

			set $service_port   "80";

			set $location_path  "/document";

			set $global_rate_limit_exceeding n;

			

			rewrite_by_lua_block {

				lua_ingress.rewrite({

					force_ssl_redirect = false,

					ssl_redirect = true,

					force_no_ssl_redirect = false,

					preserve_trailing_slash = false,

					use_port_in_redirects = false,

					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },

				})

				balancer.rewrite()

				plugins.run()

			}

			

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any

			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`

			# other authentication method such as basic auth or external auth useless - all requests will be allowed.

			#access_by_lua_block {

			#}

			

			header_filter_by_lua_block {

				lua_ingress.header()

				plugins.run()

			}

			

			body_filter_by_lua_block {

				plugins.run()

			}

			

			log_by_lua_block {

				balancer.log()

				

				monitor.call()

				

				plugins.run()

			}

			

			port_in_redirect off;

			

			set $balancer_ewma_score -1;

			set $proxy_upstream_name "ingress-nginx-document-80";

			set $proxy_host          $proxy_upstream_name;

			set $pass_access_scheme  $scheme;

			

			set $pass_server_port    $server_port;

			

			set $best_http_host      $http_host;

			set $pass_port           $pass_server_port;

			

			set $proxy_alternative_upstream_name "";

			

			client_max_body_size                    1m;

			

			proxy_set_header Host                   $best_http_host;

			

			# Pass the extracted client certificate to the backend

			

			# Allow websocket connections

			proxy_set_header                        Upgrade           $http_upgrade;

			

			proxy_set_header                        Connection        $connection_upgrade;

			

			proxy_set_header X-Request-ID           $req_id;

			proxy_set_header X-Real-IP              $remote_addr;

			

			proxy_set_header X-Forwarded-For        $remote_addr;

			

			proxy_set_header X-Forwarded-Host       $best_http_host;

			proxy_set_header X-Forwarded-Port       $pass_port;

			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			

			proxy_set_header X-Scheme               $pass_access_scheme;

			

			# Pass the original X-Forwarded-For

			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			

			# mitigate HTTPoxy Vulnerability

			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

			proxy_set_header Proxy                  "";

			

			# Custom headers to proxied server

			

			proxy_connect_timeout                   5s;

			proxy_send_timeout                      60s;

			proxy_read_timeout                      60s;

			

			proxy_buffering                         off;

			proxy_buffer_size                       4k;

			proxy_buffers                           4 4k;

			

			proxy_max_temp_file_size                1024m;

			

			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			

			proxy_cookie_domain                     off;

			proxy_cookie_path                       off;

			

			# In case of errors try the next upstream server before returning an error

			proxy_next_upstream                     error timeout;

			proxy_next_upstream_timeout             0;

			proxy_next_upstream_tries               3;

			

			proxy_pass http://upstream_balancer;

			

			proxy_redirect                          off;

			

		}

		

		location /video/ {

			

			set $namespace      "ingress";

			set $ingress_name   "nginx-ingress";

			set $service_name   "nginx-video";

			set $service_port   "80";

			set $location_path  "/video";

			set $global_rate_limit_exceeding n;

			

			rewrite_by_lua_block {

				lua_ingress.rewrite({

					force_ssl_redirect = false,

					ssl_redirect = true,

					force_no_ssl_redirect = false,

					preserve_trailing_slash = false,

					use_port_in_redirects = false,

					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },

				})

				balancer.rewrite()

				plugins.run()

			}

			

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any

			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`

			# other authentication method such as basic auth or external auth useless - all requests will be allowed.

			#access_by_lua_block {

			#}

			

			header_filter_by_lua_block {

				lua_ingress.header()

				plugins.run()

			}

			

			body_filter_by_lua_block {

				plugins.run()

			}

			

			log_by_lua_block {

				balancer.log()

				

				monitor.call()

				

				plugins.run()

			}

			

			port_in_redirect off;

			

			set $balancer_ewma_score -1;

			set $proxy_upstream_name "ingress-nginx-video-80";

			set $proxy_host          $proxy_upstream_name;

			set $pass_access_scheme  $scheme;

			

			set $pass_server_port    $server_port;

			

			set $best_http_host      $http_host;

			set $pass_port           $pass_server_port;

			

			set $proxy_alternative_upstream_name "";

			

			client_max_body_size                    1m;

			

			proxy_set_header Host                   $best_http_host;

			

			# Pass the extracted client certificate to the backend

			

			# Allow websocket connections

			proxy_set_header                        Upgrade           $http_upgrade;

			

			proxy_set_header                        Connection        $connection_upgrade;

			

			proxy_set_header X-Request-ID           $req_id;

			proxy_set_header X-Real-IP              $remote_addr;

			

			proxy_set_header X-Forwarded-For        $remote_addr;

			

			proxy_set_header X-Forwarded-Host       $best_http_host;

			proxy_set_header X-Forwarded-Port       $pass_port;

			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			

			proxy_set_header X-Scheme               $pass_access_scheme;

			

			# Pass the original X-Forwarded-For

			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			

			# mitigate HTTPoxy Vulnerability

			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

			proxy_set_header Proxy                  "";

			

			# Custom headers to proxied server

			

			proxy_connect_timeout                   5s;

			proxy_send_timeout                      60s;

			proxy_read_timeout                      60s;

			

			proxy_buffering                         off;

			proxy_buffer_size                       4k;

			proxy_buffers                           4 4k;

			

			proxy_max_temp_file_size                1024m;

			

			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			

			proxy_cookie_domain                     off;

			proxy_cookie_path                       off;

			

			# In case of errors try the next upstream server before returning an error

			proxy_next_upstream                     error timeout;

			proxy_next_upstream_timeout             0;

			proxy_next_upstream_tries               3;

			

			proxy_pass http://upstream_balancer;

			

			proxy_redirect                          off;

			

		}

		

		location = /video {

			

			set $namespace      "ingress";

			set $ingress_name   "nginx-ingress";

			set $service_name   "nginx-video";

			set $service_port   "80";

			set $location_path  "/video";

			set $global_rate_limit_exceeding n;

			

			rewrite_by_lua_block {

				lua_ingress.rewrite({

					force_ssl_redirect = false,

					ssl_redirect = true,

					force_no_ssl_redirect = false,

					preserve_trailing_slash = false,

					use_port_in_redirects = false,

					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },

				})

				balancer.rewrite()

				plugins.run()

			}

			

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any

			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`

			# other authentication method such as basic auth or external auth useless - all requests will be allowed.

			#access_by_lua_block {

			#}

			

			header_filter_by_lua_block {

				lua_ingress.header()

				plugins.run()

			}

			

			body_filter_by_lua_block {

				plugins.run()

			}

			

			log_by_lua_block {

				balancer.log()

				

				monitor.call()

				

				plugins.run()

			}

			

			port_in_redirect off;

			

			set $balancer_ewma_score -1;

			set $proxy_upstream_name "ingress-nginx-video-80";

			set $proxy_host          $proxy_upstream_name;

			set $pass_access_scheme  $scheme;

			

			set $pass_server_port    $server_port;

			

			set $best_http_host      $http_host;

			set $pass_port           $pass_server_port;

			

			set $proxy_alternative_upstream_name "";

			

			client_max_body_size                    1m;

			

			proxy_set_header Host                   $best_http_host;

			

			# Pass the extracted client certificate to the backend

			

			# Allow websocket connections

			proxy_set_header                        Upgrade           $http_upgrade;

			

			proxy_set_header                        Connection        $connection_upgrade;

			

			proxy_set_header X-Request-ID           $req_id;

			proxy_set_header X-Real-IP              $remote_addr;

			

			proxy_set_header X-Forwarded-For        $remote_addr;

			

			proxy_set_header X-Forwarded-Host       $best_http_host;

			proxy_set_header X-Forwarded-Port       $pass_port;

			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			

			proxy_set_header X-Scheme               $pass_access_scheme;

			

			# Pass the original X-Forwarded-For

			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			

			# mitigate HTTPoxy Vulnerability

			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

			proxy_set_header Proxy                  "";

			

			# Custom headers to proxied server

			

			proxy_connect_timeout                   5s;

			proxy_send_timeout                      60s;

			proxy_read_timeout                      60s;

			

			proxy_buffering                         off;

			proxy_buffer_size                       4k;

			proxy_buffers                           4 4k;

			

			proxy_max_temp_file_size                1024m;

			

			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			

			proxy_cookie_domain                     off;

			proxy_cookie_path                       off;

			

			# In case of errors try the next upstream server before returning an error

			proxy_next_upstream                     error timeout;

			proxy_next_upstream_timeout             0;

			proxy_next_upstream_tries               3;

			

			proxy_pass http://upstream_balancer;

			

			proxy_redirect                          off;

			

		}

		

		location / {

			

			set $namespace      "ingress";

			set $ingress_name   "nginx-ingress";

			set $service_name   "nginx-deployment";

			set $service_port   "80";

			set $location_path  "/";

			set $global_rate_limit_exceeding n;

			

			rewrite_by_lua_block {

				lua_ingress.rewrite({

					force_ssl_redirect = false,

					ssl_redirect = true,

					force_no_ssl_redirect = false,

					preserve_trailing_slash = false,

					use_port_in_redirects = false,

					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },

				})

				balancer.rewrite()

				plugins.run()

			}

			

			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any

			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`

			# other authentication method such as basic auth or external auth useless - all requests will be allowed.

			#access_by_lua_block {

			#}

			

			header_filter_by_lua_block {

				lua_ingress.header()

				plugins.run()

			}

			

			body_filter_by_lua_block {

				plugins.run()

			}

			

			log_by_lua_block {

				balancer.log()

				

				monitor.call()

				

				plugins.run()

			}

			

			port_in_redirect off;

			

			set $balancer_ewma_score -1;

			set $proxy_upstream_name "ingress-nginx-deployment-80";

			set $proxy_host          $proxy_upstream_name;

			set $pass_access_scheme  $scheme;

			

			set $pass_server_port    $server_port;

			

			set $best_http_host      $http_host;

			set $pass_port           $pass_server_port;

			

			set $proxy_alternative_upstream_name "";

			

			client_max_body_size                    1m;

			

			proxy_set_header Host                   $best_http_host;

			

			# Pass the extracted client certificate to the backend

			

			# Allow websocket connections

			proxy_set_header                        Upgrade           $http_upgrade;

			

			proxy_set_header                        Connection        $connection_upgrade;

			

			proxy_set_header X-Request-ID           $req_id;

			proxy_set_header X-Real-IP              $remote_addr;

			

			proxy_set_header X-Forwarded-For        $remote_addr;

			

			proxy_set_header X-Forwarded-Host       $best_http_host;

			proxy_set_header X-Forwarded-Port       $pass_port;

			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

			

			proxy_set_header X-Scheme               $pass_access_scheme;

			

			# Pass the original X-Forwarded-For

			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

			

			# mitigate HTTPoxy Vulnerability

			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

			proxy_set_header Proxy                  "";

			

			# Custom headers to proxied server

			

			proxy_connect_timeout                   5s;

			proxy_send_timeout                      60s;

			proxy_read_timeout                      60s;

			

			proxy_buffering                         off;

			proxy_buffer_size                       4k;

			proxy_buffers                           4 4k;

			

			proxy_max_temp_file_size                1024m;

			

			proxy_request_buffering                 on;

			proxy_http_version                      1.1;

			

			proxy_cookie_domain                     off;

			proxy_cookie_path                       off;

			

			# In case of errors try the next upstream server before returning an error

			proxy_next_upstream                     error timeout;

			proxy_next_upstream_timeout             0;

			proxy_next_upstream_tries               3;

			

			proxy_pass http://upstream_balancer;

			

			proxy_redirect                          off;

			

		}

		

	}

	## end server hitlocalk3s.com

	

	# backend for when default-backend-service is not configured or it does not have endpoints

	server {

		listen 8181 default_server reuseport backlog=4096;

		listen [::]:8181 default_server reuseport backlog=4096;

		set $proxy_upstream_name "internal";

		

		access_log off;

		

		location / {

			return 404;

		}

	}

	

	# default server, used for NGINX healthcheck and access to nginx stats

	server {

		listen 127.0.0.1:10246;

		set $proxy_upstream_name "internal";

		

		keepalive_timeout 0;

		gzip off;

		

		access_log off;

		

		location /healthz {

			return 200;

		}

		

		location /is-dynamic-lb-initialized {

			content_by_lua_block {

				local configuration = require("configuration")

				local backend_data = configuration.get_backends_data()

				if not backend_data then

				ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)

				return

				end

				

				ngx.say("OK")

				ngx.exit(ngx.HTTP_OK)

			}

		}

		

		location /nginx_status {

			stub_status on;

		}

		

		location /configuration {

			client_max_body_size                    21M;

			client_body_buffer_size                 21M;

			proxy_buffering                         off;

			

			content_by_lua_block {

				configuration.call()

			}

		}

		

		location / {

			content_by_lua_block {

				ngx.exit(ngx.HTTP_NOT_FOUND)

			}

		}

	}

}



stream {

	lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;";

	

	lua_shared_dict tcp_udp_configuration_data 5M;

	

	init_by_lua_block {

		collectgarbage("collect")

		

		-- init modules

		local ok, res

		

		ok, res = pcall(require, "configuration")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		configuration = res

		end

		

		ok, res = pcall(require, "tcp_udp_configuration")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		tcp_udp_configuration = res

		tcp_udp_configuration.prohibited_localhost_port = '10246'

		

		end

		

		ok, res = pcall(require, "tcp_udp_balancer")

		if not ok then

		error("require failed: " .. tostring(res))

		else

		tcp_udp_balancer = res

		end

	}

	

	init_worker_by_lua_block {

		tcp_udp_balancer.init_worker()

	}

	

	lua_add_variable $proxy_upstream_name;

	

	log_format log_stream '[$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time';

	

	access_log /var/log/nginx/access.log log_stream ;

	

	error_log  /var/log/nginx/error.log notice;

	

	upstream upstream_balancer {

		server 0.0.0.1:1234; # placeholder

		

		balancer_by_lua_block {

			tcp_udp_balancer.balance()

		}

	}

	

	server {

		listen 127.0.0.1:10247;

		

		access_log off;

		

		content_by_lua_block {

			tcp_udp_configuration.call()

		}

	}

	

	# TCP services

	

	# UDP services

	

	# Stream Snippets

	

}





# configuration file /etc/nginx/mime.types:



types {

    text/html                                        html htm shtml;

    text/css                                         css;

    text/xml                                         xml;

    image/gif                                        gif;

    image/jpeg                                       jpeg jpg;

    application/javascript                           js;

    application/atom+xml                             atom;

    application/rss+xml                              rss;



    text/mathml                                      mml;

    text/plain                                       txt;

    text/vnd.sun.j2me.app-descriptor                 jad;

    text/vnd.wap.wml                                 wml;

    text/x-component                                 htc;



    image/avif                                       avif;

    image/png                                        png;

    image/svg+xml                                    svg svgz;

    image/tiff                                       tif tiff;

    image/vnd.wap.wbmp                               wbmp;

    image/webp                                       webp;

    image/x-icon                                     ico;

    image/x-jng                                      jng;

    image/x-ms-bmp                                   bmp;



    font/woff                                        woff;

    font/woff2                                       woff2;



    application/java-archive                         jar war ear;

    application/json                                 json;

    application/mac-binhex40                         hqx;

    application/msword                               doc;

    application/pdf                                  pdf;

    application/postscript                           ps eps ai;

    application/rtf                                  rtf;

    application/vnd.apple.mpegurl                    m3u8;

    application/vnd.google-earth.kml+xml             kml;

    application/vnd.google-earth.kmz                 kmz;

    application/vnd.ms-excel                         xls;

    application/vnd.ms-fontobject                    eot;

    application/vnd.ms-powerpoint                    ppt;

    application/vnd.oasis.opendocument.graphics      odg;

    application/vnd.oasis.opendocument.presentation  odp;

    application/vnd.oasis.opendocument.spreadsheet   ods;

    application/vnd.oasis.opendocument.text          odt;

    application/vnd.openxmlformats-officedocument.presentationml.presentation

                                                     pptx;

    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

                                                     xlsx;

    application/vnd.openxmlformats-officedocument.wordprocessingml.document

                                                     docx;

    application/vnd.wap.wmlc                         wmlc;

    application/wasm                                 wasm;

    application/x-7z-compressed                      7z;

    application/x-cocoa                              cco;

    application/x-java-archive-diff                  jardiff;

    application/x-java-jnlp-file                     jnlp;

    application/x-makeself                           run;

    application/x-perl                               pl pm;

    application/x-pilot                              prc pdb;

    application/x-rar-compressed                     rar;

    application/x-redhat-package-manager             rpm;

    application/x-sea                                sea;

    application/x-shockwave-flash                    swf;

    application/x-stuffit                            sit;

    application/x-tcl                                tcl tk;

    application/x-x509-ca-cert                       der pem crt;

    application/x-xpinstall                          xpi;

    application/xhtml+xml                            xhtml;

    application/xspf+xml                             xspf;

    application/zip                                  zip;



    application/octet-stream                         bin exe dll;

    application/octet-stream                         deb;

    application/octet-stream                         dmg;

    application/octet-stream                         iso img;

    application/octet-stream                         msi msp msm;



    audio/midi                                       mid midi kar;

    audio/mpeg                                       mp3;

    audio/ogg                                        ogg;

    audio/x-m4a                                      m4a;

    audio/x-realaudio                                ra;



    video/3gpp                                       3gpp 3gp;

    video/mp2t                                       ts;

    video/mp4                                        mp4;

    video/mpeg                                       mpeg mpg;

    video/quicktime                                  mov;

    video/webm                                       webm;

    video/x-flv                                      flv;

    video/x-m4v                                      m4v;

    video/x-mng                                      mng;

    video/x-ms-asf                                   asx asf;

    video/x-ms-wmv                                   wmv;

    video/x-msvideo                                  avi;

}

0 replies

jasonwilliams14 · 2022-03-20T22:56:36Z

jasonwilliams14
Mar 20, 2022
Collaborator

@rthamrin Unfortunately, you are using a different NGINX Ingress controller project.
The Ingress you are using is this project:

https://kubernetes.github.io/ingress-nginx/deploy/

This github repo is for NGINXINC project.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load balancing to a backed websocket service using "least_conn" #1960

{{title}}

Replies: 7 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Load balancing to a backed websocket service using "least_conn" #1960

vkbodanki Sep 13, 2021

Replies: 7 comments · 4 replies

jasonwilliams14 Sep 29, 2021 Collaborator

vkbodanki Sep 29, 2021 Author

jasonwilliams14 Oct 1, 2021 Collaborator

vkbodanki Oct 1, 2021 Author

jasonwilliams14 Oct 4, 2021 Collaborator

vkbodanki Oct 5, 2021 Author

jasonwilliams14 Oct 5, 2021 Collaborator

vkbodanki Oct 9, 2021 Author

jasonwilliams14 Oct 15, 2021 Collaborator

rthamrin Mar 19, 2022

jasonwilliams14 Mar 20, 2022 Collaborator

vkbodanki
Sep 13, 2021

Replies: 7 comments 4 replies

jasonwilliams14
Sep 29, 2021
Collaborator

vkbodanki Sep 29, 2021
Author

jasonwilliams14
Oct 1, 2021
Collaborator

vkbodanki Oct 1, 2021
Author

jasonwilliams14
Oct 4, 2021
Collaborator

vkbodanki Oct 5, 2021
Author

jasonwilliams14
Oct 5, 2021
Collaborator

vkbodanki Oct 9, 2021
Author

jasonwilliams14
Oct 15, 2021
Collaborator

rthamrin
Mar 19, 2022

jasonwilliams14
Mar 20, 2022
Collaborator