Replies: 4 comments 12 replies
-
|
@xcfw Hello there. Have you injected the istio sidecar into the NGINX Ingress controller pod? can you provdie the output of a describe on the NGINX ingress controller pod? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
I added service back, and traffic now goes not through virtualservice, but service: > k get svc -n testing
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
counter ClusterIP 10.44.11.227 <none> 80/TCP 49d
web ClusterIP 10.44.19.178 <none> 80/TCP 12hDespite upstreams:
- name: web
service: web
port: 80
use-cluster-ip: true action:
proxy:
upstream: web
requestHeaders:
set:
- name: Host
value: web.staging.svc.cluster.localwhen k8s service is enabled with this: # Source: example/charts/web/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: web
labels:
helm.sh/chart: web-0.1.0
app.kubernetes.io/name: web
app.kubernetes.io/instance: powr
app.kubernetes.io/version: "1.0.0"
app.kubernetes.io/managed-by: Helm
spec:
ports:
- name: http-web
port: 80
targetPort: 3000
selector:
app.kubernetes.io/name: web
app.kubernetes.io/instance: powr
type: ClusterIPI get bad results with curl: > echo ":::canary:::"; for i in $(seq 1 10); do sleep 1s;
curl -b "userAffinity=canary" https://www.example-website.com/version.html; echo "\n";
done; echo "\n\n:::main:::"; for i in $(seq 1 10); do sleep 1s;
curl -b "userAffinity=main" https://www.example-website.com/version.html; echo "\n";
done;
:::canary:::
v1
v2
v1
v2
v2
v2
v1
v1
v1
v2
:::main:::
v1
v2
v2
v1
v1
v2
v1
v2
v2
v1And because this behaviour is similar if I delete istio, and use just k8s service without service mesh at all, and because logs show that traffic is not passed through envoy with k8s service enabled I thought about ways to force nginx to use envoy and send traffic via istio's virtualservice: click here# Source: example/charts/web/templates/istionet.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: web
labels:
helm.sh/chart: web-0.1.0
app.kubernetes.io/name: web
app.kubernetes.io/instance: example
app.kubernetes.io/version: "1.0.0"
app.kubernetes.io/managed-by: Helm
spec:
hosts:
- web
http:
- match:
- headers:
Cookie:
regex: .*userAffinity=main.*
route:
- destination:
host: web
subset: main
- match:
- headers:
Cookie:
regex: .*userAffinity=canary.*
route:
- destination:
host: web
subset: canary
- route:
- destination:
host: web
subset: main
weight: 99
- destination:
host: web
subset: canary
weight: 1
headers:
response:
add:
Set-Cookie: userAffinity=canary; Max-Age=3600000 |
Beta Was this translation helpful? Give feedback.
-
|
here's the updated raw nginx vhost.conf when service is enabled: vhost.confkubectl exec -it -n nginx deploy/nginx-ingress-controller -- cat /etc/nginx/conf.d/vs_testing_web.confupstream vs_testing_web_web {
zone vs_testing_web_web 256k;
random two least_conn;
server 10.44.19.178:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
listen [::]:80;
server_name www.example-website.com;
set $resource_type "virtualserver";
set $resource_name "web";
set $resource_namespace "testing";
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate /etc/nginx/secrets/wildcard;
ssl_certificate_key /etc/nginx/secrets/wildcard;
if ($scheme = 'http') {
return 301 https://$host$request_uri;
}
server_tokens "on";
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
types_hash_bucket_size 64;
# Proxy tuning
proxy_buffer_size 256k;
proxy_buffers 4 512k;
proxy_busy_buffers_size 512k;
proxy_connect_timeout 180s;
proxy_read_timeout 180s;
proxy_send_timeout 180s;
# SSL Tweaks
resolver 1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 valid=60s;
resolver_timeout 2s;
ssl_buffer_size 4k;
ssl_protocols TLSv1.3;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_session_timeout 1d;
# ssl_prefer_server_ciphers off;
# ssl_stapling on;
# ssl_stapling_verify on;
# Gzip compression tweaks
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 9;
gzip_http_version 1.1;
gzip_proxied any;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss application/atom+xml image/svg+xml text/javascript;
gzip_vary on;
# CSP and HTTP tweaks
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Origin-Instance "gke";
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# CSP and some other headers added here but omited
# Top level rules
location = / {
proxy_pass https://www.example-langing.com/homepage;
add_header X-Origin-Instance "gke-next-static";
}
# Plugin landing pages rules
location ~ /(.*)-website-app(.*)$ {
proxy_pass https://www.example-langing.com/$1$2;
add_header X-Origin-Instance "gke-next-static";
}
location ~ "/settings" {
set $service "web";
add_header Content-Security-Policy "" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header X-Origin-Instance "gke" always;
proxy_hide_header X-Frame-Options;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 200m;
proxy_buffering on;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "web.testing.svc.cluster.local";
proxy_pass http://vs_testing_web_web;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location / {
set $service "web";
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 200m;
proxy_buffering on;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "web.testing.svc.cluster.local";
proxy_pass http://vs_testing_web_web;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
} |
Beta Was this translation helpful? Give feedback.
-
That is expected behavior. We enumerate the kubernetes service to find the endpoint IPs and populate that in the I am not an istio expert and would have to research more on the Hope explains a bit better. |
Beta Was this translation helpful? Give feedback.
-
|
any news or ideas how could I debug? |
Beta Was this translation helpful? Give feedback.
-
|
Ingress watches for EndpointSlices and Services which are under the K8s Service API endpoint. |
Beta Was this translation helpful? Give feedback.
-
|
https://docs.nginx.com/nginx-ingress-controller/tutorials/nginx-ingress-istio/ |
Beta Was this translation helpful? Give feedback.
-
|
I followed that tutorial precisely, and still no luck, if I bring back service, virtualservice doesn't work. More details here |
Beta Was this translation helpful? Give feedback.
-
|
@xcfw are you still using the istio |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure what Istio changed that is causing an incompatibility with the K8s Service object which would break any relationship for an ingress controller. The historic implementation from object to object is: |
Beta Was this translation helpful? Give feedback.
-
|
closing as we do not support VS at the moment |
Beta Was this translation helpful? Give feedback.
-
Hello everyone! I'm trying to configure nginx to work with istio's virtualservice to achieve cookie-based canary deployment routing and whenever I deploy manifests below I get 502 error in logs of ingress and nothing in istio-sidecar logs. My guess is that traffic from ingress is not sent via sidecar and therefore can't reach virtualservice.
I know that config is very specific, and still kindly asking for a help with figuring out what part of it is wrong and hopeful to get a direction where to search for a solution.
I use
nginx/nginx-ingress:3.3.2ingress installed via helm and google'sgcr.io/gke-release/asm/proxyv2:1.16.7-asm.10for the sidecar.Here below my manifests:
How deployment and Virtualservers are configured
Main website helm rendered manifests:
How nginx deployment is configured
NGINX with helm resulting yamls:
Beta Was this translation helpful? Give feedback.
All reactions