hello-k8s: ECONNREFUSED when trying to access redis state-store from node-app

[filenda]

Expected Behavior

Node app should save order id 42 on redis

Actual Behavior

request to http://localhost:3500/v1.0/state/statestore/order failed, reason: connect ECONNREFUSED 127.0.0.1:3500

Steps to Reproduce the Problem

follow along with the tutorial

[filenda]

tried redis-master.default.svc.cluster.local:6379 and redis-master:6379 as rediHost value on redis.yaml but wasn't happy either way

[wcs1only]

In the error you are getting, this indicates that the node app is failing to connect to daprd. The daprd logs should be able to tell us more. I take it this is for the hello-kubernetes example? In which case, can you run kubectl logs --selector=app=node -c daprd --tail=-1 And paste the results?

Also, what method did you use to install redis? Did you use helm as suggested in the statestore pubsub guide?
https://docs.dapr.io/getting-started/configure-state-pubsub/#tabs-3-kubernetes

[amsoedal]

@wcs1only I'm reproducing a similar issue... in my case they pythonapp can't talk to the node app. This is what I see:

>  kubectl logs --selector=app=node -c daprd --tail=-1
error: container daprd is not valid for pod nodeapp-84867c5867-54wnr

I also see this when looking at the logs for the sidecar injector:

>  kubectl logs -l app=dapr-sidecar-injector -n dapr-system
time="2021-06-07T15:33:32.38244903Z" level=info msg="Healthz server is listening on :8080" instance=dapr-sidecar-injector-869c7966bd-zc62v scope=dapr.injector type=log ver=unknown
time="2021-06-07T15:33:32.442138926Z" level=info msg="Sidecar injector is listening on :4000, patching Dapr-enabled pods" instance=dapr-sidecar-injector-869c7966bd-zc62v scope=dapr.injector type=log ver=unknown
2021/06/07 15:37:09 http: TLS handshake error from 10.244.1.1:62250: EOF
2021/06/07 15:37:09 http: TLS handshake error from 10.240.0.5:38193: EOF
2021/06/07 15:37:55 http: TLS handshake error from 10.244.1.1:15732: EOF
2021/06/07 15:39:26 http: TLS handshake error from 10.240.0.5:50362: EOF
2021/06/07 15:41:27 http: TLS handshake error from 10.240.0.5:13316: EOF
2021/06/07 15:46:37 http: TLS handshake error from 10.244.1.1:48388: EOF
2021/06/07 15:52:24 http: TLS handshake error from 10.244.1.1:23716: EOF
2021/06/07 15:53:18 http: TLS handshake error from 10.244.1.1:55153: EOF

Any ideas on what this could be?

[wcs1only]

@amsoedal So in your case, it looks like the sidecar injector is failing TLS handshakes with the kubernetes control plane. The EOF error suggests that the peer closed the connection before the handshake could complete. This could be a lot of things, but the most likely is some kind of certificate mismatch. The effect of all this is that the Dapr sidecar container is not being injected, it looks like.

Is this a fresh kubernetes cluster with a fresh Dapr install? The first thing that I would try (provided this is not a cluster that is running production Dapr apps) is:

dapr uninstall -k
dapr init -k 
dapr status -k

[filenda]

@wcs1only

Whenever I send the command suggested I get the same response as @amsoedal

kubectl logs --selector=app=node -c daprd --tail=-1
error: container daprd is not valid for pod nodeapp-59bc6c5c4d-2hmjm

And if I try the command on the hello k8s tutorial:

kubectl logs --selector=app=node -c node --tail=-1

Node App listening on port 3000!
DAPR_HTTP_PORT: 3500
DAPR_GRPC_PORT: undefined
DAPR_HTTP_PORT: 3500
DAPR_GRPC_PORT: undefined
DAPR_HTTP_PORT: 3500
DAPR_GRPC_PORT: undefined
DAPR_HTTP_PORT: 3500
DAPR_GRPC_PORT: undefined
DAPR_HTTP_PORT: 3500
DAPR_GRPC_PORT: undefined

Also, if I type

kubectl logs --selector=app=python -c python --tail=-1

I get an infinite loop of :

Failed to establish a new connection: [Errno 111] Connection refused'))
HTTPConnectionPool(host='localhost', port=3500): Max retries exceeded with url: /v1.0/invoke/nodeapp/method/neworder (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f848ecc1450>: Failed to establish a new connection: [Errno 111] Connection refused'))

Even further:

kubectl logs --selector=app=node --tail=-1

Got a new order! Order ID: 1
{ FetchError: request to http://localhost:3500/v1.0/state/statestore failed, reason: connect ECONNREFUSED 127.0.0.1:3500
    at ClientRequest.<anonymous> (/app/node_modules/node-fetch/index.js:133:11)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at Socket.socketErrorListener (_http_client.js:401:9)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:73:8)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)
  name: 'FetchError',
  message: 'request to http://localhost:3500/v1.0/state/statestore failed, reason: connect ECONNREFUSED 127.0.0.1:3500',
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED' }

[wcs1only]

@filenda

Ok, that does indeed look like the sidecar is not being injected as all. Are you getting a similar TLS handshake error in the sidecar logs?

kubectl logs -l app=dapr-sidecar-injector -n dapr-system

Some other commands that might tell us something:

dapr status -k

This shows us that dapr is healthy.

kubectl describe pod -l app=node

Should tell us if the pod was injected, but didn't come up for some reason.

Also, just so I know, what flavor of Kubernetes are you using?

[filenda]

@wcs1only

kubectl logs -l app=dapr-sidecar-injector -n dapr-system
The above command give me no response

dapr status -k
NAME NAMESPACE HEALTHY STATUS REPLICAS VERSION AGE CREATED
dapr-operator dapr-lab True Running 1 1.1.0 43m 2021-06-14 09:53.24
dapr-sentry dapr-lab True Running 1 1.1.0 43m 2021-06-14 09:53.26
dapr-placement-server dapr-lab True Running 1 1.1.0 33d 2021-05-11 22:02.41
dapr-sidecar-injector dapr-lab True Running 1 1.1.0 43m 2021-06-14 09:53.29
dapr-dashboard dapr-lab True Running 1 0.6.0 43m 2021-06-14 09:53.22

and last:
kubectl describe pod -l app=node

Name:         nodeapp-59bc6c5c4d-5kwr8
Namespace:    dapr-lab
Priority:     0
Node:         ip-omited-by-me-us-west-2.compute.internal/omited-by-me
Start Time:   Mon, 14 Jun 2021 09:53:31 -0300
Labels:       app=node
              pod-template-hash=59bc6c5c4d
Annotations:  dapr.io/app-id: nodeapp
              dapr.io/app-port: 3000
              dapr.io/enabled: true
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.13.110.249
IPs:
  IP:           10.13.110.249
Controlled By:  ReplicaSet/nodeapp-59bc6c5c4d
Containers:
  node:
    Container ID:   docker://76d229a2cf1cfc3f61fbecbe93ea70c76307f85fa514cfb47608cc00f9981132
    Image:          vfilenga/dapr-lab-k8s-nodeapp:latest
    Image ID:       docker-pullable://vfilenga/dapr-lab-k8s-nodeapp@sha256:b307616abb6b60d4f98bc9eea765b664e145d605661469aab3ff39eba738377d
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 14 Jun 2021 09:54:11 -0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qq82t (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-qq82t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qq82t
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               43m   default-scheduler  Successfully assigned dapr-lab/nodeapp-59bc6c5c4d-5kwr8 to ip-10-13-108-252.us-west-2.compute.internal
  Warning  FailedCreatePodSandBox  43m   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e577c14a2b9d5a9a585d291f62f167911bbadfad54edb0d5cfcbb694a8ece97a" network for pod "nodeapp-59bc6c5c4d-5kwr8": networkPlugin cni failed to set up pod "nodeapp-59bc6c5c4d-5kwr8_dapr-lab" network: add cmd: failed to assign an IP address to container
  Normal   SandboxChanged          43m   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                 43m   kubelet            Pulling image "vfilenga/dapr-lab-k8s-nodeapp:latest"
  Normal   Pulled                  42m   kubelet            Successfully pulled image "vfilenga/dapr-lab-k8s-nodeapp:latest" in 28.973816903s
  Normal   Created                 42m   kubelet            Created container node
  Normal   Started                 42m   kubelet            Started container node

The only strange thing you might find is that it is pulling from my custom image "vfilenga/dapr-lab-k8s-nodeapp:latest"

These are the only silly changes I had to make to try and get it running (git diff output) The dappr gRPC port and HTTP port were empty, I fixed the http one with this modification, but gRPC seems to be still empty:

diff --git a/hello-kubernetes/deploy/node.yaml b/hello-kubernetes/deploy/node.yaml
index 23185a6..a32e546 100644
--- a/hello-kubernetes/deploy/node.yaml
+++ b/hello-kubernetes/deploy/node.yaml
@@ -36,7 +36,7 @@ spec:
     spec:
       containers:
       - name: node
-        image: dapriosamples/hello-k8s-node:latest
+        image: vfilenga/dapr-lab-k8s-nodeapp:latest
         ports:
         - containerPort: 3000
         imagePullPolicy: Always
diff --git a/hello-kubernetes/deploy/redis.yaml b/hello-kubernetes/deploy/redis.yaml
index de9be2a..44cf816 100644
--- a/hello-kubernetes/deploy/redis.yaml
+++ b/hello-kubernetes/deploy/redis.yaml
@@ -12,7 +12,7 @@ spec:
   # Redis password with your own Secret's name. For more information,
   # see https://docs.dapr.io/operations/components/component-secrets .
   - name: redisHost
-    value: redis-master:6379
+    value: redis-master.default.svc.cluster.local:6379
   - name: redisPassword
     secretKeyRef:
       name: redis
diff --git a/hello-kubernetes/node/app.js b/hello-kubernetes/node/app.js
index 70f6106..35d67f7 100644
--- a/hello-kubernetes/node/app.js
+++ b/hello-kubernetes/node/app.js
@@ -11,7 +11,7 @@ const app = express();
 app.use(bodyParser.json());

 // These ports are injected automatically into the container.
-const daprPort = process.env.DAPR_HTTP_PORT;
+const daprPort = process.env.DAPR_HTTP_PORT || 3500;
 const daprGRPCPort = process.env.DAPR_GRPC_PORT;

 const stateStoreName = `statestore`;
diff --git a/secretstore/deploy/node.yaml b/secretstore/deploy/node.yaml
index 38b4df5..062ebce 100644
--- a/secretstore/deploy/node.yaml
+++ b/secretstore/deploy/node.yaml
@@ -36,7 +36,7 @@ spec:
     spec:
       containers:
       - name: node
-        image: dapriosamples/secretstorenode:latest
+        image: vfilenga/dapr-lab-secretstore-nodeapp:latest
         env:
         - name: SECRET_STORE
           value: "kubernetes"
diff --git a/secretstore/node/app.js b/secretstore/node/app.js
index 8090053..2d80870 100644
--- a/secretstore/node/app.js
+++ b/secretstore/node/app.js
@@ -21,7 +21,7 @@ const secretsUrl = `http://localhost:${daprPort}/v1.0/secrets`;
 const port = 3000;

 app.get('/getsecret', (_req, res) => {
-    const url = `${secretsUrl}/${secretStoreName}/${secretName}?metadata.namespace=default`
+    const url = `${secretsUrl}/${secretStoreName}/${secretName}?metadata.namespace=dapr-lab`
     console.log("Fetching URL: %s", url)
     fetch(url)
     .then(res => res.json())

It is an EKS cluster!

[filenda]

can it be something related to the custom namespace that I created (dapr-lab) for the tutorial?

A guy from my ops team was able to reach the state store address from outside the dapr-lab namespace

[wcs1only]

It is possible, though running your own namespace should work. As for the sidecar logs you'll need to modify the command slightly, assuming you installed the dapr control plane to the dapr-lab namespace:

kubectl logs -l app=dapr-sidecar-injector -n dapr-lab

Also, FYI: This issue queue isn't checked daily, so you might find yourself waiting a bit for replies. We have a community discord that might get you some quicker response times:

https://aka.ms/dapr-discord

[amulyavarote]

Closing this because of inactivity. Please feel free to reopen it if needed.

[paulyuk]

Hi @filenda were you able to get Dapr + Redis working? I'd love to see you get to a successful place.

[paulafahmy]

I have exactly the same issue, I'm getting the error "error: container daprd is not valid for pod nodeapp-6b65576f5-sqrnp" and it also looks like the sidecar is not being injected at all
What should I do?