ACL binding rules for Workload identites is not working as expected.

[sijo13]

ACL binding rules for Workload identites is not working as expected.

---

In our infrastructure, we are using an open-source version of Nomad and Consul.

We have enforced a restriction in the naming convention of the service, i.e., it should be in the format -. This has been done so that we do not end up having the same identical service name belonging to different jobs running in different namespaces.

restriction is enforced by creating a policy for teams in consul that only allows write access to services having the prefix "namespace-"".

service_prefix "platforms-" {
  policy = "write"
}

We would like to enable WI in Nomad and Consul, and the same restriction needs to be enforced on the service name naming convention. because if we don't put the restriction, different teams can have the same service name running in different namespaces in Nomad, which would cause issues.

We tried adding expressions in the acl binding rule selector field. However, it's not working as expected.

Reproduction Steps

Please find the authmethod and binding rule configuration
Auth method:

Type:          jwt
Description:   JWT auth method for Nomad services and workloads
Config:
{
  "BoundAudiences": [
    "consul.test.io"
  ],
  "ClaimMappings": {
    "nomad_job_id": "nomad_job_id",
    "nomad_namespace": "nomad_namespace",
    "nomad_service": "nomad_service",
    "nomad_task": "nomad_task"
  },
  "JWKSURL": "http://127.0.0.1:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": [
    "RS256"
  ]
}

Binding rule:

8eba3cd0-12ca-8c60-e69f-7cb08b48c814:
   AuthMethod:   nomad-workloads
   Description:  Binding rule for Nomad tasks
   BindType:     role
   BindName:     ${value.nomad_namespace}-team
   Selector:     "nomad_service" not in value
f5a6e1c4-aeac-0d08-ed95-9c7dee49528d:
   AuthMethod:   nomad-workloads
   Description:  Binding rule for services registered from Nomad
   BindType:     service
   BindName:     ${value.nomad_service}
   Selector:     "nomad_service" in value and value.nomad_service contains value.nomad_namespace

Nomad config file

bind_addr = "127.0.0.1"
advertise {
  http = "127.0.0.1"
  rpc  = "127.0.0.1"
  serf = "127.0.0.1"
}
ports {
  http = 4646
  rpc  = 4647
  serf = 4648
}
datacenter = "local"
region     = "london"
disable_update_check = true
ui {
  enabled = true
  label {
    text = "Environment: TEST DataCenter: gb2"
  }
}
plugin "docker" {
  config {
    endpoint = "unix:///var/run/docker.sock"
  }
}
log_level            = "INFO"
log_rotate_duration  = "24h"
log_rotate_max_files = 7
log_json             = true
acl {
  enabled = true
}
client {
  enabled = true
  servers = [
    "127.0.0.1:4647"
  ]
  node_class = "default"
  node_pool = "platforms"
}
server {
  enabled          = true
  bootstrap_expect = 1
  encrypt          = <redacted>
  server_join {
    retry_join = [
      "127.0.0.1:4648"
    ]
  }
}
consul {
  address = "http://127.0.0.1:8500"
  token = <redacted>
  service_identity {
  aud = ["consul.test.io"]
  ttl = "1h"
}
task_identity {
  aud = ["consul.test.io"]
  ttl = "1h"
}
}

Nomad Job file

job "nginx" {
  namespace = "platforms"
  node_pool = "platforms"
  group "nginx-group" {
    count = 1
    network {
      port "http" {
        to = "80"
      }
    }
    task "nginx-task" {
     driver = "docker"
     config {
        image = "nginx:latest"
        ports = ["http"]

    }

      service {
        name     = "platforms-nginx-service"
        port     = "http"
        provider = "consul"

        check {
          type     = "http"
          port     = "http"
          path     = "/"
          interval = "2s"
          timeout  = "2s"
        }
      }
    }
  }
}

When trying to run above Job, it fails with an error:

Setup Failure: failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for service platforms-nginx-service: Unexpected response code: 403 (Permission denied)

Please note that if I use the below selector in acl binding where namespace name is hardcoded, it works as expected.

Selector: "nomad_service" in value and value.nomad_service contains "platforms"

However, if we replace platforms with value.nomad_namespace, it fails.

-->

Consul and nomad version info
consul : v1.19.2
noamd : v1.9.3