Some *internal* tasks in HPC compute node not being able to use the prefect server's api

[annshress]

With logging api

--- Error logging to API ---
Client error '403 Forbidden' for url 'https://prefect2.hedwig-workflow-api.niaiddev.net/api/logs/'

With task set state api

Traceback (most recent call last):
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/distributed/worker.py", line 3039, in apply_function_async
    result = await function(*args, **kwargs)
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/engine.py", line 1650, in begin_task_run
    state = await orchestrate_task_run(
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/engine.py", line 1966, in orchestrate_task_run
    state = await propose_state(client, terminal_state, task_run_id=task_run.id)
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/engine.py", line 2351, in propose_state
    response = await set_state_and_handle_waits(set_state)
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/engine.py", line 2338, in set_state_and_handle_waits
    response = await set_state_func()
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/client/orchestration.py", line 2068, in set_task_run_state
    response = await self._client.post(
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/client/base.py", line 312, in send
    response.raise_for_status()
  File "/gs1/home/hedwig_dev/dev/lib/python3.10/site-packages/prefect/client/base.py", line 164, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url 'https://prefect2.hedwig-workflow-api.niaiddev.net/api/task_runs/56337c2c-f49d-4812-bd34-a58c17b7745c/set_state'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403