How to route /docs path in litserve behind a proxy?

[Mayurji]

I have hosted litserve as kubernetes(EKS) deployment with a service, now it is further connected to a proxy with Virtual service CRD and gateway.

In eks deployment,

• Model: the url works 0.0.0.0:4000/predict after port forwarding.

• Docs: The url works 0.0.0.0:4000/docs after port forwarding.

In EKS Service, the above url works, mapping 4000:4000, and then port forwarding.

Now, Istio's virtual service has prefix set as "modV1" and I am able to hit the model api as

domain-name/modV1/predict

But /docs api doesn't work from virtual service,

domain-name/modV1/docs

How to update or direct the /docs route in litserve for proxy?

[aniketmaurya]

hi @Mayurji, LitServe only exposes the endpoints which can be put behind a reverse proxy. It doesn't interfere with proxy directly, you might have to check the proxy that you are using on how to do this.

[Mayurji]

@aniketmaurya , We have seen similar issues happening with Fastapi, which is also the backend for litserve. But it seems like, in fastapi they are able to update the root_path as mentioned in the below url. Can we update root_path for litserve as well.

URL: https://stackoverflow.com/questions/78649231/fastapi-fetch-error-failed-to-load-openapi-json-404-error-when-deployed-in

[aniketmaurya]

@aniketmaurya , We have seen similar issues happening with Fastapi, which is also the backend for litserve. But it seems like, in fastapi they are able to update the root_path as mentioned in the below url. Can we update root_path for litserve as well.

URL: https://stackoverflow.com/questions/78649231/fastapi-fetch-error-failed-to-load-openapi-json-404-error-when-deployed-in

All the APIs by default starts with root path "/", are you able to set that to your proxy?

[Mayurji]

@aniketmaurya , We cannot keep the root_path as proxy, we have multiple models under a common domains with different prefix.

For different models, '/predict' works while '/docs' doesn't:

https://domainName/modelV1/predict
https://domainName/modelV1/docs

https://domainName/modelV2/predict
https://domainName/modelV2/docs

[aniketmaurya]

hi @Mayurji, /docs is supposed to be used only for development so unfortunately we will not be prioritizing this at the moment.

[bhimrazy]

A somewhat similar issue: #270