You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
add a new 30s timeout for fetching web page content for the crawler
the async reqwest::get function by default does not have a timeout and thus we could potentially stuck waiting
forever.
add new max_delay_secs parameter to the /health endpoint to allow user specify a maximum delay in seconds and
the API will return error code 503 if the delay for the latest file timestamp (any file) is greater than the specified
value.
this allows better monitoring for the API's health without sending heartbeats.