Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CadcTapClient.query should not use automatic retries #178

Open
andamian opened this issue Sep 29, 2021 · 1 comment
Open

CadcTapClient.query should not use automatic retries #178

andamian opened this issue Sep 29, 2021 · 1 comment

Comments

@andamian
Copy link
Contributor

andamian commented Sep 29, 2021

if that's a sync query, there is a get|post to create the "job" and a redirect to a url that runs it; the http GET after the redirect runs the job (QUEUED -> EXECUTING) and if that times out you can't retry it because it already started (and you lost your connection to the output stream)... so that error is caused by retrying the GET (not supported)

Retry should start from the top by issuing another query.

Observed error on long queries: WARNING:RetrySession:Resending request in 30s ... ERROR:: unexpected exception: java.lang.RuntimeException: ca.nrc.cadc.uws.server.JobPhaseException: cannot execute job i6yog5u4yga33kl7 when phase = EXECUTING

@andamian
Copy link
Contributor Author

andamian commented Sep 29, 2021

More details:

cadc-tap query -f tsv --debug --timeout=60 -s ivo://cadc.nrc.ca/ad "select 'cadc:CFHT', count(distinct fileName) from archive_files where archiveName = 'CFHT' and fileName not like 'CFHT_%' and fileName not like 'WIRDS_%'"
DEBUG:WsCapabilities:Resolved URL: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/reg/resource-caps
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443
DEBUG:urllib3.connectionpool:https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443 "GET /reg/resource-caps HTTP/1.1" 200 11847
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ws-cadc.canfar.net:443
DEBUG:urllib3.connectionpool:https://ws-cadc.canfar.net:443 "GET /ac/capabilities HTTP/1.1" 200 None
INFO:cadctap.core:host for service ivo://cadc.nrc.ca/ad is ws-cadc.canfar.net
DEBUG:cadctap.core:use -n option
DEBUG:WsCapabilities:Resolved URL: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/reg/resource-caps
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443
DEBUG:urllib3.connectionpool:https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443 "GET /reg/resource-caps HTTP/1.1" 200 11847
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ws-cadc.canfar.net:443
DEBUG:urllib3.connectionpool:https://ws-cadc.canfar.net:443 "GET /ac/capabilities HTTP/1.1" 200 None
DEBUG:BaseWsClient:Creating session.
DEBUG:RetrySession:Sending request <PreparedRequest [POST]> to server.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ws-cadc.canfar.net:443
DEBUG:urllib3.connectionpool:https://ws-cadc.canfar.net:443 "POST /ac/login HTTP/1.1" 200 875
DEBUG:WsCapabilities:Resolved URL: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/reg/resource-caps
DEBUG:WsCapabilities:Read cached content of /Users/gaudet/.config/cadc-registry/alt-domains/www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/resource-caps
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443
DEBUG:urllib3.connectionpool:https://ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443 "GET /ad/capabilities HTTP/1.1" 200 None
DEBUG:cadctap.core:Service has no support for permissions
DEBUG:cadctap.core:QUERY fileds: {'LANG': 'ADQL', 'QUERY': "select 'cadc:CFHT', count(distinct fileName) from archive_files where archiveName = 'CFHT' and fileName not like 'CFHT_%' and fileName not like 'WIRDS_%'", 'FORMAT': 'tsv'}
DEBUG:BaseWsClient:Creating session.
DEBUG:RetrySession:Sending request <PreparedRequest [POST]> to server.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443
DEBUG:urllib3.connectionpool:https://ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443 "POST /ad/sync?LANG=ADQL&QUERY=select+%27cadc%3ACFHT%27%2C+count%28distinct+fileName%29+from+archive_files+where+archiveName+%3D+%27CFHT%27+and+fileName+not+like+%27CFHT_%25%27+and+fileName+not+like+%27WIRDS_%25%27&FORMAT=tsv HTTP/1.1" 303 0
DEBUG:RetrySession:Sending request <PreparedRequest [GET]> to server.
DEBUG:urllib3.connectionpool:https://ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443 "GET /ad/sync/x93u8p8nqaez9tl0/run HTTP/1.1" 504 92
WARNING:RetrySession:Resending request in 30s ...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443
DEBUG:urllib3.connectionpool:https://ws.cadc-ccda.hia-iha.nrc-cnrc.gc.ca:443 "GET /ad/sync/x93u8p8nqaez9tl0/run HTTP/1.1" 500 150
ERROR:: unexpected exception: java.lang.RuntimeException: ca.nrc.cadc.uws.server.JobPhaseException: cannot execute job x93u8p8nqaez9tl0 when phase = EXECUTING

Pat: The second attempt to GET /sync/{jobid}/run should not fail with a 500... that should be handled and be a 4xx (hopefully something more specific that 400 is approp), so that's a bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant