Implement graceful handling of WDQS rate-limiting #48
Conversation
![codeclimate[bot]](https://avatars.githubusercontent.com/in/9403?s=60&v=4){kind=small}
| def refresh_labels(app_label, model, ids=None, queued_at=None): | ||
| @celery.shared_task(bind=True, queue='wdqs') | ||
| @utils.queries_wikidata | ||
| def refresh_labels(self, app_label, model, ids=None, queued_at=None, rate_limiting_handler=None): |
There was a problem hiding this comment.
Function refresh_labels has 6 arguments (exceeds 4 allowed). Consider refactoring.
| wdqs_rate_limiting = Signal(['retry_after']) | ||
|
|
||
|
|
||
| def templated_wikidata_query(query_name: str, context: dict, |
There was a problem hiding this comment.
Function templated_wikidata_query has a Cognitive Complexity of 15 (exceeds 5 allowed). Consider refactoring.
| rate_limiting_handler(False) | ||
|
|
||
|
|
||
| def queries_wikidata(task_func): |
There was a problem hiding this comment.
Function queries_wikidata has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.
| @mock.patch('time.sleep') | ||
| def testRetriesIfTooManyRequests(self, time_sleep, urlopen): | ||
| retry_after = 10 | ||
| urlopen.side_effect = HTTPError(url=None, |
There was a problem hiding this comment.
Identical blocks of code found in 2 locations. Consider refactoring.
| @mock.patch('time.sleep') | ||
| def testSuspendsConsuming(self, time_sleep, urlopen): | ||
| retry_after = 10 | ||
| urlopen.side_effect = HTTPError(url=None, |
There was a problem hiding this comment.
Identical blocks of code found in 2 locations. Consider refactoring.
alexdutton
force-pushed
the
wdqs-rate-limiting
branch
from
6756e86
to
a75cadb
Compare
Codecov Report
@@ Coverage Diff @@
## master #48 +/- ##
==========================================
+ Coverage 74.25% 75.69% +1.44%
==========================================
Files 50 51 +1
Lines 1503 1576 +73
==========================================
+ Hits 1116 1193 +77
+ Misses 387 383 -4
Continue to review full report at Codecov.
|
alexdutton
force-pushed
the
wdqs-rate-limiting
branch
from
a75cadb
to
dfedd3e
Compare
Has tests, and seems to work locally too.
Closes #47.