#275 Add Python Coverage Report #282

WGierke · 2018-01-07T09:06:28Z

As discussed in #275, it would be helpful to generate a coverage report of the unit tests in which the total coverage and missing lines can be seen. This PR adds Coverage.py which keeps track of which LOC haven't been tested after running the tests. It generates a htmlcov directory which includes the HTML files that should be served afterwards to correctly render the report. Examples of such reports for the current master are here: prediction, interface.

To persist this report generated by Travis, the project manager need to set up Travis artifacts.

CLA

I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well

isms · 2018-01-07T19:58:51Z

tests/test_docker.sh

-docker-compose -f local.yml run prediction pytest -rsx
+# Coverage should ignore pip packets, files including tests and pytest
+docker-compose -f local.yml run prediction coverage run --branch --omit=/usr/local/lib/python3.6/dist-packages/*,src/tests/*,/usr/local/bin/pytest /usr/local/bin/pytest -rsx
+docker-compose -f local.yml run prediction coverage html


IIRC, this generates a directory htmlcov/ but does not do anything else. We probably want coverage report, right?

Yes, coverage html generates this directory which, when being served, offers an interactive report like here which is IMO easier to navigate and understand. However, if it's difficult to persist the whole directory by Travis, it should also be fine to just pass the report command instead, which creates an output similar to:

Name Stmts Miss Cover Missing ------------------------------------------------------- my_program.py 20 4 80% 33-35, 39

lamby · 2018-01-09T03:21:26Z

Wouldn't it be better to push to an external service like coveralls (or similar). This way we get nice GitHub integration where we say "your PR decreases coverage by 2%". I believe some of these services even let you block merges (!!) if you decrease :)

WGierke · 2018-01-09T07:05:48Z

@lamby I tried this "spike" for a couple of reasons. First, I don't have the rights to add such an integration like coveralls to the repo :-) Second, one would always be required to first push to GitHub to receive a test report if I understand it right. While this not only takes way longer than running the report generation locally, there are also tests that always time out on Travis which means that one can never fully rely on the report generated online. However, I understand your point.

lamby · 2018-01-10T02:08:39Z

@isms Any thoughts? :)

WGierke · 2018-01-10T11:41:37Z

Apparently codecov.io supports Coverage.py. I'm currently trying to add an upload of the generated report to codecov.io on my fork. This way we can have both - a nice HTML version that can be generated locally (if need be) and a repository integration that shows us the changed coverage that would be induced by merging a branch.

isms · 2018-01-10T13:28:12Z

My thought is that a text report in the build is all we're looking for to close the issue. As @WGierke mentioned an integration is easy and just a matter of configuring the build tool.

Anyway, fiddling with Travis is out of scope for contributors, so my recommendation is still just to generate the text report as part of the testing commands and leave it there. (We may or may not loop in a third party service this late in the game.)

WGierke · 2018-01-10T14:45:11Z

@lamby @isms Now, we just generate the standard report. If someone wants to add that the report should be uploaded to codecov.io: it worked for me by running coverage xml and uploading it using bash <(curl -s https://codecov.io/bash) -t THE-CODECOV-TOKEN to codecov. Maybe it's not necessary to specify the token when uploading from a Travis build but I'm not sure.
PS: the report from the current build:
prediction service:

Name                                                      Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------------------------------
__init__.py                                                   0      0      0      0   100%
config.py                                                    22      0      0      0   100%
src/__init__.py                                               0      0      0      0   100%
src/algorithms/__init__.py                                    0      0      0      0   100%
src/algorithms/classify/__init__.py                           0      0      0      0   100%
src/algorithms/classify/src/__init__.py                       0      0      0      0   100%
src/algorithms/classify/src/gtr123_model.py                 151      1     26      1    99%
src/algorithms/classify/trained_model.py                      4      0      0      0   100%
src/algorithms/identify/__init__.py                           0      0      0      0   100%
src/algorithms/identify/prediction.py                       190    153     58      0    15%
src/algorithms/identify/src/__init__.py                       0      0      0      0   100%
src/algorithms/identify/src/gtr123_model.py                 284    229     68      1    16%
src/algorithms/identify/trained_model.py                     43     26     18      1    36%
src/algorithms/segment/__init__.py                            0      0      0      0   100%
src/algorithms/segment/src/__init__.py                        0      0      0      0   100%
src/algorithms/segment/src/evaluate.py                       31      0      0      0   100%
src/algorithms/segment/src/models/__init__.py                 0      0      0      0   100%
src/algorithms/segment/src/models/segmentation_model.py      38     10      0      0    74%
src/algorithms/segment/src/models/simple_3d_model.py         39      7      2      0    78%
src/algorithms/segment/trained_model.py                      28      0      6      0   100%
src/factory.py                                               19      4      4      2    74%
src/preprocess/__init__.py                                    0      0      0      0   100%
src/preprocess/crop_dicom.py                                 21      1      8      1    93%
src/preprocess/crop_patches.py                               33      4     16      5    82%
src/preprocess/errors.py                                      5      0      2      1    86%
src/preprocess/extract_lungs.py                             171     88     63      2    47%
src/preprocess/generators.py                                313     59    172     25    78%
src/preprocess/improved_lung_segmentation.py                284     17    100      9    92%
src/preprocess/load_ct.py                                    91      6     28      2    93%
src/preprocess/lung_segmentation.py                         139    105     36      1    20%
src/preprocess/preprocess_ct.py                              78     14     46     14    77%
src/views.py                                                 32      0     10      1    98%
-------------------------------------------------------------------------------------------
TOTAL                                                      2016    724    663     66    62%

interface service:

Name                          Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------
backend/__init__.py               0      0      0      0   100%
backend/api/__init__.py           0      0      0      0   100%
backend/api/serializers.py       65     16      2      1    75%
backend/api/urls.py              28     10      4      0    56%
backend/api/views.py             79      7      3      0    91%
backend/cases/__init__.py         0      0      0      0   100%
backend/cases/enums.py           65      0     18      0   100%
backend/cases/factories.py       18      0      0      0   100%
backend/cases/models.py          84      0      4      0   100%
backend/images/__init__.py        0      0      0      0   100%
backend/images/factories.py      15      0      6      0   100%
backend/images/models.py         84      1     23      0    99%
config/__init__.py                0      0      0      0   100%
config/settings/__init__.py       0      0      0      0   100%
config/settings/base.py          33      2      0      0    94%
config/settings/local.py         12      0      0      0   100%
config/urls.py                   10      4      4      1    50%
manage.py                        16      6      2      1    61%
---------------------------------------------------------------
TOTAL                           509     46     66      3    90%

lamby · 2018-01-11T03:06:32Z

tests/test_docker.sh


 # run the backend API tests
-docker-compose -f local.yml run interface python manage.py test
+# Coverage should ignore pip packets, files including migrations and tests
+docker-compose -f local.yml run interface coverage run --branch --omit=/usr/local/lib/python3.6/dist-packages/*,**/migrations/*.py,**/test*.py manage.py test


Can we make the python version not be hardcoded here? perhaps with */dist-packages/* ?

Good point. Thanks, I adapted it.

lamby · 2018-01-12T02:11:43Z

Thanks!

WGierke mentioned this pull request Jan 7, 2018

Unit tests for Python backend code #275

Open

WGierke force-pushed the coverage_report branch 2 times, most recently from 0280aec to 645c3ac Compare January 7, 2018 12:07

isms suggested changes Jan 7, 2018

View reviewed changes

WGierke force-pushed the coverage_report branch from 645c3ac to fde777f Compare January 10, 2018 13:54

lamby suggested changes Jan 11, 2018

View reviewed changes

WGierke force-pushed the coverage_report branch 2 times, most recently from e9b5cb1 to 7510427 Compare January 11, 2018 08:39

Create coverage reports

8deb4c5

WGierke force-pushed the coverage_report branch from 7510427 to 8deb4c5 Compare January 11, 2018 09:30

lamby merged commit c9337eb into drivendataorg:master Jan 12, 2018

WGierke changed the title ~~Add Python Coverage Report~~ #275 Add Python Coverage Report Jan 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#275 Add Python Coverage Report #282

#275 Add Python Coverage Report #282

WGierke commented Jan 7, 2018 •

edited

Loading

isms Jan 7, 2018

WGierke Jan 7, 2018

lamby commented Jan 9, 2018

WGierke commented Jan 9, 2018

lamby commented Jan 10, 2018

WGierke commented Jan 10, 2018

isms commented Jan 10, 2018

WGierke commented Jan 10, 2018

lamby Jan 11, 2018

WGierke Jan 11, 2018

lamby commented Jan 12, 2018

#275 Add Python Coverage Report #282

#275 Add Python Coverage Report #282

Conversation

WGierke commented Jan 7, 2018 • edited Loading

CLA

isms Jan 7, 2018

Choose a reason for hiding this comment

WGierke Jan 7, 2018

Choose a reason for hiding this comment

lamby commented Jan 9, 2018

WGierke commented Jan 9, 2018

lamby commented Jan 10, 2018

WGierke commented Jan 10, 2018

isms commented Jan 10, 2018

WGierke commented Jan 10, 2018

lamby Jan 11, 2018

Choose a reason for hiding this comment

WGierke Jan 11, 2018

Choose a reason for hiding this comment

lamby commented Jan 12, 2018

WGierke commented Jan 7, 2018 •

edited

Loading