Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecate "running notebooks": switching to nbval #42

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Deprecate "running notebooks": switching to nbval #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d3560d9
Quick sketch of switching to nbval to discover issues/plan how it wou…
ceball May 10, 2020
3cf47aa
lint
ceball May 10, 2020
73a8095
Fix setup syntax.
ceball May 10, 2020
9b84caf
lint
ceball May 10, 2020
796f3cd
Update test_skip_run.
ceball May 11, 2020
5eba584
Clarify deprecated options.
ceball May 11, 2020
ea827a3
Typo
ceball May 11, 2020
f674050
code typo, ha ha I should run the tests or lint check or all those th…
ceball May 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 87 additions & 52 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,7 @@
nbsmoke
=======

Basic notebook smoke tests: Do they run ok? Do they contain lint?

----

This `Pytest`_ plugin was generated with `Cookiecutter`_ along with `@hackebrot`_'s `Cookiecutter-pytest-plugin`_ template.

Static checking of notebooks (e.g. do they contain lint?)


Installation
Expand All @@ -29,23 +24,14 @@ You can install nbsmoke via `pip`_ from `PyPI`_::

$ pip install nbsmoke

Or you can install nbsmoke via `conda`_ from `anaconda.org`_::
Or via `conda`_ from `anaconda.org`_::

$ conda install -c pyviz/label/dev -c conda-forge nbsmoke
$ conda install nbsmoke


Usage
-----

Check all notebooks run without errors::

$ pytest --nbsmoke-run

Check all notebooks run without errors, and store html to look at
afterwards::

$ pytest --nbsmoke-run --store-html=/scratch

Lint check notebooks::

$ pytest --nbsmoke-lint
Expand All @@ -56,27 +42,18 @@ Lint failures as warnings only::

Instead of all files in a directory, you can specify a list e.g.::

$ pytest --nbsmoke-run notebooks/Untitled*.ipynb
$ pytest --nbsmoke-lint notebooks/Untitled*.ipynb

If you want to restrict pytest to running only your notebook tests, use `-k`, e.g.::

$ pytest --nbsmoke-run -k ".ipynb"
$ pytest --nbsmoke-lint -k ".ipynb"

TODO: add ``--nbsmoke-verify`` docs!

Additional options are available by standard pytest 'ini'
configuration in setup.cfg, pytest.ini, or tox.ini::

[pytest]
# when running, seconds allowed per cell (see nbconvert timeout)
nbsmoke_cell_timeout = 600

# notebooks to skip running; one case insensitive re to match per line
nbsmoke_skip_run = ^.*skipme\.ipynb$
^.*skipmetoo.*$

# case insensitive re to match for file to be considered notebook;
# defaults to ``^.*\.ipynb``
it_is_nb_file = ^.*\.something$

# flakes you don't want to hear about (regex)
nbsmoke_flakes_to_ignore = .*hvplot.* imported but unused.*

Expand All @@ -91,30 +68,19 @@ configuration in setup.cfg, pytest.ini, or tox.ini::
nbsmoke supports ``# noqa`` comments to mark that something
should be ignored during lint checking.

The ``nbsmoke_skip_run`` list in a project's config can be ignored by
passing ``--ignore-nbsmoke-skip-run`` (useful if sometimes you want to
run all notebooks for a project where many are typically skipped).


What's the point?
-----------------

Although more sophisticated testing of notebooks is possible (e.g. see
nbval), just checking that notebooks run from start to finish without
error in a fresh kernel (or on a neutral CI service) can be useful
during development. Practical experience of working on several
projects with notebooks confirms this, but that's all the evidence I
have.
Checking notebooks for lint can find things like undefined names
faster than by running them.

Checking notebooks for lint might seem trivial/pointless, but it
frequently uncovers unused names (typically unused imports). It's also
quite common to find python 2 vs 3 problems, and sometimes undefined
names - in a way that's faster than running the notebook (over
multiple versions of python).
TODO: be able to switch linter? Or explicitly call the "lint"
pyflakes.

Unused imports/names themselves might seem trivial, but they can
hinder understanding of a notebook by readers, or add dependencies
that are not required.
Things that aren't errors, such as unused imports/names, might seem
trivial, but they can hinder understanding of a notebook by readers,
or add dependencies that are not required.

Hopefully you don't have mysterious (unused) imports in your notebook,
but if you do, you can add ``# noqa: explanation`` to stop flake
Expand All @@ -126,6 +92,78 @@ simple promise: it will never complain about style, and it will try
very, very hard to never emit false positives."


Deprecated usage
----------------

nbsmoke used to support checking that notebooks run without error, and
could save the generated html. However, we now recommend using nbval
instead. ``--nbsmoke-run`` is still available, but it just calls
nbval; eventually all options related to ``--nbsmoke-run`` will be
removed from nbsmoke.

Before (deprecated)---check all notebooks run without errors::

$ pytest --nbsmoke-run

After---use nbval instead::

$ pytest --nbval-lax


Note about kernel used to run notebook
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default, nbsmoke used the current environment's kernel, whereas
nbval uses the kernel stored in the notebook by default. To obtain
nbsmoke's behavior, pass ``--current-env``. See also:
https://github.com/computationalmodelling/nbval/issues/140


Cell timeout
~~~~~~~~~~~~

Before (deprecated)---pytest configuration options in setup.cfg,
pytest.ini, or tox.ini::

[pytest]
# when running, seconds allowed per cell (see nbconvert timeout)
nbsmoke_cell_timeout = 600


After---nbval has the ``--nbval-cell-timeout`` option. Specify at the
command line, or add to pytest's options (in one of the above files)::

[pytest]
addopts = --nbval-cell-timeout=600


Skipping notebooks
~~~~~~~~~~~~~~~~~~

Before (no longer supported)::

# notebooks to skip running; one case insensitive re to match per line
nbsmoke_skip_run = ^.*skipme\.ipynb$
^.*skipmetoo.*$


After---use pytest's own test selection and skipping
functionality. You can ignore certain files using ``--ignore`` or
``--ignore-glob`` at the command line, or add to pytest's options (in
one of the above files)::

[pytest]
addopts = --ignore=path/to/skipme.ipynb
--ignore=path/of/skipmetoo.ipynb


Alternatively, for more complex scenarios or to explicitly get "skip"
in your test results, see pytest's ``-k`` option or use a
``conftest.py`` file. nbsmoke has an example of using ``conftest.py``
in its own test suite (``test_skip_run`` in
https://github.com/pyviz-dev/nbsmoke/blob/master/tests/test_run.py).


Contributing
------------

Expand Down Expand Up @@ -153,11 +191,8 @@ Issues
If you encounter any problems, please `file an issue`_ (ideally
including a copy of any problematic notebook).

.. _`Cookiecutter`: https://github.com/audreyr/cookiecutter
.. _`@hackebrot`: https://github.com/hackebrot
.. _`BSD-3`: http://opensource.org/licenses/BSD-3-Clause
.. _`cookiecutter-pytest-plugin`: https://github.com/pytest-dev/cookiecutter-pytest-plugin
.. _`file an issue`: https://github.com/pyviz/nbsmoke/issues
.. _`file an issue`: https://github.com/pyviz-dev/nbsmoke/issues
.. _`pytest`: https://github.com/pytest-dev/pytest
.. _`tox`: https://tox.readthedocs.io/en/latest/
.. _`pip`: https://pypi.python.org/pypi/pip/
Expand Down
159 changes: 53 additions & 106 deletions nbsmoke/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,8 @@
# -*- coding: utf-8 -*-

# Note: created with cookiecutter by someone with no experience of how
# to make a pytest plugin. Please question anything related to the
# pytest integration!

import re
import os
import io
import contextlib

import pytest
import nbformat
import nbconvert
from nbconvert.preprocessors import ExecutePreprocessor

try:
from version import Version
Expand All @@ -30,10 +20,6 @@

def pytest_addoption(parser):
group = parser.getgroup('nbsmoke')
group.addoption(
'--nbsmoke-run',
action="store_true",
help="Run notebooks using nbconvert to check for exceptions.")

group.addoption(
'--nbsmoke-lint',
Expand All @@ -55,111 +41,72 @@ def pytest_addoption(parser):
action="store_true",
help="Verify notebooks")

parser.addini('nbsmoke_flakes_to_ignore', "flake messages to ignore during nbsmoke's flake checking")
parser.addini('nbsmoke_flakes_cell_magics_blacklist', "cell magics you don't want to see - i.e. treat as lint.")
parser.addini('nbsmoke_flakes_line_magics_blacklist', "line magics you don't want to see - i.e. treat as lint")


##################
### DEPRECATED ###
# remove in 0.6
group.addoption(
'--store-html',
action="store",
dest='store_html',
default='',
help="When running, store rendered-to-html notebooks in the supplied path.")
'--nbsmoke-run',
action="store_true",
help="**DEPRECATED: Use nbval instead** Run notebooks using nbconvert to check for exceptions.")

parser.addini('nbsmoke_cell_timeout', "nbsmoke's nbconvert cell timeout")
parser.addini('nbsmoke_cell_timeout', "**DEPRECATED: Use nbval instead** nbsmoke's nbconvert cell timeout")

####
# TODO: hacks to work around pyviz team desire to not use pytest's markers
parser.addini('nbsmoke_skip_run', 're to skip (multi-line; one pattern per line)')
parser.addini('nbsmoke_skip_run', '**DEPRECATED: Use a pytest option such as --ignore, --ignore-glob, -k, or conftest.py** re to skip (multi-line; one pattern per line)')
group.addoption(
'--ignore-nbsmoke-skip-run',
action="store_true",
help="Ignore any skip list in the ini file (allows to run all nbs if desired)")
help="**DEPRECATED: Use a pytest option such as --ignore, --ignore-glob, -k, or conftest.py** Ignore any skip list in the ini file (allows to run all nbs if desired)")
####

# TODO: remove/rename/see pytest python_files
parser.addini('it_is_nb_file', 're to determine whether file is notebook')

parser.addini('nbsmoke_flakes_to_ignore', "flake messages to ignore during nbsmoke's flake checking")

parser.addini('nbsmoke_flakes_cell_magics_blacklist', "cell magics you don't want to see - i.e. treat as lint.")
parser.addini('nbsmoke_flakes_line_magics_blacklist', "line magics you don't want to see - i.e. treat as lint")


@contextlib.contextmanager
def cwd(d):
orig = os.getcwd()
os.chdir(d)
try:
yield
finally:
os.chdir(orig)



###################################################


class RunNb(pytest.Item):

def repr_failure(self, excinfo):
return excinfo.exconly(True)

def runtest(self):
self._skip()
with io.open(self.name,encoding='utf8') as nb:
notebook = nbformat.read(nb, as_version=4)

# TODO: which kernel? run in pytest's or use new one (make it option)
_timeout = self.parent.parent.config.getini('nbsmoke_cell_timeout')
kwargs = dict(timeout=int(_timeout) if _timeout!='' else 300,
allow_errors=False,
# or sys.version_info[1] ?
kernel_name='python')

ep = ExecutePreprocessor(**kwargs)
with cwd(os.path.dirname(self.name)): # jupyter notebook always does this, right?
ep.preprocess(notebook,{})

# TODO: clean up this option handling
if self.parent.parent.config.option.store_html != '':
he = nbconvert.HTMLExporter()
# could maybe use this for chance of testing the html? but not the aim of this project
#he.template_file = 'basic'
html, resources = he.from_notebook_node(notebook)
with io.open(os.path.join(self.parent.parent.config.option.store_html,os.path.basename(self.name)+'.html'),'w',encoding='utf8') as f:
f.write(html)

def _skip(self):
_skip_patterns = self.parent.parent.config.getini('nbsmoke_skip_run')
if not self.parent.parent.config.option.ignore_nbsmoke_skip_run:
for pattern in _skip_patterns.splitlines():
if re.match(pattern,self.nodeid.split("::")[0],re.IGNORECASE):
pytest.skip()

##################


class IPyNbFile(pytest.File):
def __init__(self, fspath, parent=None, config=None, session=None, dowhat=RunNb):
self._dowhat = dowhat
def __init__(self, type_, fspath, parent=None, config=None, session=None):
self._type = type_
super(IPyNbFile,self).__init__(fspath, parent=parent, config=None, session=None)

def collect(self):
yield self._dowhat(str(self.fspath), self)

yield self._type(str(self.fspath), self)


def pytest_collect_file(path, parent):
if not path.fnmatch("*.ipynb"):
return

opt = parent.config.option
# TODO: Make this pattern standard/configurable.
# match .ipynb except .nbval.ipynb
it_is_nb_file = parent.config.getini('it_is_nb_file')
if it_is_nb_file == '':
#"^((?!\.nbval).)*\.ipynb$"
it_is_nb_file = r"^.*\.ipynb"
if re.match(it_is_nb_file,path.strpath,re.IGNORECASE):
if opt.nbsmoke_run or opt.nbsmoke_lint or opt.nbsmoke_verify:
# TODO express via the options system if you ever figure it out
# Hmm, should be able to do all - clean up!
assert (opt.nbsmoke_run ^ opt.nbsmoke_lint) ^ opt.nbsmoke_verify
if opt.nbsmoke_run:
dowhat = RunNb
elif opt.nbsmoke_lint:
dowhat = LintNb
elif opt.nbsmoke_verify:
dowhat = VerifyNb
return IPyNbFile(path, parent, dowhat=dowhat)

# TODO: you have to pick one - can't currently run and lint and
# verify (though you should be able to)

if opt.nbsmoke_run:
import warnings
warnings.warn("--nbsmoke-run is deprecated: please use nbval (--nbval-lax) instead.", DeprecationWarning)

import sys
import nbval.plugin

if '--current-env' not in sys.argv:
opt.current_env = True

if '--nbval-cell-timeout' not in sys.argv:
timeout = parent.config.getini('nbsmoke_cell_timeout')
if timeout != '':
opt.nbval_cell_timeout = timeout

skip_patterns = parent.config.getini('nbsmoke_skip_run')
if skip_patterns.strip() != '':
if not '--ignore-nbsmoke-skip-run' in sys.argv:
raise ValueError("nbsmoke_skip_run regex no longer supported; use pytest one of pytest's own options instead: -k, --ignore, --ignore-glob, conftest.py.")

return nbval.plugin.IPyNbFile(path, parent)

elif opt.nbsmoke_lint:
return IPyNbFile(LintNb, path, parent)
elif opt.nbsmoke_verify:
return IPyNbFile(VerifyNb, path, parent)
Loading