Deprecate "running notebooks": switching to nbval

[ceball]

Closes #1
Closes #20
Closes #31

• use --nbval-lax to replace --nbsmoke-run.
• recommend to use --nbval-lax instead of --nbsmoke-run
• drop support for saving html
• notebook is defined as fnmatch *.ipynb (no longer configurable) - fixes Picks up non-notebook files #6
• no more nbsmoke_skip_run config: user should use pytest itself to achieve that (fixes Specifying skipped notebooks needs improvement #11)

To do:

• nbval does not support config file? use pytest options/config
• nbval does not support skipping a notebook (except by comment on every cell) use pytest options/config
• figure out why I am getting a weird error (see the comment below) when running nbval! Happens to me locally and on travis.

Also:

• update project metadata

[ceball]

Seem to be getting the below for all the run tests while using nbval :(

Initial error?

self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
cls = <class 'jupyter_client.manager.KernelManager'>
    def get(self, obj, cls=None):
        try:
>           value = obj._trait_values[self.name]
E           KeyError: 'session'
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:528: KeyError

And another while it's being handled?

traitlets.traitlets.TraitError: The 'session' trait of a KernelManager instance must be a Session, but a value of class 'jupyter_client.session.Session' (i.e. <jupyter_client.session.Session object at 0x7f66394a7dd8>) was specified.

Full thing:

__________________ ERROR at setup of testing123.ipynb::Cell 0 __________________
self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
cls = <class 'jupyter_client.manager.KernelManager'>
    def get(self, obj, cls=None):
        try:
>           value = obj._trait_values[self.name]
E           KeyError: 'session'
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:528: KeyError
During handling of the above exception, another exception occurred:
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f66398977b8>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
    @classmethod
    def from_call(cls, func, when, reraise=None) -> "CallInfo":
        #: context of invocation: one of "setup", "call",
        #: "teardown", "memocollect"
        start = time()
        excinfo = None
        try:
>           result = func()
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/_pytest/runner.py:244: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/_pytest/runner.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_HookCaller 'pytest_runtest_setup'>, args = ()
kwargs = {'item': <IPyNbCell Cell 0>}, notincall = set()
    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()
        if self.spec and self.spec.argnames:
            notincall = (
                set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
            )
            if notincall:
                warnings.warn(
                    "Argument(s) {} which are declared in the hookspec "
                    "can not be found in this hook call".format(tuple(notincall)),
                    stacklevel=2,
                )
>       return self._hookexec(self, self.get_hookimpls(), kwargs)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/hooks.py:286: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_pytest.config.PytestPluginManager object at 0x7f6638f76d30>
hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/p...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6638f84748>>]
kwargs = {'item': <IPyNbCell Cell 0>}
    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/manager.py:93: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
hook = <_HookCaller 'pytest_runtest_setup'>
hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/p...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6638f84748>>]
kwargs = {'item': <IPyNbCell Cell 0>}
    def traced_hookexec(hook, hook_impls, kwargs):
        before(hook.name, hook_impls, kwargs)
        outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
        after(outcome, hook.name, hook_impls, kwargs)
>       return outcome.get_result()
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/manager.py:337: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pluggy.callers._Result object at 0x7f6638fa15c0>
    def get_result(self):
        """Get the result(s) for this hook call.
    
        If the hook was marked as a ``firstresult`` only a single value
        will be returned otherwise a list of results.
        """
        __tracebackhide__ = True
        if self._excinfo is None:
            return self._result
        else:
            ex = self._excinfo
            if _py3:
>               raise ex[1].with_traceback(ex[2])
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/callers.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cls = <class 'pluggy.callers._Result'>
func = <function PluginManager.add_hookcall_monitoring.<locals>.traced_hookexec.<locals>.<lambda> at 0x7f6639897950>
    @classmethod
    def from_call(cls, func):
        __tracebackhide__ = True
        result = excinfo = None
        try:
>           result = func()
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/callers.py:52: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/manager.py:335: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/p...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6638f84748>>]
kwargs = {'item': <IPyNbCell Cell 0>}
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
        methods,
        kwargs,
>       firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    )
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/manager.py:87: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/p...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6638f84748>>]
caller_kwargs = {'item': <IPyNbCell Cell 0>}, firstresult = False
    def _multicall(hook_impls, caller_kwargs, firstresult=False):
        """Execute a call into multiple python functions/methods and return the
        result(s).
    
        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    "hook call must provide argument %r" % (argname,)
                                )
    
                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
                        res = hook_impl.function(*args)
                        if res is not None:
                            results.append(res)
                            if firstresult:  # halt further impl calls
                                break
            except BaseException:
                excinfo = sys.exc_info()
        finally:
            if firstresult:  # first result hooks return a single value
                outcome = _Result(results[0] if results else None, excinfo)
            else:
                outcome = _Result(results, excinfo)
    
            # run all wrapper post-yield blocks
            for gen in reversed(teardowns):
                try:
                    gen.send(outcome)
                    _raise_wrapfail(gen, "has second yield")
                except StopIteration:
                    pass
    
>           return outcome.get_result()
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/callers.py:208: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pluggy.callers._Result object at 0x7f6638fa1748>
    def get_result(self):
        """Get the result(s) for this hook call.
    
        If the hook was marked as a ``firstresult`` only a single value
        will be returned otherwise a list of results.
        """
        __tracebackhide__ = True
        if self._excinfo is None:
            return self._result
        else:
            ex = self._excinfo
            if _py3:
>               raise ex[1].with_traceback(ex[2])
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/callers.py:80: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/p...xture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7f6638f84748>>]
caller_kwargs = {'item': <IPyNbCell Cell 0>}, firstresult = False
    def _multicall(hook_impls, caller_kwargs, firstresult=False):
        """Execute a call into multiple python functions/methods and return the
        result(s).
    
        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    "hook call must provide argument %r" % (argname,)
                                )
    
                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
>                       res = hook_impl.function(*args)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/pluggy/callers.py:187: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
item = <IPyNbCell Cell 0>
    def pytest_runtest_setup(item):
        _update_current_test_var(item, "setup")
>       item.session._setupstate.prepare(item)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/_pytest/runner.py:123: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_pytest.runner.SetupState object at 0x7f6638f84be0>
colitem = <IPyNbCell Cell 0>
    def prepare(self, colitem):
        """ setup objects along the collector chain to the test-method
            and teardown previously setup objects."""
        needed_collectors = colitem.listchain()
        self._teardown_towards(needed_collectors)
    
        # check if the last collection node has raised an error
        for col in self.stack:
            if hasattr(col, "_prepare_exc"):
                exc = col._prepare_exc
                raise exc
        for col in needed_collectors[len(self.stack) :]:
            self.stack.append(col)
            try:
                col.setup()
            except TEST_OUTCOME as e:
                col._prepare_exc = e
>               raise e
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/_pytest/runner.py:376: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_pytest.runner.SetupState object at 0x7f6638f84be0>
colitem = <IPyNbCell Cell 0>
    def prepare(self, colitem):
        """ setup objects along the collector chain to the test-method
            and teardown previously setup objects."""
        needed_collectors = colitem.listchain()
        self._teardown_towards(needed_collectors)
    
        # check if the last collection node has raised an error
        for col in self.stack:
            if hasattr(col, "_prepare_exc"):
                exc = col._prepare_exc
                raise exc
        for col in needed_collectors[len(self.stack) :]:
            self.stack.append(col)
            try:
>               col.setup()
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/_pytest/runner.py:373: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <IPyNbFile testing123.ipynb>
    def setup(self):
        """
        Called by pytest to setup the collector cells in .
        Here we start a kernel and setup the sanitize patterns.
        """
    
        if self.parent.config.option.current_env:
            kernel_name = CURRENT_ENV_KERNEL_NAME
        else:
            kernel_name = self.nb.metadata.get(
                'kernelspec', {}).get('name', 'python')
>       self.kernel = RunningKernel(kernel_name, str(self.fspath.dirname))
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/nbval/plugin.py:233: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <nbval.kernel.RunningKernel object at 0x7f6638f8a898>
kernel_name = 'python', cwd = '/tmp/pytest-of-travis/pytest-0/test_run_good0'
    def __init__(self, kernel_name, cwd=None):
        """
        Initialise a new kernel
        specify that matplotlib is inline and connect the stderr.
        Stores the active kernel process and its manager.
        """
    
        self.km, self.kc = start_new_kernel(
            kernel_name=kernel_name,
            stderr=open(os.devnull, 'w'),
>           cwd=cwd,
        )
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/nbval/kernel.py:88: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
startup_timeout = 60, kernel_name = 'python'
kwargs = {'cwd': '/tmp/pytest-of-travis/pytest-0/test_run_good0', 'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}
km = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
    def start_new_kernel(startup_timeout=60, kernel_name='python', **kwargs):
        """Start a new kernel, and return its Manager and Client"""
        logger.debug('Starting new kernel: "%s"' % kernel_name)
        km = KernelManager(kernel_name=kernel_name,
                           kernel_spec_manager=NbvalKernelspecManager())
>       km.start_kernel(**kwargs)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/nbval/kernel.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
kw = {'cwd': '/tmp/pytest-of-travis/pytest-0/test_run_good0', 'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}
    def start_kernel(self, **kw):
        """Starts a kernel on this host in a separate process.
    
        If random ports (port=0) are being used, this method must be called
        before the channels are created.
    
        Parameters
        ----------
        `**kw` : optional
             keyword arguments that are passed down to build the kernel_cmd
             and launching the kernel (e.g. Popen kwargs).
        """
>       kernel_cmd, kw = self.pre_start_kernel(**kw)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/jupyter_client/manager.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
kw = {'cwd': '/tmp/pytest-of-travis/pytest-0/test_run_good0', 'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}
    def pre_start_kernel(self, **kw):
        """Prepares a kernel for startup in a separate process.
    
        If random ports (port=0) are being used, this method must be called
        before the channels are created.
    
        Parameters
        ----------
        `**kw` : optional
             keyword arguments that are passed down to build the kernel_cmd
             and launching the kernel (e.g. Popen kwargs).
        """
        if self.transport == 'tcp' and not is_local_ip(self.ip):
            raise RuntimeError("Can only launch a kernel on a local interface. "
                               "This one is not: %s."
                               "Make sure that the '*_address' attributes are "
                               "configured properly. "
                               "Currently valid addresses are: %s" % (self.ip, local_ips())
                               )
    
        # write connection file / get default ports
>       self.write_connection_file()
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/jupyter_client/manager.py:248: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
    def write_connection_file(self):
        """Write connection info to JSON dict in self.connection_file."""
        if self._connection_file_written and os.path.exists(self.connection_file):
            return
    
        self.connection_file, cfg = write_connection_file(self.connection_file,
>           transport=self.transport, ip=self.ip, key=self.session.key,
            stdin_port=self.stdin_port, iopub_port=self.iopub_port,
            shell_port=self.shell_port, hb_port=self.hb_port,
            control_port=self.control_port,
            signature_scheme=self.session.signature_scheme,
            kernel_name=self.kernel_name
        )
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/jupyter_client/connect.py:469: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
cls = <class 'jupyter_client.manager.KernelManager'>
    def __get__(self, obj, cls=None):
        """Get the value of the trait by self.name for the instance.
    
        Default values are instantiated when :meth:`HasTraits.__new__`
        is called.  Thus by the time this method gets called either the
        default value or a user defined value (they called :meth:`__set__`)
        is in the :class:`HasTraits` instance.
        """
        if obj is None:
            return self
        else:
>           return self.get(obj, cls)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:556: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
cls = <class 'jupyter_client.manager.KernelManager'>
    def get(self, obj, cls=None):
        try:
            value = obj._trait_values[self.name]
        except KeyError:
            # Check for a dynamic initializer.
            dynamic_default = self._dynamic_default_callable(obj)
            if dynamic_default is None:
                raise TraitError("No default value found for %s trait of %r"
                                 % (self.name, obj))
>           value = self._validate(obj, dynamic_default())
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:535: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
value = <jupyter_client.session.Session object at 0x7f66394a7dd8>
    def _validate(self, obj, value):
        if value is None and self.allow_none:
            return value
        if hasattr(self, 'validate'):
>           value = self.validate(obj, value)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:591: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
value = <jupyter_client.session.Session object at 0x7f66394a7dd8>
    def validate(self, obj, value):
        if isinstance(value, self.klass):
            return value
        else:
>           self.error(obj, value)
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:1677: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <traitlets.traitlets.Instance object at 0x7f664d23deb8>
obj = <jupyter_client.manager.KernelManager object at 0x7f6638fa07f0>
value = <jupyter_client.session.Session object at 0x7f66394a7dd8>
    def error(self, obj, value):
        kind = type(value)
        if six.PY2 and kind is InstanceType:
            msg = 'class %s' % value.__class__.__name__
        else:
            msg = '%s (i.e. %s)' % ( str( kind )[1:-1], repr( value ) )
    
        if obj is not None:
            e = "The '%s' trait of %s instance must be %s, but a value of %s was specified." \
                % (self.name, class_of(obj),
                   self.info(), msg)
        else:
            e = "The '%s' trait must be %s, but a value of %r was specified." \
                % (self.name, self.info(), msg)
    
>       raise TraitError(e)
E       traitlets.traitlets.TraitError: The 'session' trait of a KernelManager instance must be a Session, but a value of class 'jupyter_client.session.Session' (i.e. <jupyter_client.session.Session object at 0x7f66394a7dd8>) was specified.
/home/travis/build/pyviz-dev/nbsmoke/.tox/dev/lib/python3.7/site-packages/traitlets/traitlets.py:1524: TraitError

[ceball]

Apart from the issue above (which I am also having locally and on travis with nbval's own test suite, in a separate piece of work), there's another issue: nbval has its own code for interacting with the kernel, and it does not behave as I would expect.

E.g. if you have some random junk magic like %this_magic_does_not_exist in a cell - which would cause lab, notebook, ipython, etc to stop processing with an error - nbval silently swallows that, marks the cell execution as successful, and moves to the next cell (so even if the next line in the cell is raise ValueError, that is also missed). nbsmoke's run check was supposed to be checking "will this notebook work for someone in jupyter notebook/lab/ipython?" so that's a problem. There might be other such problems in nbval - I'm not clear how up to date the code within nbval is for communicating with the kernel, and I'm not clear how much test coverage there is for "basic run checks" in nbval.

nbsmoke uses the jupyter org's own nbconvert, which in turns uses the jupyter org's own nbclient (originally part of nbconvert) for communicating with kernel. My next step is probably to see what the chances are of unifying things a bit more. (Would not be worth it for nbsmoke alone, but another project I use also uses parts of nbval...)

[jbednar]

Sounds like it would be worth filing an issue on nbval asking for it to behave like Jupyter does in other cases, to increase its value for testing purposes.

[ceball]

Yes, I have a few PRs and issues open on nbval already. To summarize where they are headed (in my mind):

1. I think it would be good if nbval used nbclient and had less of its own custom handling of kernel communications (but I haven't looked into the details yet - I might be wrong!).
2. It would be nice if nbval supported first of all a simple "does it run through" use case, and then people could move onto output checking afterwards.

Regarding point 2, here's an extract copy/pasted - please say if you have a better idea/other thoughts about it:

I haven't yet investigated pytest options, but I wanted to note I failed to appreciate that --nbval-lax is not just running the notebook - it also checks output for cells with # NBVAL_CHECK_OUTPUT.

To me, it seems like there are three levels at which someone might want to use nbval:

1. Does the notebook execute all the way through without exception? (This is the thing I am most interested in. I set the bar low :) ).
2. Does the notebook fulfil 1, plus: do certain, specified outputs match?
3. Do all outputs match (also fulfils 1 and 2)?

The below depends a bit on what I find out about pytest options, but I wanted to note down my thoughts anyway.

[...]

Ideally I'd like something like --nbval with an --output-check option that defaults to none, e.g.

1. pytest --nbval (--output-check defaults to none; currently: not available)
2. pytest --nbval --output-check=marked (currently: pytest --nbval-lax)
3. pytest --nbval --output-check=all (currently: pytest --nbval)

Don't know what the options should be called yet, partly because I don't know how pytest handles arguments, plus there's backwards compatibility to consider.