Run playgrounds and check whether they throw uncaught exceptions

[ferranpujolcamins]

If I run swift MyPlayground.playground/Contents.swift where MyPlayground is a simple playground with just this content:

struct E: Error {}
throw E()

I can tell that the playground raised an uncaught exception by inspecting the exit code of the command.

I guess the same idea can be adapted to the playgrounds nef generates.

This would be useful because I could add XCTest assertions on my playground and nef would check for me that they all pass (just like python's doctest)

For simple projects it's useful to just have the documentation be also the unit tests.

Maybe this behavior can be guarded with a new command-line argument.

[miguelangel-dev]

Hi @ferranpujolcamins,

Sorry for the delay in my answer, I will try to respond to you, and I will try to give you visibility about some ideas we have about it.

This would be useful because I could add XCTest assertions on my playground and nef would check for me that they all pass...

I suppose you refer to show the result of code execution as we can find in the python's doctest

...
Trying:
    factorial(5)
Expecting:
    120
ok <---
Trying:
    [factorial(n) for n in range(6)]
Expecting:
    [1, 1, 2, 6, 24, 120]
ok <---
Trying:
    [factorial(long(n)) for n in range(6)]
Expecting:
    [1, 1, 2, 6, 24, 120]
ok <---
...

Currently, using nef you only can validate a playground (with 3rd-party libraries), but you can not evaluate the code. It is a nice-to-have because you could add the result of each example, or even verify the result as you commented.

It is something we have in mind, but it is not a trivial feature, because, in a general way, we could only execute the playground and check it runs correctly, but in advance, we will want to evaluate piece of code, and the party will start, we need to provide the whole dependencies, functions or any ref. we need for this snippets, and the order is essential; it works as an interpreter... Anyway, after the nef-refactor I comment you in the other issue, I will come back at this point, and I will give you more information about the ideas, roadmap...

While this evaluation-feature comes to live, you can add tests to your documentation and run it locally. I know, it is not the best option because you can not automated it in CI (for example), but you can verify them manually.

Maybe you can find useful some utils nef provides; if you have a function like this one:

func increment(_ value: Int) -> Int {
    value + 1
}

You can add a case test, in this way:

import XCTest

class ExampleCase: XCTestCase {
    func testIncrement_Success() {
        let random = Int.random(in: 0 ... 1_000_000)
        XCTAssert(increment(random) > random)
    }
}

and execute it with nef-utils

Nef.run(testCase: ExampleCase.self)

even you can hide it in the render output, using the nef command hidden

// nef:begin:hidden
....
// nef:end

Output example:

Test Suite 'ExampleCase' started at 2020-01-21 16:34:42.667
Test Case '-[__lldb_expr_7.ExampleCase testIncrement_Success]' started.
Test Case '-[__lldb_expr_7.ExampleCase testIncrement_Success]' passed (0.005 seconds).
Test Suite 'ExampleCase' passed at 2020-01-21 16:34:42.674.
	 Executed 1 test, with 0 failures (0 unexpected) in 0.005 (0.006) seconds
🔅 Test Suite 'ExampleCase' finished successfully.

In this screenshot, you can see the code locally executed together with the output of nef markdown

[miguelangel-dev]

This issue will be continued on #176