jock: don't run tests repeatedly

[marcusmiguel]

test-jocks runs at every step of the loop, causing all tests to be executed repeatedly. This PR addresses the issue by wrapping them in traps and ensuring each test runs only once.

[tacryt-socryp]

Hi, great find! If you just preface the ^- (list ...) statement with the ~+ rune, it won't re-execute every time because the expression will be memoized. A bit cleaner, but the same idea!

[marcusmiguel]

Actually, memoization was the first solution I tried, but it didn't work (and still doesn't) for some reason. I attempted placing ~+ at the beginning of +test-jocks and at the call site (e.g., (lent ~+(test-jocks))).

One potential issue with memoization is that if one test crashes at runtime, the other tests won't run, which may or may not be the desired behavior.

[marcusmiguel]

A cleaner solution might be dynamically importing all tests from that folder (perhaps bringing the /~ rune to %choo).

[eamsden]

Actually, memoization was the first solution I tried, but it didn't work (and still doesn't) for some reason. I attempted placing ~+ at the beginning of +test-jocks and at the call site (e.g., (lent ~+(test-jocks))).

If there is any change in the subject then that will break memoization, since the memoization key is the pair of subject and formula.

Very often an explicit tisgar which brings into scope only and exactly what's necessary will fix memoization.

@marcusmiguel

[sigilante]

I'm interested in /~ affordance, but need to figure out the plumbing. Rust side knows about the filesystem but NockApp doesn't.

[tacryt-socryp]

@sigilante I'd like to wait on merging this until we have ~+ working here, for simplicity.

[marcusmiguel]

Very often an explicit tisgar which brings into scope only and exactly what's necessary will fix memoization.

Thanks, @eamsden! I didn’t know about this. However:

Even if I simplify the subject to be:

++  test-jocks:
  =>  test-let-edit
  ~+
  ~&  .    ::  <test-let-edit-arms [* <hoon-compiler>]>
  ^-  (list [term tang])
  :~  [%test-let-edit-tokens test-tokenize]
      [%test-let-edit-jeam test-jeam]
      [%test-let-edit-mint test-mint]
  ==

and test-let-edit to be:

  |%
  ++  test-tokenize
    ~&  'Calling +test-jocks always causes this to run'
    [%foo *tang]
  ++  test-jeam
    ~&  'And this'
    [%bar *tang]
  ++  test-mint
    ~&  'This one too...'
    [%baz *tang]
  --

It still doesn’t memoize properly.

[marcusmiguel]

Just realized that ~+ works if you call +test-jocks outside +mole. I will go ahead and remove virtualization, then.

[sigilante]

So, to clarify, ~+ is supposed to be working as of this latest commit?

[marcusmiguel]

That's right, I removed +mole and now it works.

[sigilante]

@marcusmiguel status update: after I fix List/Set syntax and make proper recursive types, then I'll circle back around to fixing the whole NockApp testing framework and incorporate this work.

[sigilante]

It seems to me that we want to keep +mole if we aren't sure if tests will work (during development).

[marcusmiguel]

I don't think that this PR changes anything except for making +test-all run much faster and printing individual test results during the loop (it's empty when it succeeds). It only affects +test-all, and if a single test crashes, the entire event will crash—even with virtualization—because we run +test-jocks once outside +mole.

@sigilante