Test suite against the RPC API

[aatifsyed]

Background

Lotus does not test its RPC API.

Forest performs comparison tests, which generates calls, and checks that the response from Forest matches the one from Lotus.

Motivation

As part of the work around the Common Node API, we'd love to allow users/implementers to:

• Validate that the JSON that they work with matches the OpenRPC.
  • The tooling for this work is mostly complete
• Validate the implementation of an RPC endpoint - that's what this discussion will be about.

We can assume that Systems Under Test will have "stateless modes", where filecoin state can be loaded, and isolated from the current network.

The overall goal is for developers to write "Given, When, Then" style tests.
See also https://www.youtube.com/watch?v=PmIf77thO_c

[aatifsyed]

Should the harness care about the OpenRPC spec or not?
i.e should calls be written like this?

call("Filecoin.CountYaks", [1, "foo"])

Or this?

call(CountYaks(count=1, name="foo"))

[aatifsyed]

I think the distinction that I'm trying to get across is types - the first API is untyped, slinging raw json:

1. Easier to implement
2. Easier to prototype if you're e.g writing a new method

The second API is typed, i.e there is some struct in the harness that corresponds to the schema in the spec.

1. Requires codegen to the harness
2. Is more robust
3. "Spec first" writing style

[rvagg]

I do like descriptive, "count" and "name" showing up in test descriptors is pretty nice rather than raw values. But do we have tooling to help with this already or is this a from-scratch exercise?

[rvagg]

fwiw I've been tinkering with an extension to the IPLD Schema language that gets us a bit closer to being able to codegen proper types in the various languages one might care about. Current version of it here is aiming to be able to generate output that might be consumed by go-state-types and builtin-actors: https://github.com/ipld/js-ipld-schema/blob/rvagg/codegen/test/fixtures/gen/filecoin_miner_types.md - is a test fixture that has the schema at the top and the codegen'd types below (PR WIP here). I'm not sure how useful this ends up being on the other side of an OpenRPC API/spec but perhaps there's some overlap we could consider.

[aatifsyed]

Rod, I've written an MVP test harness here: ChainSafe/filecoin-common-node-api#20
Do you want to take it for a spin?

[rvagg]

Cool, I have some questions but I'll open an issue over there about it.

[aatifsyed]

What (if any) language should we choose? Go? Rust? Python? Typescript?

[rvagg]

-1 for Python for me, just because the barrier is larger. Otherwise I don't mind too much and perhaps it depends on available tooling. JavaScript and/or TypeScript may be a good option for tooling, flexibility and expressiveness, but it may limit the number of contributors given that we're in a Go+Rust ecosystem. So one of those two would probably be best.

[aatifsyed]

Can you think of some sample tests that the harness should handle?
e.g:

• push some messages to the Mpool, and estimate gas

[rvagg]

I imagine that mostly this will be run in CI, against a devnet with a particular pre-loaded state. So we should be able to make queries to all of the APIs in the the "Common Node API" spec and get ~predictable results. I wouldn't also mine having something that we can also leverage to make queries against live networks, including mainnet, such that we can probe for current state and make a chain of queries against that and have some broad expectations for the state. For example - ask for the latest head, walk back a few epochs and query a particular tipset for its messages, events, and walk through the state using the various ways of inspecting state and do some cross-referencing of results where there's overlap.

We've been discussing having something like for ongoing performance and conformance testing, how can we achieve this against mainnet, or calibnet, i.e. more than just a devenet with predictable state? So if we have a framework for making calls just to do basic conformance against a devnet instance, it would be neat to be able to also use it to do more sophsticated things.