Home

The matter we are trying to solve is fundamentally simple:

Create an API that mimics the features and functionality of the Stripe API in a deterministic manner regardless of the parallelism of the tests.

The reason we cannot run tests against the official Stripe API is because it will not provide a deterministic environment for testing. If two tests occur at the same time, there is no guarantee about the data and the tests may provide incorrect results.

The API we endeavor to provide is composed of two parts: the library API and the HTTP API.

The library API is a programmatic library that allows users to establish the state of the system prior to testing functionality. The library API manipulates the data-under-test within the bounds of the execution environment (i.e., it never makes an HTTP call).

The HTTP API is a replica of the Stripe API, but without validation logic. Validation errors are served a pre-determined set of parameters are used. These parameters are the ones described in the Stripe Testing document. This keeps the API deterministic about how it executes. Further parameters may be created to trigger error responses that are not otherwise covered by the Stripe document.

The data system is partitioned—we will refer to partitions interchangeably with instances—in order to allow for parallel testing, and each of the APIs must specify which partition they are targeting. The library API does this by taking a struct representing the instance as the first parameter of the majority of its calls. The instance struct contains a pointer to the process managing the data partition. The HTTP API instead uses a port number to identify the data partition. When an instance is started, it binds to a random port to serve the HTTP API. This port always uses the data partition for its instance.

Through partitioning, we don't have to worry about multiple tests running at the same time. If partition A and partition B are created, a customer is inserted into partition A, and then all customers are listed for partition B, the result from partition B is an empty list. The partitions are completely isolated from each other, so changing one does not change any other. This is essential to deterministic testing. If these partitions weren't isolated, one could never guarantee that a test is succeeding based on its logic alone.

The way we accomplish partitioning is by booting a new data manager process for each partition. The data manager should most likely be a GenServer, though there are other ways to accomplish it. When a user wants to test against the API, the user starts a new process using a library call: FakeStripe.new/0. This call will return a %FakeStripe{} struct that contains a reference to the data manager that the library can use later (probably a PID) and a port number that can be used with the HTTP API.

The user must maintain the struct for the duration of the test. It is essential to any calls made with the library API. It is expected that the user will use the library API to setup the state of the test while the actual functionality under test will use the HTTP API. When the test suite has finished running against a particular instance, the instance should be stopped using FakeStripe.stop/1, passing the struct as the parameter. This shuts down all the relevant processes and unbinds from the network port.

Miscellaneous Points

• The library has initially used a method which creates a new Plug instance by starting a cowboy handler and than passing it to Plug. This was done because it was how Bypass worked. However, after much thought, it would probably be better to use Phoenix for this because ultimately we will end up re-creating a lot of Phoenix's work.
• The HTTP API is designed to identify the data partition using a port. This requires every parallel test to start a new instance which in turns binds to a random port. Under most situations this should be fine, but ideally we should be able to have a single server running on a single port. The way that this could happen is to move partition identification to the API key. That way, when a new instance is started, it doesn't start a new socket/cowboy handler/etc…it just generates an unique API key that identifies the partition it is linked with.