Multiple test/experiment

[peterfoeng]

Hi,

I am wondering how can we setup multiple experiments with multiple variants, is this possible? I am trying out the following code but only one cookie is being setup:

var abHomepageExperiment = require('express-ab').test('Homepage', {cookie: {name: 'ABHomepage'}});
app.use(abHomepageExperiment ('A', 0.1));
app.use(abHomepageExperiment ('B', 0.9));

var abDetails = require('express-ab').test('Details', {cookie: {name: 'ABDetails'}});
app.use(abDetails ('C', 0.1));
app.use(abDetails ('D', 0.9));

Many thanks!

[omichelsen]

Interesting example, haven't used it for all requests like this before. The problem is, that once you have been put into a test group for a particular request, it will not also put you in another test.

In this case, because you are running these for all requests, it can only be in one. Also because the percentage is 100% for the first test, no-one will ever be put in your second test.

The original idea for this library was to switch between routes, but I like your use case of running some general tests across all routes. Will probably need to add this as a feature though. Input is welcome :)

[rubik]

@peterfoeng @omichelsen We are running multivariate tests with this library, and it's working well. If you are running multiple tests and they interfere with one another, you need to set the experiment up as a MVT. Otherwise the interactions are not calculated properly.

We do that with express-ab by initializing all the combinations in an object, like this:

    this.core.MVT = ab.test("search-pricing", {id: "<optimize-id>"});
    this.core.variants = {
      "A:original-search,original-pricing": "0-0",
      "B:original-search,custom-pricing": "0-1",
      "C:first-search,original-pricing": "1-0",
      "D:first-search,custom-pricing": "1-1",
      "E:second-search,original-pricing": "2-0",
      "F:second-search,custom-pricing": "2-1",
    };

In this example we have three variants for the first test and two variants for the second. Then, for each route in the experiment, we instantiate it for all the variants:

  for (let i in this.core.variants) {
    this.core.app.get("/", this.core.MVT(i), function(req, res) {
      // bla bla
    });
  }

Inside each route, we retrieve the variant ID from res.locals.ab and we pass the variant data to the client.