diff --git a/notebooks/ERUPT basics.ipynb b/notebooks/ERUPT basics.ipynb index 03a7fcbb..df3d3bbb 100644 --- a/notebooks/ERUPT basics.ipynb +++ b/notebooks/ERUPT basics.ipynb @@ -10,7 +10,27 @@ } }, "source": [ - "# ERUPT under simulated random assignment" + "# Combining exploration and exploitation" + ] + }, + { + "cell_type": "markdown", + "id": "4ab88c3f-d9d9-4d91-809d-afa60d62c521", + "metadata": {}, + "source": [ + "Have you ever wanted to conduct A/B/N tests in a case where you had a prior opinion on which variant could be better suited to which customer?\n", + "\n", + "Were you torn between exploration (learning from A/B/N test results) and exploitation (showing each customer the variant you already think is most suited for them)?\n", + "\n", + "Did you know that you can do both at once? \n", + "\n", + "By making variant assignments random but biased towards what you think will work best for each customer, you can get the best of both worlds. You can observe the actual impact (for example, conversion rates) of your biased assignments and at the same time calculate, *from the same experiment*, an unbiased, model-free estimate of what the conversion rate for a purely random assignment would have been, using a mathematical technique called ERUPT or policy value. This means that beyond the usual learnings from an A/B/N test you now have a precise, unbiased estimate of the benefit that biasing the assignment has brought. \n", + "\n", + "Suppose you're not really sure about your prior beliefs. In that case, you can also turn this around: run a fully randomized experiment, then use ERUPT to calculate from that experiment an unbiased estimate of what the impact of _any other assignment policy_ would have been! \n", + "\n", + "Summing up, if your actual variant assignment policy is at all stochastic, whether fully random or with a probability depending on customer's characteristics, after running the experiment you can use ERUPT to get an unbiased estimate of what the outcome of ANY OTHER ASSIGNMENT POLICY for the same experiment would have been! \n", + "\n", + "This notebook shows you how." ] }, {