Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

erupt basics update: upload new version #309

Merged
merged 2 commits into from
Aug 20, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion notebooks/ERUPT basics.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,27 @@
}
},
"source": [
"# ERUPT under simulated random assignment"
"# Combining exploration and exploitation"
]
},
{
"cell_type": "markdown",
"id": "4ab88c3f-d9d9-4d91-809d-afa60d62c521",
"metadata": {},
"source": [
"Have you ever wanted to conduct A/B/N tests in a case where you had a prior opinion on which variant could be better suited to which customer?\n",
"\n",
"Were you torn between exploration (learning from A/B/N test results) and exploitation (showing each customer the variant you already think is most suited for them)?\n",
"\n",
"Did you know that you can do both at once? \n",
"\n",
"By making variant assignments random but biased towards what you think will work best for each customer, you can get the best of both worlds. You can observe the actual impact (for example, conversion rates) of your biased assignments and at the same time calculate, *from the same experiment*, an unbiased, model-free estimate of what the conversion rate for a purely random assignment would have been, using a mathematical technique called ERUPT or policy value. This means that beyond the usual learnings from an A/B/N test you now have a precise, unbiased estimate of the benefit that biasing the assignment has brought. \n",
"\n",
"Suppose you're not really sure about your prior beliefs. In that case, you can also turn this around: run a fully randomized experiment, then use ERUPT to calculate from that experiment an unbiased estimate of what the impact of _any other assignment policy_ would have been! \n",
"\n",
"Summing up, if your actual variant assignment policy is at all stochastic, whether fully random or with a probability depending on customer's characteristics, after running the experiment you can use ERUPT to get an unbiased estimate of what the outcome of ANY OTHER ASSIGNMENT POLICY for the same experiment would have been! \n",
"\n",
"This notebook shows you how."
]
},
{
Expand Down
Loading