13 - PSI and Moose - Encrypted Computation for Data Sharing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Private Set Intersection and Moose\n",
    "\n",
    "This notebook takes a common use case: sharing data between several parties in order to compute something together.\n",
    "\n",
    "In this case, you will use [Facebook's PrivateID library](https://github.com/facebookresearch/Private-ID#private-id-1) to compute a join between the two datasets and then [Moose from the tf-encrypted team](https://github.com/tf-encrypted/moose) to compute the values using encrypted data.\n",
    "\n",
    "For this particular example, the players are two companies with some overlap in customers. They would like to find customers that are of interest for new product offerings based on the spending categories of their combined customer bases. They can then use this result to determine what product offerings would be most appealing to which customers and reach out for more customer research.\n",
    "\n",
    "This notebook was heavily based on [Yann Dupis' notebooks](https://github.com/yanndupis) with additional support and Moose fixes from [Jason Mancuso](https://github.com/jvmncs). I encourage you to follow their work on GitHub to keep up with the latest!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate fake datasets\n",
    "\n",
    "In this example, you'll have Alice and Bob play the two data scientists at each company. They need to first have realistic data, so here you'll generate some example data. Feel free to play around with the generator to test out new ideas!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "np.random.seed(1234)\n",
    "\n",
    "_DATA_DIR = pathlib.Path(\"./data\")\n",
    "\n",
    "def generate_mock_dataset(sample_size, n_features, sample_frac, \n",
    "                          scaler=100, positive=True, precision=4):\n",
    "    x =  np.random.randn(sample_size, n_features).round(precision) * scaler\n",
    "    user_id = np.array(range(sample_size))\n",
    "    df = pd.DataFrame(x, columns=[f\"x_{i}\" for i in range(n_features)])\n",
    "    df.insert(loc=0, column='user_id', value=user_id)\n",
    "    df =  df.sample(frac=sample_frac)\n",
    "    if positive:\n",
    "        df = df.applymap(lambda x: x * -1 if x < 0 else x)\n",
    "    return df\n",
    "\n",
    "alice_df = generate_mock_dataset(10, 20, 0.6)\n",
    "bob_df = generate_mock_dataset(10, 1, 0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>x_0</th>\n",
       "      <th>x_1</th>\n",
       "      <th>x_2</th>\n",
       "      <th>x_3</th>\n",
       "      <th>x_4</th>\n",
       "      <th>x_5</th>\n",
       "      <th>x_6</th>\n",
       "      <th>x_7</th>\n",
       "      <th>x_8</th>\n",
       "      <th>...</th>\n",
       "      <th>x_10</th>\n",
       "      <th>x_11</th>\n",
       "      <th>x_12</th>\n",
       "      <th>x_13</th>\n",
       "      <th>x_14</th>\n",
       "      <th>x_15</th>\n",
       "      <th>x_16</th>\n",
       "      <th>x_17</th>\n",
       "      <th>x_18</th>\n",
       "      <th>x_19</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>20.26</td>\n",
       "      <td>65.60</td>\n",
       "      <td>19.34</td>\n",
       "      <td>55.34</td>\n",
       "      <td>131.82</td>\n",
       "      <td>46.93</td>\n",
       "      <td>67.56</td>\n",
       "      <td>181.70</td>\n",
       "      <td>18.31</td>\n",
       "      <td>...</td>\n",
       "      <td>39.78</td>\n",
       "      <td>33.74</td>\n",
       "      <td>104.76</td>\n",
       "      <td>104.59</td>\n",
       "      <td>86.37</td>\n",
       "      <td>12.21</td>\n",
       "      <td>12.47</td>\n",
       "      <td>32.28</td>\n",
       "      <td>84.17</td>\n",
       "      <td>239.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>47.14</td>\n",
       "      <td>119.10</td>\n",
       "      <td>143.27</td>\n",
       "      <td>31.27</td>\n",
       "      <td>72.06</td>\n",
       "      <td>88.72</td>\n",
       "      <td>85.96</td>\n",
       "      <td>63.65</td>\n",
       "      <td>1.57</td>\n",
       "      <td>...</td>\n",
       "      <td>115.00</td>\n",
       "      <td>99.19</td>\n",
       "      <td>95.33</td>\n",
       "      <td>202.13</td>\n",
       "      <td>33.41</td>\n",
       "      <td>0.21</td>\n",
       "      <td>40.55</td>\n",
       "      <td>28.91</td>\n",
       "      <td>132.12</td>\n",
       "      <td>154.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>53.68</td>\n",
       "      <td>74.38</td>\n",
       "      <td>32.02</td>\n",
       "      <td>91.62</td>\n",
       "      <td>85.97</td>\n",
       "      <td>22.60</td>\n",
       "      <td>62.88</td>\n",
       "      <td>18.65</td>\n",
       "      <td>95.25</td>\n",
       "      <td>...</td>\n",
       "      <td>7.26</td>\n",
       "      <td>55.06</td>\n",
       "      <td>93.82</td>\n",
       "      <td>123.91</td>\n",
       "      <td>13.97</td>\n",
       "      <td>22.30</td>\n",
       "      <td>212.37</td>\n",
       "      <td>12.23</td>\n",
       "      <td>140.94</td>\n",
       "      <td>142.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>7.62</td>\n",
       "      <td>56.64</td>\n",
       "      <td>3.61</td>\n",
       "      <td>207.50</td>\n",
       "      <td>24.78</td>\n",
       "      <td>89.72</td>\n",
       "      <td>13.68</td>\n",
       "      <td>1.83</td>\n",
       "      <td>75.54</td>\n",
       "      <td>...</td>\n",
       "      <td>84.10</td>\n",
       "      <td>144.58</td>\n",
       "      <td>140.20</td>\n",
       "      <td>10.09</td>\n",
       "      <td>54.82</td>\n",
       "      <td>14.46</td>\n",
       "      <td>35.40</td>\n",
       "      <td>3.55</td>\n",
       "      <td>56.57</td>\n",
       "      <td>154.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>29.12</td>\n",
       "      <td>56.65</td>\n",
       "      <td>50.36</td>\n",
       "      <td>28.53</td>\n",
       "      <td>48.43</td>\n",
       "      <td>136.35</td>\n",
       "      <td>78.11</td>\n",
       "      <td>46.80</td>\n",
       "      <td>122.46</td>\n",
       "      <td>...</td>\n",
       "      <td>87.55</td>\n",
       "      <td>171.07</td>\n",
       "      <td>45.08</td>\n",
       "      <td>74.92</td>\n",
       "      <td>20.39</td>\n",
       "      <td>18.22</td>\n",
       "      <td>68.07</td>\n",
       "      <td>181.85</td>\n",
       "      <td>4.71</td>\n",
       "      <td>39.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>46.44</td>\n",
       "      <td>356.35</td>\n",
       "      <td>132.11</td>\n",
       "      <td>15.26</td>\n",
       "      <td>16.45</td>\n",
       "      <td>43.01</td>\n",
       "      <td>76.74</td>\n",
       "      <td>98.49</td>\n",
       "      <td>27.08</td>\n",
       "      <td>...</td>\n",
       "      <td>7.98</td>\n",
       "      <td>40.00</td>\n",
       "      <td>102.79</td>\n",
       "      <td>58.47</td>\n",
       "      <td>81.66</td>\n",
       "      <td>8.19</td>\n",
       "      <td>34.48</td>\n",
       "      <td>52.83</td>\n",
       "      <td>106.90</td>\n",
       "      <td>51.19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id    x_0     x_1     x_2     x_3     x_4     x_5    x_6     x_7  \\\n",
       "1        1  20.26   65.60   19.34   55.34  131.82   46.93  67.56  181.70   \n",
       "0        0  47.14  119.10  143.27   31.27   72.06   88.72  85.96   63.65   \n",
       "8        8  53.68   74.38   32.02   91.62   85.97   22.60  62.88   18.65   \n",
       "2        2   7.62   56.64    3.61  207.50   24.78   89.72  13.68    1.83   \n",
       "5        5  29.12   56.65   50.36   28.53   48.43  136.35  78.11   46.80   \n",
       "4        4  46.44  356.35  132.11   15.26   16.45   43.01  76.74   98.49   \n",
       "\n",
       "      x_8  ...    x_10    x_11    x_12    x_13   x_14   x_15    x_16    x_17  \\\n",
       "1   18.31  ...   39.78   33.74  104.76  104.59  86.37  12.21   12.47   32.28   \n",
       "0    1.57  ...  115.00   99.19   95.33  202.13  33.41   0.21   40.55   28.91   \n",
       "8   95.25  ...    7.26   55.06   93.82  123.91  13.97  22.30  212.37   12.23   \n",
       "2   75.54  ...   84.10  144.58  140.20   10.09  54.82  14.46   35.40    3.55   \n",
       "5  122.46  ...   87.55  171.07   45.08   74.92  20.39  18.22   68.07  181.85   \n",
       "4   27.08  ...    7.98   40.00  102.79   58.47  81.66   8.19   34.48   52.83   \n",
       "\n",
       "     x_18    x_19  \n",
       "1   84.17  239.10  \n",
       "0  132.12  154.69  \n",
       "8  140.94  142.30  \n",
       "2   56.57  154.57  \n",
       "5    4.71   39.48  \n",
       "4  106.90   51.19  \n",
       "\n",
       "[6 rows x 21 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alice_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Private-ID\n",
    "\n",
    "Now it's time to run [Private-ID](https://github.com/facebookresearch/Private-ID#private-id-1) on Alice and Bob's User IDs. You will want to install Private-ID in a parent folder near this repository and run these commands from the location where you installed Private ID. Make sure to update the input and output file paths should your location be different!\n",
    "\n",
    "```\n",
    "env RUST_LOG=info cargo run --bin private-id-server -- \\\n",
    "    --host 0.0.0.0:10009 \\\n",
    "    --input ../practical-data-privacy/data/alice_id.csv \\\n",
    "    --output ../practical-data-privacy/data/alice_keys.csv \\\n",
    "    --no-tls\n",
    "    \n",
    "\n",
    "env RUST_LOG=info cargo run --bin private-id-client -- \\\n",
    "    --company localhost:10009 \\\n",
    "    --input ../practical-data-privacy/data/bob_id.csv \\\n",
    "    --output ../practical-data-privacy/data/bob_keys.csv  \\\n",
    "    --no-tls\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "alice_id = alice_df[\"user_id\"]\n",
    "alice_id.to_csv(_DATA_DIR / \"alice_id.csv\", index=False, header=False)\n",
    "\n",
    "bob_id = bob_df[\"user_id\"]\n",
    "bob_id.to_csv(_DATA_DIR / \"bob_id.csv\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is below the outputs from Private-ID. As you can see they both contain the same list of keys ordered the same way but for some of them they have a User Id mapped to them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>92E45747889DB6D24755537D369BF68D384DC9549F1D77...</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 key  user_id\n",
       "0  144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...      2.0\n",
       "1  5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...      8.0\n",
       "2  72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...      1.0\n",
       "3  72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...      4.0\n",
       "4  8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...      NaN\n",
       "5  92E45747889DB6D24755537D369BF68D384DC9549F1D77...      5.0\n",
       "6  F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...      NaN\n",
       "7  F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...      0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alice_keys = pd.read_csv(_DATA_DIR / \"alice_keys.csv\", names=[\"key\", \"user_id\"])\n",
    "\n",
    "alice_keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>92E45747889DB6D24755537D369BF68D384DC9549F1D77...</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 key  user_id\n",
       "0  144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...      2.0\n",
       "1  5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...      8.0\n",
       "2  72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...      1.0\n",
       "3  72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...      NaN\n",
       "4  8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...      7.0\n",
       "5  92E45747889DB6D24755537D369BF68D384DC9549F1D77...      5.0\n",
       "6  F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...      9.0\n",
       "7  F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...      0.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bob_keys = pd.read_csv(_DATA_DIR / \"bob_keys.csv\", names=[\"key\", \"user_id\"])\n",
    "\n",
    "bob_keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then merge the Private-ID output to Alice and Bob's datasets so in the datasets we have the all the keys from Private-ID with the corresponding User ID and features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "alice_df = pd.merge(alice_keys, alice_df, on='user_id', how='left')\n",
    "bob_df = pd.merge(bob_keys, bob_df, on='user_id', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>user_id</th>\n",
       "      <th>x_0</th>\n",
       "      <th>x_1</th>\n",
       "      <th>x_2</th>\n",
       "      <th>x_3</th>\n",
       "      <th>x_4</th>\n",
       "      <th>x_5</th>\n",
       "      <th>x_6</th>\n",
       "      <th>x_7</th>\n",
       "      <th>...</th>\n",
       "      <th>x_10</th>\n",
       "      <th>x_11</th>\n",
       "      <th>x_12</th>\n",
       "      <th>x_13</th>\n",
       "      <th>x_14</th>\n",
       "      <th>x_15</th>\n",
       "      <th>x_16</th>\n",
       "      <th>x_17</th>\n",
       "      <th>x_18</th>\n",
       "      <th>x_19</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.62</td>\n",
       "      <td>56.64</td>\n",
       "      <td>3.61</td>\n",
       "      <td>207.50</td>\n",
       "      <td>24.78</td>\n",
       "      <td>89.72</td>\n",
       "      <td>13.68</td>\n",
       "      <td>1.83</td>\n",
       "      <td>...</td>\n",
       "      <td>84.10</td>\n",
       "      <td>144.58</td>\n",
       "      <td>140.20</td>\n",
       "      <td>10.09</td>\n",
       "      <td>54.82</td>\n",
       "      <td>14.46</td>\n",
       "      <td>35.40</td>\n",
       "      <td>3.55</td>\n",
       "      <td>56.57</td>\n",
       "      <td>154.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>53.68</td>\n",
       "      <td>74.38</td>\n",
       "      <td>32.02</td>\n",
       "      <td>91.62</td>\n",
       "      <td>85.97</td>\n",
       "      <td>22.60</td>\n",
       "      <td>62.88</td>\n",
       "      <td>18.65</td>\n",
       "      <td>...</td>\n",
       "      <td>7.26</td>\n",
       "      <td>55.06</td>\n",
       "      <td>93.82</td>\n",
       "      <td>123.91</td>\n",
       "      <td>13.97</td>\n",
       "      <td>22.30</td>\n",
       "      <td>212.37</td>\n",
       "      <td>12.23</td>\n",
       "      <td>140.94</td>\n",
       "      <td>142.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>20.26</td>\n",
       "      <td>65.60</td>\n",
       "      <td>19.34</td>\n",
       "      <td>55.34</td>\n",
       "      <td>131.82</td>\n",
       "      <td>46.93</td>\n",
       "      <td>67.56</td>\n",
       "      <td>181.70</td>\n",
       "      <td>...</td>\n",
       "      <td>39.78</td>\n",
       "      <td>33.74</td>\n",
       "      <td>104.76</td>\n",
       "      <td>104.59</td>\n",
       "      <td>86.37</td>\n",
       "      <td>12.21</td>\n",
       "      <td>12.47</td>\n",
       "      <td>32.28</td>\n",
       "      <td>84.17</td>\n",
       "      <td>239.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>46.44</td>\n",
       "      <td>356.35</td>\n",
       "      <td>132.11</td>\n",
       "      <td>15.26</td>\n",
       "      <td>16.45</td>\n",
       "      <td>43.01</td>\n",
       "      <td>76.74</td>\n",
       "      <td>98.49</td>\n",
       "      <td>...</td>\n",
       "      <td>7.98</td>\n",
       "      <td>40.00</td>\n",
       "      <td>102.79</td>\n",
       "      <td>58.47</td>\n",
       "      <td>81.66</td>\n",
       "      <td>8.19</td>\n",
       "      <td>34.48</td>\n",
       "      <td>52.83</td>\n",
       "      <td>106.90</td>\n",
       "      <td>51.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>92E45747889DB6D24755537D369BF68D384DC9549F1D77...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>29.12</td>\n",
       "      <td>56.65</td>\n",
       "      <td>50.36</td>\n",
       "      <td>28.53</td>\n",
       "      <td>48.43</td>\n",
       "      <td>136.35</td>\n",
       "      <td>78.11</td>\n",
       "      <td>46.80</td>\n",
       "      <td>...</td>\n",
       "      <td>87.55</td>\n",
       "      <td>171.07</td>\n",
       "      <td>45.08</td>\n",
       "      <td>74.92</td>\n",
       "      <td>20.39</td>\n",
       "      <td>18.22</td>\n",
       "      <td>68.07</td>\n",
       "      <td>181.85</td>\n",
       "      <td>4.71</td>\n",
       "      <td>39.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>47.14</td>\n",
       "      <td>119.10</td>\n",
       "      <td>143.27</td>\n",
       "      <td>31.27</td>\n",
       "      <td>72.06</td>\n",
       "      <td>88.72</td>\n",
       "      <td>85.96</td>\n",
       "      <td>63.65</td>\n",
       "      <td>...</td>\n",
       "      <td>115.00</td>\n",
       "      <td>99.19</td>\n",
       "      <td>95.33</td>\n",
       "      <td>202.13</td>\n",
       "      <td>33.41</td>\n",
       "      <td>0.21</td>\n",
       "      <td>40.55</td>\n",
       "      <td>28.91</td>\n",
       "      <td>132.12</td>\n",
       "      <td>154.69</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 key  user_id    x_0     x_1  \\\n",
       "0  144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...      2.0   7.62   56.64   \n",
       "1  5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...      8.0  53.68   74.38   \n",
       "2  72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...      1.0  20.26   65.60   \n",
       "3  72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...      4.0  46.44  356.35   \n",
       "4  8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...      NaN    NaN     NaN   \n",
       "5  92E45747889DB6D24755537D369BF68D384DC9549F1D77...      5.0  29.12   56.65   \n",
       "6  F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...      NaN    NaN     NaN   \n",
       "7  F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...      0.0  47.14  119.10   \n",
       "\n",
       "      x_2     x_3     x_4     x_5    x_6     x_7  ...    x_10    x_11    x_12  \\\n",
       "0    3.61  207.50   24.78   89.72  13.68    1.83  ...   84.10  144.58  140.20   \n",
       "1   32.02   91.62   85.97   22.60  62.88   18.65  ...    7.26   55.06   93.82   \n",
       "2   19.34   55.34  131.82   46.93  67.56  181.70  ...   39.78   33.74  104.76   \n",
       "3  132.11   15.26   16.45   43.01  76.74   98.49  ...    7.98   40.00  102.79   \n",
       "4     NaN     NaN     NaN     NaN    NaN     NaN  ...     NaN     NaN     NaN   \n",
       "5   50.36   28.53   48.43  136.35  78.11   46.80  ...   87.55  171.07   45.08   \n",
       "6     NaN     NaN     NaN     NaN    NaN     NaN  ...     NaN     NaN     NaN   \n",
       "7  143.27   31.27   72.06   88.72  85.96   63.65  ...  115.00   99.19   95.33   \n",
       "\n",
       "     x_13   x_14   x_15    x_16    x_17    x_18    x_19  \n",
       "0   10.09  54.82  14.46   35.40    3.55   56.57  154.57  \n",
       "1  123.91  13.97  22.30  212.37   12.23  140.94  142.30  \n",
       "2  104.59  86.37  12.21   12.47   32.28   84.17  239.10  \n",
       "3   58.47  81.66   8.19   34.48   52.83  106.90   51.19  \n",
       "4     NaN    NaN    NaN     NaN     NaN     NaN     NaN  \n",
       "5   74.92  20.39  18.22   68.07  181.85    4.71   39.48  \n",
       "6     NaN    NaN    NaN     NaN     NaN     NaN     NaN  \n",
       "7  202.13  33.41   0.21   40.55   28.91  132.12  154.69  \n",
       "\n",
       "[8 rows x 22 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alice_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>user_id</th>\n",
       "      <th>x_0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>89.36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>142.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...</td>\n",
       "      <td>7.0</td>\n",
       "      <td>52.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>92E45747889DB6D24755537D369BF68D384DC9549F1D77...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>30.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>40.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>87.34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 key  user_id     x_0\n",
       "0  144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...      2.0    0.31\n",
       "1  5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...      8.0   89.36\n",
       "2  72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...      1.0  142.85\n",
       "3  72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...      NaN     NaN\n",
       "4  8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...      7.0   52.65\n",
       "5  92E45747889DB6D24755537D369BF68D384DC9549F1D77...      5.0   30.27\n",
       "6  F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...      9.0   40.11\n",
       "7  F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...      0.0   87.34"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bob_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In each dataset, we create a flag `user_id_available` by checking if the user id is missing or not. For the column feature x_0, we replace the missing values with 0. The value doesn't matter since these records will be filtered out when taking the intersection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>user_id</th>\n",
       "      <th>x_0</th>\n",
       "      <th>user_id_available</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>89.36</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>142.85</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...</td>\n",
       "      <td>7.0</td>\n",
       "      <td>52.65</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>92E45747889DB6D24755537D369BF68D384DC9549F1D77...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>30.27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>40.11</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>87.34</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 key  user_id     x_0  \\\n",
       "0  144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...      2.0    0.31   \n",
       "1  5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...      8.0   89.36   \n",
       "2  72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...      1.0  142.85   \n",
       "3  72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...      0.0    0.00   \n",
       "4  8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...      7.0   52.65   \n",
       "5  92E45747889DB6D24755537D369BF68D384DC9549F1D77...      5.0   30.27   \n",
       "6  F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...      9.0   40.11   \n",
       "7  F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...      0.0   87.34   \n",
       "\n",
       "   user_id_available  \n",
       "0                  1  \n",
       "1                  1  \n",
       "2                  1  \n",
       "3                  0  \n",
       "4                  1  \n",
       "5                  1  \n",
       "6                  1  \n",
       "7                  1  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bob_df['user_id_available'] = np.where(bob_df['user_id'].isnull(), 0, 1)\n",
    "bob_df = bob_df.fillna(0)\n",
    "bob_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>user_id</th>\n",
       "      <th>x_0</th>\n",
       "      <th>x_1</th>\n",
       "      <th>x_2</th>\n",
       "      <th>x_3</th>\n",
       "      <th>x_4</th>\n",
       "      <th>x_5</th>\n",
       "      <th>x_6</th>\n",
       "      <th>x_7</th>\n",
       "      <th>...</th>\n",
       "      <th>x_11</th>\n",
       "      <th>x_12</th>\n",
       "      <th>x_13</th>\n",
       "      <th>x_14</th>\n",
       "      <th>x_15</th>\n",
       "      <th>x_16</th>\n",
       "      <th>x_17</th>\n",
       "      <th>x_18</th>\n",
       "      <th>x_19</th>\n",
       "      <th>user_id_available</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.62</td>\n",
       "      <td>56.64</td>\n",
       "      <td>3.61</td>\n",
       "      <td>207.50</td>\n",
       "      <td>24.78</td>\n",
       "      <td>89.72</td>\n",
       "      <td>13.68</td>\n",
       "      <td>1.83</td>\n",
       "      <td>...</td>\n",
       "      <td>144.58</td>\n",
       "      <td>140.20</td>\n",
       "      <td>10.09</td>\n",
       "      <td>54.82</td>\n",
       "      <td>14.46</td>\n",
       "      <td>35.40</td>\n",
       "      <td>3.55</td>\n",
       "      <td>56.57</td>\n",
       "      <td>154.57</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>53.68</td>\n",
       "      <td>74.38</td>\n",
       "      <td>32.02</td>\n",
       "      <td>91.62</td>\n",
       "      <td>85.97</td>\n",
       "      <td>22.60</td>\n",
       "      <td>62.88</td>\n",
       "      <td>18.65</td>\n",
       "      <td>...</td>\n",
       "      <td>55.06</td>\n",
       "      <td>93.82</td>\n",
       "      <td>123.91</td>\n",
       "      <td>13.97</td>\n",
       "      <td>22.30</td>\n",
       "      <td>212.37</td>\n",
       "      <td>12.23</td>\n",
       "      <td>140.94</td>\n",
       "      <td>142.30</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>20.26</td>\n",
       "      <td>65.60</td>\n",
       "      <td>19.34</td>\n",
       "      <td>55.34</td>\n",
       "      <td>131.82</td>\n",
       "      <td>46.93</td>\n",
       "      <td>67.56</td>\n",
       "      <td>181.70</td>\n",
       "      <td>...</td>\n",
       "      <td>33.74</td>\n",
       "      <td>104.76</td>\n",
       "      <td>104.59</td>\n",
       "      <td>86.37</td>\n",
       "      <td>12.21</td>\n",
       "      <td>12.47</td>\n",
       "      <td>32.28</td>\n",
       "      <td>84.17</td>\n",
       "      <td>239.10</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>46.44</td>\n",
       "      <td>356.35</td>\n",
       "      <td>132.11</td>\n",
       "      <td>15.26</td>\n",
       "      <td>16.45</td>\n",
       "      <td>43.01</td>\n",
       "      <td>76.74</td>\n",
       "      <td>98.49</td>\n",
       "      <td>...</td>\n",
       "      <td>40.00</td>\n",
       "      <td>102.79</td>\n",
       "      <td>58.47</td>\n",
       "      <td>81.66</td>\n",
       "      <td>8.19</td>\n",
       "      <td>34.48</td>\n",
       "      <td>52.83</td>\n",
       "      <td>106.90</td>\n",
       "      <td>51.19</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>92E45747889DB6D24755537D369BF68D384DC9549F1D77...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>29.12</td>\n",
       "      <td>56.65</td>\n",
       "      <td>50.36</td>\n",
       "      <td>28.53</td>\n",
       "      <td>48.43</td>\n",
       "      <td>136.35</td>\n",
       "      <td>78.11</td>\n",
       "      <td>46.80</td>\n",
       "      <td>...</td>\n",
       "      <td>171.07</td>\n",
       "      <td>45.08</td>\n",
       "      <td>74.92</td>\n",
       "      <td>20.39</td>\n",
       "      <td>18.22</td>\n",
       "      <td>68.07</td>\n",
       "      <td>181.85</td>\n",
       "      <td>4.71</td>\n",
       "      <td>39.48</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>47.14</td>\n",
       "      <td>119.10</td>\n",
       "      <td>143.27</td>\n",
       "      <td>31.27</td>\n",
       "      <td>72.06</td>\n",
       "      <td>88.72</td>\n",
       "      <td>85.96</td>\n",
       "      <td>63.65</td>\n",
       "      <td>...</td>\n",
       "      <td>99.19</td>\n",
       "      <td>95.33</td>\n",
       "      <td>202.13</td>\n",
       "      <td>33.41</td>\n",
       "      <td>0.21</td>\n",
       "      <td>40.55</td>\n",
       "      <td>28.91</td>\n",
       "      <td>132.12</td>\n",
       "      <td>154.69</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 key  user_id    x_0     x_1  \\\n",
       "0  144418D81867C352A318B3C8BE1FC84BE959FC51FADF4E...      2.0   7.62   56.64   \n",
       "1  5CDC4E5348AA5E4DECC5F0AAB4DFB73D09C2BD4E4ED28B...      8.0  53.68   74.38   \n",
       "2  72A1523C8A9575605E886810AAB545E9957EB22DE5CCBC...      1.0  20.26   65.60   \n",
       "3  72F5125B88854DDAF88DA9D225A348AC8B4E3DCE6F097F...      4.0  46.44  356.35   \n",
       "4  8A623CCEE51534754EAE9D65F66ADE66BF51E241783FDA...      0.0   0.00    0.00   \n",
       "5  92E45747889DB6D24755537D369BF68D384DC9549F1D77...      5.0  29.12   56.65   \n",
       "6  F225409B2CBC3F22CD8EBCACDE75447816E81BD9DB363E...      0.0   0.00    0.00   \n",
       "7  F438B71D793E7CD9C2A6A0DF5B0AE501F84833525237A2...      0.0  47.14  119.10   \n",
       "\n",
       "      x_2     x_3     x_4     x_5    x_6     x_7  ...    x_11    x_12    x_13  \\\n",
       "0    3.61  207.50   24.78   89.72  13.68    1.83  ...  144.58  140.20   10.09   \n",
       "1   32.02   91.62   85.97   22.60  62.88   18.65  ...   55.06   93.82  123.91   \n",
       "2   19.34   55.34  131.82   46.93  67.56  181.70  ...   33.74  104.76  104.59   \n",
       "3  132.11   15.26   16.45   43.01  76.74   98.49  ...   40.00  102.79   58.47   \n",
       "4    0.00    0.00    0.00    0.00   0.00    0.00  ...    0.00    0.00    0.00   \n",
       "5   50.36   28.53   48.43  136.35  78.11   46.80  ...  171.07   45.08   74.92   \n",
       "6    0.00    0.00    0.00    0.00   0.00    0.00  ...    0.00    0.00    0.00   \n",
       "7  143.27   31.27   72.06   88.72  85.96   63.65  ...   99.19   95.33  202.13   \n",
       "\n",
       "    x_14   x_15    x_16    x_17    x_18    x_19  user_id_available  \n",
       "0  54.82  14.46   35.40    3.55   56.57  154.57                  1  \n",
       "1  13.97  22.30  212.37   12.23  140.94  142.30                  1  \n",
       "2  86.37  12.21   12.47   32.28   84.17  239.10                  1  \n",
       "3  81.66   8.19   34.48   52.83  106.90   51.19                  1  \n",
       "4   0.00   0.00    0.00    0.00    0.00    0.00                  0  \n",
       "5  20.39  18.22   68.07  181.85    4.71   39.48                  1  \n",
       "6   0.00   0.00    0.00    0.00    0.00    0.00                  0  \n",
       "7  33.41   0.21   40.55   28.91  132.12  154.69                  1  \n",
       "\n",
       "[8 rows x 23 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alice_df['user_id_available'] = np.where(alice_df['user_id'].isnull(), 0, 1)\n",
    "alice_df = alice_df.fillna(0)\n",
    "alice_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Moose Computation\n",
    "\n",
    "To the Moose computation we will feed Alice's `user_id_available` flag and feature and Bob's `user_id_available` flag and feature.\n",
    "\n",
    "We do the intersection using a logical And operation between `user_id_available` flags from Alice and Bob, filter the datasets where the logical And returns 1, then compute a metric privately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_id_available_a = alice_df.user_id_available.values\n",
    "user_id_available_b = bob_df.user_id_available.values\n",
    "\n",
    "alice_features_for_computation = alice_df.drop(['key', 'user_id', 'user_id_available'], axis=1)\n",
    "bob_features_for_computation = bob_df.drop(['key', 'user_id', 'user_id_available'], axis=1)\n",
    "\n",
    "x_a = alice_features_for_computation.values\n",
    "x_b = bob_features_for_computation.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(_DATA_DIR / \"x_a\", x_a)\n",
    "np.save(_DATA_DIR / \"x_b\", x_b)\n",
    "np.save(_DATA_DIR / \"user_id_available_a\", user_id_available_a)\n",
    "np.save(_DATA_DIR / \"user_id_available_b\", user_id_available_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymoose as pm\n",
    "\n",
    "FIXED = pm.fixed(24, 40)\n",
    "\n",
    "alice = pm.host_placement(name=\"alice\")\n",
    "bob = pm.host_placement(name=\"bob\")\n",
    "carole = pm.host_placement(name=\"carole\")\n",
    "rep = pm.replicated_placement(name=\"rep\", players=[alice, bob, carole])\n",
    "mirrored = pm.mirrored_placement(name=\"mirrored\", players=[alice, bob, carole])\n",
    "\n",
    "@pm.computation\n",
    "def psi_and_agg():    \n",
    "    with alice:\n",
    "        x_a = pm.load(\"x_a\", dtype=pm.float64)\n",
    "        user_id_available_a = pm.load(\"user_id_available_a\", dtype=pm.bool_)\n",
    "\n",
    "    with bob:\n",
    "        x_b = pm.load(\"x_b\", dtype=pm.float64)\n",
    "        user_id_available_b = pm.load(\"user_id_available_b\", dtype=pm.bool_)\n",
    "\n",
    "        # # Compute logical And between user_id_available from Alice and Bob.\n",
    "        # # If it returns 1, it means the User ID was in Alice and Bob's datasets\n",
    "        exist_in_alice_and_bob_bool = pm.logical_and(\n",
    "            user_id_available_a, user_id_available_b\n",
    "        )\n",
    "\n",
    "        # # Filter Bob's feature to keep only records where exist_in_alice_and_bob_bool returned 1\n",
    "        x_b_sub = pm.select(x_b, axis=0, index=exist_in_alice_and_bob_bool)\n",
    "        x_b_sub = pm.cast(x_b_sub, dtype=FIXED)\n",
    "\n",
    "    with alice:\n",
    "        # Filter Alice's feature to keep only records where exist_in_alice_and_bob_bool returned 1\n",
    "        x_a_sub = pm.select(x_a,  axis=0, index=exist_in_alice_and_bob_bool)\n",
    "        x_a_sub = pm.cast(x_a_sub, dtype=FIXED)\n",
    "\n",
    "    with mirrored:\n",
    "        ten_percent = pm.constant(0.1, dtype=FIXED)\n",
    "        \n",
    "    with rep:\n",
    "        # Aggregation: average ratio between sum of x_a_sub & x_b_sub\n",
    "        spend_per_category = x_a_sub + x_b_sub        \n",
    "        spend_per_user = pm.sum(spend_per_category, axis=1)\n",
    "        category_percent = spend_per_category / pm.expand_dims(spend_per_user, axis=1)\n",
    "        res = pm.greater(category_percent, ten_percent)\n",
    "\n",
    "    with alice:\n",
    "        res = pm.cast(res, dtype=pm.float64)\n",
    "        res = pm.save(\"agg_result\", res)\n",
    "\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Aggregation result with Moose [[0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0. 0. 0. 0. 0. 0. 1.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "executors_storage = {\n",
    "    \"alice\": {\"x_a\": x_a, \"user_id_available_a\": user_id_available_a.astype(np.bool_)},\n",
    "    \"bob\": {\"x_b\": x_b, \"user_id_available_b\": user_id_available_b.astype(np.bool_)}\n",
    "}\n",
    "\n",
    "runtime = pm.LocalMooseRuntime(\n",
    "    identities=[\"alice\", \"bob\", \"carole\"],\n",
    "    storage_mapping=executors_storage,\n",
    ")\n",
    "\n",
    "runtime.set_default()\n",
    "\n",
    "_ = psi_and_agg()\n",
    "\n",
    "agg_result = runtime.read_value_from_storage(\"alice\", \"agg_result\")\n",
    "print(\"Aggregation result with Moose\", agg_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In plaintext\n",
    "\n",
    "\n",
    "inner_bool = np.logical_and(user_id_available_a, user_id_available_b)\n",
    "x_a_sub = x_a[np.where(inner_bool==1)]\n",
    "x_b_sub = x_b[np.where(inner_bool==1)]\n",
    "\n",
    "spend_per_category = x_a_sub + x_b_sub\n",
    "spend_per_user = spend_per_category.sum(axis=1)\n",
    "category_percent = spend_per_category / np.expand_dims(spend_per_user, axis=1)\n",
    "plaintext_result = np.greater(category_percent, 0.1).astype(int) \n",
    "plaintext_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True],\n",
       "       [ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True],\n",
       "       [ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True],\n",
       "       [ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True],\n",
       "       [ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "         True,  True]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plaintext_result == agg_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenges\n",
    "\n",
    "- Adapt the computation so that it also returns the result to Bob\n",
    "- Try out a new computation -- what if they wanted to find the categories where their shared customers spend the least amount of money?\n",
    "- Add more rows and see how size affects the computation.\n",
    "- Try out another Private-ID or Moose computation by building your own example. Feel free to contribute examples to the reader-examples folder!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "fd7cee8bcd971dcf8831d6be0fcf4125937c319bc3875e6dd50d059cbfec2fbc"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}