Stateful Crawl with CookieStore Sharing for Parallel Browsers

[MohammadMahdiJavid]

Hi,

I have a question regarding stateful crawls with multiple browsers (num_browsers > 1) in OpenWPM. I'm interested in achieving the ability to share a profile, settings, or CookieStore among these parallel browsers. The goal is to ensure that when a website (e.g., foo.com) sets a cookie, it is propagated across all concurrently running browsers.

I'm unsure if this functionality is already supported in OpenWPM, and if not, I would appreciate guidance on how to implement it. Additionally, I'm open to any advice on optimizing the speed of stateful analysis beyond the use of multiple parallel browsers and website segmentation.

Thanks for your assistance!

[vringar]

Hey,

what you are asking for is unfortunately impossible to the best of my knowledge. Each browser has its own profile and to achieve what you want, you'd need to merge the profiles after each website visit. However, merging the profile is impossible.
The only other thing, I could imagine is creating your own browser.cookies.onChanged listener, hosting a little server and opening a bidirectional stream in the WebExtension. Then the listener could send any change to the server and the server would send it to all browsers. However, this would require quite a lot of code changes, as you would need each WebExtension to connect to the server and also would need a whole separate server.

Unfortunately, I don't know of any way to speed up the crawls, if you need the full fidelity. You might be able to save on memory if you run the crawl headless but certain OpenGL features will not work in this mode iirc.

Hope this helps

[MohammadMahdiJavid]

Hey,

I appreciate the great idea

But how about using multiple tabs in a single browser?

If we want to enhance openwpm and implement this how should it be done?

I'm grateful for the help you provide 🙏

[vringar]

But how about using multiple tabs in a single browser?

That is a super interesting idea and would certainly allow for significant speedup. However, the assumption that a single browser is only performing a single visit is embedded in every part of the design of OpenWPM. (This isn't to say it is impossible, just that it will require significant effort.)

Pros:

• Faster due to parallelism inside each browser
• Shared Cookies and other tracking data

Cons:

• Shared Failure domain - If one website manages to lock up the browser, all visits in that browser fail
• Reduced repeatability - Since concurrent visits may influence the state of one another retrying a website at a later time may change results significantly (But the same also applies to a lesser extend when running without reset=true.)
• Attributability - It may get significantly harder to figure out who triggered which event

Areas that would need redesign:

• BrowserManager - Currently each browser manager executes one command sequence at a time and has full ownership of its browser. If there were multiple command sequences running concurrently the following points would need addressing:
  • Can you guarantee that the Selenium commands (which operates on a per browser basis) of each CS don't disrupt each other
  • Do you create a new "CommandExecutor" that manages a tab or do you try to interleave execution in the BM
  • How do you handle failure in a CS if multiple are running
• WebExtension:
  • Ensure that each instrument tracks tab_id somewhere (this gets tricky with iFrames I think)
  • Maintain a mapping of tab to visit_id
  • Send data to the appropriate visit_id
• TaskManager - The execute_command_sequence would need some rethinking on how to express groupings of CSs or just schedule CSs as we would rn

I'm sorry if this sounds discouraging. I think it would be super cool to do this, but I would have to be super hands-off on this as even the limited amount of debugging time I invested in #1068 has caused me to fall behind in my studies.