ROSIEBot

The Robotic Open Science Indexing Engine

Static mirroring utility for the Open Science Framework, maintained by the Center for Open Science.

Visit the COS Github for more innovations in the openness, integrity, and reproducibility of scientific research.

Project Overview

Installation

This software requires Python 3.5 for the aiohttp library. If desired, create a virtualenv:

From scratch:

pip install virtualenv

pip install virtualenvwrapper

Create virtualenv 'rosie':

mkvirtualenv rosie --python=python3.5

Switch into the rosie environment for the first time:

workon rosie

Clone the repository into your development folder:

git clone https://github.com/zamattiac/ROSIEBOt.git

Navigate into the new folder ( cd ROSIEBot )

pip install -r requirements.txt to install dependency libraries in the virtualenv.

Enter/exit the virtual environment:

workon rosie/deactivate

ROSIEisms

OSF Pages by Category

Project	Registration	User	Institution
Dashboard	Dashboard	Profile	Dashboard
Files	Files
Wiki	Wiki
Analytics	Analytics
Registrations
Forks	Forks

Our Process

• Crawling: getting lists of all the URLs to visit
• Scraping: visiting all those URLs and saving their content to the mirror
• Resuming: continuing the crawl/scrape process if it stops in the middle
• Verifying: making sure all the files are present and in acceptable condition
• Compiling active: getting a list from the API about existing pages

Using the Command Line Interface

Running cli.py

The python file cli.py needs to be run in the command line in the rosie virtualenv. This project is optimized for Mac.

Every command consists of the following and the flag for one mode:

python cli.py

See python cli.py --help for some further usage assistance.

Mode flags:

--compile_active

Make a taskfile of all the currently active pages on the OSF. This is useful primarily for --delete, which requires such a file to remove no-longer-existant pages from the mirror.

####--scrape

Crawl and scrape the site. Must include date marker --dm=<DATE>, where <DATE> is the date of last scrape in the form YYYY-MM-DDTHH:MM:SS.000, eg. 1970-06-15T00:00:00.000

One must specify which categories to scrape:

• --nodes (projects)
• --registrations
• --users
• --institutions

Any or all can be added.

If the nodes flag is used, one must specify which project pages to include:

• -d : dashboard
• -f : files page
• -w : wiki pages
• -a : analytics
• -r : list of registrations of the project
• -k: list of forks of the project

--resume

Pick up where a normal process left off in case of an unfortunate halt. The normal process creates and updates a .json task file with its status, and this must be included with the flag --tf=<FILENAME>. The filename will be of the form YYYYMMDDHHMM.json and should be visible in the ROSIEBot directory.

--verify

Verify the completeness of the mirror. See below for steps. This process also requires a .json file in the form described in the resume step, and --rn=<INT>, where <INT> is the desired number of retries.

Verification Steps

1. Verify that each URL found by the crawler has a corresponding file on the mirror.

2. Compare the size of each file to the minimum possible size for a complete page.

3. Rescrape failed pages and try again.

--delete

Remove anything inside a category folder that isn't listed on the API. Requires a compile_active-produced taskfile.

python cli.py --delete --ctf=<TASKFILE>

--index

Creates a search engine index.

Note: Do not run until the static folder is in place in the archive.

Using search: the search button on each page should be replaced with a link to /search.html

Hosting a Mirror

Scraped pages require a static folder inside the mirror. Please get a fresh copy from the OSF repo and place directly inside archive/.

Once static is in place, run python cli.py --index to set up search utility.

Simple local server setup (does not preserve original archive organization, but does use OSF organization)

This option creates a flat copy of the archive without categorical folders. Nginx configuration is required otherwise.

Make sure whatever utilities you desire (e.g. verify, index) have been run before the copy is made.

Run bash scripts/host_locally.sh from the ROSIEBot root. Here is your mirror.

Packaging the archive

zip -r archive.zip archive/ zip -r flat-archive.zip flat-archive/

Using Nginx to host (preserving structure)

Including the following location lines provides necessary routing for a non-flat mirror.

See How to set up prerender step 2 for Nginx information, bearing in mind that some parts do not apply

server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        root /path/to/archive;
        # index index.html index.htm;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ /registration/$uri/ /profile/$uri/ /project/$uri/ /project/$uri/home /registration/$uri/home =404;
                # index index.html index.htm;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules
        }

        location /static/ {
                alias /path/to/archive/static/;
        }
}

Authenticating your mirror

(Future)

---

How to set up prerender for a local OSF

---