Skip to content

Latest commit

 

History

History
42 lines (34 loc) · 1.08 KB

README.md

File metadata and controls

42 lines (34 loc) · 1.08 KB
Raw

Wiki leaks files spider

What's this?

  • this is a web crawler that used to download files from https://file.wikileaks.org/file/

  • by setting the key words that you wants to search for. the program will download files whose name include the key words.

  • if you can not connect to the website, you can also set a proxy using socks or http protocol.

Requirement

  • python 3.5+
  • beautifuly soup
  • requests
pip install -r reqirements.txt

Usage

usage: Spider.py [-h] [-K KEY] [-D DIR] [-P PROXY]

Wikileaks file spider

optional arguments:
  -h, --help            show this help message and exit
  -K KEY, --key KEY     keywords for spider
  -D DIR, --dir DIR     directory to save files
  -P PROXY, --proxy PROXY
                        set the spider porxy
  • Example
python Spider.py -K "a b c" -D ./files -P socks5h://127.0.0.1:1080

  • Prompt

    • if you want to use proxy ,socks5 protocol, please use url like this:
    socks5h://127.0.0.1:1080

    not socks5://!!! otherwise this may cause ssl error.