Skip to content

project-polymorph/web_downloader

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Feb 13, 2025
d7059b0 · Feb 13, 2025

History

27 Commits
Nov 11, 2024
Feb 13, 2025
Feb 12, 2025
Nov 11, 2024
Nov 11, 2024
Nov 11, 2024
Jan 21, 2025
Jan 22, 2025
Feb 13, 2025
Nov 11, 2024
Jan 22, 2025
Nov 11, 2024
Nov 11, 2024
Jan 21, 2025
Nov 11, 2024
Nov 21, 2024
Nov 21, 2024

Repository files navigation

downloader

This is part of the chinese transgender digital archive project.

Scripts and results for searching and downloading webpages.

Search

  • puppeteer: search for webpages using puppeteer.
  • serper: search for webpages using serper
  • googlecustom: search for webpages using google custom search json API
  • google: search for webpages using google python library

Run ./gen_links to summary all links into a yml file.

download

See download.

Currently, support webpages and pdfs.

LICENSE

MIT

About

Scripts to search and download from websites

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published