Code

Crawl mcot title date#262

Code

Open

nattjn wants to merge 33 commits intomainOpenThaiGPT/openthaigpt-pretraining:mainfrom crawl_mcot_title_dateOpenThaiGPT/openthaigpt-pretraining:crawl_mcot_title_dateCopy head branch name to clipboard

+348

Commits

Commits on Jul 3, 2023

• test

  

  Nattarika

  committed
  52ecf42

  Copy full SHA for 52ecf42

• Add .py

  

  Nattarika

  committed
  69ce172

  Copy full SHA for 69ce172

Commits on Jul 5, 2023

• Modify get_title_date and change some variables to CONST

  nattjn

  committed
  cc8b3f5

  Copy full SHA for cc8b3f5

• Modify functions get_title_date and get_info

  nattjn

  committed
  cb824a0

  Copy full SHA for cb824a0

Commits on Jul 6, 2023

• Modify time sleep and text_response

  nattjn

  committed
  8500408

  Copy full SHA for 8500408

• remove CONST TOP_STORIES_URL and add it to script instead

  nattjn

  committed
  093914c

  Copy full SHA for 093914c

• Crawl top news from MFA and save as CSV

  nattjn

  committed
  fd9f7f7

  Copy full SHA for fd9f7f7

• Edit import

  nattjn

  committed
  8a1b490

  Copy full SHA for 8a1b490

• Modify process_title_date so it can be used for all mfa news

  nattjn

  committed
  81625f6

  Copy full SHA for 81625f6

• Change file name and create a new one to crawl government achievements

  nattjn

  committed
  64aed5a

  Copy full SHA for 64aed5a

• Modify import for top_news and add a new script to crawl mfa gov_achievements

  nattjn

  committed
  57c781b

  Copy full SHA for 57c781b

• Add script to crawl press release

  nattjn

  committed
  a417dce

  Copy full SHA for a417dce

• Add scripts to crawl press release, embassy and consulate and other news

  nattjn

  committed
  9ca1cee

  Copy full SHA for 9ca1cee

Commits on Jul 11, 2023

• Add time delay as an argument (Done)

  nattjn

  committed
  4397688

  Copy full SHA for 4397688

• Add test cases for crawl_gov_achievements

  nattjn

  committed
  a023155

  Copy full SHA for a023155

• Correct the function used for test cases [Done, ready for review]

  nattjn

  committed
  45ac726

  Copy full SHA for 45ac726

Commits on Jul 12, 2023

• Edit test function's name

  nattjn

  committed
  8afcf8d

  Copy full SHA for 8afcf8d

• Delete crawl_news.py

  nattjn

  authored
  fdbbf4d

  Copy full SHA for fdbbf4d

Commits on Jul 14, 2023

• Seperate the inside content to 2 additional functions

  nattjn

  committed
  cd8e3bc

  Copy full SHA for cd8e3bc

• Fix test cases

  nattjn

  committed
  d509673

  Copy full SHA for d509673

Commits on Jul 17, 2023

• Delete crawl_gov_achievements.py

  nattjn

  authored
  ac19a28

  Copy full SHA for ac19a28

• Delete embassy_cosulate.py

  nattjn

  authored
  32fe340

  Copy full SHA for 32fe340

• Delete gov_achievements.py

  nattjn

  authored
  3254035

  Copy full SHA for 3254035

• Delete other_news.py

  nattjn

  authored
  84c9e4a

  Copy full SHA for 84c9e4a

• Delete press_release.py

  nattjn

  authored
  e8f114a

  Copy full SHA for e8f114a

• Delete speeches.py

  nattjn

  authored
  c878f7e

  Copy full SHA for c878f7e

• Delete top_news.py

  nattjn

  authored
  71cf69c

  Copy full SHA for 71cf69c

• Delete crawlgov_test.py

  nattjn

  authored
  a8f24fe

  Copy full SHA for a8f24fe

Commits on Jul 18, 2023

• pull the wrong filed and push crawl gov_achievements

  nattjn

  committed
  00d7559

  Copy full SHA for 00d7559

• Merge branch 'gov_achivements_test' of https://github.com/OpenThaiGPT/openthaigpt-pretraining into gov_achivements_test

  nattjn

  committed
  be4e17c

  Copy full SHA for be4e17c

• Add code for crawling government achievements

  nattjn

  committed
  abfa562

  Copy full SHA for abfa562

• Add script for crawling government achievements

  nattjn

  committed
  07ba1e8

  Copy full SHA for 07ba1e8

• Add code to crawl title and date for mcot

  nattjn

  committed
  f9bb0e8

  Copy full SHA for f9bb0e8