Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crawl mcot title date #262

Open
wants to merge 33 commits into
base: main
Choose a base branch
from
Open

Crawl mcot title date #262

wants to merge 33 commits into from

Conversation

nattjn
Copy link
Collaborator

@nattjn nattjn commented Jul 18, 2023

Why this PR

Why we need this PR? To crawl titles and dates for news in MCOT

Changes

  • Add function to retry if get response403
  • Add functions to process and crawl titles and dates

Related Issues

Close #

Checklist

  • PR should be in the Naming convention
  • Assign yourself in to Assigneees
  • Tag related issues
  • Constants name should be ALL_CAPITAL, function name should be snake_case, and class name should be CamelCase
  • complex function/algorithm should have Docstring
  • 1 PR should not have more than 200 lines changes (Exception for test files). If more than that please open multiple PRs
  • At least PR reviewer must come from the task's team (model, eval, data)

Nattarika and others added 30 commits July 3, 2023 22:09
test
52ecf42
Add .py
69ce172
@nattjn
Modify get_title_date and change some variables to CONST
cc8b3f5
@nattjn
Modify functions get_title_date and get_info
cb824a0
@nattjn
Modify time sleep and text_response
8500408
@nattjn
remove CONST TOP_STORIES_URL and add it to script instead
093914c
@nattjn
Crawl top news from MFA and save as CSV
fd9f7f7
@nattjn
Edit import
8a1b490
@nattjn
Modify process_title_date so it can be used for all mfa news
81625f6
@nattjn
Change file name and create a new one to crawl government achievements
64aed5a
@nattjn
Modify import for top_news and add a new script to crawl mfa gov_achi…
57c781b 
…evements
@nattjn
Add script to crawl press release
a417dce
@nattjn
Add scripts to crawl press release, embassy and consulate and other news
9ca1cee
@nattjn
Add time delay as an argument (Done)
4397688
@nattjn
Add test cases for crawl_gov_achievements
a023155
@nattjn
Correct the function used for test cases [Done, ready for review]
45ac726
@nattjn
Edit test function's name
8afcf8d
@nattjn
Delete crawl_news.py
fdbbf4d
@nattjn
Seperate the inside content to 2 additional functions
cd8e3bc
@nattjn
Fix test cases
d509673
@nattjn
Delete crawl_gov_achievements.py
ac19a28
@nattjn
Delete embassy_cosulate.py
32fe340
@nattjn
Delete gov_achievements.py
3254035
@nattjn
Delete other_news.py
84c9e4a
@nattjn
Delete press_release.py
e8f114a
@nattjn
Delete speeches.py
c878f7e
@nattjn
Delete top_news.py
71cf69c
@nattjn
Delete crawlgov_test.py
a8f24fe
@nattjn
pull the wrong filed and push crawl gov_achievements
00d7559
@nattjn
Merge branch 'gov_achivements_test' of https://github.com/OpenThaiGPT…
be4e17c 
…/openthaigpt-pretraining into gov_achivements_test
@nattjn
Copy link
Collaborator Author

nattjn commented Jul 18, 2023

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ArthurMinovsky ArthurMinovsky ArthurMinovsky approved these changes

@boat1603 boat1603 Awaiting requested review from boat1603

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nattjn @ArthurMinovsky