Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add more crawler test&Make crawler more scalable #3

Merged
merged 10 commits into from
Oct 26, 2020
Merged

Add more crawler test&Make crawler more scalable #3

merged 10 commits into from
Oct 26, 2020

Conversation

Andrewmatilde
Copy link
Contributor

@Andrewmatilde Andrewmatilde commented Oct 26, 2020 •
edited
Loading

What is changed and how it works?

  • Split query and loop fetch from the raw code, add demo test.
  • Add more comments to share how to write github api v4 crawler.

@Andrewmatilde
Add better test
f2c455b 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
Add better test
83c94a1 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
Merge remote-tracking branch 'origin/main' into main
8b39abb 
# Conflicts:
#	crawler/crawler_test.go
@Andrewmatilde
Add tokens pool
63154a8 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
Turn crawler part into a better version
fbfab1d 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
Add some comment.
ef51287 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
complete crawler part.
e0f7ac1 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
complete crawler part.
d2ee706 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
complete crawler test part.
d22cf72 
Signed-off-by: Andrewmatilde <[email protected]>
@Andrewmatilde
complete crawler test part.
2848906 
Signed-off-by: Andrewmatilde <[email protected]>
zhouqiang-cl
zhouqiang-cl approved these changes Oct 26, 2020
View reviewed changes
Copy link
Contributor

@zhouqiang-cl zhouqiang-cl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhouqiang-cl zhouqiang-cl merged commit f9fa55f into PingCAP-QE:main Oct 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhouqiang-cl zhouqiang-cl zhouqiang-cl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Andrewmatilde @zhouqiang-cl