Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BigQuery] [GCS] Better ways to deal with google client mock #1117

Open
ShihChun-H opened this issue Sep 24, 2024 · 15 comments · May be fixed by instill-ai/pipeline-backend#775
Open

[BigQuery] [GCS] Better ways to deal with google client mock #1117

ShihChun-H opened this issue Sep 24, 2024 · 15 comments · May be fixed by instill-ai/pipeline-backend#775
Assignees
Labels
component feature New feature or request hacktoberfest hacktoberfest2024 Component improvement issues for Hacktoberfest 2024 help-wanted Help from the community is appreciated improvement Improvement on existing features instill core Label

Comments

@ShihChun-H
Copy link
Member

ShihChun-H commented Sep 24, 2024

Issue Description

Current State

  • Go packages of google does not provide interface.

Why We Want to Change?

  • So, it is a bit hard to mock client to write isolated test codes

Proposed Change

  • Research if there are better ways to deal with google client mock

Rules for the Component Hackathon

  • Each issue will only be assigned to one person/team at a time.
  • You can only work on one issue at a time.
  • To express interest in an issue, please comment on it and tag @kuroxx, allowing the Instill AI team to assign it to you.
  • Ensure you address all feedback and suggestions provided by the Instill AI team.
  • If no commits are made within five days, the issue may be reassigned to another contributor.
  • Join our Discord to engage in discussions and seek assistance in #hackathon channel. For technical queries, you can tag @chuang8511.

Component Contribution Guideline | BigQuery Documentation | GCS Documentation | Official Go Tutorial

@ShihChun-H ShihChun-H added help-wanted Help from the community is appreciated improvement Improvement on existing features feature New feature or request instill core component hacktoberfest2024 Component improvement issues for Hacktoberfest 2024 labels Sep 24, 2024
@naaa760
Copy link

naaa760 commented Oct 3, 2024

Hello @ShihChun-H

I want to work on this issue, Please assign me!
Thankyou!

@ShihChun-H
Copy link
Member Author

Hi @naaa760, The issue has been assigned to you! Cheers!

@shubhamchasing
Copy link

@ShihChun-H It seems this issue is still To be claimed

@ShihChun-H
Copy link
Member Author

Hi @shubhamchasing , thanks for the reminder 🙏

@naaa760
Copy link

naaa760 commented Oct 4, 2024

Thank you for assigning me. I will solve this issue in a few hours! :)

@naaa760
Copy link

naaa760 commented Oct 4, 2024

Hello @ShihChun-H
would you mind, Can you provide me with a little more information about this issue?

@chuang8511
Copy link
Member

Hi @naaa760
Thanks for your time.
Concretely speaking, now, we do not have test case for bigquery component.
When we developed it, we do not do it because we have not found a way to mock or stub BigQuery Server.
So, in this ticket, we would like you to add the test code for BigQuery component.

@naaa760
Copy link

naaa760 commented Oct 4, 2024

@chuang8511

Thankyou!
for letting me know! :)

@ShihChun-H
Copy link
Member Author

Hi @naaa760, I'm following up to check on any progress made regarding this issue. Could you please provide an update? Thanks 🙏

@kuroxx
Copy link
Collaborator

kuroxx commented Oct 15, 2024

Hey @naaa760 how's it going? Just checking to see if you've had a new PR update since the last one?

@PrathameshTugaonkar
Copy link

@kuroxx I would like to work on this.
Thank You!

@kuroxx
Copy link
Collaborator

kuroxx commented Oct 21, 2024

Hey @PrathameshTugaonkar thank you for your interest!

@naaa760 since there has been no further activity, I will assign this issue to @PrathameshTugaonkar now, thanks!

@PrathameshTugaonkar
Copy link

I am upto here on this - https://github.com/PrathameshTugaonkar/pipeline-backend/blob/main/pkg/component/data/bigquery/v0/main_test.go

Could you check this, @kuroxx , to see if it's the expected outcome? I will add the test cases accordingly?

@chuang8511
Copy link
Member

Hi @PrathameshTugaonkar ,
Thanks for your contribution and question!

For (m *mockInput) Read (m *mockOutput) Write (m *mockError) Error, we actually have the way to mock already.
You can check this directory.

Or, you can take a look on json operator test code as an example.

The main problem in this ticket is - the SDK provided by Google does not design bigquery.Client as an interface.
It make us not able to easily mock the client behaviours.

Now, I have 2 possible solution.

  1. Make the interface in our execution code. So, we can isolate the part that interacts with external API.
  2. Previously, I saw there is a way to build a mock bigquery server when testing.

I think both ways are ok for me. 1. is easier. 2. is more solid in the perspective of integration test.

Ideally, we would like to do 2. But, if it is too difficult, 1. is also acceptable.

@kuroxx kuroxx linked a pull request Oct 30, 2024 that will close this issue
@kuroxx
Copy link
Collaborator

kuroxx commented Nov 4, 2024

Hey @PrathameshTugaonkar how's this going?

I wanted to let you know that we will need a PR by the end of this week (8th Nov) since we are closing this event.

Please submit:

to ensure your contribution is counted!

Alternatively, if you cannot complete this within the time frame but would still like to contribute, you are more than welcome to but please note it would not be within the scope of Hacktoberfest 2024.

Thank you and look forward to your contribution! ✨

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component feature New feature or request hacktoberfest hacktoberfest2024 Component improvement issues for Hacktoberfest 2024 help-wanted Help from the community is appreciated improvement Improvement on existing features instill core Label
Projects
Status: In Progress
7 participants