[BigQuery] [GCS] Better ways to deal with google client mock

[ShihChun-H]

Issue Description

Current State

• Go packages of google does not provide interface.

Why We Want to Change?

• So, it is a bit hard to mock client to write isolated test codes

Proposed Change

• Research if there are better ways to deal with google client mock

---

Rules for the Component Hackathon

• Each issue will only be assigned to one person/team at a time.
• You can only work on one issue at a time.
• To express interest in an issue, please comment on it and tag @kuroxx, allowing the Instill AI team to assign it to you.
• Ensure you address all feedback and suggestions provided by the Instill AI team.
• If no commits are made within five days, the issue may be reassigned to another contributor.
• Join our Discord to engage in discussions and seek assistance in #hackathon channel. For technical queries, you can tag @chuang8511.

---

Component Contribution Guideline | BigQuery Documentation | GCS Documentation | Official Go Tutorial

[naaa760]

Hello @ShihChun-H

I want to work on this issue, Please assign me!
Thankyou!

[ShihChun-H]

Hi @naaa760, The issue has been assigned to you! Cheers!

[shubhamchasing]

@ShihChun-H It seems this issue is still To be claimed

[ShihChun-H]

Hi @shubhamchasing , thanks for the reminder 🙏

[naaa760]

Thank you for assigning me. I will solve this issue in a few hours! :)

[naaa760]

Hello @ShihChun-H
would you mind, Can you provide me with a little more information about this issue?

[chuang8511]

Hi @naaa760
Thanks for your time.
Concretely speaking, now, we do not have test case for bigquery component.
When we developed it, we do not do it because we have not found a way to mock or stub BigQuery Server.
So, in this ticket, we would like you to add the test code for BigQuery component.

[naaa760]

@chuang8511

Thankyou!
for letting me know! :)

[ShihChun-H]

Hi @naaa760, I'm following up to check on any progress made regarding this issue. Could you please provide an update? Thanks 🙏

[kuroxx]

Hey @naaa760 how's it going? Just checking to see if you've had a new PR update since the last one?

[PrathameshTugaonkar]

@kuroxx I would like to work on this.
Thank You!

[kuroxx]

Hey @PrathameshTugaonkar thank you for your interest!

@naaa760 since there has been no further activity, I will assign this issue to @PrathameshTugaonkar now, thanks!

[PrathameshTugaonkar]

I am upto here on this - https://github.com/PrathameshTugaonkar/pipeline-backend/blob/main/pkg/component/data/bigquery/v0/main_test.go

Could you check this, @kuroxx , to see if it's the expected outcome? I will add the test cases accordingly?

[chuang8511]

Hi @PrathameshTugaonkar ,
Thanks for your contribution and question!

For (m *mockInput) Read (m *mockOutput) Write (m *mockError) Error, we actually have the way to mock already.
You can check this directory.

Or, you can take a look on json operator test code as an example.

The main problem in this ticket is - the SDK provided by Google does not design bigquery.Client as an interface.
It make us not able to easily mock the client behaviours.

Now, I have 2 possible solution.

1. Make the interface in our execution code. So, we can isolate the part that interacts with external API.
2. Previously, I saw there is a way to build a mock bigquery server when testing.

I think both ways are ok for me. 1. is easier. 2. is more solid in the perspective of integration test.

Ideally, we would like to do 2. But, if it is too difficult, 1. is also acceptable.

[kuroxx]

Hey @PrathameshTugaonkar how's this going?

I wanted to let you know that we will need a PR by the end of this week (8th Nov) since we are closing this event.

Please submit:

• Your PR for this ticket
• This Google form

to ensure your contribution is counted!

Alternatively, if you cannot complete this within the time frame but would still like to contribute, you are more than welcome to but please note it would not be within the scope of Hacktoberfest 2024.

Thank you and look forward to your contribution! ✨