Replies: 2 comments
-
To (hopefully) answer my own question. You can use something like:
Not entirely sure of the performance yet, but hopefully all teh heavy lifting is done with c-level logic from the bq request through to the pg request |
Beta Was this translation helpful? Give feedback.
0 replies
-
Use |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have tables in BQ that i'd like to copy into Postgres (outside of GCP unfortunately).
The following code can read a table from BigQuery in batches of
pyarrow.RecordBatch
, but I'm not sure of the most optimal way to get those into postgres, or if there is a totally different approach. We are using Prefect (v1, currently) if that's relevant.Ideally I'd be upserting into postgres rather than inserting, but i'll take inserts for now if that's the only option.
I'm interested in a combination of good speed and memory/disk performance, though I don't have any hard numbers in terms of requirements (basically we will be doing a lot of copying datasets from BQ to PG so want it to be sensibly optimised).
Any thoughts very welcome, thanks!
Beta Was this translation helpful? Give feedback.
All reactions