Getting a segmentation fault error when loading full table

[helio-sb]

What language are you using?

Python

What version are you using?

0.4.0

What database are you using?

PostgreSQL

What dataframe are you using?

Pandas

Can you describe your bug?

I was trying to use connectorX to retrieve data from a table we have running on AWS RDS Postgress instance. We are retrieving the full table in our query. It works great in the QA environment with 100K records in the table , but in the environments where we have 11 Milliion records , I got a segmentation Fault error.

What are the steps to reproduce the behavior?

trying to load a large amount of data .

Database setup if the error only happens on specific data or data type

customerId , phone, email, data_1, data_2, data_3

total : 12 columns - String format

Example query / code

select * from table1
ordered by customer_id

What is the error?

zsh: segmentation fault python3.12 localRun.py

[wangxiaoying]

Hi @helio-sb ,

To narrow down the problem.

1. Is your data updating (the number of rows in the table is changing) when fetching the data?
2. Does the seg fault occurs at the beginning or after running for a while?
3. Are you enabling partitioning (setting the partition_on and partition_num) when the error occurs?
4. Also, will the error seg fault error stay there if you change the dataframe type to arrow (set the return_type=arrow)?

[helio-sb]

hey @wangxiaoying , thanks for the reply..
1 . no , no changes in the data
2. running after a while, around 1-2 minutes.
3. no specific partition setting, just default options.
4. I will give a try and let you know the results

[wangxiaoying]

hey @wangxiaoying , thanks for the reply.. 1 . no , no changes in the data 2. running after a while, around 1-2 minutes. 3. no specific partition setting, just default options. 4. I will give a try and let you know the results

Thank you! Also, it would be great if you can also test on an older version like pip install connectorx==0.3.4a3.