High Memory Usage During MySQL Table Import (DuckDB 0.10.0, Ubuntu)

[jaldaadithya]

What happens?

When I try to create a table in the duckdb from the mysql table which was around 15 GB, the duckdb process is taking entire system memory (around 30 GB) and the system has 32GB RAM.

To Reproduce

I attached mysql using mysql_scanner

1. ATTACH 'host=host user=user port=3306 database=duckdb password=password socket=/var/run/mysqld/mysqld.sock' AS mysql_db (TYPE mysql_scanner,READ_ONLY);
2. SET threads = 4;
   SET memory_limit = '4GB';
   SET max_memory = '4GB';
   SET mysql_tinyint1_as_boolean=false;
   SET old_implicit_casting = true;
3. CREATE OR REPLACE TABLE table AS FROM mysql_db.table;
   the above table was around 15GB.
4. I tried using LD_PRELOAD too.

OS:

Ubuntu

MySQL Version:

8.0.36

DuckDB Version:

0.10.0

DuckDB Client:

jdbc

Full Name:

Adithya

Affiliation:

OvalEdge

Have you tried this on the latest main branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree

[yinheli]

Same issue here, How to reduce the memory usage while import? @Mytherin

[ulissescappato]

Same issue here...

Thanks,

[mendelfriedman]

I'm seeing the same issue using mysql_query() and copy to file.

Any solutions here?

[dangkaka]

same issue here, any solutions?