has_table() got an unexpected argument 'info_cache' : when using df.to_sql() with SQLAlchemy + impala

[Brynhildr-sudo]

df.to_sql() needs an SQLAlchemy connection object. So impala's own dbapi cannot be used directly. But while using impala with SQLAlchemy and trying to create/insert into a table using df.to_sql(), the following error is thrown:
has_table() got an unexpected keyword argument 'info_cache'

Here's the traceback:

Traceback (most recent call last):
  File "/home/user/data_upload_service/batch_job/data_frame_writer.py", line 27, in write_data_frame_to_hive_table
    data_frame.to_sql(table_name, con=conn, if_exists="append", index=False, method="multi",chunksize=5000)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 333, in wrapper
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/core/generic.py", line 3087, in to_sql
    return sql.to_sql(
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/sql.py", line 842, in to_sql
    return pandas_sql.to_sql(
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/sql.py", line 2008, in to_sql
    table = self.prep_table(
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/sql.py", line 1912, in prep_table
    table.create()
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/sql.py", line 984, in create
    if self.exists():
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/sql.py", line 970, in exists
    return self.pd_sql.has_table(self.name, self.schema)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/sql.py", line 2041, in has_table
    return insp.has_table(name, schema or self.meta.schema)
  File "/home/user/.local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 439, in has_table
    return self.dialect.has_table(
TypeError: has_table() got an unexpected keyword argument 'info_cache'

it looks like this issue arises because of the absence of arbitrary keyword arguments in has_table() under impala/sqlalchemy.py:

impyla/impala/sqlalchemy.py

Line 231 in 747e1da

def has_table(self, connection, table_name, schema=None):

Versions used:

• impyla: v0.20.0
• sqlalchemy: 20.0.37
• python: 3.9.18