Problems with 4.1.1 TypeError: concat_tables() got an unexpected keyword argument 'use_threads'

[andersonzanichelli]

This error occurs when I use pandas.read_sql(my_query, dtb_connection) function.

[odykstra]

I'm getting this issue as well when using fetchall_arrow() that boils down to concat_tables()

[odykstra]

File /databricks/python/lib/python3.10/site-packages/pyRangers/table_tools.py:145, in query_delta(query, to_pandas, use_jdbc, warehouse, fetch_results, multi_queries)
142 print(e)
144 if fetch_results:
--> 145 arrow_table = cursor.fetchall_arrow()
146 cursor.close()
147 connection.close()
File /databricks/python/lib/python3.10/site-packages/databricks/sql/client.py:1199, in Cursor.fetchall_arrow(self)
1197 self._check_not_closed()
1198 if self.active_result_set:
-> 1199 return self.active_result_set.fetchall_arrow()
1200 else:
1201 raise ProgrammingError(
1202 "There is no active result set",
1203 session_id_hex=self.connection.get_session_id_hex(),
1204 )
File /databricks/python/lib/python3.10/site-packages/databricks/sql/result_set.py:363, in ThriftResultSet.fetchall_arrow(self)
360 partial_result_chunks.append(partial_results)
361 self._next_row_index += partial_results.num_rows
--> 363 result_table = concat_table_chunks(partial_result_chunks)
364 # If PyArrow is installed and we have a ColumnTable result, convert it to PyArrow Table
365 # Valid only for metadata commands result set
366 if isinstance(result_table, ColumnTable) and pyarrow:
File /databricks/python/lib/python3.10/site-packages/databricks/sql/utils.py:883, in concat_table_chunks(table_chunks)
881 return ColumnTable(result_table, table_chunks[0].column_names)
882 else:
--> 883 return pyarrow.concat_tables(table_chunks, use_threads=True)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/pyarrow/table.pxi:2299, in pyarrow.lib.concat_tables()

[jprakash-db]

Can you share what version of arrow, pandas and numpy are you using ? The error you are mentioning would occur in pyarrow<13 . But the min recommended version of pyarrow is 14 for the connector

[vikrantpuppala]

Fixed by #684