Rework read methods to utilize metadata

Why these changes are being introduced:

This commit is a culmination of work to elevate metadata about
ETL records to the point it can be used to improve the speed
and efficiency of data queries.

While the signature of the read methods will remain mostly
the same, it exposes a 'where' clause that accepts raw SQL to
filter the results, allowing for more advanced querying beyond
the simple key/value DatasetFilters.

Additionally, and equally important, data retrieval is now
coming directly from DuckDB instead of more low level pyarrow
dataset reads.  Overall complexity remains about the same, but
we have shifted focus into DuckDB table and view preperation
and SQL construction, which also pays dividends in other contexts.

It it anticipated this will set us up well for other data we may
add to the TIMDEX dataset, e.g. vector embeddings or fulltext,
which we may want to query and retrieve.

How this addresses that need:

As before, all read methods eventually call
TIMDEXDataset.read_batches_iter() which now performs a two-part
process of first quickly querying metadata records, then using that
information to prune heavier data retrieved.

SQLAlchemy is used to provide model DuckDB tables and views such
that we can preserve the simpler key/value DatasetFilters, e.g.
source='libguides' or run_type='daily', which will likely
represent the majority of the public API needs by converting those
key/value pairs into a SQL WHERE clause programatically.  This is
done without the need for complex string interpolation and
escaping.

The overall input and output signatures are largely the same, but
the underlying approach to querying the ETL parquet records now
utilizes DuckDB much more heavily, while also providing a SQL
'escape hatch' if the keyword filters don't suffice.

Side effects of this change:
* None!  Transmog and TIM can call TDA in the same way as before.
The underlying approach is different, but the signatures are
mostly the same.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-529

Author: ghukill

Pipfile

@@ -9,6 +9,8 @@ boto3 = "*"
 duckdb = "*"
 pandas = "*"
 pyarrow = "*"
+sqlalchemy = "*"
+duckdb-engine = "*"
 
 [dev-packages]
 black = "*"

Pipfile.lock

@@ -59,6 +59,7 @@ TDA_BATCH_READ_AHEAD=# Number of batches to optimistically read ahead when batch
 TDA_FRAGMENT_READ_AHEAD=# Number of fragments to optimistically read ahead when batch reaching from a dataset; pyarrow default is 4
 TDA_DUCKDB_MEMORY_LIMIT=# Memory limit for DuckDB connection
 TDA_DUCKDB_THREADS=# Thread limit for DuckDB connection
+TDA_DUCKDB_JOIN_BATCH_SIZE=# Batch size for metadata + data joins, 100k default and recommended
 ```
 
 ## Local S3 via MinIO
@@ -98,6 +99,8 @@ WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3
 
 ### Reading Data
 
+See [docs/reading.md](docs/reading.md) for an in-depth guide and Mermaid diagram.
+
 First, import the library:
 ```python
 from timdex_dataset_api import TIMDEXDataset

README.md

@@ -1,5 +1,8 @@
 import logging
 import os
+import warnings
+
+from duckdb_engine import DuckDBEngineWarning
 
 
 def configure_logger(
@@ -28,6 +31,13 @@ def configure_logger(
         for warning_logger_name in warning_only_loggers.split(","):
             logging.getLogger(warning_logger_name).setLevel(logging.WARNING)
 
+    # suppress a SQLAlchemy duckdb_engine warning
+    warnings.filterwarnings(
+        "ignore",
+        category=DuckDBEngineWarning,
+        message=r".*doesn't yet support reflection on indices.*",
+    )
+
     return logger