Skip to content

Commit d1af981

Browse files
committed
Use DuckDB secret vs PRAGMA for MinIO
Why these changes are being introduced: While PRAGMA configurations worked for direct parquet file reading in a local MinIO S3 instance, it did not work for DuckDB ATTACH statements. How this addresses that need: By using a DuckDB secret, setting the MinIO S3 endpoint and credentials there, other statments like ATTACH work as expected. This is overall a better approach anyways! The PRAGMA approach seemed like the only option originally, but this edge case of ATTACH forced revisiting it and this use of secrets does in fact work. Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TIMX-515
1 parent e1fb022 commit d1af981

1 file changed

Lines changed: 17 additions & 13 deletions

File tree

timdex_dataset_api/metadata.py

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -115,24 +115,28 @@ def _configure_s3_connection(self) -> None:
115115
if os.getenv("MINIO_S3_ENDPOINT_URL"):
116116
self.conn.execute(
117117
f"""
118-
PRAGMA s3_endpoint='{urlparse(os.environ["MINIO_S3_ENDPOINT_URL"]).netloc}';
119-
PRAGMA s3_access_key_id='{os.environ["MINIO_USERNAME"]}';
120-
PRAGMA s3_secret_access_key='{os.environ["MINIO_PASSWORD"]}';
121-
PRAGMA s3_use_ssl=false;
122-
PRAGMA s3_url_style='path';
123-
"""
118+
create or replace secret minio_s3_secret (
119+
type s3,
120+
endpoint '{urlparse(os.environ["MINIO_S3_ENDPOINT_URL"]).netloc}',
121+
key_id '{os.environ["MINIO_USERNAME"]}',
122+
secret '{os.environ["MINIO_PASSWORD"]}',
123+
region 'us-east-1',
124+
url_style 'path',
125+
use_ssl false
126+
);
127+
"""
124128
)
125129

126130
else:
127131
self.conn.execute(
128132
"""
129-
create or replace secret secret (
130-
type s3,
131-
provider credential_chain,
132-
chain 'sso;env;config',
133-
refresh true
134-
);
135-
"""
133+
create or replace secret aws_s3_secret (
134+
type s3,
135+
provider credential_chain,
136+
chain 'sso;env;config',
137+
refresh true
138+
);
139+
"""
136140
)
137141

138142
def _create_full_dataset_table(self) -> None:

0 commit comments

Comments
 (0)