Commit 37c9275
committed
Property for ETL records data
Why these changes are being introduced:
This is a small change now, that will lead to a larger change later.
The TIMDEX dataset is getting more structure, and this means we will
want to initialize a TIMDEXDataset instance with the root of the dataset,
but then internally there will be more opinionation about where files
should be read and written to.
How this addresses that need:
A new property 'data_records_root' is added to TIMDEXDataset that
mirrors similar properties in TIMDEXDatasetMetadata. This informs any
operations that need to read or write ETL records where precisely they
are in the dataset.
At this time only .write() utilizes it, but in a future ticket the
load method will be heavily reworked (if not outright removed) and
this property will be fully integrated.
This is needed now to continue updates to TIMDEXMetadataDataset
for TIMX-530.
Side effects of this change:
* Initialization of TIMDEXDataset should provide the true dataset root,
not point to /data/records. The pipeline lambda currently does this,
but will be updated in TIMX-531.
Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-530
* https://mitlibraries.atlassian.net/browse/TIMX-5311 parent ff2aff0 commit 37c9275
1 file changed
Lines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
129 | 133 | | |
130 | 134 | | |
131 | 135 | | |
| |||
370 | 374 | | |
371 | 375 | | |
372 | 376 | | |
373 | | - | |
| 377 | + | |
374 | 378 | | |
375 | 379 | | |
376 | 380 | | |
| |||
0 commit comments