Commit 14c110d
committed
Move dataset location parsing to init
Why these changes are being introduced:
When loading a dataset for reading, or preparing to write, the
appropriate pyarrow filesystem must be set and the location normalized
(e.g. removing s3:// prefix).
How this addresses that need:
* Move call of self.parse_location to init as always required
* An invalid location will fail early before an attempt to read or write
* Previously missing, add filesystem to write method to support S3
writes
Side effects of this change:
* None
Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-4151 parent 45c5d2f commit 14c110d
1 file changed
Lines changed: 6 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
132 | 133 | | |
133 | 134 | | |
134 | 135 | | |
135 | | - | |
136 | 136 | | |
137 | | - | |
| 137 | + | |
138 | 138 | | |
139 | 139 | | |
140 | 140 | | |
141 | | - | |
| 141 | + | |
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| |||
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
192 | | - | |
| 192 | + | |
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
| |||
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
206 | | - | |
| 206 | + | |
207 | 207 | | |
208 | 208 | | |
| 209 | + | |
209 | 210 | | |
210 | 211 | | |
211 | 212 | | |
| |||
0 commit comments