It looks like we only got half a day of data for 02/05/2023 and no data for 03/05/2023 with this dataset. Is there an ongoing issue here?
- NodalPricesAndVolumes - Data issue
NodalPricesAndVolumes - Data issue
- 64 Views
- Last post 4 weeks ago
Yep, should be sorted on Monday (8 May). A couple of extra columns that we weren't expecting came though in the SPD case files following the go-live of phase 4 of RTP (27 April). The resulting schema drift caused some problems that the guys have sorted out this past week. The files pushed through to EMI will hopefully be restored on Monday.
Hi Phil, thanks for that. I've seen that the files have regenerated but the file for 7 May hasn't come in yet this morning. Was there a change to the timing of when these arrive?
We're seeing another half day of data come through. Is this going to be a regular occurance going forward? Need to know so that we can account for it in our loading. Thanks
This has now been corrected and no, it should not be an ongoing issue.
We're also seeing that the files now only show up around 12 noon when they usually got updated first thing in the mornings. Is this intended to match up with when final prices are published?
No, not related to when final prices are published.
We pull the nodal prices and volumes data out of the RTD SPD case files. The file we publish on EMI each day should be there by 7am each morning, and that file should contain data for the preceding day, i.e. the day ending at midnight, oe approx 7 hours earlier than the time we publish on EMI.
For the past couple of weeks we've been encountering occasional and seemingly random errors with the Databricks autoloader tool that we've been trying to get to the bottom of. It's only been impacting the load of RTD data. We think we've now got it sorted. The noon or early afternoon publish times you've noticed are when we've had to re-run the EMI publishing job on account of the autoloader errors.