It looks like we only got half a day of data for 02/05/2023 and no data for 03/05/2023 with this dataset. Is there an ongoing issue here?
- Wholesale
- NodalPricesAndVolumes - Data issue
NodalPricesAndVolumes - Data issue
- 143 Views
- Last post 14 May 2023
Phil Bishop
posted this
14 May 2023
Hi Perry
No, not related to when final prices are published.
We pull the nodal prices and volumes data out of the RTD SPD case files. The file we publish on EMI each day should be there by 7am each morning, and that file should contain data for the preceding day, i.e. the day ending at midnight, oe approx 7 hours earlier than the time we publish on EMI.
For the past couple of weeks we've been encountering occasional and seemingly random errors with the Databricks autoloader tool that we've been trying to get to the bottom of. It's only been impacting the load of RTD data. We think we've now got it sorted. The noon or early afternoon publish times you've noticed are when we've had to re-run the EMI publishing job on account of the autoloader errors.
Phil
- #Permalink
- 1
- 0
- 0
- Liked by
-
Phil Bishop
posted this
05 May 2023
Yep, should be sorted on Monday (8 May). A couple of extra columns that we weren't expecting came though in the SPD case files following the go-live of phase 4 of RTP (27 April). The resulting schema drift caused some problems that the guys have sorted out this past week. The files pushed through to EMI will hopefully be restored on Monday.
- #Permalink
- 0
- 0
- 0
ptan
posted this
07 May 2023
Hi Phil, thanks for that. I've seen that the files have regenerated but the file for 7 May hasn't come in yet this morning. Was there a change to the timing of when these arrive?
- #Permalink
- 0
- 0
- 0
ptan
posted this
10 May 2023
Hi Phil,
We're seeing another half day of data come through. Is this going to be a regular occurance going forward? Need to know so that we can account for it in our loading. Thanks
- #Permalink
- 0
- 0
- 0
Phil Bishop
posted this
11 May 2023
This has now been corrected and no, it should not be an ongoing issue.
Phil
- #Permalink
- 0
- 0
- 0
ptan
posted this
14 May 2023
Hi Phil,
We're also seeing that the files now only show up around 12 noon when they usually got updated first thing in the mornings. Is this intended to match up with when final prices are published?
Regards,
Perry
- #Permalink
- 0
- 0
- 0
Find discussions by tag
See all
