NodalPricesAndVolumes - Data issue

661 Views
Last post 14 May 2023

ptan posted this 03 May 2023

It looks like we only got half a day of data for 02/05/2023 and no data for 03/05/2023 with this dataset. Is there an ongoing issue here?

#Permalink
0
0
0

6 Comments

Order by: Standard | Newest | Votes

Admin

Phil Bishop posted this 05 May 2023

Yep, should be sorted on Monday (8 May). A couple of extra columns that we weren't expecting came though in the SPD case files following the go-live of phase 4 of RTP (27 April). The resulting schema drift caused some problems that the guys have sorted out this past week. The files pushed through to EMI will hopefully be restored on Monday.

#Permalink
0
0
0

ptan posted this 07 May 2023

Hi Phil, thanks for that. I've seen that the files have regenerated but the file for 7 May hasn't come in yet this morning. Was there a change to the timing of when these arrive?

#Permalink
0
0
0

ptan posted this 10 May 2023

Hi Phil,

We're seeing another half day of data come through. Is this going to be a regular occurance going forward? Need to know so that we can account for it in our loading. Thanks

#Permalink
0
0
0

Admin

Phil Bishop posted this 11 May 2023

This has now been corrected and no, it should not be an ongoing issue.

Phil

#Permalink
0
0
0

ptan posted this 14 May 2023

Hi Phil,

We're also seeing that the files now only show up around 12 noon when they usually got updated first thing in the mornings. Is this intended to match up with when final prices are published?

Regards,

Perry

#Permalink
0
0
0

Admin

Phil Bishop posted this 14 May 2023

Hi Perry

No, not related to when final prices are published.

We pull the nodal prices and volumes data out of the RTD SPD case files. The file we publish on EMI each day should be there by 7am each morning, and that file should contain data for the preceding day, i.e. the day ending at midnight, oe approx 7 hours earlier than the time we publish on EMI.

For the past couple of weeks we've been encountering occasional and seemingly random errors with the Databricks autoloader tool that we've been trying to get to the bottom of. It's only been impacting the load of RTD data. We think we've now got it sorted. The noon or early afternoon publish times you've noticed are when we've had to re-run the EMI publishing job on account of the autoloader errors.

Phil

#Permalink
1
0
0

Liked by

Find discussions by tag

See all

NodalPricesAndVolumes - Data issue

Categories

Find discussions by tag

This week's high earners

Popular discussions