Real-time pricing on 1 November 2022 - impact on EMI reports and datasets

  • This discussion is locked, please start a new discussion.
  • 3.9K Views
  • Last post 07 November 2023
Phil Bishop posted this 13 October 2022

Real-time pricing (RTP) goes live on 1 November 2022. The implementation of RTP will necessitate changes to a lot of the data received, processed, and published by the EMI team. If you are a regular user of Wholesale reports or Wholesale datasets, you might want to keep an eye on this discussion for regular updates over the next few weeks.

Changes are going to be made with very little notice. Some reports or datasets may be stopped altogether on 1 November and restored in the following days or weeks.

vSPD and GDX files

The vSPD model is undergoing substantial change and testing in readiness for real-time pricing. A new version of vSPD will very soon be made available. It will not be backward compatible with the current version. After 1 November 2022, if you want to run simulations that span 30 Oct / 1 Nov 2022, you’ll need to use both versions of vSPD, i.e. the current and the new versions.

GDX files are currently produced from the ex post final pricing (FP) SPD cases. FP GDX files will cease to exist after 1 November and in their place GDX files will be produced from the real-time dispatch (RTD) cases. There will be an indeterminate number of these per day but typically there will be well over 200. These new RTD GDX files will be published beginning 1 Nov. At some point after 1 November 2022, we will change the GDX processing to package all RTD GDXs for a day (midnight to midnight) into a single GDX file, which will be published early in the morning (2-3am’ish) of the following day. The daily RTD GDX file, when we start publishing it, will be approximately six times larger than the current FP GDX files.

Dispatch and pricing

A new wholesale dataset folder will be created called DispatchAndPricing. Beginning on 1 November, the following datasets will be published into subfolders of this new folder as follows:

  • DispatchEnergyPrices – Dispatch energy prices: the ‘new’ real-time energy prices that will replace the current real-time (or 5-min) prices published here. The current real-time prices will cease to be produced from 1 Nov.
  • DispatchReservePrices – Dispatch reserve prices: the 'new' real-time reserve prices.
  • GDX: the RTD GDX files as described above.
  • CaseFiles – RTD case files: the RTD case files.
  • InterimPrices – Interim prices: from 1 November 2022, interim prices will be calculated as a time-weighted average of the 'new' real-time prices. Interim prices will pertain to the entire trading period. 
  • FinalPrices – Final prices: if no pricing error claim is lodged pertaining to interim prices, then interim prices will be declared final by 2pm the following business day. Whereas interim prices will be published by trading period, i.e. one file per trading period, final prices will be published by day, i.e. one file per day containing prices for all trading periods from the previous day.   

vSPD online

vSPD online will be disabled on 1 November until such time as we get it rebuilt to work with both the new and old versions of vSPD.

Final pricing dataset

The final pricing dataset will cease to be populated beginning on 1 November 2022. The entire folder will likely be archived within the Dispatch and pricing folder described above.    

  • Liked by
  • Josh Smith
Order by: Standard | Newest | Votes
Phil Bishop posted this 17 October 2022

A new version of vSPD has been posted to the wholesale tools page. The same version can also be extracted directly from GitHub. This most recent version of vSPD, v3.1.0, will be the last version compatible with GDX files up to and including trading day 31 October 2022. Functionally, v3.1.0 is no different to the version that preceded it so there is no real urgency or need to update your vSPD instance. What v3.1.0 does include is a lot of structure that has been deployed to SPD over recent months in readiness for real-time pricing on 1 November 2022.  

In the next few days we will publish vSPD v4.0.0 and some sample GDX files that will be of a format applicable post 1 November 2022. Note that v4.0.0 will not be backward compatible with GDX files that pre-date 1 November 2022.

  • Liked by
  • Matthew Keir
Phil Bishop posted this 26 October 2022

RTD SPD case files and RTD GDX files are now being published.

ptan posted this 01 November 2022

Hi Phil, I was previously loading data from the NodalPricesAndVolumes files from the FinalPrices dataset and it appears to have been stopped. Is there an analog we can use as a replacement with the RTP changes?

Phil Bishop posted this 01 November 2022

No replacement for NodalPricesAndVolumes just yet but there very soon will be - likely next week. However, it will be about six times the size each day, as it will be constructed from the RTD case files.  Do you use the MW values from the file or do you convert to MWh? 

Phil Bishop posted this 01 November 2022

Interim and final energy and reserve prices as calculated as per real-time pricing are now available.

 

ptan posted this 01 November 2022

Righto, we load in the MW values. Will the file format stay the same when it restarts?

Phil Bishop posted this 01 November 2022

not quite the same format, no. At a minimum we need identify each case within the trading period. Also, given the RTD cases are not generated at a uniform interval, the length (in minutes and seconds) of each interval needs to be used to convert MW to MWh - hence my question earlier. So we'll probably show MW and MWh for all volumes. 

  • Liked by
  • ptan
ptan posted this 13 November 2022

Hi Phil, Is there any update on when the new NodalPricesandVolumes files will arrive?

Phil Bishop posted this 13 November 2022

Later this week or beginning of next week. We had a lot of staff absences last week so got a little behind schedule.

  • Liked by
  • ptan
Phil Bishop posted this 14 November 2022

vSPD v4.0.0 has been uploaded to the vSPD tools page. It can also be found on GitHub. Note that with v4x, backward compatibility has been severed - v4x and onwards will only work with GDX files from 1 Nov 2022 onwards.

Phil Bishop posted this 18 November 2022

Dispatch energy and dispatch reserve prices are now being published each day on EMI.

ptan posted this 29 November 2022

Hi Phil, will the new NodalPricesandVolumes files arrive soon?

 

ptan posted this 09 February 2023

Hi Phil, I'm seeing the new NodalPricesandVolumes folder in the DispatchAndPricing folder. Are we close to seeing those files being populated?

Phil Bishop posted this 09 February 2023

very close. The files going back to 1 Nov were actually generated about a week ago but I noticed a few glitches. The guys are just working through some tweaks to the underlying dataset and then we'll be in business. Hopefully early next week! And cleared offers won't be far behind either.

 

  • Liked by
  • ptan
ptan posted this 20 February 2023

Hi Phil, any update on the timeline for these files?

Show more posts

This discussion is locked, please start a new discussion.