Real-time pricing on 1 November 2022 - impact on EMI reports and datasets

  • Last post 5 weeks ago
Phil Bishop posted this 13 October 2022

Real-time pricing (RTP) goes live on 1 November 2022. The implementation of RTP will necessitate changes to a lot of the data received, processed, and published by the EMI team. If you are a regular user of Wholesale reports or Wholesale datasets, you might want to keep an eye on this discussion for regular updates over the next few weeks.

Changes are going to be made with very little notice. Some reports or datasets may be stopped altogether on 1 November and restored in the following days or weeks.

vSPD and GDX files

The vSPD model is undergoing substantial change and testing in readiness for real-time pricing. A new version of vSPD will very soon be made available. It will not be backward compatible with the current version. After 1 November 2022, if you want to run simulations that span 30 Oct / 1 Nov 2022, you’ll need to use both versions of vSPD, i.e. the current and the new versions.

GDX files are currently produced from the ex post final pricing (FP) SPD cases. FP GDX files will cease to exist after 1 November and in their place GDX files will be produced from the real-time dispatch (RTD) cases. There will be an indeterminate number of these per day but typically there will be well over 200. These new RTD GDX files will be published beginning 1 Nov. At some point after 1 November 2022, we will change the GDX processing to package all RTD GDXs for a day (midnight to midnight) into a single GDX file, which will be published early in the morning (2-3am’ish) of the following day. The daily RTD GDX file, when we start publishing it, will be approximately six times larger than the current FP GDX files.

Dispatch and pricing

A new wholesale dataset folder will be created called DispatchAndPricing. Beginning on 1 November, the following datasets will be published into subfolders of this new folder as follows:

  • DispatchEnergyPrices – Dispatch energy prices: the ‘new’ real-time energy prices that will replace the current real-time (or 5-min) prices published here. The current real-time prices will cease to be produced from 1 Nov.
  • DispatchReservePrices – Dispatch reserve prices: the 'new' real-time reserve prices.
  • GDX: the RTD GDX files as described above.
  • CaseFiles – RTD case files: the RTD case files.
  • InterimPrices – Interim prices: from 1 November 2022, interim prices will be calculated as a time-weighted average of the 'new' real-time prices. Interim prices will pertain to the entire trading period. 
  • FinalPrices – Final prices: if no pricing error claim is lodged pertaining to interim prices, then interim prices will be declared final by 2pm the following business day. Whereas interim prices will be published by trading period, i.e. one file per trading period, final prices will be published by day, i.e. one file per day containing prices for all trading periods from the previous day.   

vSPD online

vSPD online will be disabled on 1 November until such time as we get it rebuilt to work with both the new and old versions of vSPD.

Final pricing dataset

The final pricing dataset will cease to be populated beginning on 1 November 2022. The entire folder will likely be archived within the Dispatch and pricing folder described above.    

  • Liked by
  • Josh Smith
Order by: Standard | Newest | Votes
ptan posted this 5 weeks ago

Hi Phil,

Any update for when the cleared offers files will arrive?



guanzGE posted this 25 April 2023

Hi Phil,

Thank you for the update. 



Phil Bishop posted this 25 April 2023

Hi Ziming

As you are no doubt already aware, the RTD and PRSS GDX files are published here but they are not yet packaged up into a single daily GDX file. Given our work priorities, I'd say getting them into a single file is still some weeks away - if it's not done by 30 June, it will be Aug/Sep before it happens.


guanzGE posted this 21 April 2023

Hi Phil,

I wonder when the daily RTD GDX files will be published.



Phil Bishop posted this 28 February 2023

Of your two options, I'd probabaly do the latter. I can't really tell you what to do here - you need to figure out what works for your particular purpose.

The generation MW from dispatch cases, which are used by the SO to craft dispatch instructions, are generally relevant for longer than the 5-min interval. I'm pretty sure a dispatch instruction stays in place until another one is issued that supersedes it.    

ptan posted this 28 February 2023

Okay fair enough that excel is mucking the data up. So to calculate the MW for that 5 minute period, how would I use both cases for that? Do I take a raw average or something more complicated like taking the timestamps from the caseID and recomputing a weighted average?

Phil Bishop posted this 28 February 2023

Can you please use a text editor to inspect the file (or anything other than Excel). I think you'll find the two case IDs are not identical; rather, they are 11012022110620615 and 11012022110620616 - in which case you'll want to take both rows. I've only spent a few minutes looking into this but it appears to be a (not uncommon) situation where more than one RTD case was generated within a 5-min interval, i.e. the SO needed to redispatch somebody.

Please let me know if you remain confused.



ptan posted this 28 February 2023

Hi Phil,

Crunching the data, I've found an odd occurance where there are two lines for the same IntervalDateTime (See 1/11/2022, T19:20:00) where the caseID is the same for the two lines but the file has different load, initial, gen and location factor values. Which line should we be taking in this instance?

ptan posted this 27 February 2023

I've just got a few main questions for now that we were hoping you could help with:

1. How is the interval defined for the new files? Guessing that the megawatt reading would be for the period starting from the intervaldatetime until right before the next one on the list? Also how would this work when the PoC lacks a reading at 00:00:00 - do we get the prev day's file to get a reading?

2. What's the initialmegawatts column? It wasn't in the previous iteration of the file.

3. Where is the location factor relative to? HAY2201 was our best guess.

Phil Bishop posted this 27 February 2023

see the blurb at

Let me know if you need more. I was going to list and define each column from the CSV file but they all seemed self-explanatory so I didn't bother.


ptan posted this 27 February 2023

Thanks we can see the files now. Where would I find the documentation if its been uploaded?

Phil Bishop posted this 21 February 2023

You should see nodal prices and volumes based on RTD data appear here this morning, going back to 1 Nov 2022. I'll add some documentation in the next day or two. Be aware that I've had no time to do any significant testing/QA on what they guys have produced. That said, I think the files are reliable. Please let me know if you think anything looks awry.   

Hopefully cleared offers should follow in a week or so.


  • Liked by
  • ptan
ptan posted this 20 February 2023

Hi Phil, any update on the timeline for these files?

Phil Bishop posted this 09 February 2023

very close. The files going back to 1 Nov were actually generated about a week ago but I noticed a few glitches. The guys are just working through some tweaks to the underlying dataset and then we'll be in business. Hopefully early next week! And cleared offers won't be far behind either.


  • Liked by
  • ptan
ptan posted this 09 February 2023

Hi Phil, I'm seeing the new NodalPricesandVolumes folder in the DispatchAndPricing folder. Are we close to seeing those files being populated?

Show more posts