Real-time pricing on 1 November 2022 - impact on EMI reports and datasets

  • This discussion is locked, please start a new discussion.
  • 3.7K Views
  • Last post 07 November 2023
Phil Bishop posted this 13 October 2022

Real-time pricing (RTP) goes live on 1 November 2022. The implementation of RTP will necessitate changes to a lot of the data received, processed, and published by the EMI team. If you are a regular user of Wholesale reports or Wholesale datasets, you might want to keep an eye on this discussion for regular updates over the next few weeks.

Changes are going to be made with very little notice. Some reports or datasets may be stopped altogether on 1 November and restored in the following days or weeks.

vSPD and GDX files

The vSPD model is undergoing substantial change and testing in readiness for real-time pricing. A new version of vSPD will very soon be made available. It will not be backward compatible with the current version. After 1 November 2022, if you want to run simulations that span 30 Oct / 1 Nov 2022, you’ll need to use both versions of vSPD, i.e. the current and the new versions.

GDX files are currently produced from the ex post final pricing (FP) SPD cases. FP GDX files will cease to exist after 1 November and in their place GDX files will be produced from the real-time dispatch (RTD) cases. There will be an indeterminate number of these per day but typically there will be well over 200. These new RTD GDX files will be published beginning 1 Nov. At some point after 1 November 2022, we will change the GDX processing to package all RTD GDXs for a day (midnight to midnight) into a single GDX file, which will be published early in the morning (2-3am’ish) of the following day. The daily RTD GDX file, when we start publishing it, will be approximately six times larger than the current FP GDX files.

Dispatch and pricing

A new wholesale dataset folder will be created called DispatchAndPricing. Beginning on 1 November, the following datasets will be published into subfolders of this new folder as follows:

  • DispatchEnergyPrices – Dispatch energy prices: the ‘new’ real-time energy prices that will replace the current real-time (or 5-min) prices published here. The current real-time prices will cease to be produced from 1 Nov.
  • DispatchReservePrices – Dispatch reserve prices: the 'new' real-time reserve prices.
  • GDX: the RTD GDX files as described above.
  • CaseFiles – RTD case files: the RTD case files.
  • InterimPrices – Interim prices: from 1 November 2022, interim prices will be calculated as a time-weighted average of the 'new' real-time prices. Interim prices will pertain to the entire trading period. 
  • FinalPrices – Final prices: if no pricing error claim is lodged pertaining to interim prices, then interim prices will be declared final by 2pm the following business day. Whereas interim prices will be published by trading period, i.e. one file per trading period, final prices will be published by day, i.e. one file per day containing prices for all trading periods from the previous day.   

vSPD online

vSPD online will be disabled on 1 November until such time as we get it rebuilt to work with both the new and old versions of vSPD.

Final pricing dataset

The final pricing dataset will cease to be populated beginning on 1 November 2022. The entire folder will likely be archived within the Dispatch and pricing folder described above.    

  • Liked by
  • Josh Smith
Order by: Standard | Newest | Votes
guanzGE posted this 07 November 2023

Hi Phil,

Sorry, I have just found that I was using a very old version of GAMS that couldn't use the new feature.

Regards,

Ziming Guan

Phil Bishop posted this 07 November 2023

I don't think that's an error - I think it's a somewhat new feature in GAMS that enables set membership to be initialised more flexibly. In any event, best if Tuong answers this one rather than me.

Phil

guanzGE posted this 05 November 2023

Thanks, Phil. I downloaded vSPD 5.0 released last month on Github. It returned some errors, e.g., the "<" after the set element in caseDefn(ca,cn<,rundt<) in vSPDmodel.gms. 

Regards,

Ziming Guan

Phil Bishop posted this 05 November 2023

See the latest release (v5.0.0) at https://github.com/ElectricityAuthority/vSPD.

I haven't yet updated the vSPD page on EMI but will do so when I get a few minutes. 

guanzGE posted this 05 November 2023

Hi Phil,

The daily GDX files for pricing have been published and I wonder when the vSPD 5.0 will be availble. 

Regards,

Ziming Guan

ptan posted this 09 May 2023

Hi Phil,

Any update for when the cleared offers files will arrive?

Regards,

Perry

guanzGE posted this 25 April 2023

Hi Phil,

Thank you for the update. 

Regards,

Ziming

Phil Bishop posted this 25 April 2023

Hi Ziming

As you are no doubt already aware, the RTD and PRSS GDX files are published here but they are not yet packaged up into a single daily GDX file. Given our work priorities, I'd say getting them into a single file is still some weeks away - if it's not done by 30 June, it will be Aug/Sep before it happens.

Phil  

guanzGE posted this 21 April 2023

Hi Phil,

I wonder when the daily RTD GDX files will be published.

Regards,

Ziming

Phil Bishop posted this 28 February 2023

Of your two options, I'd probabaly do the latter. I can't really tell you what to do here - you need to figure out what works for your particular purpose.

The generation MW from dispatch cases, which are used by the SO to craft dispatch instructions, are generally relevant for longer than the 5-min interval. I'm pretty sure a dispatch instruction stays in place until another one is issued that supersedes it.    

ptan posted this 28 February 2023

Okay fair enough that excel is mucking the data up. So to calculate the MW for that 5 minute period, how would I use both cases for that? Do I take a raw average or something more complicated like taking the timestamps from the caseID and recomputing a weighted average?

Phil Bishop posted this 28 February 2023

Can you please use a text editor to inspect the file (or anything other than Excel). I think you'll find the two case IDs are not identical; rather, they are 11012022110620615 and 11012022110620616 - in which case you'll want to take both rows. I've only spent a few minutes looking into this but it appears to be a (not uncommon) situation where more than one RTD case was generated within a 5-min interval, i.e. the SO needed to redispatch somebody.

Please let me know if you remain confused.

Phil

 

ptan posted this 28 February 2023

Hi Phil,

Crunching the data, I've found an odd occurance where there are two lines for the same IntervalDateTime (See 1/11/2022, T19:20:00) where the caseID is the same for the two lines but the file has different load, initial, gen and location factor values. Which line should we be taking in this instance?

ptan posted this 27 February 2023

I've just got a few main questions for now that we were hoping you could help with:

1. How is the interval defined for the new files? Guessing that the megawatt reading would be for the period starting from the intervaldatetime until right before the next one on the list? Also how would this work when the PoC lacks a reading at 00:00:00 - do we get the prev day's file to get a reading?

2. What's the initialmegawatts column? It wasn't in the previous iteration of the file.

3. Where is the location factor relative to? HAY2201 was our best guess.

Phil Bishop posted this 27 February 2023

see the blurb at https://www.emi.ea.govt.nz/Wholesale/Datasets/DispatchAndPricing/NodalPricesAndVolumes.

Let me know if you need more. I was going to list and define each column from the CSV file but they all seemed self-explanatory so I didn't bother.

 

Show more posts

This discussion is locked, please start a new discussion.