• Real-time pricing on 1 November 2022 - impact on EMI reports and datasets

Real-time pricing on 1 November 2022 - impact on EMI reports and datasets

  • This discussion is locked, please start a new discussion.
  • 3.1K Views
  • Last post 07 November 2023
Phil Bishop posted this 13 October 2022

Real-time pricing (RTP) goes live on 1 November 2022. The implementation of RTP will necessitate changes to a lot of the data received, processed, and published by the EMI team. If you are a regular user of Wholesale reports or Wholesale datasets, you might want to keep an eye on this discussion for regular updates over the next few weeks.

Changes are going to be made with very little notice. Some reports or datasets may be stopped altogether on 1 November and restored in the following days or weeks.

vSPD and GDX files

The vSPD model is undergoing substantial change and testing in readiness for real-time pricing. A new version of vSPD will very soon be made available. It will not be backward compatible with the current version. After 1 November 2022, if you want to run simulations that span 30 Oct / 1 Nov 2022, you’ll need to use both versions of vSPD, i.e. the current and the new versions.

GDX files are currently produced from the ex post final pricing (FP) SPD cases. FP GDX files will cease to exist after 1 November and in their place GDX files will be produced from the real-time dispatch (RTD) cases. There will be an indeterminate number of these per day but typically there will be well over 200. These new RTD GDX files will be published beginning 1 Nov. At some point after 1 November 2022, we will change the GDX processing to package all RTD GDXs for a day (midnight to midnight) into a single GDX file, which will be published early in the morning (2-3am’ish) of the following day. The daily RTD GDX file, when we start publishing it, will be approximately six times larger than the current FP GDX files.

Dispatch and pricing

A new wholesale dataset folder will be created called DispatchAndPricing. Beginning on 1 November, the following datasets will be published into subfolders of this new folder as follows:

  • DispatchEnergyPrices – Dispatch energy prices: the ‘new’ real-time energy prices that will replace the current real-time (or 5-min) prices published here. The current real-time prices will cease to be produced from 1 Nov.
  • DispatchReservePrices – Dispatch reserve prices: the 'new' real-time reserve prices.
  • GDX: the RTD GDX files as described above.
  • CaseFiles – RTD case files: the RTD case files.
  • InterimPrices – Interim prices: from 1 November 2022, interim prices will be calculated as a time-weighted average of the 'new' real-time prices. Interim prices will pertain to the entire trading period. 
  • FinalPrices – Final prices: if no pricing error claim is lodged pertaining to interim prices, then interim prices will be declared final by 2pm the following business day. Whereas interim prices will be published by trading period, i.e. one file per trading period, final prices will be published by day, i.e. one file per day containing prices for all trading periods from the previous day.   

vSPD online

vSPD online will be disabled on 1 November until such time as we get it rebuilt to work with both the new and old versions of vSPD.

Final pricing dataset

The final pricing dataset will cease to be populated beginning on 1 November 2022. The entire folder will likely be archived within the Dispatch and pricing folder described above.    

Final pricing
Real-time prices
EMI news
  • Liked by
  • Josh Smith
Order by: Standard | Newest | Votes
Phil Bishop posted this 17 October 2022

A new version of vSPD has been posted to the wholesale tools page. The same version can also be extracted directly from GitHub. This most recent version of vSPD, v3.1.0, will be the last version compatible with GDX files up to and including trading day 31 October 2022. Functionally, v3.1.0 is no different to the version that preceded it so there is no real urgency or need to update your vSPD instance. What v3.1.0 does include is a lot of structure that has been deployed to SPD over recent months in readiness for real-time pricing on 1 November 2022.  

In the next few days we will publish vSPD v4.0.0 and some sample GDX files that will be of a format applicable post 1 November 2022. Note that v4.0.0 will not be backward compatible with GDX files that pre-date 1 November 2022.

  • Liked by
  • Matthew Keir
Phil Bishop posted this 26 October 2022

RTD SPD case files and RTD GDX files are now being published.

ptan posted this 01 November 2022

Hi Phil, I was previously loading data from the NodalPricesAndVolumes files from the FinalPrices dataset and it appears to have been stopped. Is there an analog we can use as a replacement with the RTP changes?

Phil Bishop posted this 01 November 2022

No replacement for NodalPricesAndVolumes just yet but there very soon will be - likely next week. However, it will be about six times the size each day, as it will be constructed from the RTD case files.  Do you use the MW values from the file or do you convert to MWh? 

Phil Bishop posted this 01 November 2022

Interim and final energy and reserve prices as calculated as per real-time pricing are now available.

 

ptan posted this 01 November 2022

Righto, we load in the MW values. Will the file format stay the same when it restarts?

Phil Bishop posted this 01 November 2022

not quite the same format, no. At a minimum we need identify each case within the trading period. Also, given the RTD cases are not generated at a uniform interval, the length (in minutes and seconds) of each interval needs to be used to convert MW to MWh - hence my question earlier. So we'll probably show MW and MWh for all volumes. 

  • Liked by
  • ptan
ptan posted this 13 November 2022

Hi Phil, Is there any update on when the new NodalPricesandVolumes files will arrive?

Phil Bishop posted this 13 November 2022

Later this week or beginning of next week. We had a lot of staff absences last week so got a little behind schedule.

  • Liked by
  • ptan
Phil Bishop posted this 14 November 2022

vSPD v4.0.0 has been uploaded to the vSPD tools page. It can also be found on GitHub. Note that with v4x, backward compatibility has been severed - v4x and onwards will only work with GDX files from 1 Nov 2022 onwards.

Phil Bishop posted this 18 November 2022

Dispatch energy and dispatch reserve prices are now being published each day on EMI.

ptan posted this 29 November 2022

Hi Phil, will the new NodalPricesandVolumes files arrive soon?

 

ptan posted this 09 February 2023

Hi Phil, I'm seeing the new NodalPricesandVolumes folder in the DispatchAndPricing folder. Are we close to seeing those files being populated?

Phil Bishop posted this 09 February 2023

very close. The files going back to 1 Nov were actually generated about a week ago but I noticed a few glitches. The guys are just working through some tweaks to the underlying dataset and then we'll be in business. Hopefully early next week! And cleared offers won't be far behind either.

 

  • Liked by
  • ptan
ptan posted this 20 February 2023

Hi Phil, any update on the timeline for these files?

Phil Bishop posted this 21 February 2023

You should see nodal prices and volumes based on RTD data appear here this morning, going back to 1 Nov 2022. I'll add some documentation in the next day or two. Be aware that I've had no time to do any significant testing/QA on what they guys have produced. That said, I think the files are reliable. Please let me know if you think anything looks awry.   

Hopefully cleared offers should follow in a week or so.

Phil 

  • Liked by
  • ptan
ptan posted this 27 February 2023

Thanks we can see the files now. Where would I find the documentation if its been uploaded?

Phil Bishop posted this 27 February 2023

see the blurb at https://www.emi.ea.govt.nz/Wholesale/Datasets/DispatchAndPricing/NodalPricesAndVolumes.

Let me know if you need more. I was going to list and define each column from the CSV file but they all seemed self-explanatory so I didn't bother.

 

ptan posted this 27 February 2023

I've just got a few main questions for now that we were hoping you could help with:

1. How is the interval defined for the new files? Guessing that the megawatt reading would be for the period starting from the intervaldatetime until right before the next one on the list? Also how would this work when the PoC lacks a reading at 00:00:00 - do we get the prev day's file to get a reading?

2. What's the initialmegawatts column? It wasn't in the previous iteration of the file.

3. Where is the location factor relative to? HAY2201 was our best guess.

ptan posted this 28 February 2023

Hi Phil,

Crunching the data, I've found an odd occurance where there are two lines for the same IntervalDateTime (See 1/11/2022, T19:20:00) where the caseID is the same for the two lines but the file has different load, initial, gen and location factor values. Which line should we be taking in this instance?

Phil Bishop posted this 28 February 2023

Can you please use a text editor to inspect the file (or anything other than Excel). I think you'll find the two case IDs are not identical; rather, they are 11012022110620615 and 11012022110620616 - in which case you'll want to take both rows. I've only spent a few minutes looking into this but it appears to be a (not uncommon) situation where more than one RTD case was generated within a 5-min interval, i.e. the SO needed to redispatch somebody.

Please let me know if you remain confused.

Phil

 

ptan posted this 28 February 2023

Okay fair enough that excel is mucking the data up. So to calculate the MW for that 5 minute period, how would I use both cases for that? Do I take a raw average or something more complicated like taking the timestamps from the caseID and recomputing a weighted average?

Phil Bishop posted this 28 February 2023

Of your two options, I'd probabaly do the latter. I can't really tell you what to do here - you need to figure out what works for your particular purpose.

The generation MW from dispatch cases, which are used by the SO to craft dispatch instructions, are generally relevant for longer than the 5-min interval. I'm pretty sure a dispatch instruction stays in place until another one is issued that supersedes it.    

guanzGE posted this 21 April 2023

Hi Phil,

I wonder when the daily RTD GDX files will be published.

Regards,

Ziming

Phil Bishop posted this 25 April 2023

Hi Ziming

As you are no doubt already aware, the RTD and PRSS GDX files are published here but they are not yet packaged up into a single daily GDX file. Given our work priorities, I'd say getting them into a single file is still some weeks away - if it's not done by 30 June, it will be Aug/Sep before it happens.

Phil  

guanzGE posted this 25 April 2023

Hi Phil,

Thank you for the update. 

Regards,

Ziming

ptan posted this 09 May 2023

Hi Phil,

Any update for when the cleared offers files will arrive?

Regards,

Perry

guanzGE posted this 05 November 2023

Hi Phil,

The daily GDX files for pricing have been published and I wonder when the vSPD 5.0 will be availble. 

Regards,

Ziming Guan

Phil Bishop posted this 05 November 2023

See the latest release (v5.0.0) at https://github.com/ElectricityAuthority/vSPD.

I haven't yet updated the vSPD page on EMI but will do so when I get a few minutes. 

guanzGE posted this 05 November 2023

Thanks, Phil. I downloaded vSPD 5.0 released last month on Github. It returned some errors, e.g., the "<" after the set element in caseDefn(ca,cn<,rundt<) in vSPDmodel.gms. 

Regards,

Ziming Guan

Phil Bishop posted this 07 November 2023

I don't think that's an error - I think it's a somewhat new feature in GAMS that enables set membership to be initialised more flexibly. In any event, best if Tuong answers this one rather than me.

Phil

guanzGE posted this 07 November 2023

Hi Phil,

Sorry, I have just found that I was using a very old version of GAMS that couldn't use the new feature.

Regards,

Ziming Guan

This discussion is locked, please start a new discussion.

Categories
Find discussions by tag
See all
This week's high earners
Popular discussions

Our 1273 members have posted 1170 times in 327 discussions

Latest members:

23 users are currently online.

EA tel: + 64 4 460 8860 EMI email: emi@ea.govt.nz Legal information Glossary

© 2024 Copyright material on this site is protected by copyright owned by the Electricity Authority or its licensors.