New datasets on EMI – Forecast prices and real-time prices

  • Last post 09 December 2019
Phil Bishop posted this 09 December 2019

We recently started publishing forecast prices and real-time prices on EMI datasets. The price files for each schedule type are gathered up into a daily zip file and are published once per day. Publication of these data are designed to support research and analytical requirements, rather than real-time operations. See the relevant folders in EMI datasets for more information. When the back-fill is complete, all of these price series will go back to 20 October 2018.

In the first quarter of 2020, we should be able to get some new APIs up and running so that these data are also available in a more customisable manner, as well as much closer to real-time. The new APIs that we plan to stand up next year will replace the current RTP and RTD APIs. The current APIs were commissioned a few years ago now, primarily to demonstrate the strength of demand for wholesale market information via APIs. The demand clearly exists, and building APIs for publicly available data such as prices is now on the WITS development programme. However, the WITS APIs are still some time away.

Order by: Standard | Newest | Votes
Phil Bishop posted this 09 December 2019

All wholesale prices, whether they be forecast prices (i.e. pre-dispatch), real-time prices (i.e. 5-minute prices), or ex post final prices are generated using the scheduling, pricing, and dispatch (SPD) model. Understanding how prices are 'predicted' is not a trivial undertaking. If you're starting with no knowledge of linear programming or wholesale electricity markets, then buckle up - you're in for quite a ride.

Some places to start would be:

  • Part 13 of the Code - search for words like scheduling, pricing, the dispatch process, constraints - basically just read all of Part 13
  • Transpower's scheduling and dispatch page
  • The summary of SPD inputs by schedule - from Transpower's SPD page
  • Any introductory linear programming / mixed integer programming text book - SPD is essentially a reasonably large transportation problem looking to satisfy a fixed demand at least cost from a set of fixed-cost supply options. The energy needs to move through a capacitated network to get from the supply locations to the demand locations. Then a whole lot of other stuff gets layered on top of that basic transportation problem in the form of constraints.
  • You can download vSPD, the Authority's GAMS-based, audited, mathematical replica of SPD and check out the file called vSPDmodel.gms - it describes the entire SPD model in symbolic form.
  • You could go to vSPD-online and actually solve vSPD for yourself. Having replicated the final prices from SPD, you can then modify generator offers and/or the fixed demand and solve for a new set of prices.
    • If you're going to use vSPD-online, you'll probably want to read about vSPD GDX files.

Good luck!


paulatsydney posted this 09 December 2019

Great news.


Is there any document explains how are the forecast prices predicted?