New datasets on EMI – More final pricing data now available

  • 1.6K Views
  • Last post 26 October 2016
Phil Bishop posted this 24 October 2016

We have made two new daily CSV files available for download. Both files contain data that is already published in the form of SPD case files, among other places on EMI. The CSV files simply make the data more accessible and easier for users to download and make use of.

Cleared offers can be found here, and load, generation and price can be downloaded from here. In both cases, the files can also be accessed directly from our FTP server.

Load, generation and price data are already available elsewhere – from the Grid generation trends, Grid demand trends, and Wholesale energy prices reports on EMI; and from the Final_prices folder on EMI datasets. This new file is useful in that it gathers together in one place the Load (MW), Generation (MW) and Price ($/MWh) for all points of connection and trading periods where either load or generation is not zero. For the avoidance of any doubt, this data is extracted directly from the final pricing SPD case files without any transformation. Hence, intermittent generation such as wind will appear as negative load.

The cleared offer data are also extracted directly from the final pricing SPD case files without any transformation. The fields in these files are:

  • Date, i.e. trading date
  • TradingPeriod
  • Island
  • PointOfConnection, more preciselyPNode
  • Trader, i.e. the 4-character trader code
  • Type, i.e. offer type – ENOF = energy offers, ILRO = interruptible load reserve offers, PLRO = partially loaded spinning reserve offers, TWRO = tail water depressed reserve offers, and ENDL = dispatchable load
  • ClearedEnergy (MW)
  • ClearedFIR (MW), cleared fast instantaneous or 6-second reserves
  • ClearedSIR (MW), cleared sustained instantaneous or 60-second reserves.

Finally, note that we use the term 'point of connection' rather loosely in the header of these new CSV files.

Phil Bishop posted this 26 October 2016

Some further comments about the availability of this data and the file naming conventions are warranted.

Both the cleared offers CSV files and the load, generation and price files are made available on a daily basis and are placed in folders arranged by year and month. Additionally, the most recent ten daily files will also be available in their respective root directories. This should assist users who want to use scripted processes to obtain the files, i.e. they will only need to point to a single location and the script won't require editing each month to point to a different location.

The most recent information we have for any trading day will be made available at 2:00pm of the day following the trading day. But users should understand that SPD final pricing case files may be received more frequently than this. Furthermore, we don't necessarily know the status - provisional, interim, or final - of final prices at the time the final pricing case file is received. See also the discussion here. Note that the status attaches to published prices, not an SPD case file.

The file naming convention is as follows:

  • Cleared offers: YYYYMMDD_ClearedOffersx_X.csv
  • Load, generation and prices: YYYYMMDD_LoadGenerationPricex_X.csv

where:

  • YYYY denotes year
  • MM denotes month
  • DD denotes day
  • The lower case x, which is generally not present, denotes that SPD and vSPD have yielded different prices for at least one node in at least one trading period
  • The upper case X denotes the status of final prices and may take on the values of P, I, F, or there may be no value at all, where:
    • P stands for provisional
    • I stands for interim
    • F stands for final
    • The absence of an upper case X means that no status has yet been assigned by the pricing manager.

Finally, and for the sake of clarity, the cleared offers and load are inputs to SPD while generation and price are SPD outputs. We extract all of this data - SPD inputs and outputs - from SPD case files. The status indicator - provisional, interim, or final - refers only to the status of prices and is included in the file names to be consistent with the approach used in naming GDX files.