EMI dataset changes – Current bids and offers data to be decommissioned and replaced on 28 Aug 2017

  • 1.5K Views
  • Last post 20 September 2017
Phil Bishop posted this 13 August 2017

All users of the bids or offers data files that we publish on EMI Datasets should be aware that a replacement set of files has been published. The new files are available now at ..\Datasets\Wholesale\BidsAndOffers. Users should also be aware that Datasets on EMI is just a browser-based view of our FTP server, so this post applies also to bids and offers files obtained directly from the FTP server.

On 28 August 2017, the folder called ..\Datasets\Wholesale\Bids_and_offers and all of its contents will be deleted. Users that have scripts to automatically obtain and process these files will need to modify those scripts before 28 August 2017.

The Market Analytics team has been undertaking a revamp of our data warehouse. Among other changes, a common nomenclature and set of business rules is being applied across all datasets - this will be seen in the revised structure of the bids and offers files. The files to be decommissioned also contain some anomalies that the revamp corrects. For example, duplicate records, particularly those on the days that daylight savings ends, have been eliminated.

The remainder of this post presents the new column names as well as relates the new to the old.

Bids (new column names compared with the old - old in uppercase)

  • PointOfConnection - PNODE_NAME
  • Unit - PNODE_NAME
  • OtherGridEntity - PNODE_NAME
  • Trader - TRADER_NAME
  • UTCSubmissionDate - SUBMISSION_DATE and SRC_COMIT_ID
  • UTCSubmissionTime - SUBMISSION_DATE and SRC_COMIT_ID
  • SubmissionOrder
  • IsLatest - CURRENT_FLAG
  • DispatchableYesNo
  • TradingDate - TRADING_DATE
  • TradingPeriod - TRADING_PERIOD
  • Band - BAND
  • Megawatt - MW
  • DollarsPerMegawattHour - PRICE

BID_TYPE_DESCRIPTION and CONFORMING_BID_TYPE are columns from the old files that are not relevant and have no equivalent in the new files.

Offers (new column names compared with the old - old in uppercase)

  • PointOfConnection - PNODE_NAME
  • Unit - PNODE_NAME
  • OtherGridEntity - PNODE_NAME
  • Trader - TRADER
  • ProductType - ORDER_TYPE and RESERVE_TYPE
  • ProductClass - ORDER_TYPE and RESERVE_TYPE
  • ProductSubClass - ORDER_TYPE and RESERVE_TYPE
  • ProductCategory - ORDER_TYPE and RESERVE_TYPE
  • UTCSubmissionDate - SUBMISSION_DATE
  • UTCSubmissionTime - SUBMISSION_DATE
  • SubmissionOrder
  • IsLatest
  • TradingDate - TRADING_DATE
  • TradingPeriod - TRADING_PERIOD
  • Band - BAND
  • MaximumRampUpMegawattPerHour - MAX_RAMP_UP
  • MaximumRampDownMegawattPerHour - MAX_RAMP_DOWN
  • PartiallyLoadedSpinningReservePercent - PLSR_PERCENT
  • MaximumOutputMegawatt - MAX_OUTPUT
  • Megawatt - MW
  • DollarsPerMegawattHour - PRICE

PNODE_TYPE is a column from the old files that is not relevant and has no equivalent in the new files.

Additional notes

The existing bids and offers data for the few months prior to 13 December 2012 have not been published in the new files as the data from this period is of poor quality and contains many anomalies that we are unable to resolve.

Submission dates have been provided in UTC format rather than some unspecified version of New Zealand time in order to conveniently deal with daylight saving.

SubmissionOrder provides an ordinal ranking of the submission order. For some purposes, this ordinal ranking is a more convenient means of determining order than parsing the submission time. IsLatest is a simple Y/N flag to indicate which submission is the final submission for any set of identifying keys.

The DispatchableYesNo column in the bids' files relates only to bids that are for demand that is to be dispatched by the system operator. In these cases, the trader may submit a bid and then quite late in the piece, certainly well after gate closure, decide that they no longer want the system operator to dispatch them. Rather than nullify all aspects of the previously submitted bid, the trader may simply change the dispatchable flag from yes to no. In other words, the status of this flag takes precedence over the price and quantity tranches that have previously been submitted.  

The new files are arranged in subfolders by year, i.e. the monthly subfolders have been done away with.

In our revamped data warehouse, all data structures that denote a point on the grid or a network (e.g. field names such as pnode, node, GIP, GXP, POC, etc) are passed through a function and characterised as a:

  • PointOfConnection (POC) - a varchar(7) of the form ABC1234, a
  • Unit - a varchar of the form ABCx where x is generally a single-digit integer but very occasionally might be more than a single integer, and an 
  • OtherGridEntity (OGE) - a varchar of an arbitrary length but generally no more than about 10 characters.

The Other Grid Entity is something of a catch-all category to capture all point labels we encounter that don't fit the POC or Unit mould. Generally speaking, it is used to denote one end or the other, and maybe a flow direction, of the HVDC. There are no OGEs for either bids or offers.

Similarly, the myriad of product labels that we encounter in the data we receive has been classified in a four-level hierarchical product dimension as shown in the following table. Products are not relevant for bids. The product classification is mapped to the ORDER_TYPE and RESERVE_TYPE columns from the old offer tables in the table below.

 

  • Liked by
  • bfung
Order by: Standard | Newest | Votes
Phil Bishop posted this 18 August 2017

Sorry guys - since making this post four days ago, we've slightly revised the labelling of the 4 product-related fields. They are now:

  • ProductType
  • ProductClass
  • ProductSubClass
  • ProductCategory

This change impacts only offers and has been reflected in the post above, which I have edited accordingly. The revised (new) offer CSV files are being generated as I write this. Hopefully, this hasn't inconvenienced you unduly.

Phil

18 Aug 2017

Phil Bishop posted this 20 September 2017

Hey guys - we're going to reproduce all of the bids and offers CSV files one more time. Hopefully this isn't too much of an inconvenience.

The filenames will change slightly so you'll need to tweak your scripts - files will be called YYYYMMDD_Bids.csv and YYYYMMDD_Offers.csv, i.e. we're adding an underscore.

More importantly, the files will get larger because we're writing out the first submission in full, even if some or all bands of the first submission are zero. Thereafter, it is only when some aspect of the bid or offer changes that we will write the revised bid/offer to the file.

The initial submission for bids will contain 10 bands (although typically only about three are non-zero). For energy offers there will be five bands, and for reserve offers there will be three. This is because the Code permits this many tranches or bands.

Phil

21 Sep 2017