This discussion describes some forthcoming changes to the Frequency Keeping (FK) offer data that we publish on EMI. The changes will most definitely impact those users of the FK offer data that have automated processes to download files from our Azure storage containers.
The XML and CSV files currently found in the FinalOffers and DispatchedOffers folders will cease to be updated from 4 Dec. This date will be confirmed nearer to 4 Dec.
The underlying Azure storage containers that will no longer be updated from 4 Dec are here and here.
The remainder of this discussion describes what will replace the aforementioned files. Please be aware that almost everything about the published FK offer data and the publishing regime will change on 4 Dec:
- file structure, i.e. the number of columns and the column names
- file names
- file locations
- file format, i.e. CSV versus XML
- file contents, i.e. the actual data, in the case of final offers.
Frequency keeping offers (all FK offers)
The current files containing final offers will be replaced with a file that contains all offers, including modfied offers, i.e. the new files will have more records than the current files. The records that represent final offers will be denoted with an ‘IsLatest’ flag.
The FK offer files will be published daily in a CSV format and will be named YYYYMMDD_FKOffers.csv, i.e. the word ‘final’ is dropped from the file name, as all submitted offers will be included in the files.
The daily CSV files will be collated in yearly folders found in the file path ..\Wholesale\FrequencyKeeping\Offers\.
The published file for any given trading day will be published to the same file name for three days in a row, beginning the day after the trading day. For example, the file called 20181204_FKOffers.csv will be published on 5, 6 and 7 Dec 2018.
The current monthly files that replace the daily collections will be done away with, i.e. at the end of the calendar year, each yearly folder will contain 365 CSV files (366 in leap years).
The FK offer files will be found here on EMI datasets and here in the underlying Azure storage container.
The column names and the order of the columns in the daily FK offer CSV files are as follows:
- Participant
- OfferBlock
- TradingDate
- TradingPeriod
- SubmissionDate
- SubmissionTime
- UTCSubmissionDate
- UTCSubmissionTime
- PointOfConnection
- Unit
- Block
- IsLatest
- Megawatt
- MinimumMegawatt
- MaximumMegawatt
- Dollars
These column names should be self-explanatory to current users of these data but if you have any questions, please post to the bottom of this discussion. Within the next day or two, production files will begin being published at the locations noted above, ahead of the 4 Dec go-live date.
Frequency keeping dispatched offers
The FK dispatched offer files will be published daily in a CSV format and will be named YYYYMMDD_FKDispatchedOffers.csv. The daily CSV files will be collated in yearly folders found in the file path ..\Wholesale\FrequencyKeeping\DispatchedOffers\.
The FK dispatched offer files will be found here on EMI datasets and here in the underlying Azure storage container. Unless stated otherwise, all comments made above regarding the FK offer files will apply to the FK dispatched offer files.
The column names and the order of the columns in the daily FK dispatched offer CSV files are as follows:
- Participant
- TradingDate
- TradingPeriod
- SubmissionDate
- SubmissionTime
- UTCSubmissionDate
- UTCSubmissionTime
- PointOfConnection
- Unit
- Block
- Island
- Megawatt
- Dollars
Constrained on/off costs
The FK constrained on/off cost files will remain in their current location for now and will continue to be published monthly. However, at some point in the not too distant future we will announce some revisions regarding these files on this forum.
Accessing Azure storage accounts
Finally, see here for some general discussion on accessing data from Azure storage containers.
