GDX files for vSPD

  • This discussion is locked, please start a new discussion.
  • 4K Views
  • Last post 05 April 2020
Phil Bishop posted this 07 June 2016

The input data required to run vSPD is published in the form of a daily GDX file. Our entire collection of vSPD GDX files associated with final pricing SPD cases can be downloaded from EMI Datasets, or accessed directly from the underlying Azure storage account.

GDX stands for GAMS data exchange and represents a binary file format for use with GAMS. The files are a convenient way to store the input data to be used by a GAMS-based model such as vSPD, and to import the data at model run time.

Part 13 of the Code describes how final prices are to be determined and published. 

GDX files up to 31 October 2022

Final pricing GDX files are generated from SPD final pricing case files. The naming convention for the final pricing GDX files is FP_YYYYMMDDx_X.gdx where:

  • FP denotes final pricing
  • YYYY denotes year
  • MM denotes month
  • DD denotes day
  • The lower case x, which is generally not present, denotes that SPD and vSPD have yielded different prices for at least one node in at least one trading period
  • The upper case X denotes the status of final prices and may take on the values of P, I, F, or there may be no value at all, where:
    • P stands for provisional
    • I stands for interim
    • F stands for final
    • The absence of a value means that no status has yet been assigned by the pricing manager.

YYYYMMDD together denote the trading day, midnight to midnight, to which the file relates. 

Final pricing GDX files are published on EMI Datasets immediately after we have received and processed the SPD case file. Hence, a GDX file can appear on EMI Datasets before any status (P, I or F) is assigned to it by the pricing manager. Furthermore, the pricing status indicator may not appear on the GDX file until some time after the pricing manager has determined the status. It is worth noting that the status indication on the GDX file can change but this does not neccessarily mean that a new file has been generated; rather, we simply rename the file once the new status has been obtained.

Only one GDX file per day is published. If a subsequent final pricing case is generated, we process it to produce the new GDX file, and remove the previous GDX file from EMI.  

GDX files since 1 November 2022

More to come later - for now see here.

SPD and vSPD

SPD, and it’s mathematical replica, vSPD, are large linear programming problems and thus may be degenerate. An explanation of degeneracy in LP problems is beyond the scope of this discussion. Suffice it to say, degeneracy in SPD means that there can be a tie between two candidate prices at a node that give rise to an identical (optimal) objective function value. In other words, while the solution is optimal in both cases, it is arbitrary as to which price is selected to break the tie. SPD and vSPD have not been aligned in terms of tie-breaking rules. The practical implication of degeneracy is that the model has at least one redundant constraint – this is an extremely common occurence in large models.

Any instance of vSPD and SPD producing different prices is investigated. If it is not due to degeneracy, the cause is rectified.

The Authority receives all published SPD case files. Each SPD case file contains all of the inputs required for that case and all of the outputs generated by SPD for that case.

  • Liked by
  • Sam Harvey
Order by: Standard | Newest | Votes
Nick posted this 03 April 2020

Does a number just indicate later revision of the file?  Examples are:

FP_20190710_1.gdx

FP_20190710_2.gdx

If we find an anomaly, for example FP_20190311_F.gdx containes data labelled 20190317, do you want to know?

Thanks

Nick

Phil Bishop posted this 05 April 2020

Yes, if you notice an anomaly, please notify us. I suspect somebody has attempted to correct an observed job failure by manually overriding the automated process and in so doing has used the wrong data. Regardless, we will re-run the job for 11 Mar 2019 and publish the correct file. 

Final pricing SPD cases don't contain any information about the status of prices - i.e. whether those prices are provisional, interim or final. We don't learn what status the pricing manager has assigned to a particular case until the pricing manager publishes the prices, and this occurs quite some time after we've processed the case to create the GDX file. In fact, no SPD case identifier is included with the data when the pricing manager publishes prices, but the run date and time of the case from which the published prices were derived is included. Hence, we need to match run date/times to align the price status (P, I, F) with a case ID. For this reason, we number the GDX files sequentially for any given day until we know what price status suffix to append to the GDX file name.

By the time we know the prices are final, we also know that there will be no more cases generated for that day, so we remove all GDX files without the '_F' suffix. In other words, when the dust settles, there should be one GDX file per day and it should have '_F' in the file name.

Hope this helps

Phil

This discussion is locked, please start a new discussion.