Reconciled Demand and Generation Reports

  • 1.3K Views
  • Last post 24 June 2021
tship posted this 20 September 2020

Re: Reconciled demand and generation reports

"

Reconciled demand and generation data is our most complete source of data and refers to all purchases and sales on the wholesale market that are cleared through the clearing manager. This data becomes available around the middle of the following month. More specifically, reconciled demand and generation refers to loss-adjusted purchases and sales, and represents the purchase and sale quantities upon which money changes hands between participants and the clearing manager. The loss-adjustment means that the actual consumption data submitted to the reconciliation manager by retailers and other purchasers is grossed up to account for losses in the distribution networks.

"

is this data just month0 version or does it get updated over time as say washup files get loaded ?

Thanks in advance ....

Generation
Demand
Reconciliation
Order by: Standard | Newest | Votes
Matthew Keir posted this 20 September 2020

Hi there,

That's a good question. The data does get updated with each run of reconciliation for a particular month so the impact of wash-ups is all captured.

Cheers,

Matthew

  • Liked by
  • tship
  • msouness
tship posted this 23 September 2020

Hi again Mathew,

Is there anyway of identifying if the historical data has changed, other than downloading the file and comparing it against a previous download - eg is the information stored as a dataset anywhere else on the site with a file modified date ?

We do get NZRM files on a regular basis (types listed below)  ... are the report data changes above linked to the receipt of any of the below files eg SCOREMS

FILE_TYPE
ACCYHHR
ACCYNHH
BALDATA
ESUPSUB
ICPCOMP
ICPCSUM
ICPMISS
ICPMSUM
ICPSCAL
ICPSWIT
NHHPAVE
RECCONS
RECGENR
SCOREMS
UFEFANN
UFEFMTH
UFEFTPD
UFESUMM

Thanks again,

Tim Shipley

 

 

Phil Bishop posted this 23 September 2020

What we call reconciled demand and reconciled generation comes from the GR-010 report produced by the reconciliation manager. See here for the functional spec. The two files we receive have RECCONS and RECGENR as part of their names. But if you obtain the GR-010 files, you will only get the portion of the file pertaining to your organisation.

To answer your question, no, we do not indicate which revision number is in effect at any given time. But, as Matthew indicated above, what we use will always be the most recent wash up or revision.

This will soon be a moot point as we are shortly going to start publishing the GR-010 reports in EMI datasets and we will devise a file naming convention that allows you to know what revision we're up to and when that revision was made.

Phil 

  • Liked by
  • tship
  • msouness
tship posted this 23 September 2020

Thanks Phil,

Do you have a rough date when you expect those datasets to be available (i'm looking to develop an extract to load this data into our data warehouse) ?

Tim.

Phil Bishop posted this 23 September 2020

With all that we have going on at the moment, it's a little difficult to be too precise about the timing until we get a little further into the task. But it would be safe to say that we'll have it done within the next two or three weeks.

tship posted this 23 September 2020

Perfect ... thanks Phil.

tship posted this 18 October 2020

Morning Phil,

Just wondering if there was any news on the GR-010 reports in EMI datasets availibility ...

Thanks in advance,

Tim.

Phil Bishop posted this 19 October 2020

Hi Tim - it should be ready to go live during the latter part of next week.

It's practically ready now - just need to do some testing. The developer doing this work has a bunch of other things on the go right now and is also about to take some annual leave. We'll put off go live until he gets back.

Cheers

Phil

  • Liked by
  • msouness
tship posted this 19 October 2020

Thanks Phil ...

Phil Bishop posted this 17 November 2020
  • Liked by
  • tship
  • msouness
tship posted this 14 April 2021

@Phil Bishop

Hi Phil,

I noticed that February & March's  files uploaded today look to be incomplete ....

Name                                                                                                 Date modified  File size
ReconciledInjectionAndOfftake_202101_20210223_133616.csv.gz 14 Apr 2021    39,782 KB
ReconciledInjectionAndOfftake_202102_.csv.gz                               14 Apr 2021    19,739 KB
ReconciledInjectionAndOfftake_202103_.csv.gz                               14 Apr 2021    00 KB

Could you advise ?

Thanks

msouness posted this 14 April 2021

Now that we have the allocated volumes publicly available, I'd like to suggest that submission volumes be published.  
The reason for this is to allow third party examination of the loss allocation ratios for each trader. This will help reduce UFE.

Phil Bishop posted this 14 April 2021

Hi Tim - the problem with the publishing of the reconciliation data has been rectified. Thanks, Phil. 

  • Liked by
  • tship
tship posted this 20 May 2021

Hi Phil,

Just wondering when April's data will be available ? - roughly what day of the month would you expect the previous month's data to become available going forward ... i'm configuring / testing an automated download at the moment.

Thanks once again for your help,

Tim.

Phil Bishop posted this 20 May 2021

It should def be there already. I'll ask someone to take a look at why it isn't. Cheers, Phil

  • Liked by
  • tship
tship posted this 20 May 2021
tship posted this 21 May 2021

Hi again Phil,

I can see the files there now ... but the sizes are quite variable, especially April being a lot smaller than the others  ... if i tail the bottom of that file i get rows with empty trading periods ...

==> ReconciledInjectionAndOfftake_202104_20210508_183302.csv <==

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Offtake,20.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Offtake,2.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Offtake,19.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Injection,0.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Injection,0.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Injection,0.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Injection,0.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Offtake,1.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Injection,0.0

BOB1101,COUP,NI,PEAK,2021-04-25,"","",Offtake,19.0

that also occurs in the other files, just tailing the last 10 rows of each gives the odd empty trading period ... could you take a look ?

ReconciledInjectionAndOfftake20210120210223_133616.csv.gz 20 May 2021 51,284 KB

ReconciledInjectionAndOfftake20210220210323_112841.csv.gz 20 May 2021 45,782 KB

ReconciledInjectionAndOfftake20210320210428_201230.csv.gz 20 May 2021 50,950 KB

ReconciledInjectionAndOfftake20210420210508_183302.csv.gz 21 May 2021 35,078 KB

Phil Bishop posted this 21 May 2021

Yep, it's being worked on as we speak. My guess is that it will be next week before this gets sorted out. 

 

 

  • Liked by
  • tship
tship posted this 24 May 2021
Phil Bishop posted this 24 May 2021

I'm assured they are correct although they seem to contain about 40% zero records making them quite a bit bigger than they need to be. The zero records issue will be addressed, i.e. they will be filtered out, and when that happens, I'll then check the data is correct. All that said, a cursory glance makes me think the data is already correct.

tship posted this 24 May 2021

ok thanks Phil .. i'll wait for the removal of the zero records then,  if you can advise after that ...

Phil Bishop posted this 24 May 2021

Okay, the smaller files are now there. I still haven't had a chance to check them (although I'm assured they're all good!) but you know the old saying - users are the best testers. So have it at and please let me know if you find anything to be amiss.

Cheers

Phil

 

 

 

  • Liked by
  • tship
tship posted this 22 June 2021

Hi Phil,

There is no May data available at the moment (and many of the recent trading months files have not been updated) .... could you advise ?

Thanks,

Tim.

Phil Bishop posted this 24 June 2021

The RM data for the past several years has been republished. In addition, the May 2021 file is now there.

 

tship posted this 24 June 2021

Thanks for that Phil,

I've downloaded all the changed files. Just one question, as far as versions of the data, i'm pretty sure you were uploading a changed filename when you had changed data for that trading month (ie with a differerent runtime), whereas recently you've just been refreshing the same named file ... could you advise on that ?

Cheers,

Tim.

Categories
Find discussions by tag
See all
This week's high earners
Popular discussions

Our 1273 members have posted 1170 times in 327 discussions

Latest members:

170 users are currently online.

EA tel: + 64 4 460 8860 EMI email: emi@ea.govt.nz Legal information Glossary

© 2024 Copyright material on this site is protected by copyright owned by the Electricity Authority or its licensors.