New EMI API release – Version 2 of the ICP connection data API now available

  • 2.6K Views
  • Last post 2 weeks ago
Matthew Keir posted this 25 May 2020

As part of our initiatives to enhance additional consumer choice of electricity services (ACCES) and customer access to their data, we have released version 2 of the ICP connection data API. This release follows an external review of the API which found no material issues.

The original ICP connection API (v1) will eventually be deprecated but for now will continue to operate so that services relying on it will not be interrupted. Version 1 of the API will be turned off on 1 September 2020.

Subscribers to the API should migrate to version 2 and take advantage of the additional information at their convenience, but should do so before 1 September 2020. Migrating to the new version will require a simple change to the URL call to include v2 in the call and some changes to the search call request parameters - see release notes below. Some minor tweaks will also be required to appropriately handle the additional fields and improved error responses available in v2. Users can test the new API and responses on the API platform.

The My Meter webpage has also been updated to provide access to the new registry fields released through v2 of the API.


Version 2 ICP connection data API release notes

The original ICP connection data API that was released in 2016 as part of the Authority's 'Retail Data Project (RDP)'.

Version (v2) is now available and follows further consultation in 2019 as part of the 'Additional Consumer Choice of Electricity Services (ACCES)' project. This version of the API includes additional fields along with some other enhancements.

'Get by ID' and 'Get by ID list' call changes

New fields released

  • "PropertyNameOrDescription" (string) in the Address object (called "Property Name" in the registry functional spec)
  • "InitialElectricallyConnectedDate" (string formatted as YYYY-MM-DD) in the Network object
  • "TraderSwitchInProgress" (boolean) in the Trader object
  • "ANZSICcode" (string) in the Trader object
  • "ProfileCode" (string) in the Trader object
  • "CertificationExpiryDate" (string formatted as YYYY-MM-DD) in the Meter installation object
  • "ComponentType" (string) in the Meter component object
  • "SwitchReadIndicator" (boolean) in the Meter channel object (called "Settlement Indicator" in the registry functional spec, but only indicates if the channel read may be needed in the switch process)
  • "NetworkParticipantName" (string) in Network object (included via lookup of "NetworkParticipantID" field, will repeat the ID if not found)
  • "TraderParticipantName" (string) in Trader object (included via lookup of "TraderParticipantID" field, will repeat the ID if not found)
  • "MeteringEquipmentProviderParticipantName" (string) in Metering object (included via lookup of "MeteringEquipmentProviderParticipantID" field, will repeat the ID if not found)

Moved fields

  • "ICPIdentifier" moved to first in the return order
  • "ICPStatus" moved to second in the return order

Renamed fields

  • "NetworkParticipantID" in the Network object renamed from "Network"
  • "TraderParticipantID" in the Trader object renamed from "Trader"
  • "MeteringEquipmentProviderParticipantID" in the Metering object renamed from "MeteringEquipmentProvider"

'Search' call changes

New fields released

  •  "PropertyNameOrDescription" (string) in Address object (called "Property Name" in the registry functional spec)

Renamed request parameters

  • “streetNumber” renamed from “unitOrNumber”
  • “streetName” renamed from “streetOrPropertyName”

Gating and response code changes

  • 400 'bad request' code included filtering ICPs not meeting the required format
  • Messages will be returned in the messages object with the appropriate code where available

'Get by ID' call - successful response sample (The 'Get by ID list' call follows the same format)

{

"ICPIdentifier": "string",

"ICPStatus": 0,

"Address": {

"PropertyNameOrDescription": "string",

"PhysicalAddressUnit": "string",

"PhysicalAddressNumber": "string",

"PhysicalAddressStreet": "string",

"PhysicalAddressSuburb": "string",

"PhysicalAddressTown": "string",

"PhysicalAddressRegion": "string",

"PhysicalAddressPostCode": 0,

"GPS_Easting": 0.086597829994870867,

"GPS_Northing": 0.089592538769447838

},

"Network": {

"NetworkParticipantID": "string",

"NetworkParticipantName": "string",

"POC": "string",

"ReconciliationType": "string",

"InitialElectricallyConnectedDate": "string",

"GenerationCapacity": 0.59222133055867521,

"FuelType": "string",

"DirectBilledStatus": "string"

},

"Pricing": {

"DistributorPriceCategoryCode": "string",

"DistributorLossCategoryCode": "string",

"ChargeableCapacity": 0.74166837179270262,

"DistributorInstallationDetails": "string"

},

"Trader": {

"TraderSwitchInProgress": false,

"TraderParticipantID": "string",

"TraderParticipantName": "string",

"ProfileCode": "string",

"ANZSICcode": "string",

"DailyUnmeteredkWh": "string",

"UnmeteredLoadDetails": "string"

},

"Metering": {

"MeteringEquipmentProviderParticipantID": "string",

"MeteringEquipmentProviderParticipantName": "string",

"InstallationInformation": [

{

"MeteringInstallationCategory": 0,

"MeteringInstallationType": "string",

"CertificationExpiryDate": "string",

"ComponentInformation": [

{

"ComponentType": "string",

"MeteringComponentSerialNumber": "string",

"MeterType": "string",

"AMIFlag": false,

"CompensationFactor": 0.68670529385750934,

"ChannelInformation": [

{

"MeteringComponentSerialNumber": "string",

"ChannelNumber": 0,

"RegisterContentCode": "string",

"PeriodOfAvailability": "string",

"UnitOfMeasurement": "string",

"EnergyFlowDirection": "string",

"AccumulatorType": "string",

"SwitchReadIndicator": false

}

]

}

]

}

]

},

"Messages": [

"string"

]

}


'Search' call – successful response sample

{

"Address": {

"PropertyNameOrDescription": "string",

"PhysicalAddressStreet": "string",

"PhysicalAddressSuburb": "string",

"PhysicalAddressTown": "string",

"PhysicalAddressRegion": "string",

"PhysicalAddressPostCode": 0,

"GPS_Easting": 0.11158078265068716,

"GPS_Northing": 0.53635229909462989

},

"ICPIdentifier": "string",

"ICPStatus": 0,

"Messages": [

"string"

]

}

Order by: Standard | Newest | Votes
empsean posted this 2 weeks ago

Hi empsean

We have a profile information page with a document about profiles and a list of all profiles in use: https://www.ea.govt.nz/operations/retail/reconciliation/reconciliation-profiles/ . It doesn't show what the potential outputs are though.

cheers

Nicole


Thanks Nicole, that has a couple of them and gives an explination of RPS, so for now at least that will be enough.

Nicole Gagnon posted this 2 weeks ago

Hi empsean

We have a profile information page with a document about profiles and a list of all profiles in use: https://www.ea.govt.nz/operations/retail/reconciliation/reconciliation-profiles/ . It doesn't show what the potential outputs are though.

cheers

Nicole

Clayton posted this 2 weeks ago

Hiya, I believe the profile code relates to the way in which a retailer submits energy data to the market. You can find more information on this under part 15 of the electricity code. It depends what you want to use the information for. There might be more information under the electricity registry functional specification also if you want to look at that. Regards Clayton

  • Liked by
  • empsean
  • msouness
empsean posted this 2 weeks ago

Is there anywhere that gives a list of what the "ProfileCode" is and what the potential outputs are?

I've searched the forum and API documentation but can't find anything.

I've seen a couple of variations RPS and HHR but it'd be helpful to know whats RPS is and if there are other codes in-use

Matthew Keir posted this 03 September 2020

Hi nicekiwi,

Thanks for your comments. We edited the release notes to include the change to "ICP=" that we hadn't listed - this was done some time ago but not posted back here. The information on the API platform will always be the most up to date documentation. The release notes are available in the following location. https://emi.portal.azure-api.net/docs/services/ICP-connection-data-v2/releaseChanges

We made several changes to the gating and error handling to improve consistency within v2 and to prevent rubbish being passed onto the registry, resulting in unnecessary load. The change to error messages was listed in the release notes and in the original post above. I'd encourage you to try the test feature in the API platform for v2 if you run into any issues.

v2 was released in May and v1 scheduled to be turned off on 1 September. We've extended this until 14 September to allow the last few users more time to migrate.

Cheers,

Matthew

nicekiwi posted this 02 September 2020

Another descrepincey/bug between v1 and v2 is when retreiving a single ICP in v1 /single/?icp=??? it will return a JSON array with a single object [ { ...json }]. However in v2 the same request /single/?id=??? will return a single JSON object only { ...json }. Breaking existing error handling.

nicekiwi posted this 04 August 2020

There seems to have been omission in the API v2 release notes. the "Get by ID" request no longer accepts the paramater "id", it now expects "icp" instead. e.g. '/?icp={ICP}' instead of '/?id={ICP}'. Which returns a 404 error if left unchanged. 

Clive Gifford posted this 15 July 2020

Hello Matthew

On what date did the EA decide to cripple the API in this way?

Was this change discussed with anybody outside of the EA before that decision was taken?

Has the EA communicated any details of this specific change (blocking of "special characters") to any other third party user of the API, before your post this afternoon?

How many users are registered to use the ICP Connection Data API?

Thanks,
Clive

  • Liked by
  • nicekiwi
Matthew Keir posted this 15 July 2020

Hi Clive,

Yes, special characters in the search parameters are blocked by design and this is not a bug related to decoding of the URL.

Given the region field in the search call is not a wildcard value, it’s not that useful for consumers who may not easily identify with the precise region names as recorded in the registry. Although, given this is how regions are recorded in the registry, we’ve made a quick change to allow both ‘Nelson & Bays’ and ‘Timaru & Oamaru’ through the search call. 

Cheers,

Matthew

Clive Gifford posted this 14 July 2020

Hello Matthew.

Back on 8 June you wrote (above) that the issue I raised with the street numbers was "by design".

Given that response it seems to me that the EA has been far more interested in spending time on deliberately crippling their v2 API than conducting even a modicum of proper testing of basic functionality, all while also not thoroughly and openly documenting those kinds of changes or acknowledging how much code they could be breaking. My interpretation of this behaviour is that after first not providing a proper and lawful response to my OIA request last year (for all ICP identifiers in the registry) the EA is now (rather stupidly if you ask me) trying to close the stable door after the horse has already bolted. Trying to suggest the API was built with just one narrow use case sounds like nonsense to me.

In any case, can you also please clarify if the v2 API failing to return any records to a search by address request where the region is specified "Nelson & Bays" or "Timaru & Oamaru" is also "by design" or just another bug due to (apparently) failing to decode the encoded URL given the embedded ampersand character? Of course this "feature" blocks searches for a large number of other ICP's also, but even just those two regions (if specified as part of the search) block more than 100,000.

Hmm.. maybe I shouldn't have referred to the ampersand as a "character"? Would you say it is a character, or not?

Where do I send an invoice for debugging the EA's code?

Kind regards,
Clive Gifford

Jack Callister posted this 18 June 2020

Thanks Matt, the participant list and name are exactly what I need to solve my problem. 

Clayton posted this 18 June 2020

I myself couldn't find a reliable list other than the link that Matt has given to the spreadsheet. There might be a static list available from the registry itself, but that would probably require a participant login to access. These days there seems to be a whole lot more traders.

oops I just saw Matt's comment that there is no static list. 

Hi Daniel btw :)

funcsol posted this 18 June 2020

If I was guessing, I'd say the question was for the enumeration of TraderParticipantID

Matthew Keir posted this 18 June 2020

Hi Jack,

I'm not sure I follow but will try and help.

 

The very next field will tell you the participant name related to that trader ID.

"TraderParticipantID": "string",

"TraderParticipantName": "string",

 

We added this name field to version 2 of the ICP connection data  API  because there is no static list of participants - it's always changing as some parties exit (or are acquired), and new ones start trading. 

You can, however, get an overall picture from the retail reporting on EMI or the participant identifiers list which is periodically updated.

 

Cheers,

Matthew

Jack Callister posted this 17 June 2020

Hi Matt, I'm wondering if there's documentation to find a list of possible values for fields such as TraderParticipantId?

Show more posts
Close