New EMI API release – Version 2 of the ICP connection data API now available

  • 26K Views
  • Last post 19 October 2020
Matthew Keir posted this 25 May 2020

As part of our initiatives to enhance additional consumer choice of electricity services (ACCES) and customer access to their data, we have released version 2 of the ICP connection data API. This release follows an external review of the API which found no material issues.

The original ICP connection API (v1) will eventually be deprecated but for now will continue to operate so that services relying on it will not be interrupted. Version 1 of the API will be turned off on 1 September 2020.

Subscribers to the API should migrate to version 2 and take advantage of the additional information at their convenience, but should do so before 1 September 2020. Migrating to the new version will require a simple change to the URL call to include v2 in the call and some changes to the search call request parameters - see release notes below. Some minor tweaks will also be required to appropriately handle the additional fields and improved error responses available in v2. Users can test the new API and responses on the API platform.

The My Meter webpage has also been updated to provide access to the new registry fields released through v2 of the API.


Version 2 ICP connection data API release notes

The original ICP connection data API that was released in 2016 as part of the Authority's 'Retail Data Project (RDP)'.

Version (v2) is now available and follows further consultation in 2019 as part of the 'Additional Consumer Choice of Electricity Services (ACCES)' project. This version of the API includes additional fields along with some other enhancements.

'Get by ID' and 'Get by ID list' call changes

New fields released

  • "PropertyNameOrDescription" (string) in the Address object (called "Property Name" in the registry functional spec)
  • "InitialElectricallyConnectedDate" (string formatted as YYYY-MM-DD) in the Network object
  • "TraderSwitchInProgress" (boolean) in the Trader object
  • "ANZSICcode" (string) in the Trader object
  • "ProfileCode" (string) in the Trader object
  • "CertificationExpiryDate" (string formatted as YYYY-MM-DD) in the Meter installation object
  • "ComponentType" (string) in the Meter component object
  • "SwitchReadIndicator" (boolean) in the Meter channel object (called "Settlement Indicator" in the registry functional spec, but only indicates if the channel read may be needed in the switch process)
  • "NetworkParticipantName" (string) in Network object (included via lookup of "NetworkParticipantID" field, will repeat the ID if not found)
  • "TraderParticipantName" (string) in Trader object (included via lookup of "TraderParticipantID" field, will repeat the ID if not found)
  • "MeteringEquipmentProviderParticipantName" (string) in Metering object (included via lookup of "MeteringEquipmentProviderParticipantID" field, will repeat the ID if not found)

Moved fields

  • "ICPIdentifier" moved to first in the return order
  • "ICPStatus" moved to second in the return order

Renamed fields

  • "NetworkParticipantID" in the Network object renamed from "Network"
  • "TraderParticipantID" in the Trader object renamed from "Trader"
  • "MeteringEquipmentProviderParticipantID" in the Metering object renamed from "MeteringEquipmentProvider"

'Search' call changes

New fields released

  •  "PropertyNameOrDescription" (string) in Address object (called "Property Name" in the registry functional spec)

Renamed request parameters

  • “streetNumber” renamed from “unitOrNumber”
  • “streetName” renamed from “streetOrPropertyName”

Gating and response code changes

  • 400 'bad request' code included filtering ICPs not meeting the required format
  • Messages will be returned in the messages object with the appropriate code where available

'Get by ID' call - successful response sample (The 'Get by ID list' call follows the same format)

{

"ICPIdentifier": "string",

"ICPStatus": 0,

"Address": {

"PropertyNameOrDescription": "string",

"PhysicalAddressUnit": "string",

"PhysicalAddressNumber": "string",

"PhysicalAddressStreet": "string",

"PhysicalAddressSuburb": "string",

"PhysicalAddressTown": "string",

"PhysicalAddressRegion": "string",

"PhysicalAddressPostCode": 0,

"GPS_Easting": 0.086597829994870867,

"GPS_Northing": 0.089592538769447838

},

"Network": {

"NetworkParticipantID": "string",

"NetworkParticipantName": "string",

"POC": "string",

"ReconciliationType": "string",

"InitialElectricallyConnectedDate": "string",

"GenerationCapacity": 0.59222133055867521,

"FuelType": "string",

"DirectBilledStatus": "string"

},

"Pricing": {

"DistributorPriceCategoryCode": "string",

"DistributorLossCategoryCode": "string",

"ChargeableCapacity": 0.74166837179270262,

"DistributorInstallationDetails": "string"

},

"Trader": {

"TraderSwitchInProgress": false,

"TraderParticipantID": "string",

"TraderParticipantName": "string",

"ProfileCode": "string",

"ANZSICcode": "string",

"DailyUnmeteredkWh": "string",

"UnmeteredLoadDetails": "string"

},

"Metering": {

"MeteringEquipmentProviderParticipantID": "string",

"MeteringEquipmentProviderParticipantName": "string",

"InstallationInformation": [

{

"MeteringInstallationCategory": 0,

"MeteringInstallationType": "string",

"CertificationExpiryDate": "string",

"ComponentInformation": [

{

"ComponentType": "string",

"MeteringComponentSerialNumber": "string",

"MeterType": "string",

"AMIFlag": false,

"CompensationFactor": 0.68670529385750934,

"ChannelInformation": [

{

"MeteringComponentSerialNumber": "string",

"ChannelNumber": 0,

"RegisterContentCode": "string",

"PeriodOfAvailability": "string",

"UnitOfMeasurement": "string",

"EnergyFlowDirection": "string",

"AccumulatorType": "string",

"SwitchReadIndicator": false

}

]

}

]

}

]

},

"Messages": [

"string"

]

}


'Search' call – successful response sample

{

"Address": {

"PropertyNameOrDescription": "string",

"PhysicalAddressStreet": "string",

"PhysicalAddressSuburb": "string",

"PhysicalAddressTown": "string",

"PhysicalAddressRegion": "string",

"PhysicalAddressPostCode": 0,

"GPS_Easting": 0.11158078265068716,

"GPS_Northing": 0.53635229909462989

},

"ICPIdentifier": "string",

"ICPStatus": 0,

"Messages": [

"string"

]

}

Order by: Standard | Newest | Votes
Clayton posted this 19 October 2020

Hiya, I believe the profile code relates to the way in which a retailer submits energy data to the market. You can find more information on this under part 15 of the electricity code. It depends what you want to use the information for. There might be more information under the electricity registry functional specification also if you want to look at that. Regards Clayton

  • Liked by
  • empsean
  • msouness
Jack Callister posted this 26 May 2020

I'm getting a discrepancy on the /search endpoint:

 

v1:

unitOrNumber: "13 A"

streetOrPropertyName: "Seaview Terrace"

suburbOrTown: "Auckland"

 

Responds successfully with a single ICP

 

v2:

streetNumber: "13 A"

streetName: "Seaview Terrace"

suburbOrTown: "Auckland"

 

Responds with null (removing A returns 3 results)

 

Am I missing something? Let me know if I can do anything to help investigate.

  • Liked by
  • Racerxkiwi
Matthew Keir posted this 26 May 2020

Hi Jack,

Generally, we've tightened up what can be submitted in search fields a bit. We've relaxed this slightly to accommodate the example you've found so give it another go now.

Unfortunately, the address information in the registry isn't that well standardised and we are reliant on the existing registry search capability. As such, if you search "13 A" but the field is recorded as "13A" in the registry you won't get a match (or vice versa). Your approach of searching "13" would provide the ICPs that match either numbering format and you can select the one you are after.

I hope that helps.

Cheers,

Matthew

  • Liked by
  • Racerxkiwi
Clive Gifford posted this 15 July 2020

Hello Matthew

On what date did the EA decide to cripple the API in this way?

Was this change discussed with anybody outside of the EA before that decision was taken?

Has the EA communicated any details of this specific change (blocking of "special characters") to any other third party user of the API, before your post this afternoon?

How many users are registered to use the ICP Connection Data API?

Thanks,
Clive

  • Liked by
  • nicekiwi
funcsol posted this 25 May 2020

https://drive.google.com/file/d/10vD9JLWbdsK2LlQlezWm7AYJC_yt0tZf/view?usp=sharing

I've updated my standalone excel API scraper. Feel free to get a key. If there are any catastrophic weirdnesses, Im happy to have look. 

You will need to enable content & have a version of excel that supports power query. (2016+ from recollection. )  

[EDIT: updated since V2 API changed]

Clive Gifford posted this 25 May 2020

More thorough testing of the API needed I think.

length("/ICPConnectionData/v2/list/?list_of_ICPs=") + 60*16 - 1 = 1000

BOOM!

Jack Callister posted this 26 May 2020

Cheers Matthew, that seems understandable. Thanks for the quick adjustment.

Matthew Keir posted this 26 May 2020

We've also been asked about the new "TraderSwitchInProgress" field and when this will be set to true.

Several switch files are transferred between traders and the registry during a trader-ICP switch. 

  • The gaining trader will send a notification of transfer (NT file) to the registry. When this file arrives the ‘trader switch in progress’ flag will be set to true.
  • Notifications can be exchanged through the registry that acknowledge the switch (or withdrawal), withdraw the switch, or seek agreement on the meter reading for the switch. 
  • The losing trader will send a complete switch (CS file) notification to the registry when the switch is complete. When this file arrives the ‘trader switch in progress flag’ will return to false.

Normally this process will take around a couple of business days.

More information on the switching process is in the registry functional spec

Clayton posted this 02 June 2020

Howdy,

There is a spelling mistake in one of the metering fields. You have spelt participant wrong.

Is it worth correcting the spelling? The field with the incorrect spelling is called "MeteringEquipmentProviderPartcipantName". 

Regards

Clayton

 

Matthew Keir posted this 03 June 2020

Good catch Clayton - all fixed.

Clive Gifford posted this 03 June 2020

Hello Matthew,

> GET /ICPConnectionData/v2/single/?ICP=0000002894DE149 HTTP/1.1
< {"Code":500,"Message":"API is currently unavailable to process this request."}

What is the cause of this message?

The same ICP (and others like it) can be queried with the v1 API without any mysterious messages.

Note also for others ICPs where the normal kind of results are received, the results are always in a list (even for the single ICP interface) yet this message is not in a list. This makes the interface rather ugly, and it is also contradictory to the official documentation which states: "Provide a single valid ICP to return its details. This will always return a list."

Thanks, Clive

Clive Gifford posted this 03 June 2020

Back on the 24th February, I wrote:

"...any consumer trying to find an address without a unit number or street number (and there are many thousands of those) can't find anything because the "my meter" page requires something in that input. I suggest that should be changed if the EA is serious about that page being more usable for all consumers trying to find their address"

Sometime afer that the EA changed the My Meter page to allow a wildcard ('*') character to be entered for the number field' although I don't recall that being acknowledged anywhere.

Now the v2 API has broken that again, insisting as it does on a non-blank / no wildcard input for the streetNumber to avoid the message (also not in a list):

"Please include at least a streetNumber and streetName to filter the results by. Possible query options are streetNumber, streetName, suburbOrTown or region."

Clive

Matthew Keir posted this 08 June 2020

Hi Clive,

Thanks for your comments.

The issue where you got the 'Code 500 message' for an individual ICP number has been rectified. This issue was caused by an empty field being returned from the registry for one of the new fields released in version 2.

The issue with street numbers is by design. The search call was originally added to the API so that consumers (or their agents) could search their address and find their ICP number. The ‘my meter page’ simply calls the API and, as a user of the API, you get the same access to information that the ‘my meter’ page does. We’ve tightened up what can be submitted in the search call to better meet the requirement to ‘search an address’ and return the relevant ICP number. In version 2 at least one number is required in the street number field of the address search call. This change also helps to reduce some of the load on the Registry infrastructure. Less than 2 per cent of ICPs have no street number in the address fields and many of these ICPs are business connections. When you are working with a consumer who doesn’t know their ICP number and you can’t find it via the address search, we recommend getting it from their bill. Alternatively, it’s often noted on the meter board.

Cheers,

Matthew

Jack Callister posted this 17 June 2020

Hi Matt, I'm wondering if there's documentation to find a list of possible values for fields such as TraderParticipantId?

Matthew Keir posted this 18 June 2020

Hi Jack,

I'm not sure I follow but will try and help.

 

The very next field will tell you the participant name related to that trader ID.

"TraderParticipantID": "string",

"TraderParticipantName": "string",

 

We added this name field to version 2 of the ICP connection data  API  because there is no static list of participants - it's always changing as some parties exit (or are acquired), and new ones start trading. 

You can, however, get an overall picture from the retail reporting on EMI or the participant identifiers list which is periodically updated.

 

Cheers,

Matthew

Show more posts