• New EMI API release – Version 2 of the ICP connection data API now available

New EMI API release – Version 2 of the ICP connection data API now available

  • 26K Views
  • Last post 19 October 2020
Matthew Keir posted this 25 May 2020

As part of our initiatives to enhance additional consumer choice of electricity services (ACCES) and customer access to their data, we have released version 2 of the ICP connection data API. This release follows an external review of the API which found no material issues.

The original ICP connection API (v1) will eventually be deprecated but for now will continue to operate so that services relying on it will not be interrupted. Version 1 of the API will be turned off on 1 September 2020.

Subscribers to the API should migrate to version 2 and take advantage of the additional information at their convenience, but should do so before 1 September 2020. Migrating to the new version will require a simple change to the URL call to include v2 in the call and some changes to the search call request parameters - see release notes below. Some minor tweaks will also be required to appropriately handle the additional fields and improved error responses available in v2. Users can test the new API and responses on the API platform.

The My Meter webpage has also been updated to provide access to the new registry fields released through v2 of the API.

Version 2 ICP connection data API release notes

The original ICP connection data API that was released in 2016 as part of the Authority's 'Retail Data Project (RDP)'.

Version (v2) is now available and follows further consultation in 2019 as part of the 'Additional Consumer Choice of Electricity Services (ACCES)' project. This version of the API includes additional fields along with some other enhancements.

'Get by ID' and 'Get by ID list' call changes

New fields released

  • "PropertyNameOrDescription" (string) in the Address object (called "Property Name" in the registry functional spec)
  • "InitialElectricallyConnectedDate" (string formatted as YYYY-MM-DD) in the Network object
  • "TraderSwitchInProgress" (boolean) in the Trader object
  • "ANZSICcode" (string) in the Trader object
  • "ProfileCode" (string) in the Trader object
  • "CertificationExpiryDate" (string formatted as YYYY-MM-DD) in the Meter installation object
  • "ComponentType" (string) in the Meter component object
  • "SwitchReadIndicator" (boolean) in the Meter channel object (called "Settlement Indicator" in the registry functional spec, but only indicates if the channel read may be needed in the switch process)
  • "NetworkParticipantName" (string) in Network object (included via lookup of "NetworkParticipantID" field, will repeat the ID if not found)
  • "TraderParticipantName" (string) in Trader object (included via lookup of "TraderParticipantID" field, will repeat the ID if not found)
  • "MeteringEquipmentProviderParticipantName" (string) in Metering object (included via lookup of "MeteringEquipmentProviderParticipantID" field, will repeat the ID if not found)

Moved fields

  • "ICPIdentifier" moved to first in the return order
  • "ICPStatus" moved to second in the return order

Renamed fields

  • "NetworkParticipantID" in the Network object renamed from "Network"
  • "TraderParticipantID" in the Trader object renamed from "Trader"
  • "MeteringEquipmentProviderParticipantID" in the Metering object renamed from "MeteringEquipmentProvider"

'Search' call changes

New fields released

  •  "PropertyNameOrDescription" (string) in Address object (called "Property Name" in the registry functional spec)

Renamed request parameters

  • “streetNumber” renamed from “unitOrNumber”
  • “streetName” renamed from “streetOrPropertyName”

Gating and response code changes

  • 400 'bad request' code included filtering ICPs not meeting the required format
  • Messages will be returned in the messages object with the appropriate code where available

'Get by ID' call - successful response sample (The 'Get by ID list' call follows the same format)

{

"ICPIdentifier": "string",

"ICPStatus": 0,

"Address": {

"PropertyNameOrDescription": "string",

"PhysicalAddressUnit": "string",

"PhysicalAddressNumber": "string",

"PhysicalAddressStreet": "string",

"PhysicalAddressSuburb": "string",

"PhysicalAddressTown": "string",

"PhysicalAddressRegion": "string",

"PhysicalAddressPostCode": 0,

"GPS_Easting": 0.086597829994870867,

"GPS_Northing": 0.089592538769447838

},

"Network": {

"NetworkParticipantID": "string",

"NetworkParticipantName": "string",

"POC": "string",

"ReconciliationType": "string",

"InitialElectricallyConnectedDate": "string",

"GenerationCapacity": 0.59222133055867521,

"FuelType": "string",

"DirectBilledStatus": "string"

},

"Pricing": {

"DistributorPriceCategoryCode": "string",

"DistributorLossCategoryCode": "string",

"ChargeableCapacity": 0.74166837179270262,

"DistributorInstallationDetails": "string"

},

"Trader": {

"TraderSwitchInProgress": false,

"TraderParticipantID": "string",

"TraderParticipantName": "string",

"ProfileCode": "string",

"ANZSICcode": "string",

"DailyUnmeteredkWh": "string",

"UnmeteredLoadDetails": "string"

},

"Metering": {

"MeteringEquipmentProviderParticipantID": "string",

"MeteringEquipmentProviderParticipantName": "string",

"InstallationInformation": [

{

"MeteringInstallationCategory": 0,

"MeteringInstallationType": "string",

"CertificationExpiryDate": "string",

"ComponentInformation": [

{

"ComponentType": "string",

"MeteringComponentSerialNumber": "string",

"MeterType": "string",

"AMIFlag": false,

"CompensationFactor": 0.68670529385750934,

"ChannelInformation": [

{

"MeteringComponentSerialNumber": "string",

"ChannelNumber": 0,

"RegisterContentCode": "string",

"PeriodOfAvailability": "string",

"UnitOfMeasurement": "string",

"EnergyFlowDirection": "string",

"AccumulatorType": "string",

"SwitchReadIndicator": false

}

]

}

]

}

]

},

"Messages": [

"string"

]

}

'Search' call – successful response sample

{

"Address": {

"PropertyNameOrDescription": "string",

"PhysicalAddressStreet": "string",

"PhysicalAddressSuburb": "string",

"PhysicalAddressTown": "string",

"PhysicalAddressRegion": "string",

"PhysicalAddressPostCode": 0,

"GPS_Easting": 0.11158078265068716,

"GPS_Northing": 0.53635229909462989

},

"ICPIdentifier": "string",

"ICPStatus": 0,

"Messages": [

"string"

]

}

API
EMI news
ICP data
Order by: Standard | Newest | Votes
funcsol posted this 25 May 2020

https://drive.google.com/file/d/10vD9JLWbdsK2LlQlezWm7AYJC_yt0tZf/view?usp=sharing

I've updated my standalone excel API scraper. Feel free to get a key. If there are any catastrophic weirdnesses, Im happy to have look. 

You will need to enable content & have a version of excel that supports power query. (2016+ from recollection. )  

[EDIT: updated since V2 API changed]

Clive Gifford posted this 25 May 2020

More thorough testing of the API needed I think.

length("/ICPConnectionData/v2/list/?list_of_ICPs=") + 60*16 - 1 = 1000

BOOM!

Jack Callister posted this 26 May 2020

I'm getting a discrepancy on the /search endpoint:

 

v1:

unitOrNumber: "13 A"

streetOrPropertyName: "Seaview Terrace"

suburbOrTown: "Auckland"

 

Responds successfully with a single ICP

 

v2:

streetNumber: "13 A"

streetName: "Seaview Terrace"

suburbOrTown: "Auckland"

 

Responds with null (removing A returns 3 results)

 

Am I missing something? Let me know if I can do anything to help investigate.

  • Liked by
  • Racerxkiwi
Matthew Keir posted this 26 May 2020

Hi Jack,

Generally, we've tightened up what can be submitted in search fields a bit. We've relaxed this slightly to accommodate the example you've found so give it another go now.

Unfortunately, the address information in the registry isn't that well standardised and we are reliant on the existing registry search capability. As such, if you search "13 A" but the field is recorded as "13A" in the registry you won't get a match (or vice versa). Your approach of searching "13" would provide the ICPs that match either numbering format and you can select the one you are after.

I hope that helps.

Cheers,

Matthew

  • Liked by
  • Racerxkiwi
Jack Callister posted this 26 May 2020

Cheers Matthew, that seems understandable. Thanks for the quick adjustment.

Matthew Keir posted this 26 May 2020

We've also been asked about the new "TraderSwitchInProgress" field and when this will be set to true.

Several switch files are transferred between traders and the registry during a trader-ICP switch. 

  • The gaining trader will send a notification of transfer (NT file) to the registry. When this file arrives the ‘trader switch in progress’ flag will be set to true.
  • Notifications can be exchanged through the registry that acknowledge the switch (or withdrawal), withdraw the switch, or seek agreement on the meter reading for the switch. 
  • The losing trader will send a complete switch (CS file) notification to the registry when the switch is complete. When this file arrives the ‘trader switch in progress flag’ will return to false.

Normally this process will take around a couple of business days.

More information on the switching process is in the registry functional spec. 

Clayton posted this 02 June 2020

Howdy,

There is a spelling mistake in one of the metering fields. You have spelt participant wrong.

Is it worth correcting the spelling? The field with the incorrect spelling is called "MeteringEquipmentProviderPartcipantName". 

Regards

Clayton

 

Matthew Keir posted this 03 June 2020

Good catch Clayton - all fixed.

Clive Gifford posted this 03 June 2020

Hello Matthew,

> GET /ICPConnectionData/v2/single/?ICP=0000002894DE149 HTTP/1.1
< {"Code":500,"Message":"API is currently unavailable to process this request."}

What is the cause of this message?

The same ICP (and others like it) can be queried with the v1 API without any mysterious messages.

Note also for others ICPs where the normal kind of results are received, the results are always in a list (even for the single ICP interface) yet this message is not in a list. This makes the interface rather ugly, and it is also contradictory to the official documentation which states: "Provide a single valid ICP to return its details. This will always return a list."

Thanks, Clive

Clive Gifford posted this 03 June 2020

Back on the 24th February, I wrote:

"...any consumer trying to find an address without a unit number or street number (and there are many thousands of those) can't find anything because the "my meter" page requires something in that input. I suggest that should be changed if the EA is serious about that page being more usable for all consumers trying to find their address"

Sometime afer that the EA changed the My Meter page to allow a wildcard ('*') character to be entered for the number field' although I don't recall that being acknowledged anywhere.

Now the v2 API has broken that again, insisting as it does on a non-blank / no wildcard input for the streetNumber to avoid the message (also not in a list):

"Please include at least a streetNumber and streetName to filter the results by. Possible query options are streetNumber, streetName, suburbOrTown or region."

Clive

Matthew Keir posted this 08 June 2020

Hi Clive,

Thanks for your comments.

The issue where you got the 'Code 500 message' for an individual ICP number has been rectified. This issue was caused by an empty field being returned from the registry for one of the new fields released in version 2.

The issue with street numbers is by design. The search call was originally added to the API so that consumers (or their agents) could search their address and find their ICP number. The ‘my meter page’ simply calls the API and, as a user of the API, you get the same access to information that the ‘my meter’ page does. We’ve tightened up what can be submitted in the search call to better meet the requirement to ‘search an address’ and return the relevant ICP number. In version 2 at least one number is required in the street number field of the address search call. This change also helps to reduce some of the load on the Registry infrastructure. Less than 2 per cent of ICPs have no street number in the address fields and many of these ICPs are business connections. When you are working with a consumer who doesn’t know their ICP number and you can’t find it via the address search, we recommend getting it from their bill. Alternatively, it’s often noted on the meter board.

Cheers,

Matthew

Jack Callister posted this 17 June 2020

Hi Matt, I'm wondering if there's documentation to find a list of possible values for fields such as TraderParticipantId?

Matthew Keir posted this 18 June 2020

Hi Jack,

I'm not sure I follow but will try and help.

 

The very next field will tell you the participant name related to that trader ID.

"TraderParticipantID": "string",

"TraderParticipantName": "string",

 

We added this name field to version 2 of the ICP connection data  API  because there is no static list of participants - it's always changing as some parties exit (or are acquired), and new ones start trading. 

You can, however, get an overall picture from the retail reporting on EMI or the participant identifiers list which is periodically updated.

 

Cheers,

Matthew

funcsol posted this 18 June 2020

If I was guessing, I'd say the question was for the enumeration of TraderParticipantID

Clayton posted this 18 June 2020

I myself couldn't find a reliable list other than the link that Matt has given to the spreadsheet. There might be a static list available from the registry itself, but that would probably require a participant login to access. These days there seems to be a whole lot more traders.

oops I just saw Matt's comment that there is no static list. 

Hi Daniel btw :)

Jack Callister posted this 18 June 2020

Thanks Matt, the participant list and name are exactly what I need to solve my problem. 

Clive Gifford posted this 14 July 2020

Hello Matthew.

Back on 8 June you wrote (above) that the issue I raised with the street numbers was "by design".

Given that response it seems to me that the EA has been far more interested in spending time on deliberately crippling their v2 API than conducting even a modicum of proper testing of basic functionality, all while also not thoroughly and openly documenting those kinds of changes or acknowledging how much code they could be breaking. My interpretation of this behaviour is that after first not providing a proper and lawful response to my OIA request last year (for all ICP identifiers in the registry) the EA is now (rather stupidly if you ask me) trying to close the stable door after the horse has already bolted. Trying to suggest the API was built with just one narrow use case sounds like nonsense to me.

In any case, can you also please clarify if the v2 API failing to return any records to a search by address request where the region is specified "Nelson & Bays" or "Timaru & Oamaru" is also "by design" or just another bug due to (apparently) failing to decode the encoded URL given the embedded ampersand character? Of course this "feature" blocks searches for a large number of other ICP's also, but even just those two regions (if specified as part of the search) block more than 100,000.

Hmm.. maybe I shouldn't have referred to the ampersand as a "character"? Would you say it is a character, or not?

Where do I send an invoice for debugging the EA's code?

Kind regards,
Clive Gifford

Matthew Keir posted this 15 July 2020

Hi Clive,

Yes, special characters in the search parameters are blocked by design and this is not a bug related to decoding of the URL.

Given the region field in the search call is not a wildcard value, it’s not that useful for consumers who may not easily identify with the precise region names as recorded in the registry. Although, given this is how regions are recorded in the registry, we’ve made a quick change to allow both ‘Nelson & Bays’ and ‘Timaru & Oamaru’ through the search call. 

Cheers,

Matthew

Clive Gifford posted this 15 July 2020

Hello Matthew

On what date did the EA decide to cripple the API in this way?

Was this change discussed with anybody outside of the EA before that decision was taken?

Has the EA communicated any details of this specific change (blocking of "special characters") to any other third party user of the API, before your post this afternoon?

How many users are registered to use the ICP Connection Data API?

Thanks,
Clive

  • Liked by
  • nicekiwi
nicekiwi posted this 04 August 2020

There seems to have been omission in the API v2 release notes. the "Get by ID" request no longer accepts the paramater "id", it now expects "icp" instead. e.g. '/?icp={ICP}' instead of '/?id={ICP}'. Which returns a 404 error if left unchanged. 

nicekiwi posted this 02 September 2020

Another descrepincey/bug between v1 and v2 is when retreiving a single ICP in v1 /single/?icp=??? it will return a JSON array with a single object [ { ...json }]. However in v2 the same request /single/?id=??? will return a single JSON object only { ...json }. Breaking existing error handling.

Matthew Keir posted this 03 September 2020

Hi nicekiwi,

Thanks for your comments. We edited the release notes to include the change to "ICP=" that we hadn't listed - this was done some time ago but not posted back here. The information on the API platform will always be the most up to date documentation. The release notes are available in the following location. https://emi.portal.azure-api.net/docs/services/ICP-connection-data-v2/releaseChanges

We made several changes to the gating and error handling to improve consistency within v2 and to prevent rubbish being passed onto the registry, resulting in unnecessary load. The change to error messages was listed in the release notes and in the original post above. I'd encourage you to try the test feature in the API platform for v2 if you run into any issues.

v2 was released in May and v1 scheduled to be turned off on 1 September. We've extended this until 14 September to allow the last few users more time to migrate.

Cheers,

Matthew

empsean posted this 18 October 2020

Is there anywhere that gives a list of what the "ProfileCode" is and what the potential outputs are?

I've searched the forum and API documentation but can't find anything.

I've seen a couple of variations RPS and HHR but it'd be helpful to know whats RPS is and if there are other codes in-use

Clayton posted this 19 October 2020

Hiya, I believe the profile code relates to the way in which a retailer submits energy data to the market. You can find more information on this under part 15 of the electricity code. It depends what you want to use the information for. There might be more information under the electricity registry functional specification also if you want to look at that. Regards Clayton

  • Liked by
  • empsean
  • msouness
Nicole Gagnon posted this 19 October 2020

Hi empsean

We have a profile information page with a document about profiles and a list of all profiles in use: https://www.ea.govt.nz/operations/retail/reconciliation/reconciliation-profiles/ . It doesn't show what the potential outputs are though.

cheers

Nicole

empsean posted this 19 October 2020

Hi empsean

We have a profile information page with a document about profiles and a list of all profiles in use: https://www.ea.govt.nz/operations/retail/reconciliation/reconciliation-profiles/ . It doesn't show what the potential outputs are though.

cheers

Nicole

Thanks Nicole, that has a couple of them and gives an explination of RPS, so for now at least that will be enough.

Categories
Find discussions by tag
See all
This week's high earners
Popular discussions

Our 1273 members have posted 1170 times in 327 discussions

Latest members:

30 users are currently online.

EA tel: + 64 4 460 8860 EMI email: emi@ea.govt.nz Legal information Glossary

© 2024 Copyright material on this site is protected by copyright owned by the Electricity Authority or its licensors.