• API management policy change – ICP connection data API

API management policy change – ICP connection data API

  • 1.2K Views
  • Last post 01 July 2021
Phil Bishop posted this 21 June 2021

We have previously had to modify the ICP API management policy. We are about to implement additional changes that will impose further restrictions on the number of calls able to be issued to the API. These changes will take effect at the close of business today.

The steady increase in large aggregate call volumes coming through the ICP connection data API have been continuing to adversely impact the underlying registry infrastructure. Further complicating matters, the registry manager has had to implement more restrictive firewall policies to guard against malicious activity. To preserve the integrity of their system and its responsiveness, the registry manager has further reduced the number of concurrent calls they will manage. 

We're making adjustments to our API management policy on the ICP API in line with this. These changes are designed to smooth out the load spikes our users collectively generate on the registry system.

The new management policy is below. Once user load on the registry systems has stabilised, we may revise this policy.

A management policy is set on this API restricting users to 250 calls per minute for ‘search’ and ‘get by id’ calls and the multiple ICPs ‘get by id list’ call will no longer be available. 

 

API
ICP data
  • Liked by
  • jerome
  • msouness
Order by: Standard | Newest | Votes
Matthew Keir posted this 01 July 2021

Update: 

We've worked with the registry manager to increase the capacity of calls the collective users of our ICP connection data API can submit before the 500 error occurs. This is now double what it was earlier this week and last week and when the changes in the registry were made. However, this level remains significantly below the capacity prior to the changes. The ICP connection data API has almost 400 subscribers who collectively can generate a significant volume of calls on the registry. The registry manager made changes to secure their system and preserve the functionality used by participants to operate the market. 

Please continue to refrain from placing heavy loads on the API. We are continuing to work with the registry manager to explore ways to have the registry system handle increased loads. However, it's likely that our throttling limits will have to reduce further to avoid a few users choking things up for everyone else. If further throttling is likely to cause significant issues for your application of the API please let us know via emi@ea.govt.nz. This information may be used to inform future enhancements. We'll continue to monitor failed call volumes, and may temporarily block heavy users if their volume is causing issues for everyone esle.

Thank you for bearing with us.

empsean posted this 29 June 2021

 

As noted by users above the single ICP call is still generating failed requests. A 500 error response means your call has been blocked by the registry. We’ve temporarily blocked access to the ICP connection data API for several very heavy users for the time being to free up some limited capacity while we work with our service provider to restore a better service.

If you are a heavy user, we ask that you please refrain from hitting the API with significant volumes at this time. We’ll update this discussion with more information once we're in a better position to do so.

 

What do you deam a heavy user? Shouldnt such users be rate limited further, not at the expense of others?

In addition, why not just role back the change? If the change has caused unintended consequences, and was not made with proper authorisation, role it back.
Surely it is bad practice to just let updates go out without thorough testing, without proper authorisation, and to let them stay out when they have such negative consequences? Certainly leaves the door open for the same issue to occur again in the future, potentially with much more dangerous ramifications.

 

 

 

 

  • Liked by
  • Bridget
msouness posted this 29 June 2021

Hi Matthew,

I just want to get this right - Registry users cannot mine the Registry through web services for marketing purposes - they should instead use the API to bypass this restriction?

Matthew Keir posted this 29 June 2021

Hi Malcolm,

The registry web services are governed by the registry terms and conditions. The ICP connection data API is governed by the EMI terms and conditions which publish information under the creative commons licensing which is more permissive.

Cheers,

Matthew

msouness posted this 29 June 2021

So, what you're saying here is that the Registry API could legitimately be used by marketers for determining which traders supply addresses on a street by street basis?

 

Matthew Keir posted this 29 June 2021

Hi everyone,

We appreciate your frustration with this issue. Unfortunately, we are stuck between a rock and hard place here. The API we host is entirely reliant on data from the registry hosted by our service provider. The service provider made changes that we were not notified of that severely restricted the aggregate volume of calls from the users of our ICP connection data API. This situation resulted in most API calls getting a failed response. The actions we’ve taken were to preserve those services that most likely engaged with consumers to limit disruption. In addition, multiple single calls add lag to the requests whereas the list calls which we removed do not.

As noted by users above the single ICP call is still generating failed requests. A 500 error response means your call has been blocked by the registry. We’ve temporarily blocked access to the ICP connection data API for several very heavy users for the time being to free up some limited capacity while we work with our service provider to restore a better service.

If you are a heavy user, we ask that you please refrain from hitting the API with significant volumes at this time. We’ll update this discussion with more information once we're in a better position to do so.

Market participants are reminded they have access to the registry web services directly for operational use. The ICP connection data API was released to open up access to a broader group and promote innovation in service offerings to consumers. It's use is not constrained by the registry use terms and conditions.

Cheers,

Matthew

empsean posted this 28 June 2021

When making changes, you must provide notice, as mentioned by others business tools and processes have been built around this API and makes any future planning extremely difficult if overnight you can just decide to change things without notice.

Am still getting a high number of 500 errors when using the single ICP call, any reason for this, and when we can expect this to be resolved? Imagine since you've removed the multi ICP call that the number of individual requests overall have increased.

  • Liked by
  • stait04
  • Bridget
Bridget posted this 28 June 2021

Can you please provide more robust guidance around the future of the APIs? As noted in above posts, these have been built into business processes and the current level of uncertainty is making planning difficult. Given tools have been built within the guidelines of Access Policy, which have now been rendered non-functional, do we have any assurances the single ICP API will continue to operate or should we begin migrating all processes away from these? (If this single ICP API was disrupted overnight as the multi-ICP API was, it would cause significant business impacts). We note even now it appears 5%-10% of calls to the single ICP API appear to be failing. Is this currently being investigated?

  Thanks

  • Liked by
  • stait04
mitchellm posted this 23 June 2021

The various comments of Malcolm and Clive seem sensible, and could be better workarounds than a blanket ban. If some participants seem to be making excessive use of the system they could also be individually reminded of the Registry Access Policy:

 (iii) must not attempt to gain inappropriate access to the registry, where “inappropriate access” means access:

...

• for marketing, cold calling, direct marketing, or any other form of participant initiated contact with potential customers

 (b) a participant must not access registry records for an ICP for which the participant is not responsible unless:

(i) the participant is a trader and the prospective customer at that ICP has initiated contact with the participant

  (ii) the participant was previously responsible for the ICP, and is querying or amending registry records in respect of their period of responsibility for the ICP

 (iii) the participant is an MEP and the trader responsible for the ICP has arranged for the participant to become the MEP for the ICP

 (iv) the participant is a distributor and there is an arrangement for the ICP to switch from the existing distributor to the participant

  • Liked by
  • msouness
  • andymac94
Clive Gifford posted this 22 June 2021

Hello Phil.

The explanation given makes little sense to me. The change has broken code, presumably for quite a number of users, essentially without warning. At the very least that is an unprofessional look for the EA.

The 'get by list' call previously supported up to 10 calls per minute and 50 ICPs per call, so data for something in the order of 500 ICPs per minute could be retrieved (in theory) . By turning that off you have broken code and forced users to change their code to use 'get by id' instead which in theory allows up to 250 calls per minute so potentially restrieving data for 250 ICPs per minute.

Why not just change 'get by list' to (say) 10 calls per minute and 25 ICPs per call? It seems extremely unlikely to be more efficient for 250 individual calls to 'get by id' to be processed rather than 10  to 'get by list' for 25 ICPs per call. Some changes might still be necessary for the end user but are likely to be much more straightforward.

Better still, why not do away with the arbitrary fixed throttling which fails to recognise times when the registry is lightly loaded and has plenty of spare capacity, and instead allow the calls to be processed as fast as possible with the proviso that critical registry tasks are given a higher priority?

It seems like a very bad design in this respect, and one that is likely to force some processing (using the ICP API) from times when the registry has spare capacity to spill into times when that is no longer the case. Surely that makes matters worse instead of better.

Phil Bishop posted this 22 June 2021

The change was implemented with very little notice in order to preserve the integrity of the registry system and to prevent users receiving error messages following a dramatic increase in the number of concurrent calls. 

  • Liked by
  • msouness
mitchellm posted this 21 June 2021

When was this propsed?  ‘get by id list’ is a fundamental part of the mechanism for quoting multi site customers so if it has been turned off we need advance warning to make workarounds.

 

msouness posted this 21 June 2021

Will the Authority consider publishing API call frequency by user?  This will help sort out who's driving the activity.

  • Liked by
  • jerome
  • andymac94
Categories
Find discussions by tag
See all
This week's high earners
Popular discussions

Our 1273 members have posted 1170 times in 327 discussions

Latest members:

29 users are currently online.

EA tel: + 64 4 460 8860 EMI email: emi@ea.govt.nz Legal information Glossary

© 2024 Copyright material on this site is protected by copyright owned by the Electricity Authority or its licensors.