Planned outage of EMI and real-time APIs on Tuesday, 2 October

  • 713 Views
  • Last post 11 October 2018
Phil Bishop posted this 24 September 2018

On Tuesday, 2 October 2018 we’re making some changes to the infrastructure and processes behind the EMI website and the real-time pricing APIs.

EMI will be unavailable from around 12 noon on Tuesday, although full service should progressively return throughout the afternoon, with everything back up and running by Wednesday morning.

In addition to improving robustness and performance, we’ll be deploying a new release of the EMI website with enhanced features, making it easier for users to explore the depth of information behind reports. 

The ICP connection data API and FTP site will remain unaffected. The real-time pricing API should not be impacted for long.

We apologise for any inconvenience this outage causes. Please email emi@ea.govt.nz if you notice any issues that remain unresolved on Wednesday.

Users who access the FTP server directly may also wish to follow the discussion over here. The current FTP server will continue to run in parallel with the new direct access arrangements for 30 days.

 

API
Datasets
EMI news
Order by: Standard | Newest | Votes
Matthew Keir posted this 02 October 2018

Update on the planned outage:

EMI was back up and running around 4 pm yesterday. However, we’re experiencing a couple of issues that we are working to resolve.

  • Users are getting a lot of timeouts on EMI reports and dashboards. This issue appears to be occurring due to the number of connections to the underlying databases from EMI.
  • The date format on data downloads on reports has jumped to American format (MM/DD/YYYY) we intend to revert this to NZ date format (DD/MM/YYYY).

There are other databases are still being migrated and backed up in the new environment this is creating a large load on the system and will continue over the weekend.

Let us know if you encounter any other issues from those above. More information on the new EMI features will follow; probably next week.

Thanks for your patience.

Shane Adams posted this 03 October 2018

Forward markets data stuck again ? - latest showing is 28 Sep.

 

Phil Bishop posted this 03 October 2018

Yep, all forward market EMI reports should be up to date very soon.

With the ASX data source and one or two others, we have to go get it from their SFTP server, whereas most of the data we receive gets delivered to us. The new environment we've migrated to has meant we've needed to change how the ASX data pipepline operates and we're sorting through a few teething issues with it as we speak.

Phil.

Matthew Keir posted this 03 October 2018

Update on issues:

  • Still some ongoing issues with timeouts although performance should have improved. We are continuing to tune the solution here so further improvements to speed will come.
  • Date formats on the data downloads have been corrected to NZ format. However, are forcing time element to show eg 01/08/2018 00:00:00 instead of the old site 01/08/2018. We are looking into this.
  • ASX for yesterday will be up soon.

Let us know if you spot any other issues.

Matthew Keir posted this 08 October 2018

Update on issues:

  • General performance - should be stable, although the data warehouse is getting hammered as it tries to catch up on processing the incoming data it missed during the outage. This is likely to take the next couple of days still. Things should start to get better towards the end of the week.
  • Date formats sorted.
  • ASX files still some scheduling issues we hope to resolve soon.

Dataset issues:

The following datasets are delayed while the data warehouse catches back up on processing. The should be back in the next couple of days, so please check back soon.

Please bear with us while we get these last few issues sorted. Thanks for your patience.

Matthew Keir posted this 11 October 2018

Update on issues:

We think we've resolved all the issues listed above except for a few cases of multiple files or keeping files we'd normally remove form datasets.

Please let us know if you notice anything further.

Categories
Find discussions by tag
See all
Popular discussions

Our 1394 members have posted 1235 times in 345 discussions

Latest members:

No one is currently online at the moment

EA tel: + 64 4 460 8860 EMI email: emi@ea.govt.nz Legal information Glossary

© 2024 Copyright material on this site is protected by copyright owned by the Electricity Authority or its licensors.