Frequency keeping offers published on EMI datasets

  • 219 Views
  • Last post 17 March 2022
Phil Bishop posted this 08 March 2022

On Monday 14 March 2022 we will be deploying a new process to publish daily the frequency keeping offers and frequency keeping dispatched offers files. The files from the new process will first appear on 15 March for data pertaining to 14 March.

The structure of the files - column names, column order, data types, etc - will be identical to the current files, except for one minor aspect; in columns containing an alphanumeric data type, an empty cell will contain two double quotes whereas currently the quotes are not used.

For example, instead of appearing like this:

CTCT,1,11/02/2022,1,09/02/2022,00:10:33,08/02/2022,11:10:33,,,CLU,Y,5.00,175.00,668.00,74.00

CTCT,2,11/02/2022,1,09/02/2022,00:10:33,08/02/2022,11:10:33,,,CLU,Y,10.00,175.00,668.00,148.00

records will appear like this:

CTCT,1,11/02/2022,1,09/02/2022,00:10:33,08/02/2022,11:10:33,"","",CLU,Y,5.000,175.000,668.000,74.00

CTCT,2,11/02/2022,1,09/02/2022,00:10:33,08/02/2022,11:10:33,"","",CLU,Y,10.000,175.000,668.000,148.00

We trust this does not impact your data ingestion processes, or if it does, it's a straightforward fix.

We are making this change as part of our data migration from SQL databases to a Databricks Delta Lakehouse.

Order by: Standard | Newest | Votes
hopet posted this 15 March 2022

Hi Phil,

Can you configure a discussion tag specifically for notifications on file format changes as we don't have the resourcing to actively monitor this forum.

In addition to the changes above, my understanding is the end of row breaks changed format from DOS/Windows to Linux/Unix which caused our downstream processes to reject the file.

I'd also request that we get a minumum of two weeks notice of future changes as the < 5 business day lead time on this one would have been insufficient time to coordinate resources on our side to proactively manage this change.

-Tim

 

Phil Bishop posted this 17 March 2022

Yep, we'll stick to at least two weeks' notice in future and tag such posts as EMI News, that way subscribed users will get an email alert.

Phil