Root Cause Analysis of Marlin BB Errors on 03-Sep-2020

Problem Description:

Marlin BB clients could not process the tokens returned by the Expressplay Marlin Service. 

When:

Issue started on September 3rd, 2020 at 15:35 UTC and was fixed on the same day at 20:00 UTC.

Root Cause:

This issue occurred because two X.509 certificates for Marlin expired on September 3rd, 2020, at 15:34:50 UTC. These X.509 certificates get passed to the Marlin BB clients by the service in a file called ‘ConfigurationToken’.

When the Marlin BB client tried to process the token, it could not validate the certificate chain because of the expired certificates and threw an error : -50410 ==> STR_ERROR_EXPIRATION_TIME.  

This problem was resolved by updating the X.509 certificates in the ‘ConfigurationToken’ file, and distributed updated “Configuration Token” from the ExpressPlay Marlin Service.

Why this issue was not detected in advance:

Updating the X.509 certificates was on the service to-do list when these X.509 certificates were issued 15 years ago, however there was no service alarm associated with the expirations, and the team failed to take action prior to the expiration date.

Preventive Actions: 

• We’ve updated the X.509 certificates that are valid until December 31st, 2100 (UTC).
• We will audit the entire service to ensure that all objects in the system associated with an expiration date are also associated with a corresponding service alarm that alerts the team to update the object in advance of the expiration date.

What you need to do it you are still experiencing issues with BB Service :

• If you request new token for every license request, you do not to do anything.
• If you cache the token (i.e. request a token once and reuse same token for many playback requests), you’ll need to get a new token from BB Service.