Error refresh for all Power BI dashboards using Invantive: internal error 06D79D

Questions
,
1

This morning, we received error messages for all dashboards that use an Invantive connection.

Here is an example of the error message code:

Request failed:
OData Version: 3 and 4,
Error:
The remote server returned an error:
(500) Internal Server Error.
(An internal error ‘06D79D’ occurred. Please contact your supplier.)

Manually refreshing the dashboards is also not working.

2

We received the same message for all dashboards:

Request failed: OData Version: 3 and 4, Error: The remote server returned an error: (500) Internal Server Error. (An internal error ‘06D79D’ occurred. Please contact your supplier.

3

The actual error code visible in System Messages on Invantive Cloud is:

itgenclr093
Exception while reading from stream.
Timeout during reading attempt.
Er is een onbekende fout opgetreden.

It seems limited to Invantive Cloud and related products and happens for a percentage of the users.

4

The load on back office components was too high given the demand. Capacity has been increased. Changes are currently on their way to solve remaining issues. In the course of the next 4 weeks improvements will be made to the logic to reduce the chance of this incident happening again.

See Invantive Status - Invantive-producten deels down; capaciteit overbelast

Please refresh your reports again.

5

Further analysis revealed that the available capacity was exhausted due to a coding problem.

In case a user’s account is blocked for some reason, each individual download will trigger a HTTP 401 Unauthorized. Each HTTP 401 Unauthorized triggers evaluation of various aspects so guide the user to the best solution. Each of these evaluations is quite resource consuming in terms of database use, but in general occurs quite infrequently.

In today’s incident, a very heave user’s account was blocked, while the (frequent and recurring) requests caused a pro-rato quantity of expensive evaluations is executed. This lead to exhaustion of available capacity.

The following changes have been implemented:

  • the cost of repeating the same evaluation is decreased by caching the evaluation’s outcome without reducing the user’s perceived value towards a solution,
  • additional measures were introduced to ease analysis in cause the issue might repeat in the same area of business logic, and
  • the capacity of the database server was increased by approximately 300%.

All changes will have been implemented by the end of Friday June 16, 2023.

The capacity of the database server will be brought back to lower levels, but not early than Tuesday June 20, 2023, once it is confirmed that the combination of factors no longer triggers an exhaustion of the available capacity.

The incident has no relationship to the issues caused by the ongoing introduction of One Exact Identity.