Further analysis revealed that the available capacity was exhausted due to a coding problem.
In case a user’s account is blocked for some reason, each individual download will trigger a HTTP 401 Unauthorized. Each HTTP 401 Unauthorized triggers evaluation of various aspects so guide the user to the best solution. Each of these evaluations is quite resource consuming in terms of database use, but in general occurs quite infrequently.
In today’s incident, a very heave user’s account was blocked, while the (frequent and recurring) requests caused a pro-rato quantity of expensive evaluations is executed. This lead to exhaustion of available capacity.
The following changes have been implemented:
- the cost of repeating the same evaluation is decreased by caching the evaluation’s outcome without reducing the user’s perceived value towards a solution,
- additional measures were introduced to ease analysis in cause the issue might repeat in the same area of business logic, and
- the capacity of the database server was increased by approximately 300%.
All changes will have been implemented by the end of Friday June 16, 2023.
The capacity of the database server will be brought back to lower levels, but not early than Tuesday June 20, 2023, once it is confirmed that the combination of factors no longer triggers an exhaustion of the available capacity.
The incident has no relationship to the issues caused by the ongoing introduction of One Exact Identity.