Abstract
Exact completed on July 5, 2021 the rollout of an API rate limitation in terms of the number of errors per API key, per user, per division, per endpoint, and per hour as part of a larger program of changes. Following analysis on regular application behavior, a sequel was executed on July 9, 2021 to study Exact Online API behavior when the new error rate limits are deliberately exceeded.
It was found that we were unable to exceed the new Exact Online API limits on errors raised per hour using pre-defined test cases. The expected one-hour block was never experienced. The actual status of whether the new API limit on error messages per hour has been activated remains unclear.
New API Limit on error messages per hour
A new API limit on errors raised per hour has been activated as shown on the Exact site:
No more than 10 errors per API key, per user, per company, per endpoint, and per hour. When you exceed this limit, your API key will be temporarily blocked from making further requests. The block will automatically be lifted after one hour and will gradually increase when you continue making these errors. Response code 400, 401, 403, and 404 are counted as errors.
This API limit has completed rollout on July 5. Currently, there have been no reported production stops due to the change.
Other new rate limits such as on maximum refresh token lifetime, mandatory filtering, and restrictions on the daily and minutely rate limit window are not covered in this analysis. Please consult the base topic for more information.
What is an Exact Online “API Key”?
The specification refers to an “API key”. The meaning of “API key” varies across platforms. For instance, on some platforms, it is bound to a specific user. In this case, the term “API key” probably refers to an app registration or client ID is meant. Original Exact Online documentation refers to “API key” whereas the current web frontend names it “app”. Each app is uniquely identified by a “client ID”.
Also, the specification of the limit explicitly lists “per user” as an additional dimension. Assuming the specification is not redundant, the term “API key” should not include user-identifying data, such as user name, refresh token, or access token.
Scenario
Using the Invantive SQL table NativeScalarRequests, specific events are forwarded directly to the Exact Online API servers using the authenticated channel. The Invantive SQL engine does not parse the payload as with a query such as select * from AccountsIncremental; it just hands over the URL in this case over the previously established channel.
A range of pre-defined tests has been executed on July 9, 2021. Measurements of requests are automatically registered in NativeScalarRequests by Invantive SQL and reported upon test completion.
The pre-defined tests consisted of:
- Download invalid URL triggering an HTTP 404 - Not Found,
- Disable OAuth refresh flow triggering an HTTP 401 - Unauthorized,
- Download invalid division triggering an HTTP 403 - Forbidden.
Denial of Service Attack
Based upon the specification above, exceeding the limit of 10 errors in an hour per combination of app, user, company and endpoint should raise a block on all use of the app, independent of user, company, or endpoint.
This specification signals a vector for a Denial of Service Attack. This is a different attack form from Distributed Denial of Service Attack since it should only require a dozen carefully crafted Exact Online API calls to block an individual app.
Since the test cases have not been able to trigger the limit, we have been unable to establish whether a Denial of Service Attack is possible against 3rd party apps using for instance the implicit grant flow or code grant flow with client secret sniffing as on the Exact Online iPhone app. An educated guess is that the specification is incorrectly formulated, but the thesis can not be disambiguated.
As far as known, the old Denial of Service Attack against app vendors by re-using their client ID in an Implicit Grant Flow to trigger app fees is still available. It seems however not possible to trigger app fees on any Exact Online company using the capability to specify any division in the System/Divisions API.
Conclusion
The pre-defined test cases were expected to easily exceed the documented error rate limits. However, none of 401, 403, and 404 HTTP status codes raised a block on the API key (should probably read: “client ID” or “app registration”).
No test case was able to exceed the new error rate limits and raise a block. The reason remains unclear.
Invantive SQL Test Case 404 - Not Found
set oauth-unauthorized-max-tries@eol 2 /* Default value. */
--
-- Raise 25 times an HTTP 404 on 1 endpoint.
--
declare
l_division pls_integer := 10523456;
begin
for i in 1..25 loop
insert into exactonlinerest..nativeplatformscalarrequests@eol(url)
values
('https://start.exactonline.nl/api/v1/' || l_division || '/crm/AccountsINVALID')
;
end loop;
insert into exactonlinerest..nativeplatformscalarrequests@eol(url)
values
('https://start.exactonline.nl/api/v1/' || l_division || '/crm/Accounts?$top=5')
;
end;
select successful
, http_status_code
, http_method
, url
, duration_ms
from exactonlinerest..nativeplatformscalarrequests@eol
where transaction_id >= 1
order
by transaction_id desc
The 25 errors did not trigger the error rate limit. Even after 25 errors the download of the first five accounts still works:
Invantive SQL Test Case 401 - Unauthorized
set oauth-unauthorized-max-tries@eol 0 /* Never acquire a new access token. */
--
-- Trigger HTTP 401 Unauthorized
--
declare
l_division pls_integer := 10523456;
begin
dbms_lock.sleep(600);
for i in 1..10 loop
insert into exactonlinerest..nativeplatformscalarrequests@eol(url)
values
('https://start.exactonline.nl/api/v1/' || l_division || '/crm/AccountsINVALID')
;
end loop;
insert into exactonlinerest..nativeplatformscalarrequests@eol(url)
values
('https://start.exactonline.nl/api/v1/' || l_division || '/crm/Accounts?$top=5')
;
end;
select successful
, http_status_code
, http_method
, url
, duration_ms
from exactonlinerest..nativeplatformscalarrequests@eol
where transaction_id >= 1
order
by transaction_id desc
set oauth-unauthorized-max-tries@eol 2
-- Reconnect.
select * from me@eol
First the 401’s are raised:
The query on me@eol should fail due to a block, but works fine.
As a sidenote: Invantive SQL returns currently a 0 in this scenario instead of the actual error. When replaying with Postman with an expired access token, the first call to the URL AccountsInvalid actually returns an HTTP 400 status, and the next (identical) call an HTTP 401, which cycle of HTTP 400 and HTTP 401 repeats itself indefinitely.
Invantive SQL Test Case 403 - Forbidden
set oauth-unauthorized-max-tries@eol 2
--
-- Trigger 403 Unauthorized
--
begin
--
-- Should work always. The division is ignored.
--
for l_division in 1..25 loop
insert into exactonlinerest..nativeplatformscalarrequests@eol(url)
values
('https://start.exactonline.nl/api/v1/' || l_division || '/system/Divisions')
;
end loop;
--
-- Should raise 25 HTTP 403 errors.
--
for l_division in 1..25 loop
insert into exactonlinerest..nativeplatformscalarrequests@eol(url)
values
('https://start.exactonline.nl/api/v1/' || l_division || '/crm/Accounts?$top=1')
;
end loop;
end;
select successful
, http_status_code
, http_method
, url
, duration_ms
from exactonlinerest..nativeplatformscalarrequests@eol
where transaction_id >= 1
order
by transaction_id desc
The 25 HTTP 403 error did not impact retrieval of data due to error rate limiting:
