Fully reload TransactionLinesIncremental

Questions (nl)
, ,
1

I’ve recently started using ExactOnlineREST.Incremental.TransactionLinesIncremental@eol via Invantive Bridge Online (OData4) in Excel/Power BI. My expectation was that after the initial full load, each refresh would only retrieve and process the delta: changed and deleted rows since the last time. However, what I see in the logging deviates from this. I’m wondering if this is normal behaviour or if something is wrong.

Environment

  • Invantive Bridge Online 26.0.286
  • Administration with approximately 170,000 rows in the incremental table
  • HTTP disk cache 4 hours, Bridge response cache 15 minutes

What I see in IncrementalLoadEventLogEntries

  • After the delta, almost every refresh is followed by a reset based on the row count: “Full reload of incremental cache … since the current cache row count … deviates too much from the calculated allowed minimum row count …”. Then follows “Reconstructing. Do not reuse old data. Full reload…”.
  • With each refresh, there is "Timestamp moved from to " with empty values, and “No need to update incremental results since there was no previous version”.
  • Occasionally: “Can’t find all data files for most recent download … Expecting 2 data files, found 0”.

What I see in the Session I/Os

  • The delta itself seems to work fine: sync/Financial/TransactionLines?$filter=Timestamp gt …L with only changed rows.
  • However, about 170 pages of the full table follow. As long as the disk cache is valid, they are retrieved from the cache, but a refresh still takes 60 to 80 seconds.
  • The Deleted API is requested with each refresh from Timestamp gt 0L and gt 1L, each 4 pages. I expected these to also start from the last seen timestamp.
  • Consecutive refreshes start with the same Timestamp gt …L value. So, the timestamp doesn’t seem to move after a delta, causing the delta to get larger each time.
  • When the disk cache has expired, after the delta, the entire incremental table is retrieved from Exact again (approximately 220 actual calls, over 3 to 8 minutes for just this table).

What stands out to me
Before the delta, the stored copy matches exactly with $count. After processing over 3,000 deleted IDs from the Deleted API, there are about 3,000 rows fewer. It seems that IDs reported as deleted still exist in Exact, and this causes the row count check to trigger a full reload each time. But maybe I’m interpreting this incorrectly.

My questions

  1. Is it normal that after each delta the complete table is reconstructed?
  2. Is it normal that the Deleted API is always read from timestamp 0 and that the last seen timestamp doesn’t shift?
  3. Is my suspicion about the deleted IDs correct, or is there another explanation for the row count discrepancy?
  4. Could this relate to what was mentioned in this topic about release 26.0 concerning incremental files?

If this is not the expected behaviour: how can I fix it so that only the delta is processed?

I can provide exports of the event log and the Session I/Os upon request.

2

It is not entirely possible to comprehend the partially AI-generated text.

A full reload is not normal behaviour in any case. This only occurs if the algorithm believes the dataset is incomplete.

Thousands of deleted rows is also not common.

The advice is to add a screenshot of such a reload request as described below.

Supplementary information

Is it possible to add an (anonymised) screenshot of the details of the request in Online Monitoring on Invantive Cloud as described in Meer inzicht met nieuwe Online Monitoring?

You can find the details by clicking on the download request which represents the subject of this topic.

The download request usually has an SQL statement showing the table name.

Please make sure at least the following data from both columns is visible:

  • the title bar with the request ID,
  • the status code, network size, path and timestamps in the left column,
  • the error code and error message at the very bottom of the left column,
  • the entire right column, including the SQL statement, table name and parameter values.

For example:

image
image1474×640 49.6 KB

Check the correct server and user

Carefully check that you log in to the Invantive Cloud website with the same username used to process data via, for example, Invantive Bridge Online. You will only see the requests of the Invantive Cloud user with which you are logged in to the website.

Check the correct request and details

Make sure to select the request first to display the details. Only one request should be visible in the screenshot.

Also carefully check whether the request has a path with odata4, apps or mcp. Requests with other paths are generally not relevant for this purpose.

Error message and tips via email

Additionally, the Invantive Cloud user who received the error message will usually receive an email with an error message and tips if there is an error in Power BI, Power Query, Azure Data Factory, Qlik or Tableau.

It is advisable to check the spam of the involved user for such emails sent from support@invantive.eu.