Trickle loading is a performance improvement technique to load data from the source database into the target database. It starts with a full initial load of the source database through replication on Invantive Data Replicator. The source system informs the Invantive infrastructure of each and every change.
The changes come “trickling“ in on an individual change basis. These changes to the data can be downloaded are initially applied to the initial load, leading to the full data set. From there, each new full data sets from the target database is combined with the then recent changes from the source to establish the new contents of the target tables.
Limitations
Trickle loading is available for a limited number of source tables, depending on the capabilities of the platform connected to.
The target system (backing database) for the data load can be any of the supported ANSI SQL database platforms, including SQL Server, Azure SQL Server, Oracle, MySQL and PostgreSQL.
No data mapping takes place during processing the table load process, as with other Data Replicator processes. For data mapping of rows, you can specify your data mapping creating for instance an in-memory table based upon a query, such as illustrated on Selecteer kolommen om over te halen naar SQL Server vanaf Exact Online - 4 van forums.
Configuration
Trickle loading activation depends on the source platform for the load.
For Exact Online, please consult Use Trickle Loading with Exact Online Webhooks.
Data Replicator Token
Download of the changes is secured by the so-called “Data Replicator Token”. This token is available from Invantive Support for users with an active Data Replicator subscription.
The Data Replicator Token can be stored in the repository using the alter persistent cache set SQL-statement:
alter persistent cache set token '...'
Alternatively the Data Replicator Token can be specified when downloading the row changes with the alter persistent cache download SQL-statement:
alter persistent cache download feed token '...'
The download statement also allows limiting downloading the row changes solely for a specific contract or data container ID.
Invantive OEM-partners find more tips on Enable webhook trickle loading on OEM licenses.
The full grammar of Data Replicator statements can be found in the alter persistent cache SQL-statement section in the Invantive SQL grammar.
Performance Gains
Using trickle loading, replication duration can be reduced by a factor 100 or more.
Incremental Loading
Trickle loading is functionally equivalent to incremental loading. Trickle loading receives the changes through another channel and one-by-one, whereas incremental loading retrieves the changes in bulk and through the same direct channel as queries.
For more information on incremental loading with Exact Online consult: