Updated synchronization script between Eol and SQL server

we have enhanced our script Sharing an Incremental script for sync between Exact Online and SQL Server

the script is splitted in 2 parts in the comments below (due to post char length limitation)

### Major Improvements

1. **Performance Optimization**

  • **Bulk Operations**: Replaced row-by-row DELETE loops with a single `MERGE` statement and `DELETE` statement executed via `EXECUTE NATIVE`

  • **All Divisions at Once**: Consolidated processing to handle all divisions in a single operation instead of looping per division

  • **Staging Table Reuse**: Staging tables are populated once and reused for both verification and modification steps, eliminating redundant data insertion

  • **Native SQL Verification**: Verification counts are calculated using `EXECUTE NATIVE` to avoid loading large datasets into Invantive memory

#### 2. **Data Integrity & Safety**

  • **Verification Before Modification**: Added pre-modification verification check. If `Diff After <> 0`, modifications are skipped to avoid inconsistencies (data may have changed in EOL during API call)

  • **Transaction Management**: All modifications are wrapped in `BEGIN TRY…BEGIN CATCH` blocks with explicit `BEGIN TRANSACTION`, `COMMIT`, and `ROLLBACK` statements

  • **TEST/PROD Mode**: Added mode parameter to support testing without persisting changes:

  • **TEST mode**: Executes all operations but rolls back modifications, sends email to admin only, keeps staging tables for inspection

  • **PROD mode**: Commits modifications, sends email to full recipient list, truncates staging tables

#### 3. **Enhanced Reporting**

  • **HTML Table Summary**: Replaced full log in email with a concise HTML table showing verification results per division

  • **Total Operations Column**: Added summary column showing total operations per division

  • **Improved Email Subject**: Email subject now indicates if synchronization was skipped due to verification failure

  • **Better Error Messages**: More descriptive error messages and warnings

#### 4. **Code Quality**
**Clear Structure**: Script is organized into 4 clearly defined steps:

1. EOL Data Retrieval
2. Verification
3. Modification
5. Email Reporting

- **Better Comments**: Comprehensive comments explaining each step and important logic

- **Improved Indentation**: Code is properly indented for better readability

### Technical Changes

#### Removed

- Row-by-row DELETE loops (replaced with bulk DELETE)

- Per-division modification loops (consolidated to single operation)

- Redundant data insertion (staging tables reused)

- Full log in email body (replaced with HTML summary table)

#### Added

- `MERGE` statement for INSERT/UPDATE operations

- Bulk `DELETE` statement via `EXECUTE NATIVE`

- Pre-modification verification (Diff After check)

- TEST/PROD mode support

- Transaction management with error handling

- HTML table reporting

- Post-modification verification using actual operation results

#### Fixed

- Verification logic now correctly distinguishes between inserts and updates

- DELETE verification uses actual operation result instead of recalculating from staging tables (which no longer exist after DELETE)

- EntityKey handling (already fixed in previous version, maintained here)

### Performance Impact

- **Before**: Row-by-row operations, per-division loops, data loaded into Invantive for verification

- **After**: Bulk operations, all divisions processed at once, verification via native SQL

- **Result**: Significantly faster execution, especially for large datasets

### Migration Notes

When migrating from the previous version:

1. Update table names to match your environment (staging tables and target table)

2. Configure email settings (SMTP server, credentials, recipients)

3. Set the `l_mode` parameter to ‘TEST’ for initial testing

4. Review division IDs and names in the CASE statements

5. Ensure staging tables are created before script execution (for parser recognition)

PART1

--
-- first set the cache, divisions and create the sendmail procedure
set use-http-disk-cache@eol false; -- pay attention to inventive cache. Executing the same query on short time interval will not trigger an API call
set use-http-memory-cache@eol false;
-- set incremental-force-save-always@eol true; --(starting QT v24.0.153)
use 1000001@eol/* Division 1*/, 1000002@eol/* Division 2*/, 1000003@eol/* Division 3*/, 1000004@eol/* Division 4*/, 1000005@eol/* Division 5*/, 1000006@eol /* Division 6*/
, 1000007@eol /* Division 7*/, 1000008@eol /* Division 8*/, 1000009@eol /* Division 9*/, 1000010@eol /* Division 10*/
, 1000011@eol /* Division 11*/, 1000012@eol /* Division 12*/, 1000013@eol /* Division 13*/, 1000014@eol/* Division 14*/
;
Create or Replace procedure sendmail
    ( email_subject text
    , email_body text
    , email_recipients text
    )
    as
    declare
      l_error_txt varchar2;
      l_error_cnt varchar2;
      l_email_table varchar2;
    begin
      set smtp-host-address@mail 'smtp.example.com';
      set smtp-user-name@mail 'user@example.com';
      set smtp-password@mail 'your_password_here';
      set [mail-from-name]@mail 'System';
      set [mail-from-email]@mail 'noreply@example.com';
      set smtp-enable-ssl@mail true;
      set smtp-minimum-deliver-duration-ms@mail 1000;
      set mail-body-html@mail true;
      set mail-priority@mail 0;
      begin
    -- sending email
          insert into smtp@mail
            ( toEmail
            , ccEmail
            , bccEmail
            , subject
            , body
            --, attachmentContents -- starting PROD 20.2.112 use Attachment1Contents instead
            --, attachmentName
            --, attachmentMimeType
            )
            select email_recipients
                   toEmail
            ,      cast(null as varchar2)
                   ccEmail
            ,      cast(null as varchar2)
                   bccEmail
            ,      email_subject
                   subject
            ,      email_body
                   body
            --,      null -- rfe.file_contents
            --       attachmentContents -- starting PROD 20.2.112 use Attachment1Contents instead
            --,      null -- basename('{USER_MULTI_PARTIES_FILE_NAME}')
            --       attachmentName
            --,      null -- cast(null as varchar2)
            --       attachmentMimeType
            ;
    end;
end;

-- leave a blank line above, otherweise the stored procedure conflicts with the sql block below.
-- creating necessary objects
Create or replace table timestamp_table@inmemorystorage 
  (
   sql_lastTimestamp int64
  ,sql_count int
  ,eol_count int
  ,eol_lastTimestamp int64
  ,insertupdate int
  ,to_insert int
  ,to_update int
  ,deleted int
  ,to_delete int
  ,to_modify int
  ,division int
  ,count_records_deleted int
  ,count_records_inserted int
  );
Create or replace table to_insertupdate@inmemorystorage as select * from ExactOnlineREST.Sync.SyncTransactionLines@eol LIMIT 0;
Create or replace table to_delete@inmemorystorage as select * from ExactOnlineREST.Sync.Syncdeleted@eol LIMIT 0;
-- create staging tables in SQL Server for parser recognition (structure copied from destination table)
-- Note: These tables must exist for the parser to validate BULK INSERT statements
Create or replace table staging_insertupdate@your_sql_server_datacontainer as 
  select * from target_transaction_lines_table@your_sql_server_datacontainer LIMIT 0;
Create or replace table staging_delete@your_sql_server_datacontainer 
  (EntityKey varchar(255), Division int);
-- ============================================================================
-- SCRIPT EXECUTION STARTS HERE
-- ============================================================================
-- TEST/PROD mode parameter: set to 'TEST' to skip modifications (insert/update/delete), 'PROD' to execute modifications
Declare
  l_mode varchar2 := 'TEST'; -- Change to 'TEST' to skip modifications (PROD)
  l_stmt    varchar2;
  l_debug   varchar2;
  l_sqlresult int;
  l_useless varchar2;
  l_insertScalar int :=0;
  l_log varchar2;
  l_count1 int;
  l_count2 int;
  l_count3 int;
  l_count4 int;
  l_count5 int;
  l_count6 int;
  l_countdeleted int;
  l_countinserted int;
  l_insertstmt varchar2;
  l_scriptstart datetime;
  l_checkinsertupdate int;
  l_checkdeleted int;
  l_scriptsuccess boolean;
  l_scriptduration varchar2;
  l_checkdiff int;
  l_total_expected int;
  l_total_diff_before int;
  l_total_diff_after int;
  l_total_insertupdate int;
  l_total_to_update int;
  l_total_to_insert int;
  l_total_deleted int;
  l_total_to_delete int;
  l_total_operations int;
  l_emailrecipients text;
Begin
  -- set email recipients based on mode
  if l_mode = 'TEST' then
    l_emailrecipients := 'admin@example.com';
  else
    l_emailrecipients := 'admin@example.com,user1@example.com,user2@example.com';
  end if;
  l_debug := 'script starts in ' || l_mode || ' mode. ' || to_char(sysdate, 'YYYY-MM-DD HH24:MI:SS');
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  l_scriptstart := sysdate;
  -- ========================================================================
  -- STEP 1: EOL DATA RETRIEVAL
  -- ========================================================================
  -- Fetch data from EOL API for all divisions:
  --   - Get max timestamp and count per division from SQL Server
  --   - Retrieve records to insert/update from EOL (after last timestamp)
  --   - Retrieve records to delete from EOL (after last timestamp)
  --   - Get total count per division from EOL
  -- ========================================================================
  -- Get the max timestamp per division from SQL Server
  l_debug := 'build timestamp_table@inmemorystorage with native call to sql server';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  for r in 
  (
    execute native 'select MAX(TimeStamp) as sql_lastTimestamp, COUNT(*) as sql_count, Division as division from target_transaction_lines_table group by division'
    datacontainer 'your_sql_server_datacontainer'
  )
  loop
    insert into timestamp_table@inmemorystorage (sql_lastTimestamp, sql_count, division) values (r.sql_lastTimestamp, r.sql_count, r.division);
  end loop;
  l_debug := 'timestamp_table@inmemorystorage has been filed with sqlserver values. Now updating invantive cache';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  select id into l_useless from transactionlinesincremental@eol LIMIT 1;
  l_debug := 'invantive cache updated. now looping on each division at EOL.';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  --
  --get the latest values to insert/update from eol based on timestamp and division
  --
  for r in (select * from timestamp_table@inmemorystorage)
   loop
     l_debug := 'Processing division: ' || r.division;
     dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      --
      -- find records to insert or update per division AFTER last_timestamp
      --
      create or replace table temp@inmemorystorage as select * from ExactOnlineREST.Sync.SyncTransactionLines@eol E
      where E.timestamp > r.sql_lastTimestamp and E.division = r.division;
      l_sqlresult := sqlrowcount-1;
      l_debug := 'found ' || l_sqlresult || ' records to insert/update for division ' || r.division;
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      update timestamp_table@InMemoryStorage set eol_lastTimestamp = case when (select MAX(timestamp) from temp@inmemorystorage) is null then r.sql_lastTimestamp else (select MAX(timestamp) from temp@inmemorystorage) end  where division = r.division;
      l_sqlresult := sqlrowcount;
      l_debug := 'eol_lastTimestamp value: ' || (case when (select MAX(timestamp) from temp@inmemorystorage) is null then r.sql_lastTimestamp else (select MAX(timestamp) from temp@inmemorystorage) end );
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      synchronize temp@inmemorystorage to to_insertupdate@inmemorystorage with insert all except rowid$ identified by id;
      l_sqlresult := sqlrowcount;
      l_debug := 'synchronize inserted ' || l_sqlresult || ' records into to_insertupdate@inmemorystorage';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      --
      -- find deleted uid per division BEFORE last_timestamp
      --
      create or replace table temp2@inmemorystorage as select * from ExactOnlineREST.Sync.SyncDeleted@eol E
      where E.timestamp > r.sql_lastTimestamp and E.division = r.division and Entitytype=1; -- Entitytype=1 means transactionLines;
      l_sqlresult := sqlrowcount-1;
      l_debug := 'found ' || l_sqlresult || ' records to check for deletion for division ' || r.division;
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      synchronize temp2@inmemorystorage to to_delete@inmemorystorage with insert all except rowid$ identified by id;
      l_sqlresult := sqlrowcount;
      l_debug := 'synchronize inserted ' || l_sqlresult || ' records into to_delete@inmemorystorage';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      --
      -- $count the values via ODATA @eol
      --
      insert into exactonlinerest..nativeplatformscalarrequests@eol (url, Orig_System_Reference) values ('/api/v1/' || r.division || '/financialtransaction/TransactionLines/$count', r.division);
      l_sqlresult := sqlrowcount;
      l_debug := '$count from eol for division ' || r.division || ' : ' || (select to_number(result_text) from exactonlinerest..nativeplatformscalarrequests@eol order by transaction_id DESC LIMIT 1);
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      l_insertScalar := l_insertScalar + 1;
      l_debug := 'end of processing division: ' || r.division;
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  end loop;
  l_debug := 'end of divisions loop at EOL.';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  l_stmt := '
  create or replace table eol_count@inmemorystorage as
    select to_number(Orig_System_Reference) division, to_number(result_text) eol_count from exactonlinerest..nativeplatformscalarrequests@eol
    order by transaction_id DESC LIMIT ' || l_insertScalar; -- pay attention this number has to be the exact number of divisions called in the use@eol above
  execute immediate l_stmt;
  synchronize eol_count@inmemorystorage
    to timestamp_table@inmemorystorage
    with update all except rowid$
    identified by Division;
  l_sqlresult := sqlrowcount;
  l_debug := 'updated eol_count table';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  -- ========================================================================
  -- PREPARATION: Populate staging tables for verification and modification
  -- ========================================================================
  -- Clear and populate staging tables once - they will be reused for both
  -- verification and modification steps to avoid redundant data insertion
  -- ========================================================================
  l_debug := 'now clearing staging tables and loading data once for verification and modification';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  -- clear staging tables (tables already exist, created at script start for parser recognition)
  -- in TEST mode, don't truncate to allow inspection of previous data
  if l_mode = 'PROD' then
    execute native 'TRUNCATE TABLE dbo.staging_insertupdate;
    TRUNCATE TABLE dbo.staging_delete;' datacontainer 'your_sql_server_datacontainer';
  else
    l_debug := 'TEST mode: skipping truncate of staging tables';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  end if;
  -- populate staging tables once (will be reused for verification and modification)
  bulk insert into staging_insertupdate@your_sql_server_datacontainer
           (Account, AmountDC, AmountFC, AmountVATBaseFC, AmountVATFC, Asset, CostCenter, CostUnit, Created, Creator, Currency, Date, Description, Division, Document, DueDate, EntryID, EntryNumber, ExchangeRate, ExtraDutyAmountFC, ExtraDutyPercentage, FinancialPeriod, FinancialYear, GLAccount, ID, InvoiceNumber, Item, JournalCode, LineNumber, LineType, Modified, Modifier, Notes, OffsetID, OrderNumber, PaymentDiscountAmount, PaymentReference, Project, Quantity, Status, Subscription, Timestamp, TrackingNumber, Type, VATCode, VATPercentage, VATType, YourRef)
     select Account, AmountDC, AmountFC, AmountVATBaseFC, AmountVATFC, Asset, CostCenter, CostUnit, Created, Creator, Currency, Date, Description, Division, Document, DueDate, EntryID, EntryNumber, ExchangeRate, ExtraDutyAmountFC, ExtraDutyPercentage, FinancialPeriod, FinancialYear, GLAccount, ID, InvoiceNumber, Item, JournalCode, LineNumber, LineType, Modified, Modifier, Notes, OffsetID, OrderNumber, PaymentDiscountAmount, PaymentReference, Project, Quantity, Status, Subscription, Timestamp, TrackingNumber, Type, VATCode, VATPercentage, VATType, YourRef
     from to_insertupdate@inmemorystorage;
  l_sqlresult := sqlrowcount;
  l_debug := 'populated staging_insertupdate table with ' || l_sqlresult || ' records for all divisions.';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  bulk insert into staging_delete@your_sql_server_datacontainer (EntityKey, Division)
     select EntityKey, Division from to_delete@inmemorystorage;
  l_sqlresult := sqlrowcount;
  l_debug := 'populated staging_delete table with ' || l_sqlresult || ' records for all divisions.';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  -- ========================================================================
  -- STEP 2: VERIFICATION
  -- ========================================================================
  -- Verify data consistency using EXECUTE NATIVE (avoids loading data into Invantive):
  --   - Count records to insert/update per division
  --   - Count records to delete per division
  --   - Calculate differences (Diff Before and Diff After)
  --   - If Diff After <> 0, skip modification (data may have changed in EOL during API call)
  -- ========================================================================
  l_debug := 'now entering verification step using EXECUTE NATIVE';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  -- initialize verification fields to 0 for all divisions (in case no data exists in staging tables)
  update timestamp_table@inmemorystorage set insertupdate = 0, to_update = 0, to_insert = 0, deleted = 0, to_delete = 0;
  -- calculate counts per division using EXECUTE NATIVE (avoids loading data into Invantive)
  -- first get insertupdate counts per division
  for r_verify_insert in (
    execute native 'SELECT 
      t.Division,
      COUNT(*) as insertupdate,
      SUM(CASE WHEN s.ID IS NOT NULL THEN 1 ELSE 0 END) as to_update,
      SUM(CASE WHEN s.ID IS NULL THEN 1 ELSE 0 END) as to_insert
    FROM staging_insertupdate t
    LEFT JOIN target_transaction_lines_table s ON s.ID = t.ID
    GROUP BY t.Division' datacontainer 'your_sql_server_datacontainer'
  )
  loop
    update timestamp_table@inmemorystorage set insertupdate = r_verify_insert.insertupdate, to_update = r_verify_insert.to_update, to_insert = r_verify_insert.to_insert where division = r_verify_insert.Division;
  end loop;
  -- then get delete counts per division
  for r_verify_delete in (
    execute native 'SELECT 
      d.Division,
      COUNT(*) as deleted,
      SUM(CASE WHEN s.ID IS NOT NULL THEN 1 ELSE 0 END) as to_delete
    FROM staging_delete d
    LEFT JOIN target_transaction_lines_table s ON s.ID = d.EntityKey
    GROUP BY d.Division' datacontainer 'your_sql_server_datacontainer'
  )
  loop
    update timestamp_table@inmemorystorage set deleted = r_verify_delete.deleted, to_delete = r_verify_delete.to_delete where division = r_verify_delete.Division;
  end loop;
  -- output verification results per division
  for r in (select * from timestamp_table@inmemorystorage)
  loop
    l_debug:= 'division: ' || r.division || '|sql_count: ' || nvl(r.sql_count, 0) || '|eol_count: ' || nvl(r.eol_count, 0) || '|diff before modify: ' || (nvl(r.eol_count, 0) - nvl(r.sql_count, 0))
    || '|merge found from EOL: ' || nvl(r.insertupdate, 0) || '|to update: ' || nvl(r.to_update, 0) || '|to insert: ' || nvl(r.to_insert, 0) || '|delete found from EOL: ' || nvl(r.deleted, 0) || '|to delete : ' || nvl(r.to_delete, 0)
    || '|total operations: ' || (nvl(r.insertupdate, 0) + nvl(r.deleted, 0))
    || '|diff after: ' || ((nvl(r.sql_count, 0) + nvl(r.to_insert, 0) - nvl(r.to_delete, 0)) - nvl(r.eol_count, 0) );
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  end loop;
  -- output total line
  select sum(nvl(insertupdate, 0)), sum(nvl(to_update, 0)), sum(nvl(to_insert, 0)), sum(nvl(deleted, 0)), sum(nvl(to_delete, 0)), 
    sum(nvl(eol_count, 0) - nvl(sql_count, 0)), sum((nvl(sql_count, 0) + nvl(to_insert, 0) - nvl(to_delete, 0)) - nvl(eol_count, 0))
    into l_total_insertupdate, l_total_to_update, l_total_to_insert, l_total_deleted, l_total_to_delete, l_total_diff_before, l_total_diff_after
    from timestamp_table@inmemorystorage;
  l_total_operations := l_total_insertupdate + l_total_deleted;
  l_debug := 'TOTAL|sql_count: |eol_count: |diff before modify: ' || l_total_diff_before
  || '|merge found from EOL: ' || l_total_insertupdate || '|to update: ' || l_total_to_update || '|to insert: ' || l_total_to_insert || '|delete found from EOL: ' || l_total_deleted || '|to delete : ' || l_total_to_delete
  || '|total operations: ' || l_total_operations
  || '|diff after: ' || l_total_diff_after;
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  l_debug := 'end of verification';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);

PART2

 -- ========================================================================
  -- STEP 3: MODIFICATION
  -- ========================================================================
  -- Apply modifications only if verification passed (Diff After = 0)
  -- If Diff After <> 0, data may have changed in EOL during API call,
  -- skip modification to avoid inconsistencies (script will re-run at next scheduled time)
  -- ========================================================================
  if l_total_diff_after = 0 then
    -- Verification passed: proceed with modifications
    l_debug := 'now entering modification processing (all divisions at once) - ' || l_mode || ' mode';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    -- Initialize counters
    l_countinserted := 0;
    l_countdeleted := 0;
    -- Execute MERGE and DELETE in a single transaction with BEGIN TRY
    -- MERGE handles both INSERT and UPDATE operations
    -- DELETE removes records marked for deletion
    l_debug := 'executing MERGE and DELETE statements in transaction';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    if l_mode = 'PROD' then
      -- PROD mode: Execute MERGE and DELETE, then COMMIT
      for r_result in (
      execute native 'BEGIN TRY
        BEGIN TRANSACTION;
        DECLARE @rows_affected_merge INT;
        DECLARE @rows_deleted INT;
        MERGE target_transaction_lines_table AS target
        USING staging_insertupdate AS source
        ON target.ID = source.ID
        WHEN MATCHED THEN
          UPDATE SET
            Account = source.Account, AmountDC = source.AmountDC, AmountFC = source.AmountFC, AmountVATBaseFC = source.AmountVATBaseFC, AmountVATFC = source.AmountVATFC,
            Asset = source.Asset, CostCenter = source.CostCenter, CostUnit = source.CostUnit, Created = source.Created, Creator = source.Creator,
            Currency = source.Currency, Date = source.Date, Description = source.Description, Division = source.Division, Document = source.Document,
            DueDate = source.DueDate, EntryID = source.EntryID, EntryNumber = source.EntryNumber, ExchangeRate = source.ExchangeRate,
            ExtraDutyAmountFC = source.ExtraDutyAmountFC, ExtraDutyPercentage = source.ExtraDutyPercentage, FinancialPeriod = source.FinancialPeriod,
            FinancialYear = source.FinancialYear, GLAccount = source.GLAccount, InvoiceNumber = source.InvoiceNumber, Item = source.Item,
            JournalCode = source.JournalCode, LineNumber = source.LineNumber, LineType = source.LineType, Modified = source.Modified,
            Modifier = source.Modifier, Notes = source.Notes, OffsetID = source.OffsetID, OrderNumber = source.OrderNumber,
            PaymentDiscountAmount = source.PaymentDiscountAmount, PaymentReference = source.PaymentReference, Project = source.Project,
            Quantity = source.Quantity, Status = source.Status, Subscription = source.Subscription, Timestamp = source.Timestamp,
            TrackingNumber = source.TrackingNumber, Type = source.Type, VATCode = source.VATCode, VATPercentage = source.VATPercentage,
            VATType = source.VATType, YourRef = source.YourRef
        WHEN NOT MATCHED BY TARGET THEN
          INSERT (Account, AmountDC, AmountFC, AmountVATBaseFC, AmountVATFC, Asset, CostCenter, CostUnit, Created, Creator, Currency, Date, Description, Division, Document, DueDate, EntryID, EntryNumber, ExchangeRate, ExtraDutyAmountFC, ExtraDutyPercentage, FinancialPeriod, FinancialYear, GLAccount, ID, InvoiceNumber, Item, JournalCode, LineNumber, LineType, Modified, Modifier, Notes, OffsetID, OrderNumber, PaymentDiscountAmount, PaymentReference, Project, Quantity, Status, Subscription, Timestamp, TrackingNumber, Type, VATCode, VATPercentage, VATType, YourRef)
          VALUES (source.Account, source.AmountDC, source.AmountFC, source.AmountVATBaseFC, source.AmountVATFC, source.Asset, source.CostCenter, source.CostUnit, source.Created, source.Creator, source.Currency, source.Date, source.Description, source.Division, source.Document, source.DueDate, source.EntryID, source.EntryNumber, source.ExchangeRate, source.ExtraDutyAmountFC, source.ExtraDutyPercentage, source.FinancialPeriod, source.FinancialYear, source.GLAccount, source.ID, source.InvoiceNumber, source.Item, source.JournalCode, source.LineNumber, source.LineType, source.Modified, source.Modifier, source.Notes, source.OffsetID, source.OrderNumber, source.PaymentDiscountAmount, source.PaymentReference, source.Project, source.Quantity, source.Status, source.Subscription, source.Timestamp, source.TrackingNumber, source.Type, source.VATCode, source.VATPercentage, source.VATType, source.YourRef);
        SET @rows_affected_merge = @@ROWCOUNT;
        DELETE t FROM target_transaction_lines_table t
        INNER JOIN staging_delete d ON t.ID = d.EntityKey;
        SET @rows_deleted = @@ROWCOUNT;
        COMMIT TRANSACTION;
        SELECT @rows_affected_merge as rows_affected, @rows_deleted as rows_deleted;
      END TRY
      BEGIN CATCH
        IF @@TRANCOUNT > 0 ROLLBACK TRANSACTION;
        THROW;
      END CATCH' datacontainer 'your_sql_server_datacontainer'
    )
    loop
        l_countinserted := r_result.rows_affected;
        l_countdeleted := r_result.rows_deleted;
      end loop;
    else
      -- TEST mode: Execute MERGE and DELETE, then ROLLBACK (no modifications persisted)
      for r_result in (
      execute native 'BEGIN TRY
        BEGIN TRANSACTION;
        DECLARE @rows_affected_merge INT;
        DECLARE @rows_deleted INT;
        MERGE target_transaction_lines_table AS target
        USING staging_insertupdate AS source
        ON target.ID = source.ID
        WHEN MATCHED THEN
          UPDATE SET
            Account = source.Account, AmountDC = source.AmountDC, AmountFC = source.AmountFC, AmountVATBaseFC = source.AmountVATBaseFC, AmountVATFC = source.AmountVATFC,
            Asset = source.Asset, CostCenter = source.CostCenter, CostUnit = source.CostUnit, Created = source.Created, Creator = source.Creator,
            Currency = source.Currency, Date = source.Date, Description = source.Description, Division = source.Division, Document = source.Document,
            DueDate = source.DueDate, EntryID = source.EntryID, EntryNumber = source.EntryNumber, ExchangeRate = source.ExchangeRate,
            ExtraDutyAmountFC = source.ExtraDutyAmountFC, ExtraDutyPercentage = source.ExtraDutyPercentage, FinancialPeriod = source.FinancialPeriod,
            FinancialYear = source.FinancialYear, GLAccount = source.GLAccount, InvoiceNumber = source.InvoiceNumber, Item = source.Item,
            JournalCode = source.JournalCode, LineNumber = source.LineNumber, LineType = source.LineType, Modified = source.Modified,
            Modifier = source.Modifier, Notes = source.Notes, OffsetID = source.OffsetID, OrderNumber = source.OrderNumber,
            PaymentDiscountAmount = source.PaymentDiscountAmount, PaymentReference = source.PaymentReference, Project = source.Project,
            Quantity = source.Quantity, Status = source.Status, Subscription = source.Subscription, Timestamp = source.Timestamp,
            TrackingNumber = source.TrackingNumber, Type = source.Type, VATCode = source.VATCode, VATPercentage = source.VATPercentage,
            VATType = source.VATType, YourRef = source.YourRef
        WHEN NOT MATCHED BY TARGET THEN
          INSERT (Account, AmountDC, AmountFC, AmountVATBaseFC, AmountVATFC, Asset, CostCenter, CostUnit, Created, Creator, Currency, Date, Description, Division, Document, DueDate, EntryID, EntryNumber, ExchangeRate, ExtraDutyAmountFC, ExtraDutyPercentage, FinancialPeriod, FinancialYear, GLAccount, ID, InvoiceNumber, Item, JournalCode, LineNumber, LineType, Modified, Modifier, Notes, OffsetID, OrderNumber, PaymentDiscountAmount, PaymentReference, Project, Quantity, Status, Subscription, Timestamp, TrackingNumber, Type, VATCode, VATPercentage, VATType, YourRef)
          VALUES (source.Account, source.AmountDC, source.AmountFC, source.AmountVATBaseFC, source.AmountVATFC, source.Asset, source.CostCenter, source.CostUnit, source.Created, source.Creator, source.Currency, source.Date, source.Description, source.Division, source.Document, source.DueDate, source.EntryID, source.EntryNumber, source.ExchangeRate, source.ExtraDutyAmountFC, source.ExtraDutyPercentage, source.FinancialPeriod, source.FinancialYear, source.GLAccount, source.ID, source.InvoiceNumber, source.Item, source.JournalCode, source.LineNumber, source.LineType, source.Modified, source.Modifier, source.Notes, source.OffsetID, source.OrderNumber, source.PaymentDiscountAmount, source.PaymentReference, source.Project, source.Quantity, source.Status, source.Subscription, source.Timestamp, source.TrackingNumber, source.Type, source.VATCode, source.VATPercentage, source.VATType, source.YourRef);
        SET @rows_affected_merge = @@ROWCOUNT;
        DELETE t FROM target_transaction_lines_table t
        INNER JOIN staging_delete d ON t.ID = d.EntityKey;
        SET @rows_deleted = @@ROWCOUNT;
        ROLLBACK TRANSACTION;
        SELECT @rows_affected_merge as rows_affected, @rows_deleted as rows_deleted;
      END TRY
      BEGIN CATCH
        IF @@TRANCOUNT > 0 ROLLBACK TRANSACTION;
        THROW;
      END CATCH' datacontainer 'your_sql_server_datacontainer'
    )
    loop
        l_countinserted := r_result.rows_affected;
        l_countdeleted := r_result.rows_deleted;
      end loop;
    end if;
    -- Log modification results
    l_debug := 'MERGE completed. Total rows affected: ' || l_countinserted || ' records.';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    l_debug := 'DELETE completed. Total rows deleted: ' || l_countdeleted || ' records.';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    if l_mode = 'PROD' then
      l_debug := 'PROD mode: transaction committed';
    else
      l_debug := 'TEST mode: transaction rolled back (no modifications persisted)';
    end if;
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    -- Post-modification verification: get counts per division from staging tables
    -- (for both TEST and PROD modes to verify operations)
    -- Note: We calculate count_records_inserted from staging table (before truncate)
    -- but for count_records_deleted, we use the actual DELETE result (l_countdeleted)
    -- because after DELETE, records no longer exist in target table for INNER JOIN
    l_debug := 'calculating counts per division from staging tables';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    -- Count inserts/updates per division (from staging table before truncate)
    for r_counts in (
      execute native 'SELECT 
        Division,
        COUNT(*) as count_inserted
      FROM staging_insertupdate
      GROUP BY Division' datacontainer 'your_sql_server_datacontainer'
    )
    loop
      update timestamp_table@inmemorystorage set count_records_inserted = r_counts.count_inserted where division = r_counts.Division;
    end loop;
    -- For deletes, we use the actual DELETE result (l_countdeleted) instead of recalculating
    -- because after DELETE, records no longer exist in target table
    -- Initialize count_records_deleted to 0 for all divisions, then set it based on l_countdeleted
    -- Since we delete all divisions at once, we need to distribute l_countdeleted across divisions
    -- For simplicity, we'll use to_delete as the expected count and compare with l_countdeleted globally
    update timestamp_table@inmemorystorage set count_records_deleted = 0;
    -- Cleanup staging tables (truncate instead of drop, tables must remain for parser recognition)
    -- In TEST mode, don't truncate to allow inspection of data
    if l_mode = 'PROD' then
      execute native 'TRUNCATE TABLE dbo.staging_insertupdate;
      TRUNCATE TABLE dbo.staging_delete;' datacontainer 'your_sql_server_datacontainer';
      l_debug := 'staging tables truncated';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    else
      l_debug := 'TEST mode: staging tables not truncated (data available for inspection)';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    end if;
    l_debug := 'finished the modification processing.';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    -- Final verifications before reporting
    -- verify that all insert/update records were processed (count_records_inserted should equal insertupdate)
    select sum(insertupdate - count_records_inserted) into l_checkinsertupdate from timestamp_table@inmemorystorage;
    -- verify that MERGE result matches expected total (l_countinserted should equal sum of insertupdate)
    select sum(insertupdate) into l_total_expected from timestamp_table@inmemorystorage;
    if l_countinserted != l_total_expected then
      l_checkinsertupdate := l_checkinsertupdate + (l_total_expected - l_countinserted);
      l_debug := 'WARNING: MERGE result (' || l_countinserted || ') does not match expected total (' || l_total_expected || ')';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    end if;
    -- verify that all delete records were processed
    -- Use l_countdeleted (actual DELETE result) instead of count_records_deleted
    -- because after DELETE, records no longer exist in target table for recalculation
    select sum(to_delete) into l_total_expected from timestamp_table@inmemorystorage;
    l_checkdeleted := l_total_expected - l_countdeleted;
    if l_checkdeleted != 0 then
      l_debug := 'WARNING: DELETE result (' || l_countdeleted || ') does not match expected total (' || l_total_expected || ')';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    end if;
    -- in TEST mode, don't check diff with actual data (rollback means data didn't change)
    -- in PROD mode, check that final count matches eol_count
    if l_mode = 'PROD' then
      select sum((sql_count + (insertupdate - to_update) - to_delete) - eol_count) into l_checkdiff from timestamp_table@inmemorystorage;
    else
      l_checkdiff := 0; -- skip diff check in TEST mode (data rolled back)
    end if;
    l_scriptsuccess :=  case when l_checkinsertupdate + l_checkdeleted + l_checkdiff = 0 then true else false end;
    l_debug := 'global verification on insert/update/delete: ' 
              || case when l_scriptsuccess = true then 'ALL GOOD.' else 'ERROR: inconsistencies in verification step.' end;
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  else
    -- ========================================================================
    -- Verification failed: Diff After <> 0
    -- ========================================================================
    -- Data may have changed in EOL during API call, skip modification
    -- to avoid inconsistencies (script will re-run at next scheduled time)
    -- ========================================================================
    l_debug := 'WARNING: Verification failed - Diff After is not zero (' || l_total_diff_after || '). Data may have changed in EOL during API call. Skipping modification and post-verification. Script will re-run at next scheduled time.';
    dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    l_scriptsuccess := false;
    l_countinserted := 0;
    l_countdeleted := 0;
    -- initialize count_records_inserted and count_records_deleted to 0 for email reporting
    update timestamp_table@inmemorystorage set count_records_inserted = 0, count_records_deleted = 0;
    -- cleanup staging tables in PROD mode (in TEST mode, keep them for inspection)
    if l_mode = 'PROD' then
      execute native 'TRUNCATE TABLE dbo.staging_insertupdate;
      TRUNCATE TABLE dbo.staging_delete;' datacontainer 'your_sql_server_datacontainer';
      l_debug := 'staging tables truncated';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    else
      l_debug := 'TEST mode: staging tables not truncated (data available for inspection)';
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
    end if;
  end if;
  -- ========================================================================
  -- STEP 4: EMAIL REPORTING
  -- ========================================================================
  -- Build and send email report with verification results:
  --   - Script status (success/failed/skipped)
  --   - HTML table with verification results per division
  --   - Total operations summary
  -- ========================================================================
  l_scriptduration := round((sysdate-l_scriptstart)*60*60*24,3); -- in seconds
  l_debug := 'script duration: ' || l_scriptduration || ' seconds.';
  dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
  -- Build HTML table with division verification data
  l_stmt := '<table border="1" cellpadding="5" cellspacing="0" style="border-collapse: collapse;"><tr><th>Division</th><th>Division Name</th><th>SQL Count</th><th>EOL Count</th><th>Diff Before</th><th>merge found from EOL</th><th>To Update</th><th>To Insert</th><th>delete found from EOL</th><th>To Delete</th><th>Total Operations</th><th>Diff After</th></tr>';
  for r in (select * from timestamp_table@inmemorystorage order by division)
  loop
      l_stmt := l_stmt || '<tr><td>' || r.division || '</td><td>' 
        || case 
          when r.division = 1000001 then 'Division 1'
          when r.division = 1000002 then 'Division 2'
          when r.division = 1000003 then 'Division 3'
          when r.division = 1000004 then 'Division 4'
          when r.division = 1000005 then 'Division 5'
          when r.division = 1000006 then 'Division 6'
          when r.division = 1000007 then 'Division 7'
          when r.division = 1000008 then 'Division 8'
          when r.division = 1000009 then 'Division 9'
          when r.division = 1000010 then 'Division 10'
          when r.division = 1000011 then 'Division 11'
          when r.division = 1000012 then 'Division 12'
          when r.division = 1000013 then 'Division 13'
          when r.division = 1000014 then 'Division 14'
          else 'Unknown'
        end || '</td><td>' || nvl(r.sql_count, 0) || '</td><td>' || nvl(r.eol_count, 0) || '</td><td>' 
        || (nvl(r.eol_count, 0) - nvl(r.sql_count, 0)) || '</td><td>' || nvl(r.insertupdate, 0) || '</td><td>' 
        || nvl(r.to_update, 0) || '</td><td>' || nvl(r.to_insert, 0) || '</td><td>' || nvl(r.deleted, 0) 
        || '</td><td>' || nvl(r.to_delete, 0) || '</td><td>' || (nvl(r.insertupdate, 0) + nvl(r.deleted, 0))
        || '</td><td>' 
        || ((nvl(r.sql_count, 0) + nvl(r.to_insert, 0) - nvl(r.to_delete, 0)) - nvl(r.eol_count, 0)) || '</td></tr>';
  end loop;
  -- Add total row to HTML table
  l_stmt := l_stmt || '<tr style="font-weight: bold; background-color: #f0f0f0;"><td colspan="2">TOTAL</td><td></td><td></td><td>' 
    || (select sum(nvl(eol_count, 0) - nvl(sql_count, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum(nvl(insertupdate, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum(nvl(to_update, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum(nvl(to_insert, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum(nvl(deleted, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum(nvl(to_delete, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum(nvl(insertupdate, 0) + nvl(deleted, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
    || (select sum((nvl(sql_count, 0) + nvl(to_insert, 0) - nvl(to_delete, 0)) - nvl(eol_count, 0)) from timestamp_table@inmemorystorage) || '</td></tr>';
  l_stmt := l_stmt || '</table>';
  -- Send email report
  sendmail(
      -- email subject here
      '[' || l_mode || '] synchronization script for eol transactionlines: ' || case when l_total_diff_after <> 0 then 'SKIPPED - verification failed (diff <> 0)' when l_scriptsuccess = true then 'full success!' else 'ERROR: inconsistencies in verification step.' end
      -- email body here
      ,'script mode: ' || l_mode || '<br>'
       || 'script status: ' || case when l_total_diff_after <> 0 then '<b>SKIPPED - Verification failed: Diff After is not zero (' || l_total_diff_after || '). Data may have changed in EOL during API call. Modification and post-verification were skipped. Script will re-run at next scheduled time.</b>' when l_scriptsuccess = true then 'full success.' else 'ERROR: inconsistencies in verification step.' end || '<br>'
       || 'script start time: ' || to_char(sysdate, 'YYYY-MM-DD HH24:MI:SS') || '<br>'
       || 'script duration: ' || l_scriptduration || ' seconds.' || '<br>'
       || case when l_total_diff_after = 0 then 'records deleted (to delete and for update done by insert): ' || l_countdeleted || '<br>' || 'records inserted (insert and update): ' || (select sum(count_records_inserted) from timestamp_table@inmemorystorage) || '<br>' else '' end
       || case when l_mode = 'TEST' then '<b>NOTE: Running in TEST mode - no modifications were executed.</b><br>' else '' end
       || '<br>Verification results by division:<br>'
       || l_stmt
       -- || 'full log below' || '<br>'
       -- || replace(l_log, chr(13), '<br>')
      -- recipients here
      ,l_emailrecipients
      );
    -- exception handling
    exception
      when others
      then
      l_debug := 'Error catched in exception: ' || chr(13) || sqlerrm;
      dbms_output.put_line(l_debug); l_log := l_log || sysdate || ' ' || l_debug || chr(13);
      l_scriptsuccess := false;
      l_scriptduration := round((sysdate-l_scriptstart)*60*60*24,3); -- in seconds. computed again here because exception skipped it.
      -- build HTML table with division verification data (if available)
      begin
        l_stmt := '<table border="1" cellpadding="5" cellspacing="0" style="border-collapse: collapse;"><tr><th>Division</th><th>Division Name</th><th>SQL Count</th><th>EOL Count</th><th>Diff Before</th><th>merge found from EOL</th><th>To Update</th><th>To Insert</th><th>delete found from EOL</th><th>To Delete</th><th>Total Operations</th><th>Diff After</th></tr>';
        for r in (select * from timestamp_table@inmemorystorage order by division)
        loop
          l_stmt := l_stmt || '<tr><td>' || r.division || '</td><td>' 
            || case 
            when r.division = 1000001 then 'Division 1'
            when r.division = 1000002 then 'Division 2'
            when r.division = 1000003 then 'Division 3'
            when r.division = 1000004 then 'Division 4'
            when r.division = 1000005 then 'Division 5'
            when r.division = 1000006 then 'Division 6'
            when r.division = 1000007 then 'Division 7'
            when r.division = 1000008 then 'Division 8'
            when r.division = 1000009 then 'Division 9'
            when r.division = 1000010 then 'Division 10'
            when r.division = 1000011 then 'Division 11'
            when r.division = 1000012 then 'Division 12'
            when r.division = 1000013 then 'Division 13'
            when r.division = 1000014 then 'Division 14'
            else 'Unknown'
            end || '</td><td>' || nvl(r.sql_count, 0) || '</td><td>' || nvl(r.eol_count, 0) || '</td><td>' 
            || (nvl(r.eol_count, 0) - nvl(r.sql_count, 0)) || '</td><td>' || nvl(r.insertupdate, 0) || '</td><td>' 
            || nvl(r.to_update, 0) || '</td><td>' || nvl(r.to_insert, 0) || '</td><td>' || nvl(r.deleted, 0) 
            || '</td><td>' || nvl(r.to_delete, 0) || '</td><td>' || (nvl(r.insertupdate, 0) + nvl(r.deleted, 0))
            || '</td><td>' 
            || ((nvl(r.sql_count, 0) + nvl(r.to_insert, 0) - nvl(r.to_delete, 0)) - nvl(r.eol_count, 0)) || '</td></tr>';
        end loop;
        -- add total row
        l_stmt := l_stmt || '<tr style="font-weight: bold; background-color: #f0f0f0;"><td colspan="2">TOTAL</td><td></td><td></td><td>' 
          || (select sum(nvl(eol_count, 0) - nvl(sql_count, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum(nvl(insertupdate, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum(nvl(to_update, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum(nvl(to_insert, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum(nvl(deleted, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum(nvl(to_delete, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum(nvl(insertupdate, 0) + nvl(deleted, 0)) from timestamp_table@inmemorystorage) || '</td><td>' 
          || (select sum((nvl(sql_count, 0) + nvl(to_insert, 0) - nvl(to_delete, 0)) - nvl(eol_count, 0)) from timestamp_table@inmemorystorage) || '</td></tr>';
        l_stmt := l_stmt || '</table>';
      exception
        when others then
          l_stmt := '<p>Error building verification table: ' || sqlerrm || '</p>';
      end;
          sendmail(
      -- email subject here
      '[' || l_mode || '] synchronization script for eol transactionlines: ERROR. An sql exception occured.'
      -- email body here
      ,'script mode: ' || l_mode || '<br>'
       || 'script status: ERROR. An sql exception occured.' || '<br>'
       || case when l_mode = 'PROD' then 'check the table consistency, it may be damaged as partial operation might have occured.' || '<br>' else '' end
       || 'script start time: ' || to_char(l_scriptstart, 'YYYY-MM-DD HH24:MI:SS') || '<br>'
       || 'script duration: ' || l_scriptduration || ' seconds.' || '<br>'
       || 'records deleted (to delete and for update done by insert): ' || l_countdeleted || '<br>'
       || 'records inserted (insert and update): ' || (select sum(count_records_inserted) from timestamp_table@inmemorystorage) || '<br>'
       || '<br>Verification results by division:<br>'
       || l_stmt
       -- || 'full log below' || '<br>'
       -- || replace(l_log, chr(13), '<br>')
      -- recipients here
      ,l_emailrecipients
      );
end;