Als ik de documenten via het incrementele script (zie hieronder) loopt dan duurt het erg lang.
Mijn vraag is hoe ik kan controleren of er inderdaad wel documenten binnen komen.
Allereerst heb ik in de database INVANTIVE-EOL een extra tabel aangemaakt (SettingsAttachments) die het script gebruikt (zie screenshot 1)
Vervolgens draai ik m.b.v. Data Hub het daaronder staande script gedraaid (dit was een aangedragen script dat ik enigszins heb aangepast doordat de settings-tabel permanent nodig is en de stukken tussen /…/ gaven foutmeldingen.)
Het 2e screenshot laat de huidige stand van zaken in het batchscript zien dat Data Hub in gebruik heeft.
Vervolgens wilde ik het trace-bestand bijvoegen als bijlage bij ter informatie van wat er gebeurt is tot op het moment van afbreken, daarvoor in de plaats voeg ik een deel toe van de eerste regels t/m een aantal van de in het eind vermelde regels.
use all@eol
declare
l_timestamp integer;
l_timestamp_next_min integer;
g_step_size pls_integer := 10000;
begin
select
--/*+ low_cost */
timestampmax
into l_timestamp
from SettingsAttachments@sqlserver
;
l_timestamp_next_min := l_timestamp + g_step_size;
for in
( select
--/*+ ods(false) http_disk_cache(false) http_memory_cache(false) */
sdt.document
, sdt.url
, sdt.timestamp
from SyncDocumentAttachments@eol sdt
join AttachmentByUrl@eol(sdt.url) atl
where sdt.timestamp > l_timestamp
and sdt.timestamp < l_timestamp_next_min
)
loop
if r.timestamp > l_timestamp
then
l_timestamp := r.timestamp;
end if;
-- Schrijf weg naar disk.
end loop;
--
if l_timestamp < l_timestamp_next_min
then
l_timestamp := l_timestamp_next_min;
end if;
--
update SettingsAttachments@sqlserver
set timestampmax = l_timestamp
;
end;
Customer Service API Process: False
Process ID: 13836
Parent Process ID: 9420
Parent Process Executable: cmd
Working Directory: D:\INVANTIVE-EOL\batch
Command Line: "C:\Program Files (x86)\Invantive Software BV\Invantive Data Hub 20.2\Invantive.Producer.QueryEngine.exe" /database:"Preadyz\exact-online-nl-sql-server" /file:"D:\INVANTIVE-EOL\queries\DataHub\IncrementeleDocumentenOphalen.sql" /logfile:"D:\INVANTIVE-EOL\logs\IncrementeleDocumentenOphalen.log" /logoverwrite:"" /interactive:False
Machine Full Name: PZ-SQL02
Operating system version: 10.0.17763.0
Operating system platform: Windows
Helper platform: Windows
- Is Windows: True
- Is Linux: False
- Is OSX: False
Operating system user: preadyz-admin
X64 Operating system: True
Process Executable: C:\Program Files (x86)\Invantive Software BV\Invantive Data Hub 20.2\Invantive.Producer.QueryEngine.exe
#License Checks: 1
First License Check: 12-4-2022 10:21:30 (UTC)
Last License Check: 12-4-2022 10:21:30 (UTC)
License Contract Code: L206
IIID: 96dd4b02-9a09-9e7c-5243-f24bcb08acb3
X64 Process: True
Within Power BI or Power Query: False
Product: Invantive Data Hub
Description: Invantive Data Hub is command-line driven software that is capable of executing Invantive Query Tool-compatible scripts across many database and cloud platforms. Ideal for high volume data loads and extractions of cloud applications such as Exact Online or Salesforce. It provides a headless mode which makes it ideal for use in server environments and scheduled uploads and downloads in lights-out operations.
Description Short: Invantive(R) Data Hub
Copyright: (C) Copyright 2004-2021 Invantive Software B.V., the Netherlands. All rights reserved.
Trademark: Invantive
Configuration: Release
File Location: C:\Program Files (x86)\Invantive Software BV\Invantive Data Hub 20.2\Invantive.Producer.QueryEngine.exe
File Version: 20.2.86.1889
File Version Label: 20.2.86-PROD+1889
Build Date (UTC): 3-11-2021 20:41:18
Environment variable INVANTIVE_TRACE_TO_FILE: true
Environment variable INVANTIVE_TRACE_ACTIVE: true
File Switch: 2
--- START OF TRACE MESSAGES ---
--- CURRENT TIME 12-04-2022 10:21:32 (UTC) ---
--- LOG FILE PART 1 ---
--- FLUSH OF TRACE CACHE ON SWITCH FILE WITH 2.216 UNFLUSHED CHARACTERS WHICH HAVE BEEN FLUSHED NOW DIRECTLY BEFORE THIS TEXT. ---
{"Message":"Parse data of group '' (UUID BR48DL2dWUuVrOWiOQGAZg==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":930,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"a135aeed-930a-4913-97ac-4fada17688b6"}
{"Message":"Parse data of group 'VNET-nl-data cache' (UUID GQmVvWuOIE+WUti5QQ1f1Q==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":931,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"7c265481-a3a4-4caa-991a-4e6a3eff350a"}
{"Message":"Parse data of group 'vnet' (UUID MHm8DsRf5UScH/6Xg0Q/0w==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":932,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"b50fc24d-1ba4-4ebb-a21e-bd8de1d14139"}
{"Message":"Parse data of group 'AFAS CVO NWF' (UUID bPNoORDHEUqUjaxGZQRzMA==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":933,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"96d4c5b3-6eb5-4db9-b8fa-8942fa12793c"}
{"Message":"Parse data of group '' (UUID +9HGbVx5uE6J5Mo2n+2Vuw==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":934,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"82bed1e4-5628-4e66-836b-4e7431a3fc75"}
{"Message":"Parse data of group 'data cache repository' (UUID yxIblxi36kSSvEWxnGIwuw==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":935,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"6b3c6dba-b352-4cc2-854a-ef0c7348d817"}
{"Message":"Parse data of group '' (UUID I5fy5mnzqUapNA+byG+ZTg==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":936,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"86eb2d7d-c16f-4001-9af8-b926b1933d69"}
{"Message":"Parse data of group 'exact-online-nl-sql-server' (UUID inLBRIw/mke/z4ZqWrKmwA==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":937,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"d68caae8-6d88-47a9-8a07-3f7343555e9d"}
{"Message":"Parse data of group 'eol' (UUID XsK334G9AEKvSO7z789BQA==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1279916Z","Id":938,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"3686b3ef-fb62-46b1-8949-cdf1241e6a43"}
{"Message":"Parse data of group 'sqlserver' (UUID u9S1NoUUSEmhZqhhsild1w==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":939,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"c9466fef-5d22-4921-a109-ac1055b404b1"}
{"Message":"Parse data of group 'bol' (UUID Vi1f83L0UUKhfWzMvoOOrw==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":940,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"6406432f-fb28-49cd-a367-897e3a5362e7"}
{"Message":"Parse data of group 'BOL-nl-data cache' (UUID 1PcG1j/pZ0iMIG8X0mmHLA==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":941,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"aef17863-20a0-491a-ad2b-c7b4eaf4df20"}
{"Message":"Parse data of group 'bol' (UUID PddFVcTzVk2Ba/a2sPDT6A==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":942,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"410e0db2-223d-4a3b-a84b-660ab8f95062"}
{"Message":"Parse data of group 'Business Apps' (UUID 0TqF6SBYfUuLufOElu4tPQ==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":943,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"d8c8ef3a-fe2b-49fe-a663-b50453449aa1"}
{"Message":"Parse data of group 'AFAS Online' (UUID lWv2agjdXUKqXakkiJaRBw==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":944,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"85d73591-fe11-482c-8b14-73c8dc9984c1"}
{"Message":"Parse data of group '' (UUID 0hcnDROp6kWq0o12mkxwbg==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":945,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"a6d88b49-b575-415a-81e9-282711bbfe37"}
{"Message":"Parse data of group 'Open (Public) Data' (UUID 6w2plzM+QE6YNvdwoxk9Mg==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":946,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"d3e5024b-dc58-44e6-adc7-8d454816c5e9"}
{"Message":"Parse data of group 'Kadaster' (UUID q8AFYcZwKEyIRoleqYI7uA==).","MessageCode":null,"OccurredUtc":"2022-04-12T10:21:32.1329868Z","Id":947,"ManagedThreadId":1,"SessionId":"PROP-LastUsedDatabase-f77dd786-17da-46ee-99b8-4654e0fd2f14","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"fc028de4-8b80-4388-9d5b-3e2e483023de"}
..........
t/m
..........
{"Message":"Execution log file: C:\\Users\\preadyz-admin\\Invantive\\executionlog.ndjson","MessageCode":null,"OccurredUtc":"2022-04-12T14:13:38.67397Z","Id":3005,"ManagedThreadId":1,"SessionId":"AUTO-ExecutionLogFileNamePath-db998f5a-da8a-4c5a-830a-16afd6c14aa7","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"b519b354-6820-4094-858e-3b7db5872ced"}
{"Message":"Appending 1.022 characters in 97 ms to file 'C:\\Users\\preadyz-admin\\Invantive\\executionlog.ndjson'.","MessageCode":"itgenfby002","OccurredUtc":"2022-04-12T14:13:38.7713239Z","Id":3006,"ManagedThreadId":1,"SessionId":"MTD-RegisterExecutableExit-6f28669b-a8dc-4ba7-a077-42f5dd3a09a1","PoolIdentityId":null,"CallingProviderAlias":null,"GuiAction":null,"GuiExternalIpAddress":null,"GuiLanguage":null,"GuiModuleName":null,"GuiModuleVersion":null,"GuiOsUser":null,"GuiUser":null,"MachineNameFull":"PZ-SQL02","UserLogonCodeWithDomain":"pz-sql02\\preadyz-admin","ProcessId":13836,"Uid":"bd39a8b4-f11f-4083-923b-112cfaba0b61"}
--- END OF TRACE LOGGING TO FILE ---
--- CURRENT TIME 12-04-2022 14:13:38 (UTC) ---
--- CLOSING DOWN PROCESS ---
Clock run time (ms): 13.936.336
User processor time (ms): 10.594
Privileged processor time (ms): 1.172
Total processor time (ms): 11.766
Peak physical memory size (bytes): 421.179.392
Peak virtual memory size (bytes): 5.614.112.768
--- TRACE CACHE HAD 209.762 UNFLUSHED CHARACTERS WHICH HAVE BEEN FLUSHED NOW DIRECTLY BEFORE THIS TEXT. ---
