Documenti AFAS: OutOfMemoryException

Domande (nl)
,
1

Un mio cliente riceve il seguente errore su 1 tabella. Stanno cercando di scaricare documenti con questo script SQL:

-- Nel caso in cui l'accesso a un documento risulti impossibile: ignoralo.
-- 
-- Genera un errore 404 Not Found quando non è accessibile:
--
-- L'URL OData richiesto non è stato trovato sul server AFAS Online.
-- Contattare il supporto Invantive o utilizzare un'altra tabella. 
-- (https://93926.rest.afas.online/profitrestservices/subjectconnector/9520/90b21cfe240a4213805a164886ba6927?take=50000).
--
-- Il 404 non può essere risolto dalla cache dei documenti sul Subject connector, quindi richiederà
-- circa 125 ms per ogni riga a ogni esecuzione.
--
set ignore-http-404-errors@afas true
2026-10-13 04:59:43.017 Information itgensql264: Invantive UniversalSQL statement started.
2026-10-13 04:59:43.035 Information itgendhb017: 1 Rows Affected (19 ms).
2026-10-13 04:59:43.035 Information itgendhb050: Creation date: 13-10-2026 04:59:43 (UTC)
2026-10-13 04:59:43.035 Information itgendhb053: Fill CPU time (ms): 
2026-10-13 04:59:43.035 Information itgendhb056: Fingerprint: 
2026-10-13 04:59:43.035 Information itgendhb057: SQL statement: --
-- Nel caso in cui l'accesso a un documento risulti impossibile: ignoralo.
-- 
-- Genera un errore 404 Not Found quando non è accessibile:
--
-- L'URL OData richiesto non è stato trovato sul server AFAS Online.
-- Contattare il supporto Invantive o utilizzare un'altra tabella. 
-- (https://93926.rest.afas.online/profitrestservices/subjectconnector/9520/90b21cfe240a4213805a164886ba6927?take=50000).
--
-- Il 404 non può essere risolto dalla cache dei documenti sul Subject connector, quindi richiederà
-- circa 125 ms per ogni riga a ogni esecuzione.
--
set ignore-http-404-errors@afas true
2026-10-13 04:59:43.035 Information itgendhb058: Parameter values: 
2026-10-13 04:59:43.035 Information itgendhb059: User full name: 
2026-10-13 04:59:43.035 Information itgendhb060: User email address: 
2026-10-13 04:59:43.035 Information itgendhb061: User log on code: 
2026-10-13 04:59:43.035 Information itgendhb063: Active data container provider name: Invantive SQL 25.0.95-PROD+5629
2026-10-13 04:59:43.035 Information itgendhb064: Execution steps: 0
2026-10-13 04:59:43.035 Information itgendhb044: Run SQL statement: create or replace table tstdoc@sql
as
select m.attachment_id
,      m.subject_id
,      m.file_id
,      m.file_name
,      s.filedata
,      s.mimetype
from   AfasOnline.GetConnector.profit_subject_attachments@afas m
join   Subject@afas(m.subject_id, m.file_id) s
--
-- Filtro lato server possibilmente inoltrabile.
--
where  m.file_name like 'I________.pdf' 
--
-- Controllo ottimizzato per verificare che il pattern del nome file della fattura corrisponda.
--
and    regexp_instr(m.file_name, '^I[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].pdf$') = 1
2026-10-13 04:59:43.036 Information itgensql264: Invantive UniversalSQL statement started.
2026-10-13 05:17:34.925 Warning itgendhb173: Context: c:\jobs\sql\repl.sql: create or replace table tstdoc@sql
as
select m.attachment_id...le_name, '^I[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].pdf$') = 1
2026-10-13 05:17:53.255 Error itgencun016: Error itgenclr007: External error in Invantive Data Hub.

Exception of type 'System.OutOfMemoryException' was thrown.
2026-10-13 05:17:55.405 Error itgenclr007: System.OutOfMemoryException
ValidationException
   at Invantive.Data.ParallelSelectManyEnumerator`2.System.Collections.IEnumerator.MoveNext()
   at Invantive.Sql.V1.DataSourceOrFunctionWithRotatorTree.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, SparseArrayPool sparseArrayPool, QueryObject queryObject, ParameterList parameters)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.CleanSparseArrayIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.ExtensionMethods.IteratorWithFinally`1.IteratorWithFinallyEnumerator.MoveNext()
   at Invantive.Data.ExtensionMethods.CacheFirstEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.JoinIterator.InnerLeftRightOuterJoinRegular(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow, IEnumerable`1 outerRows, IEnumerable`1 innerRows, Boolean leftIsOuter, Boolean isRhsTableFunctionOrNestedSql, SparseArray emptyInnerRow, Int64 outerRowsCount, JoinStatistics joinStatistics)+MoveNext()
   at Invantive.Sql.V1.JoinIterator.InnerLeftRightOuterJoin(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow, JoinStatistics joinStatistics, Boolean isRhsTableFunctionOrNestedSql)+MoveNext()
   at Invantive.Sql.V1.JoinIterator.JoinInternal(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.JoinIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FilterIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.ChainedFirehose.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.SelectListIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.MaterializeSparseArrayIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.ResultSet..ctor(GlobalState owner, ExecutionOptions executionOptions, IProviderManager manager, String originCode, Nullable`1 queryTreeType, IColumnDefinitionCollection fields, IEnumerable`1 rows, ResultSetAuditInformation auditInformation)
   at Invantive.Sql.V1.DdlStatementTree.CreateTable(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, FirehoseResultSet results, IDdlProvider ddlProvider, Int64 rowsAffectedStatement, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow, Nullable`1 maxBatchSizeRows)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.QueryPlan.CorrectAndCheckData(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, SparseArrayPool sparseArrayPool, IEnumerable`1 data, String sqlStatement)+MoveNext()
   at Invantive.Data.ExtensionMethods.IteratorWithFinally`1.IteratorWithFinallyEnumerator.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Invantive.Data.ConnectionManager.ExecuteAndCombineQueryPlansV1(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, QueryPlan[] queryPlans, String sqlStatement, ParameterList parameters)
   at Invantive.Data.ConnectionManager.ExecuteProviderPassthroughSqlActionTable(GlobalState owner, ExecutionOptions executionOptions, String actionSql, ParameterList parameters, SqlExecutionStep sqlExecutionStep, String callSafeNameOverrule)
--- End of stack trace from previous location ---
   at Invantive.Data.ConnectionManager.ExecuteProviderPassthroughSqlActionTable(GlobalState owner, ExecutionOptions executionOptions, String actionSql, ParameterList parameters, SqlExecutionStep sqlExecutionStep, String callSafeNameOverrule)
   at Invantive.Data.ConnectionManager.b(GlobalState owner, ExecutionOptions, SqlExecutionStep, String actionSql, ParameterList, String)
--- End of stack trace from previous location ---
   at Invantive.Data.ConnectionManager.b(GlobalState owner, ExecutionOptions, SqlExecutionStep, String actionSql, ParameterList, String)
   at Invantive.Data.ConnectionManager.PassthroughSqlActionTable(GlobalState owner, ExecutionOptions executionOptions, String actionSql, ParameterList parameters, String callSafeNameOverrule)
   at Invantive.Producer.Data.Hub.QueryEngine.ProcessStatement(GlobalState owner, ExecutionOptions executionOptions, String statement, ResultSet& resultSet, Action`3 fetchMoreRows, Action`9 executeStatement, Boolean isInInteractiveMode, ScriptingLanguageNextStepSpecification& nextStepSpecification) in c:\smoke\ws-25.0\Invantive.Data.Hub\src\Invantive.Producer.Data.Hub\QueryEngine.cs:line 2582
   at Invantive.Producer.Data.Hub.QueryEngine.ProcessStatements(GlobalState owner, ExecutionOptions executionOptions, Action`3 fetchMoreRows, Action`9 executeStatement, IEnumerable`1 statements, String sourceFileName, ScriptingLanguageNextStepSpecification& exitSpecification) in c:\smoke\ws-25.0\Invantive.Data.Hub\src\Invantive.Producer.Data.Hub\QueryEngine.cs:line 1582
2026-10-13 05:17:56.076 Information itgenqtl137: Exit program. (0)
2026-10-13 05:17:56.218 Information itgendhb105: Executed statement to exit application.
2

Probabilmente uno dei documenti è molto grande. In totale vengono elaborati circa 7,6 GB di dati JSON.

Alcune domande di approfondimento:

  • Quale versione di Data Hub è questa?
  • Il processo viene eseguito come processo a 64 bit su un sistema operativo a 64 bit?
  • Se si tratta di Data Hub con Data Replicator: l’opzione per memorizzare nella cache i dati su disco è disattivata? Vedi Error itgendhb212: External error in Invantive Data Hub. (out of memory)
  • Questo messaggio si verifica da poco tempo o già da più tempo?

Inoltre, la dimensione della pagina sembra impostata su un valore elevato.

  • È stata modificata manualmente?
  • Quali valori vengono utilizzati?
  • Aiuta impostare un valore più basso?
3
  • Versione: 25.0.95-PROD+5630, 64 bit
  • Solo Invantive Data Hub.
  • La segnalazione è presente da oggi

Per quanto ne so, non è stata modificata la dimensione della pagina.

Dove potrei eventualmente modificare questa impostazione?

4

Di solito tramite un’istruzione SQL set requested-page-size all’inizio dello script.

Anche se non ci sono stati molti cambiamenti nell’implementazione di AFAS, sono stati fatti molti lavori sul motore SQL, inclusa una notevole ottimizzazione della memoria. Potresti voler fare un test utilizzando l’ultima versione, disponibile su https://download.invantive.eu.

5

Da un’analisi emerge la seguente ipotesi.

Il create or replace table senza bulk insert e sull’intero dataset richiederà circa 8 o 16 GB di memoria di lavoro sotto la release 25.0. Questo probabilmente non rientra nella memoria di lavoro disponibile.

Il documento più grande è di circa 41 MB.

Si consiglia di passare alla release 27.0. Questa contiene ottimizzazioni per elaborare il dataset in streaming anche con create or replace. Questo riduce notevolmente la pressione sulla memoria.

La dimensione della pagina del driver AFAS non ha alcuna influenza sul problema; la dimensione della pagina è rilevante solo lato AFAS, e il connettore Subject procede per file.

Soluzioni alternative:

  • Raddoppiare la memoria di lavoro del server (soluzione temporanea).
  • Utilizzare un algoritmo incrementale.