AFAS-dokument: OutOfMemoryException

Frågor (nl)
,
1

En av mina kunder får felmeddelandet nedan på 1 tabell. De försöker ladda ner dokument med det här SQL-skriptet:

-- Om det verkar omöjligt att få åtkomst till ett dokument: ignorera det.
-- 
-- Ger 404 Hittades inte när det inte är åtkomligt:
--
-- Den begärda OData-URL:en kunde inte hittas på AFAS Online-servern.
-- Kontakta Invantive Support eller använd en annan tabell. 
-- (https://93926.rest.afas.online/profitrestservices/subjectconnector/9520/90b21cfe240a4213805a164886ba6927?take=50000).
--
-- 404:an kan inte besvaras från dokumentcachen på Subject connector, så det kommer att ta
-- ungefär 125 ms per rad vid varje körning.
--
set ignore-http-404-errors@afas true
2026-10-13 04:59:43.017 Information itgensql264: Invantive UniversalSQL statement started.
2026-10-13 04:59:43.035 Information itgendhb017: 1 Rows Affected (19 ms).
2026-10-13 04:59:43.035 Information itgendhb050: Creation date: 13-10-2026 04:59:43 (UTC)
2026-10-13 04:59:43.035 Information itgendhb053: Fill CPU time (ms): 
2026-10-13 04:59:43.035 Information itgendhb056: Fingerprint: 
2026-10-13 04:59:43.035 Information itgendhb057: SQL statement: --
-- Om det verkar omöjligt att få åtkomst till ett dokument: ignorera det.
-- 
-- Ger 404 Hittades inte när det inte är åtkomligt:
--
-- Den begärda OData-URL:en kunde inte hittas på AFAS Online-servern.
-- Kontakta Invantive Support eller använd en annan tabell. 
-- (https://93926.rest.afas.online/profitrestservices/subjectconnector/9520/90b21cfe240a4213805a164886ba6927?take=50000).
--
-- 404:an kan inte besvaras från dokumentcachen på Subject connector, så det kommer att ta
-- ungefär 125 ms per rad vid varje körning.
--
set ignore-http-404-errors@afas true
2026-10-13 04:59:43.035 Information itgendhb058: Parameter values: 
2026-10-13 04:59:43.035 Information itgendhb059: User full name: 
2026-10-13 04:59:43.035 Information itgendhb060: User email address: 
2026-10-13 04:59:43.035 Information itgendhb061: User log on code: 
2026-10-13 04:59:43.035 Information itgendhb063: Active data container provider name: Invantive SQL 25.0.95-PROD+5629
2026-10-13 04:59:43.035 Information itgendhb064: Execution steps: 0
2026-10-13 04:59:43.035 Information itgendhb044: Run SQL statement: create or replace table tstdoc@sql
as
select m.attachment_id
,      m.subject_id
,      m.file_id
,      m.file_name
,      s.filedata
,      s.mimetype
from   AfasOnline.GetConnector.profit_subject_attachments@afas m
join   Subject@afas(m.subject_id, m.file_id) s
--
-- Eventuellt vidarebefordringsbart serversidesfilter.
--
where  m.file_name like 'I________.pdf' 
--
-- Finjusterad kontroll att fakturafilnamnsmönstret matchas.
--
and    regexp_instr(m.file_name, '^I[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].pdf$') = 1
2026-10-13 04:59:43.036 Information itgensql264: Invantive UniversalSQL statement started.
2026-10-13 05:17:34.925 Warning itgendhb173: Context: c:\jobs\sql\repl.sql: create or replace table tstdoc@sql
as
select m.attachment_id...le_name, '^I[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].pdf$') = 1
2026-10-13 05:17:53.255 Error itgencun016: Error itgenclr007: External error in Invantive Data Hub.

Exception of type 'System.OutOfMemoryException' was thrown.
2026-10-13 05:17:55.405 Error itgenclr007: System.OutOfMemoryException
ValidationException
   at Invantive.Data.ParallelSelectManyEnumerator`2.System.Collections.IEnumerator.MoveNext()
   at Invantive.Sql.V1.DataSourceOrFunctionWithRotatorTree.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, SparseArrayPool sparseArrayPool, QueryObject queryObject, ParameterList parameters)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.CleanSparseArrayIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.ExtensionMethods.IteratorWithFinally`1.IteratorWithFinallyEnumerator.MoveNext()
   at Invantive.Data.ExtensionMethods.CacheFirstEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.JoinIterator.InnerLeftRightOuterJoinRegular(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow, IEnumerable`1 outerRows, IEnumerable`1 innerRows, Boolean leftIsOuter, Boolean isRhsTableFunctionOrNestedSql, SparseArray emptyInnerRow, Int64 outerRowsCount, JoinStatistics joinStatistics)+MoveNext()
   at Invantive.Sql.V1.JoinIterator.InnerLeftRightOuterJoin(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow, JoinStatistics joinStatistics, Boolean isRhsTableFunctionOrNestedSql)+MoveNext()
   at Invantive.Sql.V1.JoinIterator.JoinInternal(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.JoinIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FilterIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.ChainedFirehose.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.SelectListIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.GetDataEnumerator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SqlExecutionStep parentStep, SparseArrayPool sparseArrayPool, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.MaterializeSparseArrayIterator.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Data.ResultSet..ctor(GlobalState owner, ExecutionOptions executionOptions, IProviderManager manager, String originCode, Nullable`1 queryTreeType, IColumnDefinitionCollection fields, IEnumerable`1 rows, ResultSetAuditInformation auditInformation)
   at Invantive.Sql.V1.DdlStatementTree.CreateTable(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, FirehoseResultSet results, IDdlProvider ddlProvider, Int64 rowsAffectedStatement, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow, Nullable`1 maxBatchSizeRows)+MoveNext()
   at Invantive.Data.CompressedEnumerable`1.GetEnumerator()+MoveNext()
   at Invantive.Sql.V1.FirehoseResultSet.Iterator(GlobalState owner, ExecutionOptions executionOptions, IQueryStatePerExecution queryStatePerExecution, SparseArrayPool sparseArrayPool, SqlExecutionStep parentStep, ISparseArray outerRow, ISparseArray pSqlVariableRow, SqlColumnDefinitionCollection oldNewTriggerFields, ISparseArray oldTriggerRow, ISparseArray newTriggerRow, SqlColumnDefinitionCollection sourceTriggerFields, ISparseArray sourceTriggerRow)+MoveNext()
   at Invantive.Sql.V1.QueryPlan.CorrectAndCheckData(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, SparseArrayPool sparseArrayPool, IEnumerable`1 data, String sqlStatement)+MoveNext()
   at Invantive.Data.ExtensionMethods.IteratorWithFinally`1.IteratorWithFinallyEnumerator.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Invantive.Data.ConnectionManager.ExecuteAndCombineQueryPlansV1(GlobalState owner, ExecutionOptions executionOptions, SqlExecutionStep sqlExecutionStep, QueryPlan[] queryPlans, String sqlStatement, ParameterList parameters)
   at Invantive.Data.ConnectionManager.ExecuteProviderPassthroughSqlActionTable(GlobalState owner, ExecutionOptions executionOptions, String actionSql, ParameterList parameters, SqlExecutionStep sqlExecutionStep, String callSafeNameOverrule)
--- End of stack trace from previous location ---
   at Invantive.Data.ConnectionManager.ExecuteProviderPassthroughSqlActionTable(GlobalState owner, ExecutionOptions executionOptions, String actionSql, ParameterList parameters, SqlExecutionStep sqlExecutionStep, String callSafeNameOverrule)
   at Invantive.Data.ConnectionManager.b(GlobalState owner, ExecutionOptions, SqlExecutionStep, String actionSql, ParameterList, String)
--- End of stack trace from previous location ---
   at Invantive.Data.ConnectionManager.b(GlobalState owner, ExecutionOptions, SqlExecutionStep, String actionSql, ParameterList, String)
   at Invantive.Data.ConnectionManager.PassthroughSqlActionTable(GlobalState owner, ExecutionOptions executionOptions, String actionSql, ParameterList parameters, String callSafeNameOverrule)
   at Invantive.Producer.Data.Hub.QueryEngine.ProcessStatement(GlobalState owner, ExecutionOptions executionOptions, String statement, ResultSet& resultSet, Action`3 fetchMoreRows, Action`9 executeStatement, Boolean isInInteractiveMode, ScriptingLanguageNextStepSpecification& nextStepSpecification) in c:\smoke\ws-25.0\Invantive.Data.Hub\src\Invantive.Producer.Data.Hub\QueryEngine.cs:line 2582
   at Invantive.Producer.Data.Hub.QueryEngine.ProcessStatements(GlobalState owner, ExecutionOptions executionOptions, Action`3 fetchMoreRows, Action`9 executeStatement, IEnumerable`1 statements, String sourceFileName, ScriptingLanguageNextStepSpecification& exitSpecification) in c:\smoke\ws-25.0\Invantive.Data.Hub\src\Invantive.Producer.Data.Hub\QueryEngine.cs:line 1582
2026-10-13 05:17:56.076 Information itgenqtl137: Exit program. (0)
2026-10-13 05:17:56.218 Information itgendhb105: Executed statement to exit application.
2

Det är mogligt att ett av dokumenten är mycket stort. Totalt hanteras ungefär 7,6 GB med JSON-data.

Några uppföljningsfrågor:

  • Vilken version av Data Hub är detta?
  • Körs processen som en 64-bitars process på ett 64-bitars operativsystem?
  • Om det rör sig om Data Hub med Data Replicator: är alternativet för att cacha data på disk avstängt? Se Error itgendhb212: External error in Invantive Data Hub. (out of memory)
  • Har detta felmeddelande börjat uppstå nyligen eller har det pågått en längre tid?

Det verkar även som att sidstorleken är inställd högt.

  • Har denna ändrats manuellt?
  • Vilka värden används?
  • Hjälper det att sänka värdet?
3
  • Version: 25.0.95-PROD+5630, 64-bitars
  • Endast Invantive Data Hub.
  • Meddelandet har funnits sedan idag

Så vitt jag vet har ingenting ändrats i sidstorleken.

Var skulle jag eventuellt kunna justera detta?

4

Vanligtvis genom ett set requested-page-size SQL-uttalande i början av skriptet.

Även om det inte har skett så många ändringar i AFAS-implementeringen, har mycket arbete lagts ner på SQL-motorn, inklusive en hel del minnesoptimeringar. Du kanske vill göra ett test med den senaste versionen, som finns tillgänglig på https://download.invantive.eu.

5

Från en analys kommer följande hypotes.

create or replace table utan bulk insert och över hela datamängden kommer att kräva cirka 8 eller 16 GB arbetsminne under release 25.0. Detta ryms troligen inte i arbetsminnet.

Det största dokumentet är ungefär 41 MB.

Rekommendationen är att uppgradera till release 27.0. Där finns optimeringar som gör att även create or replace kan bearbeta datamängden med streaming. Detta minskar minnestrycket avsevärt.

Sidstorleken för AFAS-drivrutinen påverkar inte problemet; sidstorleken är endast relevant på sidan mot AFAS, och Subject-anslutningen går per fil.

Workarounds (tillfälliga lösningar):

  • Dubbla serverns arbetsminne (tillfällig lösning).
  • Använd en inkrementell algoritm.