Invantive Data Hub offers standard integration with Elastic APM.
The integration of Invantive Data Hub with Elastic APM makes it easier to control large complex environments with many processing elements.
Elastic APM is a separate third-party product from Elastic for monitoring and controlling complex IT applications.
Elastic APM Integration
Invantive Data Hub’s Elastic APM integration captures the following elements:
- Invantive Trace: insight at a deep level into which activities are executed by whom, where and when;
- SQL transactions: the execution path is measured and recorded per execution of a (P)SQL statement;
- Errors encountered: a registration is performed per error message;
- Relationships between other services for the Elastic APM service map.
Elastic APM Example
For example it is possible to see what transactions have been executed when:
including zoom to individual transactions of a SQL-statement. The example below visualizes the execution of a complex PSQL-statement with many steps, each visible including joins:
The distribution of execution times is easily available across all executions of a statement in a time period:
Also the backing Invantive Trace of a statement’s execution is only 1 click away in Elastic APM:
A view across all activity of a service or server can also be easily and quickly retrieved, and combined with alerts or dashboards:
Appsettings.Production.json
On Invantive Data Hub is for 24.0-releases since July 2024 the presence of an application configuration file appsettings.Production.json required in the installation folder (relocated to %USERPROFILE%/invantive starting 24.0.314).
This configuration file allows transactions to be recorded within a service on Elastic APM.
Data Hub without Elastic APM
The following contents of the configuration file can be used when Elastic APM is not to be used:
Contents no use
{ "Logging":
{ "LogLevel":
{ "Default": "Warning"
}
}
//
// Group to configure Invantive Data Hub.
//
, "DataHub":
{ //
// Elastic APM
//
"ElasticApm":
{ //
// Whether enabled.
//
"Enabled": false
//
// The API key used to send data to the APM server.
// Ensures that only your agents can send data to your APM server.
//
, "ApiKey": null
//
// When defined, all namespaces not starting with one of the values of this collection are ignored when determining
// Exception culprit.
//
, "ApplicationNamespaces": []
//
// Controls which baggage values are automatically attached to the given event (transactions, spans, errors). Baggage values are derived from the "baggage" header defined in
// https://www.w3.org/TR/baggage/.
// You can programmatically write and read baggage values via the Activity API.
//
, "BaggageToAttach": null
//
// For transactions that are HTTP requests, the agent can optionally capture the request body, e.g., POST variables.
// If the request has a body and this setting is disabled, the body will be shown as [REDACTED].
// Valid values are off, errors, transactions and all.
//
, "CaptureBody": "off"
//
// Configures for which content types the body should be captured.
//
, "CaptureBodyContentTypes": ["application/x-www-form-urlencoded*, text/*", "application/json*", "application/xml*"]
//
// Capture request and response headers, including cookies.
//
, "CaptureHeaders": true
//
// Whether the agent is configured to make periodic requests to the APM server to fetch and use the latest
// APM agent central configuration.
//
, "CentralConfig": false
//
// Specify which cloud provider should be assumed for metadata collection. By default, the agent will attempt to detect
// the cloud provider or, if that fails, will use trial and error to collect the metadata. Valid options are
// "aws", "gcp", and "azure".
// If this config value is set to "False", no cloud metadata will be collected.
//
, "CloudProvider": "aws"
//
// Disables the collection of certain metrics. If the name of a metric matches any of the wildcard expressions, it will
// not be collected
//
, "DisableMetrics": []
//
// Enables OpenTelemetry bridging. If this is set to true, the agent will automatically capture every
// <see cref="System.Diagnostics.Activity" /> as part of a trace captured in Elastic APM.
//
, "OpenTelemetryBridgeEnabled": true
//
// The name of the environment this service is deployed in.
//
, "Environment": "Production"
//
// A list of namespaces to exclude when reading an exception's StackTrace to determine the culprit.
// Namespaces are checked with string.StartsWith() so "System." matches all System namespaces
//
, "ExcludedNamespaces": [ "System.", "Microsoft.", "MS.", "FSharp.", "Newtonsoft.Json", "Serilog", "NLog", "Giraffe." ]
//
// Sets the minimum duration of exit spans in milliseconds. Exit spans with a duration lesser than this threshold are attempted to be
// discarded.
// If the exit span is equal or greater the threshold, it should be kept.
// In some cases exit spans cannot be discarded. For example, spans that propagate the trace context to downstream
// services, such as outgoing HTTP requests, can't be discarded.
// However, external calls that don't propagate context, such as calls to a database, can be discarded using this
// threshold. Additionally, spans that lead to an error can't be discarded.
//
, "ExitSpanMinDuration": 100.0
//
// The maximal amount of time (in seconds) events are held in queue until there is enough to send a batch.
// It's possible for a batch to contain less then <seealso cref="MaxBatchEventCount" /> events
// if there are events that need to be sent out because they were held for too long.
// A lower value will increase the load on your APM server,
// while a higher value can increase the memory pressure on your app.
// A higher value also impacts the time until transactions are indexed and searchable in Elasticsearch.
//
// * Positive number - The maximal amount of time to hold events in queue.
// * 0 - Events are not held in queue but are sent immediately.
// * Negative - Invalid and the default value is used instead.
//
, "FlushInterval": "00:00:10"
//
// Unknown.
//
, "GlobalLabels": {}
//
// Allows for the reported hostname to be manually specified. If unset, the hostname will be detected.
//
, "HostName": "ENTER_YOUR_HOST_NAME"
//
// Disables the tracing of messages from certain queues, topics exchanges.
// If the name of a queue, topic or exchange matches any of the wildcard expressions, it will
// not be traced
//
, "IgnoreMessageQueues": []
//
// The logging level for the agent.
//
, "LogLevel": 3
//
// The maximal number of events to send in a batch.
// It's possible for a batch contain less then the maximum events
// if there are events that need to be sent out because they were held for too long.
//
// * Positive number - The maximal number of of events to send in a batch.
// * 0 and negative - Invalid and the default value is used instead.
//
, "MaxBatchEventCount": 10
//
// The maximal number of events to hold in queue as candidates to be sent.
// If the queue is at its maximum capacity then the agent discards the new events
// until the queue has free space.
// * Positive number - The maximal number of of events to send in a batch.
// If MaxQueueEventCount is less than MaxBatchEventCount then
// MaxBatchEventCount is used as MaxQueueEventCount.
// * 0 and negative - Invalid and the default value is used instead.
//
, "MaxQueueEventCount": 1000
//
// Unknown.
//
, "MetricsIntervalInMilliseconds": 30000.0
//
// Whether the agent is recording.
// When set to true. the agent instruments and capture requests, tracks errors, and
// collects and sends metrics.
// When set to false, the agent does not collect data or communicate with the APM server, except to
// fetch central configuration.
// Recording can be changed during the lifetime of the application.
//
, "Recording": true
//
// Sometimes it is necessary to sanitize the data sent to Elastic APM, e.g. remove sensitive data.
// Configure a list of wildcard patterns of field names which should be sanitized.
// These apply for example to HTTP headers and application/x-www-form-urlencoded data.
//
, "SanitizeFieldNames": []
//
// A secret token to ensure that only your agents can send data to your APM server.
// Both agents and APM server have to be configured with the same secret token.
//
, "SecretToken": "ENTER_YOUR_SECRET_TOKEN"
//
// The path to the PEM-encoded certificate used by APM server. This can be used when using a certificate
// signed by a Certificate Authority (CA) that is not in the trust store, such as a self-signed certificate,
// to perform validation through certificate pinning.
//
, "ServerCert": null
//
// The URL for APM server.
//
, "ServerUrl": "https://ENTER_YOUR_APM_SERVER_NAME.apm.eu-west-1.aws.cloud.es.io:443"
//
// The URLs for APM server.
//
, "ServerUrls": null
//
// If set to true, Agent will use windows default credentials that the service running the agent has to send requests if it is needed for proxy.
//
, "UseWindowsCredentials": false
//
// The name of service instrumented by the APM agent. This is used to group all the errors and transactions
// of the service together, and is the primary filter in the Elastic APM user interface.
//
, "ServiceName": "Invantive Data Hub"
//
// A name used to differentiate between nodes in a service. If not set, data aggregations will be done
// based on a container ID (where valid) or on the reported hostname (automatically discovered).
//
, "ServiceNodeName": null
//
// The version of the service.
// If deployments are not versioned, it is recommended to set this to the commit identifier of the deployed revision,
// e.g. the output of git rev-parse HEAD.
//
, "ServiceVersion": "24.0.291"
//
// Setting this option to true will enable span compression feature.
// Span compression reduces the collection, processing, and storage overhead, and removes clutter from the UI.
// The tradeoff is that some information such as DB statements of all the compressed spans will not be collected.
//
, "SpanCompressionEnabled": true
//
// Consecutive spans that are exact match and that are under this threshold will be compressed into a single composite
// span. Measurement is in milliseconds.
// This option does not apply to composite spans. This reduces the collection, processing, and storage overhead, and
// removes clutter
// from the UI.
// The tradeoff is that the DB statements of all the compressed spans will not be collected.
//
, "SpanCompressionExactMatchMaxDuration": 100.0
//
// Consecutive spans to the same destination that are under this threshold will be compressed into a single composite
// span.
// This option does not apply to composite spans.
// This reduces the collection, processing, and storage overhead, and removes clutter from the UI. The tradeoff is that
// the DB statements of
// all the compressed spans will not be collected.
//
, "SpanCompressionSameKindMaxDuration": 100.0
//
// The agent limits stack trace collection to spans with durations equal or longer than the given value
// negative value: Disables stack trace collection for spans completely
// 0: stacktrace will be collected for all spans
// positive value n: stacktrace will be captured for spans with a duration equal or longer than n ms.
//
, "SpanStackTraceMinDurationInMilliseconds": 100.0
//
// The agent limits stack trace collection to spans with durations equal or longer than the given value
// 0: Disables stack trace collection for spans completely
// negative value: stacktrace will be collected for all spans
// positive value n: stacktrace will be captured for spans with a duration equal or longer than n ms.
//
, "SpanFramesMinDurationInMilliseconds": 100.0
//
// The number of stack frames the agent collects.
// 0: no stacktrace is collected - This also applies to spans no matter what is the value of
// SpanFramesMinDurationInMilliseconds.
// negative: all frames must be collected
// positive number n: top n frames must be collected
//
, "StackTraceLimit": -1
//
// By setting this to true the agent will ignore the sampled part of the W3C TraceContext traceparent header
// when it's false and when the upstream transaction is not coming from an Elastic APM agent.
// In practice this means that in case a caller service calls another service where this value is true,
// the agent will ignore the sampling decision of the upstream service, and it will make a new sampling decision.
// This can be useful when a caller service always sets the sampled flag to false and the agent would have no chance to
// create any sampled transaction.
//
, "TraceContextIgnoreSampledFalse": false
//
// The traceparent header of requests that are traced with our agents might have been added by a 3rd party component.
// This situation becomes more and more common as the w3c trace context gets adopted. In such cases we can end up with
// traces where part of the trace is outside of our system.
// In order to handle this properly, the agent SHOULD offer several trace continuation strategies.
// The agent SHOULD offer a configuration called trace_continuation_strategy with the following values and behavior:
// - continue: The agent takes the traceparent header as it is and applies it to the new transaction.
// - restart: The agent always creates a new trace with a new trace id. In this case the agent MUST create a span link in
// the new transaction pointing to the original traceparent.
// - restart_external: The agent first checks the tracestate header. If the header contains the es vendor flag, it's
// treated as internal, otherwise (including the case when the tracestate header is not present) it's treated as external.
// In case of external calls the agent MUST create a new trace with a new trace id and MUST create a link in the new
// transaction pointing to the original trace.
//
, "TraceContinuationStrategy": "continue"
//
// A list of patterns to match HTTP requests to ignore. An incoming HTTP request whose request line matches any of the
// patterns will not be reported as a transaction.
//
, "TransactionIgnoreUrls": []
//
// A list of patterns to be used to group incoming HTTP server transactions by matching names contain dynamic parts
// to a more suitable route name.
//
, "TransactionNameGroups": []
//
// The number of spans that are recorded per transaction.
// * 0: no spans will be collected.
// * -1: all spans will be collected.
//
, "TransactionMaxSpans": 500
//
// The sample rate for transactions.
// By default, the agent will sample every transaction (e.g. a request to your service). To reduce overhead and storage
// requirements,
// the sample rate can be set to a value between 0.0 and 1.0. The agent still records the overall time and result for
// unsampled
// transactions, but no context information, labels, or spans are recorded.
//
, "TransactionSampleRate": 1.0
//
// If true, for all outgoing HTTP requests the agent stores the traceparent in a "elastic-apm-traceparent" header
// name.
// Otherwise, it'll use the official w3c "traceparent" header name.
//
, "UseElasticTraceparentHeader": false
//
// If true, the default, the agent will use the path of the incoming HTTP request as the transaction name in situations
// when a more accurate route name cannot be determined from route data or request headers.
//
, "UsePathAsTransactionName": true
//
// The agent verifies the server's certificate if an HTTPS connection to the APM server is used.
// Verification can be disabled by setting to false.
//
, "VerifyServerCert": true
//
// A description for the configuration.
//
, "Description": "Invantive Data Hub"
}
}
}
Data Hub with Elastic APM
Basically, when using Elastic APM, the file should only be modified in the following areas:
Enabled:falsebecomestrue.HostName: provide desired name device in APM of leave empty for default.SecretToken: token from Elastic APM.ServerUrl: server URL from Elastic APM.
This data can be found in the menu at “Management” > “Integration” and then “APM”:
below:
