Where to store application events used to monitor application usage patterns and behavior?

Suppose you have an application and you have an APM platform like Datadog or Application Insights that you use to monitor the performance and the inner workings of your application. Sometimes there are application events that you want to record and check later. By analyzing these events you can spot bugs in your application, detect malicious users or understand that some of your customers are using the application in the wrong way. Example scenario: you have a message queue in your backend and sometimes there are items in the queue that cannot be processed because some business validation rules fail (e.g.: mandatory data are missing in the work item). My question is: where do you usually store these kind of application events ? Please, notice that I'm not referring to domain-level events that you want to raise and / or handle in the application code to model the business domain. I'm referring to application events to be used by operation teams to monitor the application behavior, its usage patterns and ensures that the system is behaving correctly. I have seen different approaches to solve this problem: use a structured logging system and store this information in the logs (e.g.: Serilog in the dotnet space). store events at the APM platform level. As an example, Application Insights as a built in concept of custom events with associated metadata and metrics (see here, for an example in the C# sdk). store these events in the application database, by using a custom table. This is also useful to be sure that all the events are indeed stored and to avoid sampling issues and / or losing events because of logs and traces retention policies. Each one of these approaches has some pros and cons. I'm just looking for guidance and some advice from people having experience with this topic. My favorite approach is storing this information using APM-level events; the problem with that is that not all the APM systems have the same concept of event and the same underlying data model (as an example, the concept of event in Application Insight and Datadog is quite different).

Mar 17, 2025 - 23:18

Suppose you have an application and you have an APM platform like Datadog or Application Insights that you use to monitor the performance and the inner workings of your application.

Sometimes there are application events that you want to record and check later. By analyzing these events you can spot bugs in your application, detect malicious users or understand that some of your customers are using the application in the wrong way.

Example scenario: you have a message queue in your backend and sometimes there are items in the queue that cannot be processed because some business validation rules fail (e.g.: mandatory data are missing in the work item).

My question is: where do you usually store these kind of application events ? Please, notice that I'm not referring to domain-level events that you want to raise and / or handle in the application code to model the business domain. I'm referring to application events to be used by operation teams to monitor the application behavior, its usage patterns and ensures that the system is behaving correctly.

I have seen different approaches to solve this problem:

use a structured logging system and store this information in the logs (e.g.: Serilog in the dotnet space).
store events at the APM platform level. As an example, Application Insights as a built in concept of custom events with associated metadata and metrics (see here, for an example in the C# sdk).
store these events in the application database, by using a custom table. This is also useful to be sure that all the events are indeed stored and to avoid sampling issues and / or losing events because of logs and traces retention policies.

Each one of these approaches has some pros and cons. I'm just looking for guidance and some advice from people having experience with this topic. My favorite approach is storing this information using APM-level events; the problem with that is that not all the APM systems have the same concept of event and the same underlying data model (as an example, the concept of event in Application Insight and Datadog is quite different).