Using Azure Application Insights to collect security metrics

Azure Application Insights is a monitoring and analytics service that helps you collect, analyze, and visualize performance and usage data from your applications. When integrated with Identify, Application Insights can log critical security events and errors, enabling administrators to monitor authentication attempts, detect anomalies, and respond to potential threats in near real-time.

By collecting security metrics such as login attempts, account lockouts, and unauthorized access attempts, Application Insights provides invaluable insights into your system's security posture.

This guide will walk you through:

• Setting up Azure Application Insights for security monitoring
• Identifying key security metrics for monitoring user authentication and authorization activities
• Analyzing security metrics with Application Insights
• Setting up alerts for security metrics

Setting up Azure Application Insights for security monitoring

Integrating Application Insights with Identify is as straightforward as setting up any other log target. Follow this guide to configure Application Insights and connect it to your Identify tenant for monitoring and analysis.

Ensure that Security under Enable logging features is active on the Logging page.

Key security metrics to monitor

The following are key metrics you should monitor:

• Failed Logins: Monitor failed authentication attempts to identify potential brute-force attacks or misconfigured accounts.

• Account Lockouts: Track account lockouts to detect security threats, such as repeated failed login attempts.

Azure Application Insights collects some default metrics, but for security-specific monitoring, you may need a list of events to capture specific behaviors or actions that pose security risks. For Identify, you can use these security events for your monitoring:

• Username & Password login failed:
  • Event ID: 4514
  • Action: UserNamePassword - Wrong username or password
• User Account is locked out
  • Event ID: 4516
  • Action: Account locked out

Analyzing security metrics with Application Insights

You have two options to analyze security metrics with Application Insights:

Using the Application Insights Metrics Explorer

Use the Application Insights Metrics Explorer to view the collected security data. Below are two examples to analyze the number of failed logins and account locked out:

• Analyze the number of failed logins by filtering for Event ID 4514.

Anylyze number of account lockouts by filtering for Event ID 4516

Querying metrics with Kusto Query Language (KQL)

Query number of failed logins in a given period of time:

Query number of account locked out in a given period of time:

Other common security metric queries:

• Total account locked out in the last day:

traces
| where timestamp > ago(1d) and customDimensions.Action has "Account locked out"
| summarize count()

• Total failed logins in the last day:

traces
| where timestamp > ago(1d) and customDimensions.Action has "UserNamePassword - Wrong username or password"
| summarize count()

Setting Up Alerts for security metrics

To proactively monitor security-related events, you can configure alerts in Azure Application Insights. These alerts allow you to receive notifications when specific conditions are met, such as multiple failed login attempts or hight number of account locked out, enabling your team to respond quickly to potential threats.

Defining thresholds for key security metrics

For each security metric, define specific thresholds to trigger alerts. This ensures that you receive notifications only when a metric exceeds a normal range.

Failed Logins

A high number of failed login attempts can indicate several security issues:

• Potential brute-force attacks
• Forgotten passwords causing repeated failed attempts
• Unauthorized access attempts from unfamiliar locations

Recommended Threshold: Fire an alert if failed login attempts across all users exceed a certain number (e.g., 100 attempts) within a 10-minute period.

Account lockouts

An account lockout is triggered when a user reaches the maximum allowable failed login attempts, signaling potential unauthorized access attempts or misconfigured login credentials.

Recommended Threshold: Set a system-wide alert if multiple user accounts (e.g., 10 or more) are locked out within a 30-minute window, which could indicate a larger security incident or widespread attack.

Configuring alerts in Application Insights

In Azure Application Insights, alerts are configured through the Alerts panel, where you can define conditions, thresholds, and notification options for each metric. Here’s how to set up alerts for failed login attempts and account lockouts.

Failed logins alert

• In the Azure portal, open your Application Insights resource.
• In the left-hand menu, select Alerts. Click + Create => Alert rule to start configuring an alert.

• Set Up failed login alert

  • Define the Scope: Select your Application Insights resource as the scope for the alert.
  • Add a Condition: Select Custom log search in the Signal name setting. Use a Kusto Query Language (KQL) query to track failed login events. For example:

traces
| where customDimensions.Action has "UserNamePassword - Wrong username or password"

This query will count the failed logins over a 10-minute interval. Adjust the > 100 threshold based on your security needs:

• Set Up Notifications: Configure a Quick Action or Action Group to receive notifications. For example, below defines a Quick Action configuration under the Actions tab.

• Review the alert settings, then click Create alert rule to activate the alert.

Below is the email notification that is sent when the failed login alert is triggered:

Account lockouts alert

You can use the same process described above to set up account lockout alerts.

Testing and tuning alerts

To ensure these alerts work as expected:

• Simulate failed logins: Trigger a few failed login attempts on a test account to verify that the failed login alert is accurate.
• Simulate account lockouts: Configure a test user account to make multiple failed login attempts until it is locked out, and verify that the alert is triggered.
• Fine-Tune Thresholds: Adjust thresholds based on observed activity patterns to minimize unnecessary alerts while effectively detecting security-relevant metrics.