Data collection, retention, and storage in Application Insights

When you install the Application Insights SDK in your app, it sends telemetry about your app to the cloud. As a responsible developer, you want to know exactly what data is sent, what happens to the data, and how you can keep control of it. In particular, could sensitive data be sent, where is it stored, and how secure is it?

First, the short answer:

• The standard telemetry modules that run "out of the box" are unlikely to send sensitive data to the service. The telemetry is concerned with load, performance and usage metrics, exception reports, and other diagnostic data. The main user data visible in the diagnostic reports are URLs. But your app shouldn't, in any case, put sensitive data in plain text in a URL.
• You can write code that sends more custom telemetry to help you with diagnostics and monitoring usage. (This extensibility is a great feature of Application Insights.) It would be possible, by mistake, to write this code so that it includes personal and other sensitive data. If your application works with such data, you should apply a thorough review process to all the code you write.
• While you develop and test your app, it's easy to inspect what's being sent by the SDK. The data appears in the debugging output windows of the IDE and browser.
• You can select the location when you create a new Application Insights resource. For more information about Application Insights availability per region, see Products available by region.
• Review the collected data because it might include data that's allowed in some circumstances but not others. A good example of this circumstance is device name. The device name from a server doesn't affect privacy and is useful. A device name from a phone or laptop might have privacy implications and be less useful. An SDK developed primarily to target servers would collect device name by default. This capability might need to be overwritten in both normal events and exceptions.

The rest of this article discusses these points more fully. The article is self-contained, so you can share it with colleagues who aren't part of your immediate team.

What is Application Insights?

Application Insights is a service provided by Microsoft that helps you improve the performance and usability of your live application. It monitors your application all the time it's running, both during testing and after you've published or deployed it. Application Insights creates charts and tables that show you informative metrics. For example, you might see what times of day you get most users, how responsive the app is, and how well it's served by any external services that it depends on. If there are failures or performance issues, you can search through the telemetry data to diagnose the cause. The service sends you emails if there are any changes in the availability and performance of your app.

To get this functionality, you install an Application Insights SDK in your application, which becomes part of its code. When your app is running, the SDK monitors its operation and sends telemetry to an Application Insights Log Analytics workspace, which is a cloud service hosted by Microsoft Azure. Application Insights also works for any applications, not just applications that are hosted in Azure.

Application Insights stores and analyzes the telemetry. To see the analysis or search through the stored telemetry, you sign in to your Azure account and open the Application Insights resource for your application. You can also share access to the data with other members of your team, or with specified Azure subscribers.

You can have data exported from Application Insights, for example, to a database or to external tools. You provide each tool with a special key that you obtain from the service. The key can be revoked if necessary.

Application Insights SDKs are available for a range of application types:

• Web services hosted in your own Java EE or ASP.NET servers, or in Azure
• Web clients, that is, the code running in a webpage
• Desktop apps and services
• Device apps such as Windows Phone, iOS, and Android

They all send telemetry to the same service.

What data does it collect?

There are three sources of data:

• The SDK, which you integrate with your app either in development or at runtime. There are different SDKs for different application types. There's also an SDK for webpages, which loads into the user's browser along with the page.

  • Each SDK has many modules, which use different techniques to collect different types of telemetry.
  • If you install the SDK in development, you can use its API to send your own telemetry, in addition to the standard modules. This custom telemetry can include any data you want to send.

• In some web servers, there are also agents that run alongside the app and send telemetry about CPU, memory, and network occupancy. For example, Azure VMs, Docker hosts, and Java application servers can have such agents.

• Availability overview are processes run by Microsoft that send requests to your web app at regular intervals. The results are sent to Application Insights.

What kind of data is collected?

The main categories are:

• Web server telemetry: HTTP requests. URI, time taken to process the request, response code, and client IP address. Session id.
• Webpages: Page, user, and session counts. Page load times. Exceptions. Ajax calls.
• Performance counters: Memory, CPU, IO, and network occupancy.
• Client and server context: OS, locale, device type, browser, and screen resolution.
• Exceptions and crashes: Stack dumps, build id, and CPU type.
• Dependencies: Calls to external services such as REST, SQL, and AJAX. URI or connection string, duration, success, and command.
• Availability tests: Duration of test and steps, and responses.
• Trace logs and custom telemetry: Anything you code into your logs or telemetry.

For more information, see the section Data sent by Application Insights.

How can I verify what's being collected?

If you're developing an app using Visual Studio, run the app in debug mode (F5). The telemetry appears in the Output window. From there, you can copy it and format it as JSON for easy inspection.

There's also a more readable view in the Diagnostics window.

For webpages, open your browser's debugging window. Select F12 and open the Network tab.

Can I write code to filter the telemetry before it's sent?

You'll need to write a telemetry processor plug-in.

How long is the data kept?

Raw data points (that is, items that you can query in Analytics and inspect in Search) are kept for up to 730 days. You can select a retention duration of 30, 60, 90, 120, 180, 270, 365, 550, or 730 days. If you need to keep data longer than 730 days, you can use diagnostic settings.

Data kept longer than 90 days incurs extra charges. For more information about Application Insights pricing, see the Azure Monitor pricing page.

Aggregated data (that is, counts, averages, and other statistical data that you see in metric explorer) are retained at a grain of 1 minute for 90 days.

Debug snapshots are stored for 15 days. This retention policy is set on a per-application basis. If you need to increase this value, you can request an increase by opening a support case in the Azure portal.

Who can access the data?

The data is visible to you and, if you have an organization account, your team members.

It can be exported by you and your team members and could be copied to other locations and passed on to other people.

What does Microsoft do with the information my app sends to Application Insights?

Microsoft uses the data only to provide the service to you.

Where is the data held?

You can select the location when you create a new Application Insights resource. For more information about Application Insights availability, see Products available by region.

How secure is my data?

Application Insights is an Azure service. Security policies are described in the Azure Security, Privacy, and Compliance white paper.

The data is stored in Microsoft Azure servers. For accounts in the Azure portal, account restrictions are described in the Azure Security, Privacy, and Compliance document.

Access to your data by Microsoft personnel is restricted. We access your data only with your permission and if it's necessary to support your use of Application Insights.

Data in aggregate across all our customers' applications, such as data rates and average size of traces, is used to improve Application Insights.

Could someone else's telemetry interfere with my Application Insights data?

Someone could send more telemetry to your account by using the instrumentation key. This key can be found in the code of your webpages. With enough extra data, your metrics wouldn't correctly represent your app's performance and usage.

If you share code with other projects, remember to remove your instrumentation key.

Is the data encrypted?

All data is encrypted at rest and as it moves between datacenters.

Is the data encrypted in transit from my application to Application Insights servers?

Yes. We use HTTPS to send data to the portal from nearly all SDKs, including web servers, devices, and HTTPS webpages.

Does the SDK create temporary local storage?

Yes. Certain telemetry channels will persist data locally if an endpoint can't be reached. The following paragraphs describe which frameworks and telemetry channels are affected:

• Telemetry channels that utilize local storage create temp files in the TEMP or APPDATA directories, which are restricted to the specific account running your application. This situation might happen when an endpoint was temporarily unavailable or if you hit the throttling limit. After this issue is resolved, the telemetry channel will resume sending all the new and persisted data.
• This persisted data isn't encrypted locally. If this issue is a concern, review the data and restrict the collection of private data. For more information, see Export and delete private data.
• If a customer needs to configure this directory with specific security requirements, it can be configured per framework. Make sure that the process running your application has write access to this directory. Also make sure this directory is protected to avoid telemetry being read by unintended users.

Java

The folder C:\Users\username\AppData\Local\Temp is used for persisting data. This location isn't configurable from the config directory, and the permissions to access this folder are restricted to the specific user with required credentials. For more information, see implementation.

.NET

By default, ServerTelemetryChannel uses the current user's local app data folder %localAppData%\Microsoft\ApplicationInsights or temp folder %TMP%. For more information, see implementation.

Via configuration file:

<TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel,   Microsoft.AI.ServerTelemetryChannel">
    <StorageFolder>D:\NewTestFolder</StorageFolder>
</TelemetryChannel>

Via code:

• Remove ServerTelemetryChannel from the configuration file.

• Add this snippet to your configuration:

  ServerTelemetryChannel channel = new ServerTelemetryChannel();
  channel.StorageFolder = @"D:\NewTestFolder";
  channel.Initialize(TelemetryConfiguration.Active);
  TelemetryConfiguration.Active.TelemetryChannel = channel;
  

NetCore

By default, ServerTelemetryChannel uses the current user's local app data folder %localAppData%\Microsoft\ApplicationInsights or temp folder %TMP%. For more information, see implementation.

In a Linux environment, local storage will be disabled unless a storage folder is specified.

The following code snippet shows how to set ServerTelemetryChannel.StorageFolder in the ConfigureServices() method of your Startup.cs class:

services.AddSingleton(typeof(ITelemetryChannel), new ServerTelemetryChannel () {StorageFolder = "/tmp/myfolder"});

For more information, see AspNetCore custom configuration.

Node.js

By default, %TEMP%/appInsights-node{INSTRUMENTATION KEY} is used for persisting data. Permissions to access this folder are restricted to the current user and administrators. For more information, see the implementation.

The folder prefix appInsights-node can be overridden by changing the runtime value of the static variable Sender.TEMPDIR_PREFIX found in Sender.ts.

JavaScript (browser)

HTML5 Session Storage is used to persist data. Two separate buffers are used: AI_buffer and AI_sent_buffer. Telemetry that's batched and waiting to be sent is stored in AI_buffer. Telemetry that was just sent is placed in AI_sent_buffer until the ingestion server responds that it was successfully received.

When telemetry is successfully received, it's removed from all buffers. On transient failures (for example, a user loses network connectivity), telemetry remains in AI_buffer until it's successfully received or the ingestion server responds that the telemetry is invalid (bad schema or too old, for example).

Telemetry buffers can be disabled by setting enableSessionStorageBuffer to false. When session storage is turned off, a local array is instead used as persistent storage. Because the JavaScript SDK runs on a client device, the user has access to this storage location via their browser's developer tools.

OpenCensus Python

By default, OpenCensus Python SDK uses the current user folder %username%/.opencensus/.azure/. Permissions to access this folder are restricted to the current user and administrators. For more information, see the implementation. The folder with your persisted data will be named after the Python file that generated the telemetry.

You can change the location of your storage file by passing in the storage_path parameter in the constructor of the exporter you're using.

AzureLogHandler(
  connection_string='InstrumentationKey=00000000-0000-0000-0000-000000000000',
  storage_path='<your-path-here>',
)

How do I send data to Application Insights using TLS 1.2?

To ensure the security of data in transit to the Application Insights endpoints, we strongly encourage customers to configure their application to use at least Transport Layer Security (TLS) 1.2. Older versions of TLS/Secure Sockets Layer (SSL) have been found to be vulnerable. Although they still currently work to allow backward compatibility, they aren't recommended. The industry is quickly moving to abandon support for these older protocols.

The PCI Security Standards Council has set a deadline of June 30, 2018 to disable older versions of TLS/SSL and upgrade to more secure protocols. After Azure drops legacy support, if your application or clients can't communicate over at least TLS 1.2, you wouldn't be able to send data to Application Insights. The approach you take to test and validate your application's TLS support will vary depending on the operating system or platform and the language or framework your application uses.

We do not recommend explicitly setting your application to only use TLS 1.2, unless necessary. This setting can break platform-level security features that allow you to automatically detect and take advantage of newer more secure protocols as they become available, such as TLS 1.3. We recommend that you perform a thorough audit of your application's code to check for hardcoding of specific TLS/SSL versions.

Platform/Language-specific guidance

Check what version of OpenSSL your Linux distribution is running

To check what version of OpenSSL you have installed, open the terminal and run:

openssl version -a

Run a test TLS 1.2 transaction on Linux

To run a preliminary test to see if your Linux system can communicate over TLS 1.2, open the terminal and run:

openssl s_client -connect bing.com:443 -tls1_2

Personal data stored in Application Insights

For an in-depth discussion on this issue, see Managing personal data in Log Analytics and Application Insights.

Can my users turn off Application Insights?

Not directly. We don't provide a switch that your users can operate to turn off Application Insights.

You can implement such a feature in your application. All the SDKs include an API setting that turns off telemetry collection.

Data sent by Application Insights

The SDKs vary between platforms, and there are several components that you can install. For more information, see Application Insights overview. Each component sends different data.

Classes of data sent in different scenarios

For SDKs for other platforms, see their documents.

The classes of collected data

You can switch off some of the data by editing ApplicationInsights.config.

Can I modify or update data after it has been collected?

No. Data is read-only and can only be deleted via the purge functionality. To learn more, see Guidance for personal data stored in Log Analytics and Application Insights.

Frequently asked questions

This section provides answers to common questions.

What happens to Application Insight telemetry when a server or device loses connection with Azure?

All of our SDKs, including the web SDK, include reliable transport or robust transport. When the server or device loses connection with Azure, telemetry is stored locally on the file system (Server SDKs) or in HTML5 Session Storage (Web SDK). The SDK periodically retries to send this telemetry until our ingestion service considers it "stale" (48 hours for logs, 30 minutes for metrics). Stale telemetry is dropped. In some cases, such as when local storage is full, retry won't occur.

Is personal data sent in the telemetry?

You can send personal data if your code sends such data. It can also happen if variables in stack traces include personal data. Your development team should conduct risk assessments to ensure that personal data is properly handled. Learn more about data retention and privacy.

All octets of the client web address are always set to 0 after the geolocation attributes are looked up.

The Application Insights JavaScript SDK doesn't include any personal data in its autocompletion, by default. However, some personal data used in your application might be picked up by the SDK (for example, full names in window.title or account IDs in XHR URL query parameters). For custom personal data masking, add a telemetry initializer.