Finding Illicit Activity The Old Fashioned Way

Finding bad guys doing bad things in your cloud services is a hard thing to do under even the best circumstances. There are a ton of idiosyncrasies at play, including the capabilities of the cloud applications you are using and the very unique nature of your users and the data you are storing in the cloud. Even the best automated detection systems will struggle to catch all illicit activity most of the time. The good news is that humans are great at detecting anomalies! If you are a security conscious person, Office 365 offers a wide variety of security related reports and data that you can review to manually find illicit activity. The below reports are grouped together to highlight the types of threat or breach categories the data can help mitigate.

Prerequisites

To complete the manual security reviews below, you will need to meet a minimum set of pre-requisites:

1. Be an Office 365 Customer. The data is only available for an active customer, so if you aren't already, go ahead and sign up!
2. Acquire appropriate permissions (Global Admin). Currently, the only permission set that allows for the full range of data review listed below is global administrator. We're working to make this possible with a more limited permission role in the future. If you aren't already a global admin, ask another global administrator to grant you global admin role permissions at https://portal.office.com. We recommend you perform global administrative activities using an alternate admin account that is separate from your daily use account that you do email from (for example), and that when you no longer require the permissions you remove your alt account from the role.
3. Lastly, you will need to enable Audit Recording as per the following instructions: https://support.office.com/en-us/article/Search-the-audit-log-in-the-Office-365-Security-Compliance-Center-0d4d0f35-390b-4518-800e-0c7ec95e946c?ui=en-US&rs=en-US&ad=US. Only takes a second, and will dramatically improve your security posture!

Where Do these Reports Live?

In general, the below security reports come in several forms and locations:

1. The Office 365 Security and Compliance Center. This portal contains the main ad hoc activity search interface. This interface is under Search & Investigation --> Audit Log Search

2. The Azure Management Portal. This portal contains the Azure Active Directory reports about accounts. For all of these reports, you'll need to select the domain you want to review reports for, then click the reports tab, then select the report you wish to review:

3. The Office 365 Exchange Administration Console. This portal contains all of the Exchange related reports.

4. Various SharePoint Interfaces. The sharing review recommendations below are data-focused, so you'll have to review the external sharing lists from the sites you want to make sure you have protected.

5. Varied Powershell Scripts. Office 365 has written a few PowerShell scripts to help you quickly produce data in a reviewable format (Excel, in this case).

Finding Breached Accounts

Breached accounts are the most common, and often most damaging breaches an Office 365 customer can experience. It is usually the first step an attacker will take in a larger effort to steal your money or your data. The breached account security reviews are important because most breaches require some sort of privileged access (in the form of an owned account!).

• Review Sign-Ins After Multiple Failures
  • Where is it? Azure Management Portal --> Directory --> Reports --> Sign Ins After Multiple Failures
  • What's in the report? This report shows user accounts that have successfully signed in after multiple consecutive failed sign in attempts. While there are legitimate reasons for this behavior, it also could indicate that an attacker successfully performed a brute force password cracking attack. The report itself includes the name of the user, the number of failed attempts before the successful sign in, and the time of the first successful sign in.
  • What do I look for? Any records in this report warrant at least a check-in with the user to ensure it was them. If it was not them, you should undertake a robust account breach remediation and investigation to determine where the account came from, and whether the breached account is associated with any other illicit activity.
  • How often should I review? You should review this report at least weekly.

• Review Sign-Ins from Unknown Sources
  • Where is it? Azure Management Portal --> Directory --> Reports --> Sign Ins From Unknown Sources
  • What's in the report? This report shows users who have successfully signed in while assigned a client IP address that has been recognized by Microsoft as an anonymous proxy IP address (such as for the TOR network). These proxies are often used by users that want to hide their computer's IP address, and may be used for illicit activity. The report shows the user, the number of successful sign ins, the IP address, and the time of the last successful logon.
  • What do I look for? Any records in this report warrant a check-in with the user to verify that the logon was legitimate.
  • How often should I review? You should review this report at least weekly.

• Review Sign-Ins from Multiple Geographies
  • Where is it? Azure Management Portal --> Directory --> Reports --> Sign Ins From Multiple Geographies
  • What's in the report? This report includes successful sign ins from a user where two sign ins appeared to originate from different regions and the time between sign ins makes it impossible for the user to have traveled between those regions. Essentially, the user appears to be in two places at once. There are several potential causes, including sharing passwords (which is bad), using VPN's or remote desktop, or using two devices and one device has an unusual IP address. The report includes the user, the first sign in location, the second sign in location, the time between sign ins, the estimated travel time, and the time of the 2nd sign in.
  • What do I look for? This report can contain a very high false positive rate, especially if you have a geographically disperse workforce. When you are looking for illicit activity, watch for sign ins (either first or second) from geographies where it doesn't make sense for your workforce to be signing in from. For example, if you team operates primarily out of Europe, and you see logons from Russia, but no one on your team is traveling there, and you have no vendor relationships with anyone from there, it might be illicit. Another thing to look for is sign ins at times when your users would not normally be signing in (such as a 3am). You should contact the user to determine if there is a reasonable explanation.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for User Administration Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> User administration Activities
  • What's in the report? This report will show you all of the user administration activities including new users, updated users, password requirement changes, and user deletions. The summary record shows you the date the event happened, the IP address (if available), the impacted user, and the activity performed. The detailed list will show you the actor who made the change, and all of the other details to help correlate the activity. Depending on how many users you have, this list could get very large, in which case you should export and analyze offline in a tool like Excel.
  • What do I look for? Newly created users, or out of band password changes can be signs of an attacker either taking over an account, or creating a new account from which to operate. Look for events from unusual IP addresses, or at unusual times of the day. Also look for odd usernames. If you find something unusual, you should first try to validate whether the activity is legitimate by contacting the user that made the change.
  • How often should I review? You should review this report at least weekly.

• Review Malware Detections Report
  • Where is it? https://portal.office.com/AdminPortal/Home#/reportsSecurityAndCompliance --> Malware Detections
  • What's in the report? This report shows the number of malware detections sent or received in mail before the malware action was applied (usually a delete action). The report includes a date and the number of received and sent malware instances. Details about individual malware-filtered messages are available by selecting a point in the graph.
  • What do I look for? You should think about this report as an indicator of the volume of malware that is being attempted to be transacted to and from your users. The report only shows instances where malware was correctly identified and handled, so there isn't any specific action to take. The sent volume is probably more important to flag because it means one of your users attempted to send it outbound. Think of this report as a way to understand how targeted your organization is by attackers with malware. High volumes of inbound malware means your users are more likely to get a piece of malware that slipped through protections, and you should have robust anti-virus and anti-malware protections installed on your client machines. Big spikes in volumes means that attackers may be targeting you particularly.
  • How often should I review? You should review this report at least weekly.

• Review Exchange Mailbox Activity
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Exchange Mailbox Activities
  • What's in the report? This report shows activity that could indicate a mailbox is being accessed illicitly. It includes created or received messages, moved or deleted messages, copied or purged messages, sent messages using send on behalf or send as, and all mailbox sign ins. The data include date, IP address, user, activity performed, the item affected, and any extended details. Note that you must enable mailbox auditing to have the data be recorded.
  • What do I look for? The volume of data included here could be very substantial, so focus your search on users that would be high impact if breached. Look for unusual patterns such as odd times of the day, or unusual IP addresses, and look for patterns such as high volumes of moves, purges, or deletes.
  • How often should I review? You should review this report at least every other week.

Finding Elevation of Privilege

After breaching an account, the second step in an attackers portfolio is almost always to find a way to elevate their privileges and entrench their access. Regular reviews of privileged access can tip you off to an attacker trying to get elevated privileges before they can get effectively entrenched. They key here is understanding who has privileged access in your environments in the first place, and that usually means limiting that access as much as possible in the first place.

• Review List of Global Admins
  • Where is it? https://portal.office.com/AdminPortal/Home#/users --> Filters --> Global Admins
  • What's in the report? This report shows you a list of all the users in your tenancy that currently hold global administrator privileges. Selecting any given user will bring up a pane showing more details about the user.
  • What do I look for? Look for new or unusual names. Your global administrator list should be managed very carefully with privileges only granted with a very clear need. If you see someone new, reach out to them to determine if their need is legitimate. You can also look at other roles they hold, whether they are forwarding mail, and whether they have MFA enabled (which you should be requiring!).
  • How often should I review? You should review this report at least weekly.

• Review List of Non-Global Admins
  • Where is it? https://portal.office.com/AdminPortal/Home#/users --> Filters --> User Management Admins, Billing Admins, Password Admins, Service Admins
  • What's in the report? This report shows you a list of all the users in your tenancy that hold the filtered role. Selecting a user will bring up a pane showing more details about the user.
  • What do I look for? Look for new or unusual names. While these roles are less powerful than a global admin, they do grant special privileges that can be used illicitly. If you see something unusual contact the user to confirm it is a legitimate need.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Role Administration Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Role Administration Activities
  • What's in the report? This report includes activities to add members to roles (including Global Administrator), removing users from roles, and updating company contact information.
  • What do I look for? Look for new or unusual names, unusual IP addresses, unusual times of day for operations, or any kind of promotion to global administration (which should be relatively rare). If you see global admin role assignment, contact the user to verify it is a legitimate need.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Group Administration Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Group Administration Activities
  • What's in the report? This report includes new groups, updated groups, deleted groups, plus membership changes.
  • What do I look for? Most of the activity in this report will be legitimate, and groups often have no privileges associated with them, but if an attacker is attempting to access sensitive information with regular user permissions, getting added to the right groups is an easy way to do that. Look for new or unusual names, unusual IP addresses, unusual times of day for operations.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Application Administration Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Application Administration Activities
  • What's in the report? This report includes activity around managing directory service principals, adding delegation permissions to your directory, or managing service principal credentials.
  • What do I look for? An attacker can leverage an application service principal to maintain access to sensitive information by adding and then granting a service principal privileges to desired access levels. You should maintain a clear idea of which service principals should have access to your data, and note any new or unusual additions. Look for new or unusual names, unusual IP addresses, unusual times of day for operations. Changes are attributed to specific users, so if you see something unusual verify with the user whether it is legitimate.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Directory Administration Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Directory Administration Activities
  • What's in the report? This report includes activity around managing your directory configuration, including managing partners permissions, domains, federation settings, and password policy requirements.
  • What do I look for? An attacker can gain a lot of power by changing directory configuration. Make sure you understand what your desired configuration settings should be and look for any changes to that desired state. Look for new or unusual names, unusual IP addresses, unusual times of day for operations.
  • How often should I review? You should review this report at least weekly.

• Review Account Provisioning Activity
  • Where is it? Azure Management Portal --> Directory --> Reports --> Account Provisioning Activity
  • What's in the report? This report includes a history of all attempts to provision accounts to external applications (such as third party identity management applications). It will include the date, the external application, the account, and the action taken (such as active, create, delete, etc.).
  • What do I look for? If you don't usually use a third party provider to manage accounts, any entry on the list is likely illicit. But, if you do, this is a great way to monitor transaction volumes, and look for new or unusual third party applications that are managing users. If you see something unusual, contact the provider to determine if the action is legitimate.
  • How often should I review? You should review this report at least weekly.

Finding Data Exfiltration

The most important asset that Office 365 hosts is your data, in the form of documents, email, conversations, and web content. Most attackers are targeting your data as their primary objective. If you find data exfiltration in your tenancy, you most likely have a broader account breach as well, so a positive finding in the below reports will require further investigation to make sure you discover all the ways you've been compromised and can fully fix it.

• Review Delegate Permissions
  • Where is it? https://github.com/OfficeDev/O365-InvestigationTooling/blob/master/DumpDelegatesandForwardingRules.ps1
  • What's in the report? This report is created from running a small powershell script that will go get a list of all your users, then use the Get-MailboxPermission cmdlet to create a csv of all the mailbox delegates in your tenancy.
  • What do I look for? Look for unusual names or permission grants. If you see something unusual, contact the mailbox owner to see if it is legitimate.
  • How often should I review? You should review this report at least weekly.

• Review Mail Forwarding Rules
  • Where is it? https://github.com/OfficeDev/O365-InvestigationTooling/blob/master/DumpDelegatesandForwardingRules.ps1
  • What's in the report? This report is created from running a small powershell script that will go get a list of all your users, then use the Get-InboxRule cmdlet to create a csv of all the mailbox forwarding rules to external locations in your tenancy.
  • What do I look for? Look for unusual target locations, or any kind of external addressing. Also look for forwarding rules with unusual key words in the criteria such as 'all mail with the word invoice in the subject'. Contact the mailbox owner to see if it is legitimate.
  • How often should I review? You should review this report at least weekly.

• Review Mail Transport Rules
  • Where is it? https://outlook.office365.com/ecp/ --> Rules
  • What's in the report? The summary view of the report shows you a list of all the mail transport rules you have configured for your tenancy. When you select any given rule, you'll see details of the rule in a summary pane to the right which include the qualifying criteria and action taken when the rule condition matches.
  • What do I look for? Look for new rules, or rules that have been modified to redirect mail to external domains. The number of rules should be relatively small such that you can maintain a list of known good rules. If a new rule is created, there should be a new entry in the Audit report for that event. You can search it to determine who created the rule and from where. If you see something unusual, contact the creator to determine if it is legitimate.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Sharing and Access Request Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Sharing and access request activities
  • What's in the report? This report contains activity records for managing sharing of documents, files, and folders, managing invitations, and managing access requests.
  • What do I look for? Look for new or unusual names, unusual IP addresses, unusual times of day for operations, or any kind of mass sharing events. If you see anything unusual, contact the user to verify it is legitimate activity.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Synchronization Activity
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Synchronization Activity
  • What's in the report? This report contains activity records for document synchronization in OneDrive for Business and SharePoint Online. The report will tell you if a computer was allowed to sync and download files, upload files or file changes, or was blocked from doing any of the above.
  • What do I look for? Look for new or unusual names, unusual IP addresses, unusual times of day for operations, or downloads from multiple users to a single computer. Also, if you see a repeated blocked computer from syncing you likely have either an issue with a user configuration, or a failed attempt to exfiltrate data. If you see anything unusual, contact the user to verify it is legitimate activity.
  • How often should I review? You should review this report at least weekly.

• Search Audit Log for Site Administration Activities
  • Where is it? https://protection.office.com/#/unifiedauditlog --> Activities Filter Drop Down --> Site Administration Activities
  • What's in the report? This report contains a huge set of activities related to SharePoint Online site administration. You can see group management, role management, particularly for site admins, permissions management, site collection management, and the management of specific features such as RSS feeds.
  • What do I look for? Look for new or unusual names, unusual IP addresses, unusual times of day for operations, and especially activity that grants users site administration permissions, which allow a very short path to easy data exfiltration. You can also look for patterns like the creation of new sites, followed by large data uploads or transfers from one site to another, and then mass sharing with external contacts for the newly created site. If you see anything unusual, contact the user to verify it is legitimate activity.
  • How often should I review? You should review this report at least weekly.

• For sensitive documents or folders in SharePoint, review Shared With details
  • Where is it? Site Collection Home Page --> Configuration (in upper right) --> Shared With
  • What's in the report? This report will show you all of the people that the site has been shared with.
  • What do I look for? Look for users that are external to the organization, or should not have access to the site. If the data on the site is sufficiently sensitive, the list of users with permissions should be small. Contact the site owner to determine if the access is legitimate.
  • How often should I review? You should review this report at least monthly for high value data.

• Review External Sharing Contacts for SharePoint Sites
  • Where is it? Select Document --> Share --> Shared With
  • What's in the report? This report will show you all of the people that the document has been shared with.
  • What do I look for? Look for users that are external to the organization, or should not have access to the document. If the data on the site is sufficiently sensitive, the list of users with permissions should be small. Contact the document owner to determine if the access is legitimate.
  • How often should I review? You should review this report at least monthly for high value data.

So, there you have it. We've laid out an old fashioned and time consuming approach, but the data is there to find many potential breach pathways. A better approach would be to identify the most likely breach scenarios for your tenancy and then build an automated detection solution on top of the appropriate signals. Third party applications that can integrate your on-prem, and multi-cloud applications and infrastructure are a good place to start! Meanwhile, you can review the above data on a regular schedule and maximize your chances of finding an attacker before it is too late.