Mimecast for Splunk Administrators Guide

Document created by user.zL0FB6L9lN Expert on Jul 25, 2016Last modified by user.Yo2IBgvWqr on Jun 14, 2018
Version 34Show Document
  • View in full screen mode

Mimecast for Splunk Overview

 

Author: Mimecast

App Version: 2.0.1

Vendor Products: Mimecast (S1, D1, M2, M2A)

Has index operations: false, need not install on indexers

Create an index: true, impacts storage, the app does not use report acceleration, search acceleration, data model acceleration. The app does use summary indexing to create single line CIM compliant events from multi-line events.

 

Mimecast for Splunk allows a Splunk Enterprise administrator to ingest email and audit events from Mimecast and provides a range of dashboards to showcase the types of visualizations possible with the data. Data is mapped to the Common Information Model, where relevant, to allow users to correlate Mimecast events with other data sources and in other applications.

 

Scripts and Binaries

 

  • Splunk_TA_mimecast_for_splunk_v2.py: a python script used to collect events from the Mimecast API.
  • login.py: a python script that *nix user's interact with to collect the required authentication token.
  • login.ps1: a PowerShell script that Windows user's interact with to collect the required authentication token.
  • Python Requests open source library: version 2.13.0 - used for HTTPS requests to the Mimecast API. 

 

Release Notes

 

Version 2.0.1

 

  • Added support and dashboards for Targeted Threat Protection URL Protect and Attachment Protect data types.
  • Refreshed version 1 dashboards to be more efficient, and moved these to the Sample Dashboards menu.
  • Added support for proxy settings in the modular input script.
  • Added support for Advanced Account Administration customers to access log data from all their accounts, using a single installation of the app.
  • Changed logging strategy of the modular input script from logging to file, to logging to the splunkd log.
  • Added a Troubleshooting dashboard to get easy access and display logs.
  • Simplified the app configuration and programmatic extraction of the access key and secret key values required to authorize API requests.
  • Added support for rate limiting applied by the Mimecast API.
  • Removed requirement on version 1 of the Mimecast API.
  • Improved error handling.

 

Version 1.0.4

 

  • Added support for secure storage for Mimecast Access and Secret Keys.
  • Addresses an issue where checkpoint files weren't being closed properly.

 

About this Release

 

Splunk Enterprise Versions: 6.4.x, 6.5.x, 6.6.x

Platforms: Platform independent

Vendor Products: Any Mimecast Gateway Product.

Lookup file changes: n/a

 

Support and Resources

 

 

Installation and Configuration

 

Version Support

 

Mimecast for Splunk 2.0.1 has been tested on versions 6.5.0, 6.5.1, 6.6.1 of Splunk Enterprise hosted on Windows, Linux, and Mac OSX. Other host operating systems and Splunk versions later than 6.4.0 should also work, but haven't been tested.

 

System Requirements and Prerequisites

 

Software

 

  • The app does not require any other software other than a default working Splunk installation to operate.

 

Network

 

The app uses the Mimecast API to collect data and events. You must ensure that the server hosting Splunk and the Mimecast application has outbound HTTPS access (TCP port 443) to the following hosts depending on the region where your Mimecast account is hosted:

 

RegionHost(s)
EUapi.mimecast.com AND eu-api.mimecast.com
DEapi.mimecast.com AND de-api.mimecast.com
USapi.mimecast.com AND us-api.mimecast.com
ZAapi.mimecast.com AND za-api.mimecast.com
AUapi.mimecast.com AND au-api.mimecast.com
Offshoreapi.mimecast.com AND je-api.mimecast.com

 

Mimecast Permissions

 

The app uses the Mimecast API to collect data and events. See the table below for the endpoints used, and the Mimecast administrator permissions required. For convenience, all permissions are included in the Basic Administrator role.

 

Endpoint
Permission Required
/api/login/discover-authenticationn/a
/api/email/get-email-queuesDashboard read
/api/directory/get-connectionsDirectory Sync read
/api/journaling/get-serviceJournaling read
/api/audit/get-audit-eventsLogs read
/api/audit/get-siem-logsTracking read

The Admin IP Ranges specified in the Administration Console are respected when attempting to retrieve SIEM logs. Admin IP Ranges are entered in the Administration | Account | Account Settings | User Access and Permissions section of the console.

Preparation Steps

 

The Mimecast for Splunk app requires a Mimecast Administrator authentication token's Access Key and Secret Key to be entered during the configuration of the app. By default, an authentication token expires after three days. This means that your Mimecast for Splunk app will stop being able to collect data and events after three days without manual intervention. To prevent this, create a new user and Authentication Profile defining a longer-lived authentication token as described in the process below.

Advanced Account Administration customers must repeat this process, and use a dedicated administrative user for each Mail Processing account. To additionally collect data from the Master account and any Grouping accounts configured in the Advanced Account Administration structure:

 

  • Federated Administration must be enabled throughout the structure as described in the Federated Administration guide. 
  • The process below should be repeated for an administrator from the Master account using the Federated Administration domain.

 

Look out for further guidance on Advanced Account Administration later in the guide.

Step 1: Creating a User

 

  1. Log on to the Administration Console.
  2. Select the Administration toolbar menu item.
  3. Select the Directories | Internal Directories menu item to display a list of internal domains.
  4. Select the Internal Domain where you would like to create your new user.
  5. Select the New Address button from the menu bar.
  6. Complete the new address form.
  7. Select Save and Exit to create the user.
    Keep a note of the password set, as you will use this to get your Authentication Token in Step 6.

Step 2: Adding the User to an Administrative Role

 

  1. Select the Administration toolbar menu item.
  2. Select the Account | Roles menu item to display the roles.
  3. Right click the Basic Administrator role.
  4. Select Add Users to Role menu item.
  5. Browse or search to find the User created in Step 1.
  6. Select the Tick Box to the left of the user.
  7. Select the Add Selected Users button to add the user to the role.

 

Step 3: Creating a Group and Add Your User

 

  1. Select the Administration toolbar menu item.
  2. Select the Directories | Profile Groups menu item to display the profile groups page.
  3. Select the Plus Icon in the parent folder where you would like to create the group. This creates a new group called "New Folder"
  4. To rename the group:
    1. Select the "New Folder" group.
    2. From the Edit group text box, type the name you want to give the folder (e.g. Splunk Admin).
    3. Press the Enter key.
  5. With the group selected, click on the Build drop down button.
  6. Select Add Email Addresses.
  7. Type the name of the User created in Step 1.
  8. Select Save and Exit to add the new user to the group.

 

Step 4: Creating an Authentication Profile

 

  1. Select the Administration toolbar menu item.
  2. Select the Services | Applications menu item to display the Application Settings page.
  3. Select the Authentication Profiles button.
  4. Select the New Authentication Profile button.
  5. Type a Description for the new profile.
  6. Set the Authentication TTL setting to Never Expires. This ensures that when you create your Authentication Token, it won't expire and impact the data collection of the app.
  7. Leave all other settings as their default.
  8. Select Save and Exit to create the profile.

 

Step 5: Creating an Application Setting

 

  1. Select the Administration toolbar menu item.
  2. Select the Services | Applications menu item to display the Application Settings page.
  3. Select the New Application Settings button.
  4. Type a Description.
  5. Select the Lookup button next to the Group field to select the Group you created in Step 3.
  6. Select the Lookup button next to the Authentication Profile field to select the Authentication Profile you created in Step 4.
  7. Leave all other settings as their default.
  8. Select Save and Exit to create and apply the Application Settings to your new group and user.

 

Step 6: Getting Your Authentication Token

 

Now that you have a dedicated user who'll receive an Authentication Token that won't expire, the final task is to get the Authentication Token for the user.

 

Mac OSX or *nix

 

  1. Open a terminal application.
  2. Navigate to the %SPLUNK_HOME%\etc\apps\Splunk_TA_mimecast_for_splunk_v2\bin directory.
  3. Execute the login.py script to be guided through the steps required to request the required access key and secret key.
    python login.py
  4. The script outputs the access key and secret key. Keep the values to hand, as they are required when setting up the Splunk Data Input. 

Windows

 

  1. Start a Powershell window. See the Starting Windows PowerShell page of the Microsoft documentation for full details.
  2. Navigate to the %SPLUNK_HOME%\etc\apps\Splunk_TA_mimecast_for_splunk_v2\bin directory. See the Navigating the File System page of the Microsoft documentation for full details.
  3. Execute the login.ps1 script to be guided through the steps required to request the required access key and secret key. See the How to Write and Run Scripts in the Windows PowerShell page of the Microsoft documentation for full details.
  4. The script outputs the access key and secret key, keep the values to hand as they are required when setting up the Splunk Data Input.

When copying / pasting from the Powershell window, remove any line breaks in the access key and secret key values to prevent later issues.

This process has been tested in Powershell version 4 and 5.

Step 7: Enabling logging for your account

 

  1. Select the Administration toolbar menu item.
  2. Select the Account | Account Settings menu item to display the Account Settings page.
  3. Select the Enhanced Logging section.
  4. Select the types of logs you want to enable. The choices are:
    • Inbound: logs for messages from external senders to internal recipients.
    • Outbound: logs for messages from internal senders to external recipients.
    • Internal: logs for messages between internal domains.
  5. Select Save to apply the changes.

 

Once these settings are saved, the Mimecast MTA starts logging data for your account and logs will become available for download up to 30 minutes after that.

 

Download

 

The application is available for download from SplunkBase.

 

Installation Steps

 

To install and configure this app on your supported platform, follow these steps:

  1. Log on to the Splunk web interface.
  2. Select the Cogs Icon at the top of the Apps bar to open the Apps list page.
  3. Select the Browse More Apps button.
  4. Search for Mimecast.
  5. Follow the steps to install the app.

 

Upgrading from version 1.0.4

 

  1. Disable the version 1 Data Input:
    1. Log on to the Splunk web interface.
    2. Select Data inputs from the Settings menu.
    3. From the data inputs list, select Mimecast for Splunk.
    4. Select Disable from the list of Data Inputs 
  2. Download version 2 from SplunkBase.
  3. Install version 2:
    1. Log on to the Splunk web interface.
    2. Select the Cogs Icon at the top of the Apps bar to open the Apps list page.
    3. Select Install App From the File.
    4. Upload the Splunk_TA_mimecast_for_splunk_v2.tar.gz file.
    5. The app is now installed side by side with version 1.

Do not remove version 1 until version 2 is successfully running in your environment, as the application will automatically attempt to migrate cached state tokens to prevent getting duplicate data in your environment.

Mimecast for Splunk User Guide

 

The app connects to Mimecast to collect events. To use the app, you must be familiar with configuring data inputs and searching Splunk Enterprise.

 

Configuration

 

To start collecting data from Mimecast you must configure a Splunk Data Input:

  1. Open the Mimecast for Splunk v2 app from the Splunk interface.
  2. Select the Complete your configuration link from the home page to navigate to the Data Inputs screen.
  3. Select the New button.
  4. Add a Name for the Input.
  5. Add the Email Address for the administrator created in the "Step 1: Creating a User" section.
  6. Add the Access Key and Secret Key values created for the administrator Email Address in the "Step 6: Getting Your Authentication Token" section.
  7. Optionally add details of your Proxy Server.
  8. Add the Mimecast Account Code that data should be collected for. This value can be found on the landing page of the Mimecast Administration Console.
  9. Select the More Settings checkbox.
  10. Set an interval in seconds that you want the input to check Mimecast for new data. Recommended value: 600.
  11. Select Next to complete the configuration.
Advanced Account Administration Customers

You can use this process to configure a data input for each of your mail processing accounts, using an administrator from the mail processing account itself. However, to collect events from master and / or grouping accounts, contact our support team who'll be able to guide you through the process of enabling federated administration in your Mimecast hierarchy, and the process for configuring a data input for these account types.

Using the App

 

Once you've configured a data input, Mimecast data should start being ingested into your Splunk environment. The app is comprised of a number of dashboards showcasing the data available. See the table below for a description of each dashboard and the expected update frequencies.

 

Data Types

 

DashboardDescriptionUpdate frequencyFirst runData retention
Mimecast Email ActivityDisplays messages received over time by route and spark lines for Rejections, Bounces, and Held Messages.Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Targeted Threat Protection > URL ProtectDisplays visualizations for data logged when a user clicks on a potentially malicious link in an email.Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Targeted Threat Protection > Attachment ProtectDisplays visualizations for data logged when a potentially malicious attachment is identified by the Mimecast sandbox. Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Targeted Threat Protection > Impersonation Protect

Displays visualizations for Targeted Threat Protection Impersonation protect message characteristics detected in a message.

These visualizations do not necessarily indicate that a policy action was triggered, just that one of the characteristics was detected.

Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Audit and Access > Audit LogDisplays a feed of administrator and authentication activity.Real timeData from 10 minutes before you configured the Splunk Data Input.The lifetime of your Mimecast account.
Audit and Access > Access AttemptsDisplays visualizations for access attempts made to the Mimecast account.Real timeData from 10 minutes before you configured the Splunk Data Input.The lifetime of your Mimecast account.
Sample Dashboards > Email DeliveryDisplays visualizations for messages delivered by the Mimecast MTA.Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > Email ReceiptDisplays visualizations for messages received by the Mimecast MTA.Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > TLSDisplays visualizations detailing the secure delivery and receipt of messages processed by the Mimecast MTA.Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > AV / ASDisplays visualizations for messages detected as Spam or carrying a virus.Up to every 30 minutesData from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > Service HealthA dashboard displaying inbound and outbound message queue totals, as well as the status of Directory and Journal integrations if configured,Real timeData from 1 hour before you configured the Splunk Data Input.n/a
TroubleshootingMimecast specific events from the splunkd.log file.Real timen/an/a

The current app design does not include the subject field in Journal logs.

Troubleshooting

 

The Modular Input script writes to the splunkd.log file, and a dashboard is provided in the app that filters this log for Mimecast specific events to help identify issues.

 

Dashboards aren't Updating

 

  1. Check you have a valid Splunk Data Input configured.
  2. Check the date filters are set to a time period where you're expecting to see data.
  3. Check the log file to confirm events are being added to Splunk.

 

Request Returned with Status Code 418 ()

 

If you see "Request returned with status code 418 ()" in the app logs, the access key and secret key used in your Data Input has expired. Ensure that you've followed the preparation steps to create a user with an Authentication Token that doesn't expire.

 

Example Use Cases

 

Mimecast for Splunk can be used for many reasons. For example:

  • To ingest Secure Email Gateway events into Splunk to feed the Splunk Enterprise Security or User Behaviour Analytics products.
  • To ingest Mimecast events to create custom reports on email traffic, security, and usage.
9 people found this helpful

Attachments

    Outcomes