Mimecast for Splunk Administrators Guide

Document created by user.zL0FB6L9lN Expert on Jul 25, 2016Last modified by user.zL0FB6L9lN Expert on Jul 13, 2017
Version 27Show Document
  • View in full screen mode

 

Overview

 

About Mimecast for Splunk

 

Author: Mimecast

App Version: 2.0.0

Vendor Products: Mimecast (S1, D1, M2, M2A)

Has index operations: false, need not install on indexers

Create an index: true, impacts storage, the app does not use report acceleration, search acceleration, data model acceleration. The app does use summary indexing to create single line CIM compliant events from multi-line events.

 

Mimecast for Splunk allows a Splunk Enterprise administrator to ingest email and audit events from Mimecast and provides a range of dashboards to showcase the types of visualizations possible with the data. Data is mapped to the Common Information Model where relevant to allow users to correlate Mimecast events with other data sources and in other applications.

 

Scripts and binaries

 

  • Splunk_TA_mimecast_for_splunk_v2.py: a python script used to collect events from the Mimecast API.
  • login.py: a python script that *nix user's interact with to collect the required authentication token.
  • login.ps1: a PowerShell script that Windows user's interact with to collect the required authentication token.
  • Python Requests open source library: version 2.13.0 - used for HTTPS requests to the Mimecast API. 

 

 

Release Notes

 

Version 2.0.0

 

  • Added support and dashboards for new Targeted Threat Protection URL Protect and Attachment Protect data types.
  • Refreshed version 1 dashboards to be more efficient and moved these to the Sample Dashboards menu.
  • Added support for Mimecast log data to appear in Enterprise Security dashboards.
  • Added support for proxy settings in the modular input script.
  • Added support for Advanced Account Administration customers to access log data from all their accounts using a single installation of the app.
  • Changed logging strategy of the modular input script from logging to file to logging to the splunkd log
  • Added a new Troubleshooting dashboard to get easy access and display logs.
  • Simplified app configuration and programatic extraction of the access key and secret key values required to authorize API requests.
  • Added new simplified method to get access key and secret key values required for the Data Input configuration.
  • Added support rate limiting applied at the Mimecast API.
  • Removed requirement on version 1 of the Mimecast API.
  • Improved error handling.

 

Version 1.0.4

 

  • Adds support for secure storage for Mimecast Access and Secret Keys
  • Addresses an issue where check point files were not being closed properly

 

About this release

 

Splunk Enterprise Versions: 6.4.x, 6.5.x, 6.6.x

CIM: 4.3.1 and later

Platforms: Platform independent

Vendor Products: Any Mimecast Gateway Product.

Lookup file changes: n/a

 

Support and Resources

 

Questions and answers

 

Access questions and answers specific to Mimecast for Splunk at https://community.mimecast.com 

 

Support

 

To find out about Mimecast support please visit: Email Solutions Technical Support | Mimecast.

 

Installation and Configuration

 

Version Support

 

Mimecast for Splunk 2.0.0 has been tested on versions 6.5.0, 6.5.1, 6.6.1 of Splunk Enterprise hosted on Windows, Linux and Mac OSX. Other host operating systems and Splunk versions later than 6.4.0 should also work but have not been tested.

 

System Requirements and Prerequisites

 

Software

 

The app does not require any other software other than a default working Splunk installation to operate.

 

Because this add-on runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.

 

Network

 

The app uses the Mimecast API to collect data and events. You must ensure that the server hosting Splunk and the Mimecast application has outbound HTTPS access (TCP port 443) to the following hosts depending on the region where your Mimecast account is hosted:

 

RegionHost(s)
EUapi.mimecast.com AND eu-api.mimecast.com
USapi.mimecast.com AND us-api.mimecast.com
ZAapi.mimecast.com AND za-api.mimecast.com
AUapi.mimecast.com AND au-api.mimecast.com
Offshoreapi.mimecast.com AND je-api.mimecast.com

 

Mimecast Permissions

 

The app uses the Mimecast API to collect data and events. Please see the table below for the endpoints used and the Mimecast administrator permissions required. For convenience all permissions are included in the Basic Administrator role.

 

Endpoint
Permission Required
/api/login/discover-authenticationn/a
/api/email/get-email-queuesDashboard read
/api/directory/get-connectionsDirectory Sync read
/api/journaling/get-serviceJournaling read
/api/audit/get-audit-eventsLogs read
/api/audit/get-siem-logsTracking read

 

Preparation Steps

 

The Mimecast for Splunk app requires the Access Key and Secret Key from a Mimecast Authentication token of a Mimecast administrator to be entered during the configuration of the app. By default an Authentication Token expires after 3 days, this means that your Mimecast for Splunk app would stop being able to collect data and events after 3 days without manual intervention.

 

Consequently, for the best experience you must create a new user and Authentication Profile defining a longer lived Authentication Token. The steps below describe this process:

 

Advanced Account Administration customer's should repeat this process and use a dedicated administrative user for each Mail Processing account.

 

To additionally collect data from the Master account and any Grouping accounts configured in the Advanced Account Administration structure

 

  • Federated Administration must be enabled throughout the structure as described in the Federated Administration guide. 
  • The process below should be repeated for an administrator from the Master account using the Federated Administration domain.

 

Look out for further guidance on Advanced Account Administration later in the guide.

Step 1: Create a new user

 

  1. Login to the Administration Console.
  2. Navigate to the Administration | Directories | Internal Directories menu item to display a list of internal domains.
  3. Select the internal domain where you would like to create your new user.
  4. Select the New Address button from the menu bar.
  5. Complete the new address form and select Save and Exit to create the new user.
  6. Keep a note of the password set as you will use this to get your Authentication Token in Step 6.

 

Step 2: Add the user to an Administrative Role

 

  1. While logged into the Administration Console, navigate to the Administration | Account | Roles menu item to display the Roles page.
  2. Right click the Basic Administrator role and select Add users to role.
  3. Browse or search to find the new user created in the Step 1.
  4. Select the tick box to the left of the user.
  5. Select the Add selected users button to add the user to the role.

 

Step 3: Create a new group and add your new user

 

  1. While logged into the Administration Console, navigate to the Administration | Directories | Profile Groups menu item to display the Profile groups page.
  2. Create a new group by selecting the plus icon on the parent folder where you would like to create the group. This creates a new group with the Name "New Folder"
  3. To rename the group, select the newly created "New Folder" group. Then from the Edit group text box type the name you want to give the folder, for example Splunk Admin and press the Enter key to apply the change.
  4. With the group selected select the Build drop down button and select Add Email Addresses.
  5. Type the name of the new user created in Step 1.
  6. Select Save and Exit to add the new user to the group.

 

Step 4: Create a new Authentication Profile

 

  1. While logged into the Administration Console, navigate to the Administration | Services | Applications menu item to display the Application Settings page.
  2. Select the Authentication Profiles button.
  3. Select the New Authentication Profile button.
  4. Type a Description for the new profile.
  5. Set the Authentication TTL setting to Never Expires. This will make sure that when you create your Authentication Token it will not expire and impact the data collection of the app.
  6. Leave all other settings as their default.
  7. Select Save and Exit to create the profile.

 

Step 5: Create a new Application Setting

 

  1. While logged into the Administration Console, navigate to the Administration | Services | Applications menu item to display the Application Settings page.
  2. Select the New Application Settings button.
  3. Type a Description.
  4. Use the Group Lookup button to select the Group that you created in Step 3.
  5. Use the Authentication Profile Lookup button to select the Authentication Profile created in Step 4.
  6. Leave all other settings as their default.
  7. Select Save and Exit to create and apply the Application Settings to your new group and user.

 

Step 6: Get your authentication token

 

Now that you have a dedicated user who will receive a an Authentication Token that will never expire, the final preparation task is to get the Authentication Token for the user.

 

Getting an Authentication token using Mac OSX or *nix

 

  1. Open a terminal application and navigate to the %SPLUNK_HOME%\etc\apps\Splunk_TA_mimecast_for_splunk_v2\bin directory.
  2. Execute the login.py script to be guided through the steps required to request the required access key and secret key.

    python login.py
  3. The script will output the access key and secret key, keep the values to hand as they are required when setting up the Splunk Data Input. 

 

Getting an Authentication token using Windows

 

  1. Open a Powershell window and navigate to the %SPLUNK_HOME%\etc\apps\Splunk_TA_mimecast_for_splunk_v2\bin directory.
  2. Execute the login.ps1 script to the guided through the steps required to request the required access key and secret key.
  3. The script will output the access key and secret key, keep the values to hand as they are required when setting up the Splunk Data Input.

When using Copy / Paste from the Powershell window be sure to remove any line breaks in the access key and secret key values to prevent later issues.

This process has been tested in Powershell version 4 and 5.

 

Step 7: Enable logging for your account

 

  1. While logged into the Administration Console, navigate to the Administration | Account | Account Settings menu item to display the Account Settings page.
  2. Select the Enhanced Logging section.
  3. Select the types of logs you want to enable. The choices are:
    • Inbound - logs for messages from external senders to internal recipients
    • Outbound - logs for messages from internal senders to external recipients
    • Internal - logs for messages between internal domains
  4. Select Save to apply the changes.

 

Once these settings have been saved the Mimecast MTA will start logging data for your account and logs should start to become available for download up to 30 minutes after that.

 

Download

 

The application is available for download from SplunkBase.

 

Installation Steps

 

To install and configure this app on your supported platform, follow these steps:

 

  1. Login to the Splunk web interface.
  2. Select the cogs icon at the top of the Apps bar to open the Apps list page.
  3. Select the Browse more apps button.
  4. Search for Mimecast.
  5. Follow the steps to install in the app.

 

Upgrading from version 1.0.4

 

  1. Disable the version 1 Data Input
    1. While logged in to the Splunk web interface.
    2. Select Data inputs from the Settings menu.
    3. From the data inputs list, select Mimecast for Splunk.
    4. Select Disable from the list of Data Inputs 
  2. Download version 2 from SplunkBase.
  3. Install version 2.
    1. While logged in to the Splunk web interface.
    2. Select the cogs icon at the top of the Apps bar to open the Apps list page.
    3. Select Install app from file.
    4. Upload the Splunk_TA_mimecast_for_splunk_v2.tar.gz file.
    5. The app is now installed side by side with version 1.

Do not remove version 1 until version 2 is successfully running in your environment as the application will automatically attempt to migrate cached state tokens to prevent getting duplicate data in your environment.

User Guide

 

Key Concepts for Mimecast for Splunk

 

The app connects to Mimecast to collect events, to use the app you should be familiar with configuring data inputs and searching Splunk Enterprise.

 

Configuration

 

To start collecting data from Mimecast you need to configure a Splunk Data Input. Please follow these steps:

 

  1. While logged in to the Splunk web interface open the MImecast for Splunk v2 app.
  2. Select the Complete your configuration link from the home page to navigate to the Data Inputs screen.
  3. Select the New button.
  4. Add a Name for the Input.
  5. Add the Email Address for the administrator created in the Preparation > Step 1: Create a new user section.
  6. Add the Access Key and Secret Key values created for the administrator Email Address in the Preparation > Step 6: Get your authentication token
  7. Optionally add details of your proxy server.
  8. Optionally add the Mimecast Account Code that data should be collected for. This value can be found on the landing page of the Mimecast Administration Console.

    Advanced Account Administration Customers

    - It is highly recommended that you include the Account Code for all Mail Processing accounts. This will make it easier to identify issues in the Troubleshooting logs.
    - If you are using a Federated Administrator it is required that you use enter the Account Code for Data Inputs that should connect to Master and / or Grouping accounts so that data is collected from the correct account. Failure to do so will result in the data input only collecting data from the Master account.
  9. Select the More Settings checkbox.
  10. Set an interval in seconds that you want the input to check Mimecast for new data. Recommended value: 600.
  11. Select Next to complete the configuration.

 

Using the app

 

Once you have configured a data input, Mimecast data should start being ingested into your Splunk environment. The app is comprised of a number of dashboards showcasing the data available. Please see the table below for a description of each dashboard and the expected update frequencies.

 

Data Types

 

DashboardDescriptionUpdate frequencyFirst runData retention
Mimecast Email ActivityDisplays messages received over time by route and spark lines for Rejections, Bounces, and Held Messages.Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Targeted Threat Protection > URL ProtectDisplays visualizations for data logged when a user clicks on a potentially malicious link in an email.Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Targeted Threat Protection > Attachment ProtectDisplays visualizations for data logged when a potentially malicious attachment is identified by the Mimecast sandbox. Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Targeted Threat Protection > Impersonation Protect

Displays visualizations for Targeted Threat Protection Impersonation protect message characteristics detected in a message.

These visualizations do not necessarily indicate that a policy action was triggered, just that one of the characteristics was detected.

Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Audit and Access > Audit LogDisplays a feed of administrator and authentication activity.Real time.Data from 10 minutes before you configured the Splunk Data Input.The lifetime of your Mimecast account.
Audit and Access > Access AttemptsDisplays visualizations for access attempts made to the Mimecast account.Real time.Data from 10 minutes before you configured the Splunk Data Input.The lifetime of your Mimecast account.
Sample Dashboards > Email DeliveryDisplays visualizations for messages delivered by the Mimecast MTA.Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > Email ReceiptDisplays visualizations for messages received by the Mimecast MTA.Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > TLSDisplays visualizations detailing the secure delivery and receipt of messages processed by the Mimecast MTA.Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > AV / ASDisplays visualizations for messages detected as Spam or carrying a virus.Up to every 30 minutes.Data from when you enabled Enhanced Logging on your account.7 days
Sample Dashboards > Service HealthA dashboard displaying inbound and outbound message queue totals, as well as the status of Directory and Journal integrations if configured,Real time.Data from 1 hour before you configured the Splunk Data Input.n/a
TroubleshootingMimecast specific events from the splunkd.log file.Real time.n/an/a

 

Troubleshooting

 

  • The Modular Input script writes to the splunkd.log file and a dashboard is provided in the app that filters this log for Mimecast specific events to help identify issues.

 

Common issues

 

Dashboards are not updating:

 

  1. Check that you have a valid Splunk Data Input configured.
  2. Check that the date filters are set to a time period where you are expecting to see data.
  3. Check the log file to confirm events are being added to Splunk.

 

Request returned with status code 418 ()

 

If you see Request returned with status code 418 () in the app logs, this means that the access key and secret key used in your Data Input has expired. Please ensure that you have followed the Preparation steps to create a user with an Authentication Token that does not expire.

 

Example use cases

 

Mimecast for Splunk can be used for many reasons. For example,

 

  • To ingest Secure Email Gateway events into Splunk to feed the Splunk Enterprise Security or User Behaviour Analytics products.
  • To ingest Mimecast events to create custom reports on email traffic, security, usage.
10 people found this helpful

Attachments

    Outcomes