Mimecast Data Collection Scripts for LogRhythm Administrators Guide

Document created by user.zL0FB6L9lN Employee on Mar 15, 2018Last modified by user.oxriBaJeN4 on Jul 4, 2019
Version 13Show Document
  • View in full screen mode

The Mimecast Data Collection Scripts for LogRhythm allow a LogRhythm administrator to download email and audit events from Mimecast ready for collection by the LogRhythm platform. Data is collected and parsed using LogRhythm Flat File collectors and MPE rules composed by LogRhythm.

 

System Requirements and Prerequisites

 

LogRhythm Server

 

  • Python v2.7.x installed on the server used to collect data from the Mimecast API
  • At least v7.1.433.0 of the LogRhythm KB - this release contains the Mimecast data parsing rules

 

Network

 

Data collection uses the Mimecast API. You must ensure that the server hosting the data collection scripts has outbound HTTPS access (TCP port 443) to the following hosts depending on the region where your Mimecast account is hosted:

RegionHost(s)
EUapi.mimecast.com AND eu-api.mimecast.com
DEapi.mimecast.com AND de-api.mimecast.com
USapi.mimecast.com AND us-api.mimecast.com
ZAapi.mimecast.com AND za-api.mimecast.com
AUapi.mimecast.com AND au-api.mimecast.com
Offshoreapi.mimecast.com AND je-api.mimecast.com

 

Mimecast Permissions

 

See the table below for the endpoints used by the data collection scripts and the associated Mimecast administrator permissions required. For convenience all permissions are included in the Basic Administrator role.

Endpoint
Permission Required
/api/login/discover-authenticationn/a
/api/login/loginn/a
/api/audit/get-audit-eventsLogs | Read
/api/audit/get-siem-logsTracking | Read

 

Configuring Mimecast

The data collection scripts require an Mimecast Administrator Authentication token. By default an authentication token expires after three days. This means that your scripts will stop collecting data from Mimecast after that time. For the best experience you must create a new user and Authentication Profile defining a longer lived Authentication Token. The steps below describe this process.

The preparation required in the Mimecast Administration Console involves:

  1. Creating a user.
  2. Adding the user to an administrator role.
  3. Creating a group and adding the user to it.
  4. Creating an authentication profile.
  5. Creating an appliction setting.
  6. Enable logging.

 

Creating a User

 

To create a user:

  1. Click on the Administration toolbar menu item.
  2. Select the Directories | Internal Directories menu item.
  3. Select the Internal Domain where you would like to create the user.
  4. Click on the New Address button.
  5. Complete the New Address form. See the Managing User Email Addresses page for further details.
    Keep a note of the password set, as you will use this when setting up the scripts.
  6. Click on the Save and Exit button.

 

Adding the User to an Administrator Role

 

To add the user to an administrator role:

  1. Click on the Administration toolbar menu item.
  2. Select on theAccount | Roles menu item.
  3. Right click on the Administrator Role (e.g. Basic Administrator).
  4. Select the Add Users to Role menu item.
  5. Browse for the User created in the "Creating a User" section.
  6. Select the Tick Box to the left of the user.
  7. Click on the Add Selected Users button.

 

Creating a Group and Adding the User

 

To create a group and add the user to it:

  1. Click on the Administration toolbar menu item.
  2. Select the Directories | Profile Groups menu item.
  3. Create a Group. See the Managing Groups page for further details.
    Give the group a descriptive name (e.g. LogRhythm Admin).
  4. With the group selected, Click on the Build button.
  5. Click on the Add Email Addresses button.
  6. Type the name of the User created in the "Creating a User" section.
  7. Click on the Save and Exit button..

 

Creating an Authentication Profile

2-Step authentication must be disabled for this authentication profile. If you have issues, contact our Support team.

To create an authentication profile:

  1. Click on the Administration toolbar menu item.
  2. Select the Services | Applications menu item.
  3. Click on the Authentication Profiles button.
  4. Click on the New Authentication Profile button. See the Configuring an Authentication Profile page for further details.
  5. Type a Description for the profile.
  6. Set the Authentication TTL setting to "Never Expires". This makes sure that when you create your authentication token, it will not expire and impact the data collection of the app.
  7. Leave all other settings as their default.
  8. Click on the Save and Exit button.

 

Creating an Application Setting

 

To create an application setting:

  1. Click on the Administration toolbar menu item.
  2. Select the Services | Applications menu item.
  3. Click on the New Application Settings button. See the Configuring Application Settings page for further details.
  4. Type a Description.
  5. Select the Group you created in the "Creating a Group and Adding the User" task.
  6. Select the Authentication Profile created in the "Creating an Authentication Profile" task.
  7. Leave all other settings as their default.
  8. Click on the Save and Exit button.

 

Enabling Logging

 

To enable logging on your account:

  1. Click on the Administration toolbar menu item.
  2. Select the Account | Account Settings menu item.
  3. Expand the Enhanced Logging section.
  4. Select the types of logs you want to enable. The choices are:
    • Inbound: Logs for messages from external senders to internal recipients.
    • Outbound: Logs for messages from internal senders to external recipients.
    • Internal: Logs for messages between internal domains.
  5. Click on the Save button.

 

Once these settings have been saved, the Mimecast MTA starts logging data. Log files should become available for download up to 30 minutes after that.

 

Configuring Data Collection Scripts

 

api_setup.py

 

To create an authentication token, follow the steps in the Managing API Applications page. Specify an application name of “Mimecast for LogRhythm” to obtain an Application ID, Application Key.

 

  1. Download the Mimecast Data Collection Scripts for LogRhythm from here.
  2. Extract the .ZIP File to the location on your LogRhythm server where you want the scripts to execute.
  3. Open a Command Prompt.
  4. Change the Directory to where Python is installed:
    cd C:\Python27
  5. Start the setup utilitypython.exe “PATH TO DATA COLLECTION SCRIPTS\api_setup.py”
  6. Enter your Data Directory. This is the full path to the directory where you would like to store the data download from Mimecast, and where LogRhythm collects data from.
  7. Enter the Application Id and Application Key.
  8. Enter the Base URL for your region from the table below:
  9. Enter the Email Address and Password of the user created in the "Creating a User" step above.
  10. You should see a message indicating the successful completion of the setup.
    Log in successful
    Getting account code and saving config...
    Config saved successfully.

 

Creating a Scheduled Task to Execute the Data Collectors

 

To create a scheduled task:

  1. Open the Windows Task Scheduler on the server hosting the data collection scripts.
  2. Click on the Actions | Create Task menu item.
  3. Complete the General tab with the following information:
    Field / OptionSetting
    NameProvide a name that describes the task (e.g. Mimecast MTA Log Collection).
    Run Whether User is Logged On or NotSelect this option.
  4. Complete the Triggers tab:
    1. Click on the New button.
    2. Configure the schedule settings to meet your requirements.
      We recommend running the data collection scripts daily, every 30 minutes for an indefinite time period.
    3. Click on the OK button.
  5. Complete the Actions tab:
    1. Click on the New button.
    2. Complete the dialog as follows:
      Field / OptionSetting
      ActionSelect "Start a Program" from the drop down list
      Program / ScriptSpecify the path to the python executable (e.g. C:\Python27\python.exe).
      Add Arguments (Optional)Specify the path to the Mimecast script (e.g. C:\Mimecast Data Collector for LogRhythm\Mimecast Data Collector for LogRhythm\siem_log_collector.py).
    3. Click on the OK button.
  6. Optionally configure any Conditions or Settings you want to apply.
  7. Click on the OK button to save the task.
  8. Repeat this process for the audit_log_collector.py script.

 

Once complete, the scripts will execute as scheduled. The scripts can also be run by selecting Run Now from the Task Scheduler. Your data is downloaded to the data directory specified in the setup utility.

The data collection scripts remove files not modified after seven days to save disk space.

Configuring LogRhythm

 

LogRhythm requires a LogRhythm System Monitor Agent to collect the logs. The file being collected must be viewable on the host, with the agent using a standard file name path such as /var/log/logfile.txt or C:\logs\logfile.txt.

 

To configure LogRhythm:

  1. Start the LogRhythm Console.
  2. Select the Deployment Manager button.
  3. Select the System Monitors tab.
  4. Either:
    1. Double-click the System Monitor Agent that will be collecting the Umbrella DNS Logs.
    2. Right-click on the System Monitor Agent and select the Properties menu item.
  5. Select the Agent Settings tab.
  6. Right-click anywhere in the Log Sources List.
  7. Select New from the context menu.
  8. Select the Basic Configuration tab.
  9. Select Flat-File Mimecast Audit from the "Log Message Source Type" box.
  10. Select the Flat File Settings tab.
  11. Populate the boxes on the Flat File Settings tab with the following information from our example:
    • File Path: C:\SIEMLogs\Audit
    • Date Parsing Format: Mimecast Audit Logs*
    • Is Directory must be checked
    • Recursion Depth may be set to 1 or higher
    • Compression Type must be set to none
    • Click OK in the Log Message Source Properties window.
    • Click OK in the System Monitor Agent Properties window.
  12. Repeat this process for the Mimecast Email Log Source.

 

Optionally Install Dashboards

 

Sample dashboards can be downloaded here. These can be uploaded to your LogRhythm server using the LogRhythm web portal.

 

 

Troubleshooting

 

The data collection scripts provided by Mimecast output a log file of activity for troubleshooting purposes. These logs are written to logs directory in the directory where the scripts are executed from, for example: C:\Mimecast Data Collector for LogRhythm\log. Logs are kept for 7 days.

 

These logs should be used to diagnose issues where data is not being populated in the data directory. Mimecast support will require these logs to assist you with any issues.

 

The logs will not provide any insight into LogRhythm system agents. Please consult LogRhythm documentation and / or support for issues relating to the Flat File - System Agents.

Attachments

    Outcomes