Mimecast Data Collection Scripts for LogRhythm Administrators Guide

Document created by user.zL0FB6L9lN Expert on Mar 15, 2018Last modified by user.oxriBaJeN4 on May 28, 2019
Version 7Show Document
  • View in full screen mode


The Mimecast Data Collection Scripts for LogRhythm allow a LogRhythm administrator to download email and audit events from Mimecast ready for collection by the LogRhythm platform. Data is collected and parsed using LogRhythm Flat File collectors and MPE rules composed by LogRhythm.


System Requirements and Prerequisites


LogRhythm Server


  • Python v2.7.x installed on the server used to collect data from the Mimecast API
  • At least v7.1.433.0 of the LogRhythm KB - this release contains the Mimecast data parsing rules




Data collection uses the Mimecast API. You must ensure that the server hosting the data collection scripts has outbound HTTPS access (TCP port 443) to the following hosts depending on the region where your Mimecast account is hosted:


EUapi.mimecast.com AND eu-api.mimecast.com
DEapi.mimecast.com AND de-api.mimecast.com
USapi.mimecast.com AND us-api.mimecast.com
ZAapi.mimecast.com AND za-api.mimecast.com
AUapi.mimecast.com AND au-api.mimecast.com
Offshoreapi.mimecast.com AND je-api.mimecast.com


Mimecast Permissions


Please see the table below for the endpoints used by the data collection scripts and the associated Mimecast administrator permissions required. For convenience all permissions are included in the Basic Administrator role.


Permission Required
/api/audit/get-audit-eventsLogs | Read
/api/audit/get-siem-logsTracking | Read


Preparation Steps

IMPORTANT: The data collection scripts require an Mimecast Administrator Authentication token.


By default an Authentication Tokens expire after 3 days, this means that your scripts will stop collecting data from Mimecast after 3 days.


For the best experience you must create a new user and Authentication Profile defining a longer lived Authentication Token. The steps below describe this process:


Step 1: Create a new user


  1. Login to the Administration Console.
  2. Navigate to the Administration | Directories | Internal Directories menu item to display a list of internal domains.
  3. Select the internal domain where you would like to create your new user.
  4. Select the New Address button from the menu bar.
  5. Complete the new address form and select Save and Exit to create the new user.
  6. Keep a note of the password set as you will use this when setting up the scripts.


Step 2: Add the user to an Administrative Role


  1. While logged into the Administration Console, navigate to the Administration | Account | Roles menu item to display the Roles page.
  2. Right click the Basic Administrator role and select Add users to role.
  3. Browse or search to find the new user created in the Step 1.
  4. Select the tick box to the left of the user.
  5. Select the Add selected users button to add the user to the role.


Step 3: Create a new group and add your new user


  1. While logged into the Administration Console, navigate to the Administration | Directories | Profile Groups menu item to display the Profile groups page.
  2. Create a new group by selecting the plus icon on the parent folder where you would like to create the group. This creates a new group with the Name "New Folder"
  3. To rename the group, select the newly created "New Folder" group. Then from the Edit group text box type the name you want to give the folder, for example Splunk Admin and press the Enter key to apply the change.
  4. With the group selected select the Build drop down button and select Add Email Addresses.
  5. Type the name of the new user created in Step 1.
  6. Select Save and Exit to add the new user to the group.


Step 4: Create a new Authentication Profile


  1. While logged into the Administration Console, navigate to the Administration | Services | Applications menu item to display the Application Settings page.
  2. Select the Authentication Profiles button.
  3. Select the New Authentication Profile button.
  4. Type a Description for the new profile.
  5. Set the Authentication TTL setting to Never Expires. This will make sure that when you create your Authentication Token it will not expire and impact the data collection of the app.
  6. Leave all other settings as their default.
  7. Select Save and Exit to create the profile.


Step 5: Create a new Application Setting


  1. While logged into the Administration Console, navigate to the Administration | Services | Applications menu item to display the Application Settings page.
  2. Select the New Application Settings button.
  3. Type a Description.
  4. Use the Group Lookup button to select the Group that you created in Step 3.
  5. Use the Authentication Profile Lookup button to select the Authentication Profile created in Step 4.
  6. Leave all other settings as their default.
  7. Select Save and Exit to create and apply the Application Settings to your new group and user.


Step 6: Enable logging for your account


  1. While logged into the Administration Console, navigate to the Administration | Account | Account Settings menu item to display the Account Settings page.
  2. Select the Enhanced Logging section.
  3. Select the types of logs you want to enable. The choices are:
    • Inbound - logs for messages from external senders to internal recipients
    • Outbound - logs for messages from internal senders to external recipients
    • Internal - logs for messages between internal domains
  4. Select Save to apply the changes.


Once these settings have been saved the Mimecast MTA will start logging data for your account and logs should start to become available for download up to 30 minutes after that.


You are now ready to set up the data collection scrips.


Set up Data Collection Scripts


Step 1: api_setup.py


  1. Download the Mimecast Data Collection Scripts for LogRhythm from here.
  2. Extract the zip file to the location on your LogRhythm server that you want the scripts to execute from
  3. Open a command prompt and change directory to the directory that Python is installed
    cd C:\Python27
  4. Launch the setup utilitypython.exe “PATH TO DATA COLLECTION SCRIPTS\api_setup.py”
  5. When prompted enter your data directory. This is the full path to the directory where you would like to store the data download from Mimecast and where LogRhythm will collect the data from.
  6. Enter the email address of the administrator account created earlier in Preparation Steps | Step 1: Set up your Mimecast administrator account
  7. Enter the user’s password
  8. You should see a message indicating the successful completion of the setup.
    Log in successful
    Getting account code and saving config...
    Config saved successfully.


Step 2: Create a Scheduled Task to execute the data collectors


  1. On the server hosting the data collection scripts open the Windows Task Scheduler
  2. Select to create a new Scheduled Task
  3. On the General tab
    1. provide a name, for example, Mimecast MTA Log Collection
    2. select the option to “Run whether user is logged on or not”
  4. On the Triggers tab
    1. Click New
    2. Set the schedule settings as per your requirements, we recommend running the data collection scripts daily, every 30 minutes for an indefinite time period
    3. Click OK
  5. On the Actions tab
    1. Click New
    2. Leave the Action as “Start a program”
    3. In the Program / Script textbook, enter the path to the python executable, for example, C:\Python27\python.exe
    4. In the Add arguments / optional text box, enter the path to the Mimecast script, for example, C:\Mimecast Data Collector for LogRhythm\Mimecast Data Collector for LogRhythm\siem_log_collector.py
    5. Click OK.
  6. Optionally configure any Conditions or Settings you want to apply.
  7. Click OK to save the task.
  8. Repeat this process for the audit_log_collector.py script.


Once complete the scripts should execute as scheduled or by selecting run now from the Task Scheduler and data should be downloaded to the data directory specified in the setup utility.

NOTE: the data collection scripts will remove files not modified after 7 days to save disk space.

Step 3: Configure LogRhythm

LogRhythm requires a LogRhythm System Monitor Agent be used to collect the logs. The file being collected must be viewable on the host with the agent using a standard file name path such as /var/log/logfile.txt or C:\logs\logfile.txt.
  1. Start the LogRhythm Console.
  2. Select the Deployment Manager button from the main toolbar.
  3. Select the System Monitors tab.
  4. Double-click the System Monitor Agent that will be collecting the Umbrella DNS Logs, or right-click it and select Properties from the context menu.
  5. Select the Agent Settings tab.
  6. Right-click anywhere in the Log Sources List and select New from the context menu.
  7. Select the Basic Configuration tab.
  8. Select Flat-File Mimecast Audit from the Log Message Source Type box.
  9. Select the Flat File Settings tab.

  10. Populate the boxes on the Flat File Settings tab with the following information from our example:
    1. File Path: C:\SIEMLogs\Audit
    2. Date Parsing Format: Mimecast Audit Logs*
    3. Is Directory must be checked
    4. Recursion Depth may be set to 1 or higher
    5. Compression Type must be set to none
    6. Click OK in the Log Message Source Properties window.
    7. Click OK in the System Monitor Agent Properties window.
    8. Repeat this process for the Mimecast Email Log Source.
Data should now be collected by LogRhythm.

Optional - Install Dashboards

Sample dashboards can be downloaded here.
These can be uploaded to your LogRhythm server using the LogRhythm web portal.



The data collection scripts provided by Mimecast output a log file of activity for troubleshooting purposes. These logs are written to logs directory in the directory where the scripts are executed from, for example: C:\Mimecast Data Collector for LogRhythm\log. Logs are kept for 7 days. 


These logs should be used to diagnose issues where data is not being populated in the data directory. Mimecast support will require these logs to assist you with any issues.


The logs will not provide any insight into LogRhythm system agents. Please consult LogRhythm documentation and / or support for issues relating to the Flat File - System Agents.