Carbon Black EDR Splunk App User Guide

The VMware Carbon Black EDR App for Splunk lets administrators leverage the industry’s leading EDR solution to detect and take action on endpoint activity directly from within Splunk.

If you are an administrator looking to perform a clean install, follow the sections to verify requirements, select the appropriate deployment configuration, and install.

If you are upgrading from an existing VMware EDR (or CB Response) app version, once you have verified the Requirements you should review the migration information and then come back to the Installation instructions.

If you want to know what you can do with the new app, jump directly to Getting Started.

Requirements

  • Splunk Enterprise, Splunk Cloud
  • Platform Version: 9.1, 9.0, 8.2
  • CIM Version: 5.X
  • Because this App runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.
  • Carbon Black EDR server, version 7.2 or above.
    • The app works with Carbon Black EDR clusters.
    • The Carbon Black EDR Unified View (Federated) server is not currently supported.

Installation

The previous version, 2.2.0 was deprecated on June 30, 2021 and archived on SplunkBase.

Deployment Guide

Warning: Installing the VMware Carbon Black Technology Add-on (TA) on the same node as the App is an unsupported configuration that may result in instability or errors.

Depending on your Splunk configuration and version, the VMware Carbon Black EDR App and Technology Add-on (TA) need to be installed on specific Splunk instances. See the following sections as to where each component is installed.

Installation

  1. Deploy the App and/or Add-on per the Deployment Guide above.

  2. Create the index(s) for your data (if required)

    a. One index for the Carbon Black EDR data.

    b. One index for the results of the Alert Actions. This can be the same index as used for the EDR data.

    For instructions on creating an Index see the Splunk documentation

  3. In the EDR App, navigate to the Administration -> Application Configuration menu, VMware CB EDR Base Configuration tab.

    • The options configured on this page will update settings in local/eventtypes.conf
    • VMware CB EDR Base Index: specify the index for EDR data created at Step 2a above. This is required on the searching tier.
    • VMware CB EDR Action Index: specify the index for Alert Actions created at Step 2b. This is required on the searching tier and should be a single index.

  4. [Optional] Configure a proxy if needed from the Proxies tab

  5. The Administration -> Application Health Overview menu shows errors during processing and can be very useful during troubleshooting.

  6. The Administration -> Application Usage menu provides insights into which Splunk users are using the app

Configuration

Authentication and Authorization

After the Carbon Black EDR app for Splunk is installed, you must configure it to connect to your Carbon Black EDR server by using the Carbon Black EDR REST API. For more information on the Carbon Black EDR REST API and how to generate an API key, see the Carbon Black Developer Network.

The Carbon Black EDR app for Splunk uses a Carbon Black EDR API key to do the following:

  1. Enable the sensorsearch, processsearch, and binarysearch custom commands by performing searches via the Carbon Black EDR API.

  2. Enable the Endpoint Isolation and Endpoint Un-Isolation Adaptive Response Actions by requesting endpoint isolation or un-isolation through the Carbon Black EDR API.

  3. Enable the Ban Hash Adaptive Response Action by using the Carbon Black EDR API to add an MD5 hash to the list of banned hashes.

  4. Enable the Kill Process Adaptive Response Action by using the Carbon Black EDR Live Response API to kill a process on a remote endpoint. Live Response must be enabled on the Carbon Black EDR server for this action to function; see the VMware Carbon Black EDR User Guide for more information about Live Response.

To configure the Carbon Black EDR app for Splunk to connect to your Carbon Black EDR server:

  1. Retrieve an API key for a Global Administrator user on the Carbon Black EDR server. See the authentication instructions at the Carbon Black Developer Network.

  2. In the Splunk Carbon Black EDR App, navigate to the Administration -> Application Configuration menu, API Token Configuration tab.

    a. Enter the Organization Name - this is used for Alert Action configuration

    b. Enter the URL for your Carbon Black EDR server instance in the URL field. For example, enter: https://cbserver.mycompany.com.

    c. Paste the API token into the API Secret Key field.

    d. [Optional] Select a proxy if required

  3. Click Setup to save the new configuration.

The application supports multiple API Configurations to enable data from multiple Carbon Black EDR organizations to be ingested.

Note: SSL validation is enabled by default.

For an on-prem installation of Splunk, SSL Validation can be disabled by putting the following entry in api-configs.conf in the VMware Carbon Black EDR local directory.

For Splunk Cloud, SSL Validation is required.

[ssl_info]
ssl_verify=false

Alert Actions

All available alert actions will be displayed on the Alert Actions tab on the Administration -> Application Configuration menu. Configuration details and considerations for alert actions are described in the Adaptive Response Alert Actions section.

Custom Commands

You can use custom commands in your Splunk pipeline to access Splunk’s visualization and searching capability on Carbon Black EDR data, without ingesting all of the raw endpoint data into Splunk.

The following custom commands are included in default/commands.conf.

  • edrsensorsearch: Search for sensors by IP address or hostname.
    • Example: edrsensorsearch query="ip:172.22.5.141"
  • edrprocesssearch: Search for processes in your Carbon Black EDR server.
    • Example: edrprocesssearch query="process_name:cmd.exe"
  • edrbinarysearch: Search for binaries in your Carbon Black EDR server.
    • Example: edrbinarysearch query="md5:fd3cee0bbc4e55838e65911ff19ef6f5"
Note: Do not modify any configurations in '/default'. Doing so will cause your changes to be overwritten when the app is upgraded. If required or directed to by support, create the appropriate configuration files in '/local' and include the stanza attributes that are being changed.

Changing Credentials

The Carbon Black EDR app for Splunk uses Splunk’s encrypted credential storage facility to securely store the API token for your Carbon Black EDR server.

To change the API key or Carbon Black EDR server URL after the Splunk app has been set up, use the configuration page using the Administration -> Application Configuration menu, API Token Configuration tab as set up initially in the Authentication and Authorization section.

Configuring Event Forwarder & S3 Inputs

The AWS add-on for Splunk is required for configuring S3 inputs. The add-on can be downloaded from Splunkbase. This add-on will be used to configure inputs for this Splunk app. Before configuring any inputs, you should create separate queues and S3 buckets for alert and endpoint events. A Carbon Black Event Forwarder must also be configured in order to forward data to the S3 buckets and to efficiently take in data.

Event Forwarder Configuration

An event forwarder must be created before any input can be received. This forwarder will be responsible for routing data to an S3 bucket where it can then be taken as input by Splunk. Configure your forwarder with filters to limit the amount of event data forwarded to Splunk in order to reduce costs. The forwarder installation guide for VMware CB EDR is located at Developer Network. It is recommended to follow the guide there, and then proceed with the installation and configuration of Splunk.

More details can be found at the CB Event Forwarder on Github.

Configure input in AWS Add-On

Before configuring the AWS inputs, make sure that the AWS add-on is properly installed in your Splunk environment. Get details for installing the AWS add-on from the Splunk documentation site. This documentation provides helpful information regarding the app and configuration settings.

The recommended approach to ingest Carbon Black EDR Event Forwarder data into Splunk is the SQS-based S3 data input.

Configuring an input in the AWS add-on to pull Carbon Black Cloud data using SQS-based S3:

  1. Configure your AWS account on the Configuration page in the AWS Add-on
  2. In the AWS Add-on, create a new input on the Inputs page by selecting Create New input -> Custom Data Type -> SQS-based S3

    1. Specify a name that should be used for this input

    2. Select the AWS account you configured with the AWS Add-on

    3. [Optional] If you configured IAM roles when configuring your AWS account select the created role

    4. Select the AWS Region where you configured the SQS queue and S3 bucket

    5. Select the SQS queue from the dropdown

      Note: If you don't see your SQS queue ensure you have selected the correct AWS region and the SQS queue was created in the region.

    6. Set the batch size for Splunk to pull from your SQS queue Default, should be left at: 10 messages

    7. Ensure the S3 File Decoder is set to Custom Logs

    8. Set the Source Type to vmware:cb:edr:json

    9. Ensure the index you select matches the base index configured in the VMware Carbon Black EDR base configurations.

    10. In Advanced Settings, you can increase the polling cycle for fetching messages from the SQS Default: 300 seconds

Note: If you need to reload older events and are using SQS to pull buckets, the events will not be available in the queue once they are retrieved. To view historical events or reload data, use the generic S3 option or copy the events to another prefix to copy it to the queue.

Dashboards

Builtin dashboards provide you with a quick health check on your Carbon Black EDR server, the status of your Carbon Black EDR deployment, and an overview of detected threats on your network. The following example dashboards are distributed with this app; not all of these are populated with data, depending on what events are being forwarded to Splunk via the Carbon Black Event Forwarder.

  • Binary Status: Displays information about attempts to execute banned processes, and new executables and shared libraries that are discovered by Carbon Black EDR.

  • Endpoint Status: Displays information about the total number of reported sensors, OS, and Carbon Black EDR agent version distribution across all endpoints.

  • Network Overview: Visualizes incoming and outgoing network connections that are recorded by Carbon Black EDR. This view is only populated if netconn events are forwarded via the Carbon Black Event Forwarder.

  • Process Timeline: Produces a simple timeline of events based on a Carbon Black EDR process GUID.

  • System Check: Provides a quick overview, including the number of sensors reporting alerts and the top feed and watchlist hits across the enterprise.

  • Application Health Overview: This dashboard is under the Administration menu option. Use this page to get health and status information about any alerts, events, or API errors in the Carbon Black EDR. View total_failures, messages, and severity level for each instance.

  • There are three dashboards included to support VMware Search commands:

    • Binary Search: Searches the Carbon Black EDR binary holdings via the Binary Search custom command.
    • Process Search: Searches the processes that are tracked by Carbon Black EDR via the Process Search custom command.
    • Sensor Search: Searches endpoints that are tracked by Carbon Black EDR via the Sensor Search custom command.

Adaptive Response Alert Actions

The Carbon Black EDR Splunk app includes Adaptive Response Alert Actions that allow you to take action directly from the Splunk console. The following actions can be configured to occur as either a result of automated correlation searches or on an ad-hoc basis through the Splunk Enterprise Security Incident Review page.

  • Kill Process: Kill a given process that is actively running on an endpoint that is running the Carbon Black EDR sensor. Killing processes allow the security analyst to quickly respond to attackers who are using tools that cannot otherwise be banned by hash (for example, reusing a legitimate administrative tool for malicious purposes).

    • Search Configuration
      • Device ID: Processes on this device will be listed.
      • Process: The name of the process that will be killed.

  • Ban MD5 Hash: Ban a given MD5 hash from executing on any host that is running the Carbon Black EDR sensor. The MD5 hash can be specified by a custom hash field. This feature allows incident responders to quickly react to evolving threats by keeping attackers’ tools from executing while the threat is properly remediated and the attacker is expelled from the network.

    • Search Configuration
      • Hash: The hash that will be banned.

  • Isolate Sensor: Isolate an endpoint from the network. The endpoint can be specified by a sensor ID that’s provided in Carbon Black EDR events in Splunk. Isolation is useful when malware is active on an endpoint. It lets you perform investigative tasks (for example, retrieving files or killing processes through Carbon Black Live Response) from your management console while preventing any connections to active command and control or exfiltration of sensitive data.

    • Search Configuration
      • Device ID: The ID of the device to be isolated.

  • Un-isolate Sensor: Remove the specified device from the isolated state, allowing it to communicate normally on the network.

    • Search Configuration
      • Device ID: The ID of the device to be removed from isolation.

Notes for configuring these actions:

  • The global configurations referenced below are configured under Administration -> Application Configuration on the Alert Actions tab.

  • All search results that are being passed to the alert actions are required to have a host field that corresponds to the correct Organization Name as configured in the API Token Configuration tab.

  • If you use multi-tenancy, include the host field with the corresponding value in the Splunk search query. This value is the Organization Name found in the API Token Configuration.

  • By default when a new alert is created in Splunk the parameter action. *.param.api_config = <api_config guid> will be added to the savedsearches.conf file in the VMware Carbon Black EDR local directory.

  • If you need to change credentials used on an alert action in the Application Configuration dashboard then all previously created alerts that were using the old credential needs to be changed. After updating credentials, delete the above parameter from the savedsearches.conf file for the appropriate saved search and restart Splunk.

Saved Searches

Included in this release are a significant collection of saved searches to jump-start Threat Hunting from within the Splunk environment. These are available on the Splunk menu Settings-> Knowledge -> Searches, reports and alerts.

If the searches are not visible:

  • Check the filter for App: it should be VMware Carbon Black EDR; if it is not, then select it from the drop down.
  • Check the filter for Owner: it should be set to All.

If the searches are not running:

  • Searches are disabled by default and should be enabled as needed by selecting the Actions -> Edit -> Enable for required searches.
  • Check the schedule under Actions -> Edit -> Edit Schedule.
  • Verify results are as expected using the Run option for immediate execution.

Workflow Actions

This app includes workflow actions to provide additional context from Carbon Black EDR on events that originated from any product that pushes data to your Splunk server. These context menu items include the following, and a complete list is included in the Using Workflow Actions section of Getting Started:

  • Deep links: Deep links into the Carbon Black EDR server for any event that originated from a Carbon Black EDR sensor. This allows you to access the Process Tree and other Carbon Black EDR data from a single link inside Splunk.
  • Process search by IP, MD5: Search the Carbon Black EDR server for processes that are associated with a given IP address or MD5 hash from any event in Splunk.
  • Sensor info by IP: Search the Carbon Black EDR server for detailed endpoint information that is associated with a given IP address from any event in Splunk.

More information on Workflow Actions is available here.

Getting Started

Using the VMware Carbon Black EDR App for Splunk

After the app is installed, a new icon showing the VMware Carbon Black EDR logo appears on the left-hand side of the Splunk front page. Clicking the logo brings you to the default dashboard of the Carbon Black EDR for the Splunk app. Additional dashboards include an overview of endpoint status, including a breakdown of OS and sensor versions, as well as data on the latest new binaries seen in the environment.

The Process Search, Binary Search, and Sensor Search dashboards in the VMware Searhc menu allow you to perform Carbon Black searches directly from within Splunk. These dashboards use the respective custom commands to perform the search through the REST API without ingesting the data into Splunk. The results are displayed on the same screen. You can also use Carbon Black search features using custom search commands.

Examples:

  • processsearch query=“process_name:cmd.exe”
  • binarysearch query=“md5:fd3cee0bbc4e55838e65911ff19ef6f5”
  • sensorsearch query=”ip:172.22.5.141”

Using Custom Commands

The Splunk app includes three custom commands to perform searches on the Carbon Black datastore from Splunk: binarysearch, processsearch, and sensorsearch. These three commands have corresponding views in the Carbon Black app: Binary Search, Process Search, and Sensor Search.

To use the custom commands in your Splunk searches, first make sure that you’re using the Carbon Black EDR context by invoking the search through the Splunk > Search menu in the Carbon Black EDR app. You can use any of the search commands by appending the Carbon Black EDR query as a “query” parameter. For example:

| sensorsearch query=”ip:172.22.5.141”

sends an API request to Carbon Black EDR to query for all sensors that have reported an IP address of 172.22.5.141. The result of this query can be piped through to other Splunk commands for aggregation, visualization, and correlation.

To update the base EDR index for macros and eventtypes, change [edr_base_index] in eventtypes.conf.

Using Saved Searches

Several example reports and saved searches are included in this app release. You can find a full list of these searches in Settings > Searches, Reports, and Alerts menu item from the Carbon Black EDR app. None of these are run or scheduled to run by default, and some will not return any data unless certain data types (netconns, procstarts, etc.) are forwarded via the Carbon Black Event Forwarder into Splunk.

Using Adaptive Response Alert Actions

The Carbon Black EDR app for Splunk now integrates with Splunk’s Adaptive Response framework and provides four Adaptive Response Alert Actions:

  • Isolate Endpoint
  • Un-Isolate Endpoint
  • Ban MD5 Hash
  • Kill Process

Each of these Actions can be performed either on an ad-hoc basis on a notable event surfaced in Enterprise Security, or on an automated basis as part of a Splunk Correlation Search. In addition, the Isolate Endpoint and Ban MD5 Hash actions can be invoked based on search results from any Splunk search, as long as a field is present that provides a Device ID (for Isolate Endpoint) or an MD5 hash (for Ban Hash). Currently, only events that are surfaced via the Carbon Black Event Forwarder can be used as input for the Kill Process alert action.

Using Workflow Actions

Workflow Actions allow you to pivot into Carbon Black searches from standardized fields. The Carbon Black EDR app for Splunk includes Workflow Actions with context about events in any Splunk view, including Enterprise Security’s Notable Event table.

To perform a workflow action, drilldown into an event and click the Event Actions button. The available workflow actions from this app are displayed. You can pivot directly from a field if a workflow action is available for that field.

The following Workflow Actions are included:

  • Sensor Information by IP: find detailed information about a Carbon Black EDR sensor given an IP address field.
  • Binary Search by MD5 hash: retrieve context around a binary that has a specific MD5 file hash.
  • Search for Processes contacting IP: retrieve a list of processes from Carbon Black EDR that have made a connection to or received a connection from the given IP address.
  • Search for Processes related to MD5 hash: retrieve a list of processes from Carbon Black EDR that have links to the given MD5 hash (a loaded module/DLL, the executable itself, a file write to an executable with the given MD5 hash).
  • Search for Processes contacting Domain: retrieve a list of processes from Carbon Black EDR that have made a connection to or received a connection from the given domain name.
  • Search for Processes related to filename: retrieve a list of processes from Carbon Black EDR that refer to the given filename (written/modified the file, etc.).

In addition, for events that were generated by Carbon Black EDR (forwarded into Splunk via the Carbon Black Event Forwarder), additional Workflow Actions provide deep links into the Carbon Black EDR console directly from the event in Splunk, where applicable. These deep links require the Carbon Black Event Forwarder to be configured to generate these links at event generation time (see the Carbon Black Event Forwarder configuration file for more details).

  • Deep Link to target process’s Process Analysis page
  • Deep Link to parent process’s Process Analysis page
  • Deep Link to child process’s Process Analysis page
  • Deep Link to Binary Analysis page
  • Deep Link to Sensor page

Migrating from v2.2.0 to v3.0.x

This section has details for moving from v2.2.0 of the DA-ESS-cbresponse app to v3.0.x of the vmware_cb_edr_app_for_splunk for different configurations. Please make sure to check Splunk version compatibility prior to migration.

Migration when using HEC inputs

  1. Install v3.0.x of the vmware_cb_edr_app_for_splunk from Splunkbase on the Search Tier of your environment.

  2. Install v3.0.x of the TA-vmware_cb_edr_app_for_splunk from Splunkbase on the Indexing Tier of your environment.

  3. Update/Create the Splunk HEC input to use the new sourcetype vmware:cb:edr:json

Migration when using AWS S3 inputs

  1. Install v3.0.x of the vmware_cb_edr_app_for_splunk from Splunkbase on the Search Tier of your environment.

  2. Install v3.0.x of the TA-vmware_cb_edr_app_for_splunk from Splunkbase on the Indexing Tier of your environment.

  3. Update or create the AWS S3 input to use the new sourcetype vmware:cb:edr:json

Migration when using event_bridge_output.json

  1. Install v3.0.x of the vmware_cb_edr_app_for_splunk from Splunkbase on the Search Tier of your environment.

  2. Install v3.0.x of the TA-vmware_cb_edr_app_for_splunk from Splunkbase on the Indexing Tier of your environment.

  3. Update the sourcetype setting of inputs.conf of the Universal Forwarder to vmware:cb:edr:json

Update Event types

If the old data using the bit9:carbonblack:json sourcetype is to be integrated into the new app then update eventtype vmware_cb_edrto have the older sourcetype using:

eventtype=vmware_cb_edr_base_index sourcetype IN (vmware:cb:edr:json, bit9:carbonblack:json)

This is configured on the Splunk Settings -> Knowledge -> Event types page.

Troubleshooting

Logs and Diagnostics

Internal Splunk Logs

index=_internal source=/opt/splunk/var/log/splunk/vmware_cb_edr_app_for_splunk/*

Indexed Error Logs

index=main sourcetype=vmware:cb:edr:*:error

Monitoring Console Health Checks

VMware Carbon Black EDR includes health checks in the Monitoring Console health check list, defined in default/checklist.conf.

Diagnostics Generation

Please include a support diagnostic file when creating a support ticket. Use the following command to generate the file.

$SPLUNK_HOME/bin/splunk diag --collect=app:vmware_cb_edr_app_for_splunk

Support and Resources

  • Use the CB Developer Network community forum to discuss issues and get answers from other API developers in the CB Developer Network
  • Report bugs and product issues to Broadcom Support
  • View all API and integration offerings on the Developer Network along with reference documentation, video tutorials, and how-to guides.

Last modified on September 1, 2023