Data Forwarder Configuration API

This API will be deactivated on October 31, 2024.

This API has been deprecated. Please use the Data Forwarder API v2 instead.

Migration information is available here.

Note: Event Forwarder has been renamed to Data Forwarder. The functionality has improved the new version.

Introduction

The Carbon Black Cloud Forwarder lets you send data about alerts and events to an AWS S3 bucket where it can be reconfigured to port into other applications in your security stack, such as Splunk. The Data Forwarder is recommended over APIs for obtaining large amounts of data from Carbon Black Cloud in near real time.

You can now configure a forwarder in the Carbon Black Cloud console under Settings > Data Forwarders.

Available Forwarder Types

endpoint.event
alert

Requirements

Carbon Black Cloud Endpoint Standard or Enterprise EDR
API Key with appropriate permissions
Configured S3 bucket in the same region as the forwarder

Quick Setup with Postman & S3 bucket Configuration

Watch the video tutorial or follow the step-by-step guide for enabling the Carbon Black Cloud Data Forwarder using Postman. The guide walks you through the following:

Create a bucket in your AWS Management Console
Configure an AWS S3 Bucket to allow the Data Forwarder to write events
Create New Access Level in the Carbon Black Cloud Console
Create New API Key in the Carbon Black Cloud Console
Configure the API in Postman
Create a New Event or Alert Forwarder
Monitor the Data Flow to the S3 Bucket

Alternative Bucket Setup

If you already have an AWS S3 bucket or are looking for additional ways to configure your bucket for varying levels of access and other use-cases, see here.

Data Format

S3 Data Storage Format

Currently the Carbon Black Cloud Data Forwarder only supports output to AWS S3 buckets provided by customers. The Data Forwarder outputs gzip compressed single-event-per-line JSON format (sometimes known as “JSONL”) to the customer’s S3 bucket with the following folder structure:

org_key={org_key}/year={year}/month={month}/day={day}/hour={hour}/minute={minute}/second={second}/{uuid}.jsonl.gz

Therefore, if a batch of events occurred on August 27, 2019 at 12:36:53UTC for the organization withorg_keyABCD1234 can be located in:

org_key=ABCD1234/year=2019/month=8/day=27/hour=12/minute=36/second=53/ac203140-f0e2-48fe-90cf-05eb7289f628.jsonl.gz

Data Mapping

You can find the Event and Alert schema as well as sample data in the Data Mapping Guide.

Data Filtering

EDR provides tremendous visibility into everything happening across your endpoints. This data may be prohibitively expensive in licensing or scale for some downstream SIEM, SOAR, or data lake solutions. To reduce the load, one or more filters can be applied to the events emitted by a data forwarder configuration.

Data Filtering is available for forwarders of type “endpoint.event”.

If you want to remove or modify an existing filter then edit the configuration with the desired filter change. To remove all filters for a configuration set the filter property to an empty array.

Filter Schema

Field	Definition	Data Type	Values
`attribute`	The field from the endpoint.event to filter.	String	`event_origin`, `type`, `alert_id`, or `sensor_action`
`equals`	Filters the endpoint.events based on the attribute equalling the specified value.	String	See below
`not_equals`	Filters the endpoint.events based on the attribute not equalling the specified value.	String	See below
`match_any_bits`	Filters the endpoint.events based on the attribute equalling one of the specified values in the array.	Array	See below

Supported Values

The following values are supported based on what attribute is provided and the filters that are supported.

Attribute	Values	Filter
`event_origin`	`NGAV`, `EDR`	`equals`, `not_equals`, `match_any_bits`
`type`	`endpoint.event.procstart`, `endpoint.event.procend`, `endpoint.event.apicall`, `endpoint.event.netconn`, `endpoint.event.netconn_proxy`, `endpoint.event.filemod`, `endpoint.event.regmod`, `endpoint.event.moduleload`, `endpoint.event.crossproc`, `endpoint.event.scriptload`, `endpoint.event.fileless_scriptload`	`equals`, `not_equals`, `match_any_bits`
`alert_id`	empty string `""`	`not_equals`
`sensor_action`	`ACTION_ALLOW`, `ACTION_BLOCK`, `ACTION_BREAK`, `ACTION_SUSPEND`, `ACTION_TERMINATE`	`equals`, `not_equals`, `match_any_bits`

Authentication

Use the following information for authentication, and see the Carbon Black Cloud Authentication Guide for full instructions.

Access Level: Before you create your API Key, you need to create a “Custom” Access Level for the “Data Forwarder” category
- To use all functions of the Data Forwarder, allow the following permissions: Create, Read, Update, Delete(or see each call below for individual requirements)
API Key: When you create your API Key, use the Access Level Type of “Custom”, then select the Access Level you created.
Environment: use the URL of your Carbon Black Cloud console (this is the Dashboard URL).
API Route: event_forwarder_config/v1/orgs/{org_key}/configs
- Note: when you insert your org_key, you must also remove the { } brackets.

API Calls

The following API calls can be used for either “endpoint.event” or “alert” forwarder types.

Create Forwarder

Use this call to create a new forwarder. The API will then make calls to check whether the Forwarder can write to the specified S3 Bucket using the configuration supplied. It will attempt to write a test message called healthcheck.json to the specified bucket under a subfolder called healthcheck. If the bucket is misconfigured (i.e. incorrect permissions, principle arn, etc.) or the configuration is incorrect (i.e. bucket prefix doesn’t match path specified in policy), the API will respond with a 400 error and message with information about what was incorrect and how to fix the issue.

If you want to forward both alert type data and endpoint.event type data, you should create a separate forwarder for each. The forwarder should be configured to send the data to its own subfolder in the S3 bucket using the S3 prefix property. The subfolder you configure will be automatically added to the S3 bucket.

RBAC Permissions Required

Permission (.notation name)	Operation(s)
`event-forwarder.settings`	`CREATE`

Request

POST <cbc-hostname>/event_forwarder_config/v1/orgs/{org_key}/configs

Request Body

Required fields: Name, s3_bucket_name, s3_prefix, type

{
  "name": "<string>",
  "s3_bucket_name": "<string>",
  "s3_prefix": "<string>",
  "type": "<string>",
  "enabled": <boolean>,
  "filters": [{
    "attribute": "<string>",
    "equals": "<string>",
    "not_equals": "<string>",
    "match_any_bits": [
      "<string>",
      "<string>"
    ]
  }]
}

Body Schema

Field	Definition	Data Type	Values
`name` REQUIRED	Defined name for the specific event or alert forwarder	String	N/A
`s3_bucket_name` REQUIRED	Configured unique name for s3 bucket	String	N/A
`s3_prefix` REQUIRED	Defined folder structure the forwarder will write events or alerts to	String	N/A
`type` REQUIRED	The datastream type that is to be forwarded.	Sting	`endpoint.event`, `alert`
`enabled`	Boolean flag indicating whether to create the data forwarder as enabled	Boolean	true, false
`filters`	A list of filters to apply to the data being forwarded. Use only one of `equals`, `not_equals`, or `match_any_bits` per filter in the list. Only supported when type equals `endpoint.event`	Array	`[ { "attribute": "<string>", "equals": "<string>", "not_equals": "<string>", "match_any_bits": [ "<string>", "<string>" ] } ]` See Data Filtering for more information

Response

Use the following to troubleshoot errors. For more troubleshooting tips, see below.

Code	Description	Content-Type	Content
200	Successful creation of forwarder	application/json	View example response below
400	Bad request. The JSON body was malformed, or the configuration is invalid (misconfigured bucket or configuration value)	application/json	View example response below
404	Resource not found	N/A	N/A
500	Internal server error	N/A	N/A

Example

Request

POST https://defense-eap01.conferdeploy.net/event_forwarder_config/v1/orgs/ABCD1234/configs

Request Body

{
  "name": "DevRelTest",
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "enabled": false,
  "filters": [
    {
      "attribute": "type",
      "match_any_bits": [
        "endpoint.event.netconn",
        "endpoint.event.procstart"
      ]
    },
    {
      "attribute": "event_origin",
      "equals": "EDR"
    }
  ]
}

Response for valid configuration

{
  "id": "58aa259f-f42c-11e9-bd5d-2eca45f7dc1c",
  "org_key": "ABC123",
  "name": "DevRelTest",
  "enabled": false,
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "filters": [
    {
      "attribute": "type",
      "match_any_bits": [
        "endpoint.event.netconn",
        "endpoint.event.procstart"
      ]
    },
    {
      "attribute": "event_origin",
      "equals": "EDR"
    }
  ],
  "create_time": "2019-10-21T17:58:11Z",
  "update_time": "2019-10-21T17:58:11Z"
}

Response for invalid configuration

If you receive an error indicating your configuration is invalid, either NoSuchBucket or AccessDenied, see the Troubleshooting Errors section below.

Forwarder Healthcheck

This call can be used to run a healthcheck on an Forwarder to determine if the forwarder is able to write to the specified S3 Bucket using the existing configuration. The healthcheck can indicate if the bucket is misconfigured (i.e. incorrect permissions, principle arn, etc.), if the configuration is incorrect (i.e. bucket prefix doesn’t match path specified in policy), or if the forwarder is working as expected. If successful, the healthcheck will attempt to write a test message called healthcheck.json to your S3 bucket under a subfolder called healthcheck and respond with a 200, regardless of whether the API was able to write to the bucket.

RBAC Permissions Required

Permission (.notation name)	Operation(s)
`event-forwarder.settings`	`READ`

Request

GET <cbc-hostname>/event_forwarder_config/v1/orgs/{org_key}/configs/{config_id}/health_check

Response

Use the following to troubleshoot errors. For more troubleshooting tips, see below.

Code	Description	Content-Type	Content
200	Healthcheck completed successfully	application/json	View example response below
404	Resource not found	N/A	N/A
500	Internal server error	N/A	N/A

Example

Request

GET
https://defense-eap01.conferdeploy.net/event_forwarder_config/v1/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/health_check

Successful Response

Response for a successful healthcheck that was able to write to the S3 bucket using the specified configuration:

{
  "id": "57b09141-f4e8-11e9-83de-22656feed3f2",
  "enabled": true,
  "update_time": "2019-10-22T16:59:41Z",
  "status": "Valid Bucket Configuration for Bucket: DevRelTestBucket with Prefix: test",
  "error": ""
}

Successful Response with Error Status

Example response for successful healthcheck that was unable to write to the S3 bucket using the specified configuration:

{
  "id": "57b09141-f4e8-11e9-83de-22656feed3f2",
  "enabled": true,
  "update_time": "2019-10-22T16:59:41Z",
  "status": "Invalid Bucket Configuration for Bucket: IncorrectBucket with Prefix: test",
  "error": "AccessDenied"
}

If you received a NoSuchBucket error or AccessDenied error, see the Troubleshooting Errors section below for suggested fixes.

Delete Forwarder

Use this call to delete a forwarder.

RBAC Permissions Required

Permission (.notation name)	Operation(s)
`event-forwarder.settings`	`DELETE`

Request

DELETE <cbc-hostname>/event_forwarder_config/v1/orgs/{org_key}/configs/{config_id}

Response

Use the following to troubleshoot errors. For more troubleshooting tips, see below.

Code	Description	Content-Type	Content
200	Successful deletion of forwarder	application/json	View example response below
400	Resource not found	N/A	N/A
500	Internal server error	N/A	N/A

Example

Request

DELETE https://defense-eap01.conferdeploy.net/event_forwarder_config/v1/orgs/ABCD1234/configs/58aa259f-f42c-11e9-bd5d-2eca45f7dc1c

Response

{}

Get Configured Forwarders

Get all configured forwarders and their information

RBAC Permissions Required

Permission (.notation name)	Operation(s)
`event-forwarder.settings`	`READ`

Request

GET <cbc-hostname>/event_forwarder_config/v1/orgs/{org_key}/configs

Response

Use the following to troubleshoot errors. For more troubleshooting tips, see below.

Code	Description	Content-Type	Content
200	Successful acquiring of all configured forwarders	application/json	View example response below
400	Resource not found	N/A	N/A
500	Internal server error	N/A	N/A

Example

Request

GET https://defense-eap01.conferdeploy.net/event_forwarder_config/v1/orgs/ABCD1234/configs

Response

[{
  "id": "57b09141-f4e8-11e9-83de-22656feed3f2",
  "org_key": "ABCD1234",
  "name": "DevRelTest",
  "enabled": true,
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "filters": [
    {
      "attribute": "type",
      "match_any_bits": [
        "endpoint.event.netconn",
        "endpoint.event.procstart"
      ]
    },
    {
      "attribute": "event_origin",
      "equals": "EDR"
    }
  ],
  "create_time": "2019-10-22T16:23:55Z",
  "update_time": "2019-10-22T16:59:41Z"
}]

Edit Forwarder

This call is used to edit an existing forwarder. You can edit the Enabling/Disabling functionality or change the s3 bucket name.

When editing an existing forwarder, the API will make additional calls to check whether the Carbon Black Cloud event or alert forwarder can write to the specified S3 Bucket using the configuration supplied. It will attempt to write a test message called healthcheck.json to the specified bucket under a subfolder called healthcheck. If the bucket is misconfigured (i.e. incorrect permissions, principle arn, etc.) or the configuration is incorrect (i.e. bucket prefix doesn’t match path specified in policy or bucket does not exist), the Data Forwarder Configuration API will respond with a 400 and message including information regarding what was incorrect, providing the customer with feedback that enables them to fix issues as needed.

If you want to remove filters that are applied to your configuration then update the configuration with an empty array for the filters property.

RBAC Permissions Required

Permission (.notation name)	Operation(s)
`event-forwarder.settings`	`UPDATE`

Request

PUT <cbc-hostname>/event_forwarder_config/v1/orgs/{org_key}/configs/{config_id}

Request Body

Required fields: Name, s3_bucket_name, s3_prefix, type

{
  "name": "<string>",
  "s3_bucket_name": "<string>",
  "s3_prefix": "<string>",
  "type": "<string>",
  "filters": [{
    "attribute": "<string>",
    "equals": "<string>",
    "not_equals": "<string>",
    "match_any_bits": [
      "<string>",
      "<string>"
    ]
  }]
}

Body Schema

Field	Definition	Data Type	Values
`name` REQUIRED	Defined name for the specific event or alert forwarder	String	N/A
`s3_bucket_name` REQUIRED	Configured unique name for s3 bucket	String	N/A
`s3_prefix` REQUIRED	Defined folder structure the forwarder will write events or alerts to	String	N/A
`type` REQUIRED	The datastream type that is to be forwarded.	Sting	`endpoint.event`, `alert`
`filters`	A list of filters to apply to the data being forwarded. Use only one of `equals`, `not_equals`, or `match_any_bits` per filter in the list. Only supported when type equals `endpoint.event`	Array	`[ { "attribute": "<string>", "equals": "<string>", "not_equals": "<string>", "match_any_bits": [ "<string>", "<string>" ] } ]` See Data Filtering for more information

Response

Use the following to troubleshoot errors. For more troubleshooting tips, see below.

Code	Description	Content-Type	Content
200	Successful editing of forwarder	application/json	View example response below
400	Bad request. The JSON body was malformed, or some part of the JSON body included an invalid value	application/json	View example response below
404	Resource not found	N/A	N/A
500	Internal server error	N/A	N/A

Example

Request

PUT https://defense-eap01.conferdeploy.net/event_forwarder_config/v1/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2

Request Body

{
  "name": "DevRelTest",
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "filters": [
    {
      "attribute": "alert_id",
      "not_equals": ""
    }
  ],
  "enabled": false
}

Response for valid configuration

{
  "id": "57b09141-f4e8-11e9-83de-22656feed3f2",
  "org_key": "ABCD1234",
  "name": "DevRelTest",
  "enabled": false,
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "filters": [
    {
      "attribute": "alert_id",
      "not_equals": ""
    }
  ],
  "create_time": "2019-10-22T16:23:55Z",
  "update_time": "2019-10-22T16:26:01Z"
}

Response for invalid configuration

If you receive a response for invalid configuration, see the Troubleshooting Errors section below for suggested fixes.

Request for removing filters

PUT https://defense-eap01.conferdeploy.net/event_forwarder_config/v1/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2

Request Body

{
  "name": "DevRelTest",
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "filters": [],
  "enabled": true
}

Response for removing filters

{
  "id": "57b09141-f4e8-11e9-83de-22656feed3f2",
  "org_key": "ABCD1234",
  "name": "DevRelTest",
  "enabled": true,
  "s3_bucket_name": "DevRelTestBucket",
  "s3_prefix": "test",
  "type": "endpoint.event",
  "filters": [],
  "create_time": "2019-10-22T16:23:55Z",
  "update_time": "2019-10-22T16:26:01Z"
}

Troubleshooting Errors

Error Code	Reason For Error	Suggested Fix
`NoSuchBucket`	Configuration is invalid. The specified S3 bucket name doesn’t exist	Follow the Quick Start Guide to create an S3 bucket for use by the CBC Data Forwarder
`AccessDenied`	Either: Bucket was created in the wrong region OR The policy is malformed	Ensure that your bucket is created in the same region document for policy troubleshooting and configuration instructions
`400`	The reason varied depending on the API call	Refer to the individual API call you are making to validate the JSON body
`401`	Problem using the specified token to authenticate	Check that your X-Auth-Token matches the format secret_key/api_id and that the values are correct
`404`	This could be due to a specified resource could not be found, an incorrect org key in the path, or incorrect permissions on the API key’s access level	Verify that you specified the correct values (path, org key, forwarder id, etc.) and your forwarder configuration is using the correct Authentication information

Guides & Resources

Quick Setup in Postman & S3 Bucket Configuration

Carbon Black Cloud Authentication Guide

Writing a Bucket Policy for the Data Forwarder

Carbon Black Cloud Data Forwarder Data Mapping

CBC: What URLs are used to access the APIs?

FAQ

Where do I go to get help?

Either reach out to Carbon Black Support or file an issue on the Developer Community Forum.

Do I need a Carbon Black employee to enable Data Forwarder for me?

No, the Carbon Black Cloud Data Forwarder is completely self-service. You first need to create a valid configuration to enable the CBC Data Forwarder. See the Quick Setup in Postman & S3 Bucket Configuration guide for help.

Is there a way that I can scope down the Data Forwarder’s access to my S3 resources?

Yes, you may configure your S3 policy in a variety of ways to ensure that the Data Forwarder only has permissions to certain resources. Please see the Data Forwarder and S3 Policy document for details

Is there a way to choose what types of files the Data Forwarder forwards?

No, only gzipped JSONL (.jsonl.gz) files are supported.

Is there a way to choose what specific types of events or alerts are forwarded?

Yes, see the Data Filtering section on this page

Are there other types supported besides “endpoint.event” and “alert”?

No, only event and alert forwarders are currently supported.

What does this field in my data mean?

See the Carbon Black Cloud Data Forwarder Data Mapping document for more information about data schema and sample data.

I am migrating from EDR to the Carbon Black Cloud Data Forwarder. How do the data sent differ between the two?

Refer to the Migration Guide for information on how EDR event data map to Carbon Black Cloud Data Forwarder event data. Some Carbon Black Cloud Data Forwarder fields may be outdated and the Data Mapping document should be referenced for the most up to date information.

Why am I getting an error?

See the Troubleshooting Errors section above.

When will a new copy of the alert will be sent?

When something changes on the backend.

Why don’t Events show up in the Forwarder at the same time as related Alerts?

The Events stream is not annotated until after the Carbon Black Cloud backend has issued an Alert, so it takes time for the Events service to process the data that identifies Events associated with Alerts.

Last modified on July 13, 2023

Carbon Black Cloud

Page Contents

Data Forwarder Configuration API

Introduction

Available Forwarder Types

Requirements

Quick Setup with Postman & S3 bucket Configuration

Alternative Bucket Setup

Data Format

S3 Data Storage Format

Data Mapping

Data Filtering

Filter Schema

Supported Values

Authentication

API Calls

Create Forwarder

Forwarder Healthcheck

Delete Forwarder

Get Configured Forwarders

Edit Forwarder

Troubleshooting Errors

Guides & Resources

FAQ