Data Forwarder API
Introduction
The Carbon Black Cloud Forwarder lets you send data about alerts and events to an AWS S3 bucket where it can be reconfigured to port into other applications in your security stack, such as Splunk. The Data Forwarder is recommended over APIs for obtaining large amounts of data from Carbon Black Cloud in near real time.
You can now configure a forwarder in the Carbon Black Cloud console under Settings > Data Forwarders.
Available Forwarder Types
- alert
- endpoint.event
- watchlist.hit
Requirements
- Carbon Black Cloud Endpoint Standard or Enterprise EDR
- API Key with appropriate permissions
- Configured S3 bucket in the same region as the forwarder.
- It is possible to work around this requirement using S3 Cross-Region Replication (CRR).
Guides and Resources
- Carbon Black Cloud User Guide - Settings - Data Forwarders
- Data Forwarder Integrations
- Postman Collection
Authentication
Determine whether you use Carbon Black Cloud or VMware Cloud Services Platform to manage identity and authorization, or see the Carbon Black Cloud API Access Guide for complete instructions.Carbon Black Cloud Managed Identity and Authentication
Customize your access to the Carbon Black Cloud APIs with Role-Based Access Control; All APIs and Services authenticate via API Keys. To access the data in Carbon Black Cloud via API, you must set up a key with the correct permissions for the calls you want to make and pass it in the HTTP Headers.
Environment
Available on majority of environments; Use the Carbon Black Cloud Console URL, as described here.
API Route
Replace the {cbc-hostname} and {org_key} with the URL of your Environment and the org_key for your specific Org.
- Schema information: {cbc-hostname}/data_forwarder/v2/schemas/
- Configuration: {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/
- Filter validation: {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/validate_filter/
Access Level
Before you create your API Key, you need to create a "Custom" Access Level including each category:
- Data Forwarder > Settings > event-forwarder.settings, allow permission to
CREATE, READ, UPDATE, DELETE
API Key
When creating your API Key, use the Access Level Type of "Custom" and select the Access Level you created. Details on constructing and passing the API Key in your requests are available here.
Cloud Services Platform Managed Identity and Authentication
Customize your access to the Carbon Black Cloud APIs with OAuth Access Control; API access is controlled using OAuth apps or User API Tokens. This is currently limited to the UK Point of Presence and AWS GovCloud (US).
Environment
Available on
Prod UK
and AWS GovCloud (US)
. Full list of environments is available here; Use the Carbon Black Cloud Console URL from Cloud Services Platform, as described here.
API Route
Replace the {cbc-hostname} and {org_key} with the URL of your Environment and the org_key for your specific Org.
- Schema information: {cbc-hostname}/data_forwarder/v2/schemas/
- Configuration: {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/
- Filter validation: {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/validate_filter/
Access Level
Before you create your OAuth App, you need to create a custom Role with the following permissions under IDENTITY & ACCESS MANAGEMENT > Roles > VMware Carbon Black Cloud:
- _API.Data.Forwarder:event-Forwarder.Settings, allow permission to
CREATE, READ, UPDATE, DELETE
API Authentication
The Cloud Services Platform supports several authentication options, Access Token, API Token, and for backward compatibility, X-Auth-Token. To learn about the differences or how to use the authentication methods see the Authentication Guide.
Setup for new Data Forwarder
Information on setting up the AWS S3 bucket, bucket policy, bucket encryption and more is on the Integrations > Data Forwarder page.
Data Format
S3 Data Storage Format
Currently the Carbon Black Cloud Data Forwarder only supports output to AWS S3 buckets provided by customers. The Data Forwarder outputs gzip compressed single-event-per-line JSON format (sometimes known as “JSONL”) to the customer’s S3 bucket with the following folder structure:
org_key={org_key}/year={year}/month={month}/day={day}/hour={hour}/minute={minute}/second={second}/{uuid}.jsonl.gz
Therefore, if a batch of events occurred on August 27, 2019 at 12:36:53UTC for the organization with org_key
ABCD1234, it can be located in:
org_key=ABCD1234/year=2019/month=8/day=27/hour=12/minute=36/second=53/ac203140-f0e2-48fe-90cf-05eb7289f628.jsonl.gz
Data Mapping
You can find the Event, Alert and Watchlist Hit schema as well as sample data in the Data Forwarder Data Guide.
Data Filtering
NEW Custom Query Support
EDR provides tremendous visibility into everything happening across your endpoints. This data may be prohibitively expensive in licensing or scale for some downstream SIEM, SOAR, or data lake solutions. To reduce the load, one or more filters can be applied to the events emitted by a data forwarder configuration.
For additional guidance see Syntax Tips for Custom Query Filters in the User Guide.
Data Filtering is only available for forwarders of type “endpoint.event”.
Filter Schema
Field | Definition | Data Type | Values |
---|---|---|---|
action REQUIRED
|
Whether the filter should include or exclude data from being forwarded | String | INCLUDE , EXCLUDE |
name REQUIRED
|
Defined name for the filter | String | N/A |
query REQUIRED
|
The query to match against the data | String | N/A |
enabled |
Boolean flag indicating whether the Data Forwarder will apply the filter to incoming data | Boolean | Default: true |
Query Fields
The fields documented as FILTERABLE
under endpoint.events can be used in filtering queries for both inclusion and exclusion. There are a few fields which are not supported because their values are not feasible to use as a filter for a stream of events.
All string-matching queries are case-insensitive – both your submitted values and the endpoint data will be lowercased before matching.
Event Atomicity
A query cannot include fields from different endpoint.events
types as the Data Forwarder matches against individual events not the state of a process. e.g. scriptload_name:c\\:\\\\windows\\\\system32\\\\* AND netconn_domain:*.ru
or childproc_reputation:REP_WHITE AND netconn_inbound:false
would fail to match any events as no single event can include scriptload, netconn, and childproc fields
You can combine any number of the Common Fields as long as the query does not exceed the complexity (“depth”) limit.
Wildcards
Wildcards are used in search terms to represent one or more other characters and are supported on most fields. A wildcard can be used as a leading or trailing match.
?
wildcard will match a single character*
wildcard will match 0 or more characters
See the Data Forwarder Data Guide for which fields support wildcarding.
Tokenization
Fields that support tokenization will allow partial matches on each segment where a string is split at the space character. e.g. winword.exe c:\users\mike\file.txt
has two tokens winword.exe
and c:\users\mike\file.txt
.
Do not use a trailing whitespace, “\\
” or “\\ \\
” at the end of a field query or filter. Instead, use one of the following:
- A trailing wildcard
- Tokenization to create a query
- Filter by field name as described below
e.g. for copy c:\windows\system32\cacls.exe c:\users\mike
, copy
and c\\:\\\\users\\\\mike
will match, as will c\\:\\\\windows\\\\system32\\\\*
but not c\\:\\\\windows\\\\system32
Further information is also available in the Carbon Black Cloud User Guide.
See the Data Forwarder Data Guide for which fields support tokenization.
Examples
Note: JSON requires backslashes to be escaped, which is why you must submit query filters like the following 'process_path:*\\\\brave.exe'. The Carbon Black Cloud UI console will perform the escaping for the user when making the API call. If you copy an example and paste into the UI console ensure you remove half the slashes i.e. 'process_path:*\\brave.exe'.
Replicate a filter from v1 API
event_origin:NGAV AND sensor_action:DENY AND type:endpoint.event.procstart AND alert_id:*
Use complex boolean logic
event_origin:NGAV AND (process_path:*\\\\brave.exe OR process_path:*\\\\firefox.exe OR (process_reputation:REP_KNOWN_MALWARE AND NOT sensor_action:ACTION_ALLOW))
Use wildcards
local_port:80* OR process_path:c\\:\\\\windows\\\\system32*
Use tokenization
process_cmdline:powershell.exe OR process_cmdline:*.ps1
API Calls
Config
Use the following API calls for any type of forwarder (alert, endpoint.event, or watchlist.hit).
Create Forwarder
Use this call to create a new forwarder. The API will then make calls to check whether the Forwarder can write to the specified S3 Bucket using the configuration supplied. It will attempt to write a test message called healthcheck.json
to the specified bucket under a subfolder called healthcheck
. If the bucket is misconfigured (i.e. incorrect permissions, principle arn, etc.) or the configuration is incorrect (i.e. bucket prefix doesn’t match path specified in policy), the API will respond with a 400
error and message with information about what was incorrect and how to fix the issue. See Troubleshooting Errors for more info.
If you want to forward alert type data, endpoint.event type data and watchlist hit type data, you must create a separate forwarder for each. The forwarder should be configured to send the data to its own subfolder in the S3 bucket using the S3 prefix property. The subfolder you configure will be automatically added to the S3 bucket.
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
CREATE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.create |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
POST {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs
Request Body
{
"name": "<string>",
"s3_bucket_name": "<string>",
"s3_prefix": "<string>",
"type": "<string>",
"enabled": "<boolean>"
}
Body Schema
Field | Definition | Data Type | Values |
---|---|---|---|
name REQUIRED
|
Defined name for the specific event or alert forwarder | String | N/A |
s3_bucket_name REQUIRED
|
Configured unique name for s3 bucket | String | N/A |
s3_prefix REQUIRED
|
Defined folder structure the forwarder will write events or alerts to | String | N/A |
type REQUIRED
|
The datastream type that is to be forwarded. | Sting | alert , endpoint.event , watchlist.hit |
enabled |
Boolean flag indicating whether to create the data forwarder as enabled | Boolean | Default: true |
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful creation of forwarder | application/json | View example response below |
400 | Bad request. The JSON body was malformed, or the configuration is invalid (misconfigured bucket or configuration value) | application/json |
|
404 | Resource not found | application/json | See above |
409 | Conflict. A duplicate config was found. | application/json | See above |
500 | Internal server error | application/json | See above |
Example
Request
POST https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs
Request Body
{
"name": "DevRelTest",
"s3_bucket_name": "DevRelTestBucket",
"s3_prefix": "test",
"type": "endpoint.event",
"enabled": false
}
Response for valid configuration
{
"id": "58aa259f-f42c-11e9-bd5d-2eca45f7dc1c",
"org_key": "ABC123",
"name": "DevRelTest",
"enabled": false,
"s3_bucket_name": "DevRelTestBucket",
"s3_prefix": "test",
"type": "endpoint.event",
"create_time": "2019-10-21T17:58:11Z",
"update_time": "2019-10-21T17:58:11Z"
}
Response for invalid configuration
If you receive an error indicating your configuration is invalid, either NoSuchBucket
or AccessDenied
, see the Troubleshooting Errors section below.
Forwarder Healthcheck
This call can be used to run a healthcheck on an Forwarder to determine if the forwarder is able to write to the specified S3 Bucket using the existing configuration. The healthcheck can indicate if the bucket is misconfigured (i.e. incorrect permissions, principle arn, etc.), if the configuration is incorrect (i.e. bucket prefix doesn’t match path specified in policy), or if the forwarder is working as expected. If successful, the healthcheck will attempt to write a test message called healthcheck.json
to your S3 bucket under a subfolder called healthcheck
and respond with a 200
, regardless of whether the API was able to write to the bucket.
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
READ |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.read |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
GET {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{config_id}/health_check
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Healthcheck completed successfully | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
GET
https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/health_check
Successful Response
Response for a successful healthcheck that was able to write to the S3 bucket using the specified configuration:
{
"id": "57b09141-f4e8-11e9-83de-22656feed3f2",
"enabled": true,
"update_time": "2019-10-22T16:59:41Z",
"status": "Valid Bucket Configuration for Bucket: DevRelTestBucket with Prefix: test",
"error": ""
}
Successful Response with Error Status
Example response for successful healthcheck that was unable to write to the S3 bucket using the specified configuration:
{
"id": "57b09141-f4e8-11e9-83de-22656feed3f2",
"enabled": true,
"update_time": "2019-10-22T16:59:41Z",
"status": "Invalid Bucket Configuration for Bucket: IncorrectBucket with Prefix: test",
"error": "AccessDenied"
}
If you received a NoSuchBucket
error or AccessDenied
error, see the Troubleshooting Errors section below for suggested fixes.
Get Configured Forwarders
Get all configured forwarders and their information
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
READ |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.read |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
GET {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful request for all configured forwarders | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
GET https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs
Response
[{
"id": "57b09141-f4e8-11e9-83de-22656feed3f2",
"org_key": "ABCD1234",
"name": "DevRelTest",
"enabled": true,
"s3_bucket_name": "DevRelTestBucket",
"s3_prefix": "test",
"type": "endpoint.event",
"create_time": "2019-10-22T16:23:55Z",
"update_time": "2019-10-22T16:26:01Z"
}]
Get Specific Forwarder
Get a specific forwarder’s configuration by id
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
READ |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.read |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
GET {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{id}
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successfully returned forwarder | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
GET https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2
Response
{
"id": "57b09141-f4e8-11e9-83de-22656feed3f2",
"org_key": "ABCD1234",
"name": "DevRelTest",
"enabled": true,
"s3_bucket_name": "DevRelTestBucket",
"s3_prefix": "test",
"type": "endpoint.event",
"create_time": "2019-10-22T16:23:55Z",
"update_time": "2019-10-22T16:26:01Z"
}
Edit Forwarder
This call is used to edit an existing forwarder. You can edit the Enabling/Disabling functionality or change the s3 bucket name.
When editing an existing forwarder, the API will make additional calls to check whether the Carbon Black Cloud event or alert forwarder can write to the specified S3 Bucket using the configuration supplied. It will attempt to write a test message called healthcheck.json
to the specified bucket under a subfolder called healthcheck
. If the bucket is misconfigured (e.g. incorrect permissions, principle arn, etc.) or the configuration is incorrect (e.g. bucket prefix doesn’t match path specified in policy or bucket does not exist), the Data Forwarder Configuration API will respond with a 400
and message including information regarding what was incorrect, providing the customer with feedback that enables them to fix issues as needed.
If you want to remove filters that are applied to your configuration then use the Delete Filter endpoint
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
UPDATE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.update |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
PUT {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{config_id}
Request Body
{
"name": "<string>",
"s3_bucket_name": "<string>",
"s3_prefix": "<string>",
"type": "<string>"
}
Body Schema
Field | Definition | Data Type | Values |
---|---|---|---|
name REQUIRED
|
Defined name for the specific event or alert forwarder | String | N/A |
s3_bucket_name REQUIRED
|
Configured unique name for s3 bucket | String | N/A |
s3_prefix REQUIRED
|
Defined folder structure the forwarder will write events or alerts to | String | N/A |
type REQUIRED
|
The datastream type that is to be forwarded. | Sting | alert , endpoint.event , watchlist.hit |
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful editing of forwarder | application/json | View example response below |
400 | Bad request. The JSON body was malformed, or some part of the JSON body included an invalid value | application/json |
|
404 | Resource not found | application/json | See above |
409 | Conflict. A duplicate config was found. | application/json | See above |
500 | Internal server error | application/json | See above |
Example
Request
PUT https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2
Request Body
{
"name": "DevRelTest",
"s3_bucket_name": "DevRelTestBucket",
"s3_prefix": "test",
"type": "endpoint.event",
"enabled": false
}
Response for valid configuration
{
"id": "57b09141-f4e8-11e9-83de-22656feed3f2",
"org_key": "ABCD1234",
"name": "DevRelTest",
"enabled": false,
"s3_bucket_name": "DevRelTestBucket",
"s3_prefix": "test",
"type": "endpoint.event",
"create_time": "2019-10-22T16:23:55Z",
"update_time": "2019-10-22T16:26:01Z"
}
Response for invalid configuration
If you receive a response for invalid configuration, see the Troubleshooting Errors section below for suggested fixes.
Delete Forwarder
Use this call to delete a forwarder.
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
DELETE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.delete |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
DELETE {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{config_id}
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
203 | Successful deletion of forwarder | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
DELETE https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/58aa259f-f42c-11e9-bd5d-2eca45f7dc1c
Response
No Content
Filter
The Filter API calls can be used only for “endpoint.event” forwarder type.
Validate Filter
Validate whether the filter uses compatible query syntax, field names and values
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
CREATE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.create |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
POST {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/validate_filter
Request Body
{
"action": "<string>",
"enabled": <boolean>,
"name": "<string>",
"query": "<string>"
}
Body Schema
Field | Definition | Data Type | Values |
---|---|---|---|
action REQUIRED
|
Whether the filter should include or exclude data from being forwarded | String | INCLUDE , EXCLUDE |
name REQUIRED
|
Defined name for the filter | String | N/A |
query REQUIRED
|
The query to match against incoming data | String | N/A |
enabled |
Boolean flag indicating whether the Data Forwarder will apply the filter to incoming data | Boolean | Default: true |
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Filter is valid | application/json | View example response below |
400 | Bad request. The JSON body was malformed, or the filter is invalid | application/json | View example response below |
404 | Resource not found | application/json | View example response below |
500 | Internal server error | application/json | See above |
Example
Request
POST https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/validate_filter
Request Body Valid Filter
{
"action": "INCLUDE",
"enabled": true,
"name": "Example Filter",
"query": "device_os:WINDOWS AND event_origin:NGAV"
}
Response
{
"id": "",
"name": "Example Filter",
"query": "device_os:WINDOWS AND event_origin:NGAV",
"action": "INCLUDE",
"create_time": "",
"update_time": "",
"enabled": true
}
Request Body Invalid Filter
{
"action": "INCLUDE",
"enabled": true,
"name": "Example Filter",
"query": "bad:WINDOWS"
}
Response
{
"error_code": "QUERY_UNKNOWN_FIELD",
"message": "field bad does not exist",
"args": [
"bad"
]
}
Filterable Event Schema
JSON schema document describing filterable fields, their types, and available enum values
Request
GET {cbc-hostname}/data_forwarder/v2/schemas/events?filterable=true
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Filterable Event Schema | application/json | View example response below |
400 | Bad request. Missing required query parameter | application/json |
|
404 | Resource not found | application/json | See above |
500 | Internal server error | application/json | See above |
Example
Request
GET https://defense.conferdeploy.net/data_forwarder/v2/schemas/events?filterable=true
Response
{
"$id": "https://carbonblack.com/forwarder/events/filterable.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Filterable Event Schema",
"description": "CBC Data Forwarder Filterable Fields and Enums for Events",
"type": "object",
"properties": {
"alert_id": {
"type": "string"
},
"childproc_hash": {
"type": "string"
},
"childproc_name": {
"type": "string"
},
"childproc_pid": {
"type": "number"
},
"childproc_publisher_state": {
"type": "string",
"enum": [
"FILE_SIGNATURE_STATE_INVALID",
"FILE_SIGNATURE_STATE_SIGNED",
"FILE_SIGNATURE_STATE_VERIFIED",
"FILE_SIGNATURE_STATE_NOT_SIGNED",
"FILE_SIGNATURE_STATE_UNKNOWN",
"FILE_SIGNATURE_STATE_CHAINED",
"FILE_SIGNATURE_STATE_TRUSTED",
"FILE_SIGNATURE_STATE_OS",
"FILE_SIGNATURE_STATE_CATALOG_SIGNED"
]
},
... truncated
},
"anyOf": [
{
"properties": {
"action": {
"type": "string",
"enum": [
"ACTION_INVALID",
"ACTION_FILE_CREATE",
"ACTION_FILE_WRITE",
"ACTION_FILE_DELETE",
"ACTION_FILE_LAST_WRITE",
"ACTION_FILE_MOD_OPEN",
"ACTION_FILE_RENAME",
"ACTION_FILE_UNDELETE",
"ACTION_FILE_TRUNCATE",
"ACTION_FILE_OPEN_READ",
"ACTION_FILE_OPEN_WRITE",
"ACTION_FILE_OPEN_DELETE",
"ACTION_FILE_OPEN_EXECUTE",
"ACTION_FILE_READ",
"ACTION_FILE_PENDING_DELETE",
"ACTION_FILE_PENDING_RENAME",
"ACTION_FILE_SET_SECURITY",
"ACTION_FILE_LOCK",
"ACTION_FILE_DISCOVERED",
"ACTION_FILE_LINK",
"ACTION_FILE_FIND"
]
},
"type": {
"type": "string",
"enum": [
"endpoint.event.filemod"
]
}
}
},
... truncated
]
}
Create Filter
Create a filter for the specified configuration to include or exclude data from being forwarded. Multiple Filters for the same config id apply logical OR to support separated complex conditions. The following sample shows how two include and two exclude filters would be applied (IncludeFilter1 OR IncludeFilter2) AND NOT (ExcludeFilter1 OR ExcludeFilter2)
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
CREATE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.create |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
POST {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{id}/filters
Request Body
{
"action": "<string>",
"enabled": <boolean>,
"name": "<string>",
"query": "<string>"
}
Body Schema
Field | Definition | Data Type | Values |
---|---|---|---|
action REQUIRED
|
Whether the filter should include or exclude data from being forwarded | String | INCLUDE , EXCLUDE |
name REQUIRED
|
Defined name for the filter | String | N/A |
query REQUIRED
|
The query to match against incoming data | String | N/A |
enabled |
Boolean flag indicating whether the Data Forwarder will apply the filter to incoming data | Boolean | Default: true |
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful creation of filter | application/json | View example response below |
400 | Bad request. The JSON body was malformed, or the filter is invalid | application/json |
|
404 | Resource not found | application/json | See above |
409 | Conflict. A duplicate filter was found. | application/json | See above |
500 | Internal server error | application/json | See above |
Example
Request
POST https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/filters
Request Body
{
"action": "INCLUDE",
"enabled": true,
"name": "Example Filter",
"query": "device_os:WINDOWS AND event_origin:NGAV"
}
Response
{
"action": "INCLUDE",
"create_time": "2021-07-02T16:23:55Z",
"enabled": true,
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891",
"name": "Example Filter",
"query": "device_os:WINDOWS AND event_origin:NGAV",
"update_time": "2021-07-03T16:23:55Z"
}
Bulk Filters
Create or update multiple filters for the specified configuration to include or exclude data from being forwarded. The presence of an id
field is the differentiator between Create
and Update
.
Multiple Filters for the same config id apply logical OR to support separated complex conditions. The following sample shows how two include and two exclude filters would be applied (IncludeFilter1 OR IncludeFilter2) AND NOT (ExcludeFilter1 OR ExcludeFilter2)
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
CREATE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.create |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
POST {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{id}/filters/_bulk
Request Body
[
// New Filter Input
{
"action": "<string>",
"enabled": <boolean>,
"name": "<string>",
"query": "<string>"
},
// Update Filter Input
{
"action": "<string>",
"enabled": <boolean>,
"id": "<string>",
"name": "<string>",
"query": "<string>"
}
]
Body Schema
Field | Definition | Data Type | Values |
---|---|---|---|
List[Filters] | List of filters to be created or updated | List[Filters] | Max: 25 filters |
Filter Schema
Field | Definition | Data Type | Values |
---|---|---|---|
action REQUIRED
|
Whether the filter should include or exclude data from being forwarded | String | INCLUDE , EXCLUDE |
name REQUIRED
|
Defined name for the filter | String | N/A |
query REQUIRED
|
The query to match against incoming data | String | N/A |
enabled |
Boolean flag indicating whether the Data Forwarder will apply the filter to incoming data | Boolean | Default: true |
id |
Id of an existing filter to be updated | String | N/A |
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful creation of filter | application/json | View example response below |
400 | Bad request. The JSON body was malformed, or the filter is invalid | application/json |
|
404 | Resource not found | application/json | See above |
409 | Conflict. A duplicate filter was found. | application/json | See above |
500 | Internal server error | application/json | See above |
Example
Request
POST https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/filters/_bulk
Request Body
[
{
"action": "INCLUDE",
"enabled": true,
"name": "Example Filter",
"query": "device_os:LINUX AND event_origin:EEDR",
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891"
},
{
"action": "EXCLUDE",
"enabled": true,
"name": "Example EXCLUDE Filter",
"query": "device_os:MAC"
}
]
Response
{
"results": [
{
"action": "INCLUDE",
"create_time": "2021-07-02T16:23:55Z",
"enabled": true,
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891",
"name": "Example Filter",
"query": "device_os:LINUX AND event_origin:EEDR",
"update_time": "2022-01-28T17:30:52Z"
},
{
"action": "EXCLUDE",
"create_time": "2022-01-28T17:30:52Z",
"enabled": true,
"id": "9f7ce0be-445e-4f30-9faf-9997c1cbc552",
"name": "Example EXCLUDE Filter",
"query": "device_os:MAC",
"update_time": "2022-01-28T17:30:52Z"
}
]
}
Get Filters
Get all filters for the specified configuration
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
READ |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.read |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
GET {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{id}/filters
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful request for all configured filters | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
GET https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/filters
Response
[
{
"action": "INCLUDE",
"create_time": "2021-07-02T16:23:55Z",
"enabled": true,
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891",
"name": "Example Filter",
"query": "device_os:WINDOWS AND event_origin:NGAV",
"update_time": "2021-07-03T16:23:55Z"
}
]
Get Specific Filter
Get a specific filter by id for a given configuration.
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
READ |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.read |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
GET {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{id}/filters/{id}
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful request for specified filter | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
GET https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/filters/f13e6723-7779-4fdc-a26e-5a95e66be891
Response
{
"action": "INCLUDE",
"create_time": "2021-07-02T16:23:55Z",
"enabled": true,
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891",
"name": "Example Filter",
"query": "device_os:WINDOWS AND event_origin:NGAV",
"update_time": "2021-07-03T16:23:55Z"
}
Edit Filter
Adjust an existing a filter by modifying the query, renaming the filter, changing the action, or enabling/disabling the filter.
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
UPDATE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.update |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
PUT {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{id}/filters/{id}
Request Body
{
"action": "<string>",
"enabled": <boolean>,
"name": "<string>",
"query": "<string>",
"id": "<string>",
"create_time": "<string>",
"update_time": "<string>"
}
Body Schema
Field | Definition | Data Type | Values |
---|---|---|---|
action REQUIRED
|
Whether the filter should include or exclude data from being forwarded | String | INCLUDE , EXCLUDE |
name REQUIRED
|
Defined name for the filter | String | N/A |
query REQUIRED
|
The query to match against incoming data | String | N/A |
enabled |
Boolean flag indicating whether the Data Forwarder will apply the filter to incoming data | Boolean | Default: true |
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
200 | Successful update of filter | application/json | View example response below |
400 | Bad request. The JSON body was malformed, or the filter is invalid | application/json |
|
404 | Resource not found | application/json | See above |
409 | Conflict. A duplicate filter was found. | application/json | See above |
500 | Internal server error | application/json | See above |
Example
Request
PUT https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/57b09141-f4e8-11e9-83de-22656feed3f2/filters/f13e6723-7779-4fdc-a26e-5a95e66be891
Request Body
{
"action": "INCLUDE",
"create_time": "2021-07-02T16:23:55Z",
"enabled": true,
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891",
"name": "Example Filter - Edited",
"query": "device_os:WINDOWS AND event_origin:NGAV AND process_reputation:KNOWN_MALWARE",
"update_time": "2021-07-03T16:23:55Z"
}
Response
{
"action": "INCLUDE",
"create_time": "2021-07-02T16:23:55Z",
"enabled": true,
"id": "f13e6723-7779-4fdc-a26e-5a95e66be891",
"name": "Example Filter - Edited",
"query": "device_os:WINDOWS AND event_origin:NGAV AND process_reputation:KNOWN_MALWARE",
"update_time": "2021-07-03T16:23:55Z"
}
Delete Filter
Use this call to delete a filter.
API Permissions Required
Identity Manager | Permission (.notation name) | Operation(s) | Environment |
---|---|---|---|
Carbon Black Cloud | event-forwarder.settings |
DELETE |
Majority of environments |
VMware Cloud Services Platform | _API.Data.Forwarder:event-Forwarder.Settings.delete |
N/A - included in permission name | Prod UK and AWS GovCloud (US) |
Request
DELETE {cbc-hostname}/data_forwarder/v2/orgs/{org_key}/configs/{config_id}/filters/{id}
Response
Use the following to troubleshoot errors. For more troubleshooting tips, see below.
Code | Description | Content-Type | Content |
---|---|---|---|
204 | Successful deletion of filter | application/json | View example response below |
404 | Resource not found | application/json |
|
500 | Internal server error | application/json | See above |
Example
Request
DELETE https://defense.conferdeploy.net/data_forwarder/v2/orgs/ABCD1234/configs/58aa259f-f42c-11e9-bd5d-2eca45f7dc1c/filters/f13e6723-7779-4fdc-a26e-5a95e66be891
Response
No Content
Troubleshooting Errors
Error Code | Reason For Error | Suggested Fix |
---|---|---|
NoSuchBucket |
Configuration is invalid. The specified S3 bucket name doesn’t exist | Follow the Quick Start Guide to create an S3 bucket for use by the CBC Data Forwarder |
AccessDenied OR
“Your request was unsuccessful” during new Data Forwarder configuration |
Either: Bucket was created in the wrong region OR The policy is malformed | Ensure that your bucket is created in the same region as the where the Data Forwarder is deployed. Refer to the Data Forwarder and S3 Policy document for policy troubleshooting and configuration instructions |
400 |
The reason varied depending on the API call | Refer to the individual API call you are making to validate the JSON body |
401 |
Problem using the specified token to authenticate | Check that your X-Auth-Token matches the format secret_key/api_id and that the values are correct |
404 |
This could be due to a specified resource could not be found, an incorrect org key in the path, or incorrect permissions on the API key’s access level | Verify that you specified the correct values (path, org key, forwarder id, etc.) and your forwarder configuration is using the correct Authentication information |
Guides & Resources
Quick Setup in Postman & S3 Bucket Configuration
Carbon Black Cloud Authentication Guide
Writing a Bucket Policy for the Data Forwarder
Deconstructing the Process GUID
CBC: What URLs are used to access the APIs?
FAQ
Where do I go to get help?
- Either reach out to Carbon Black Support or file an issue on the Developer Community Forum.
Do I need a Carbon Black employee to enable Data Forwarder for me?
- No, the Carbon Black Cloud Data Forwarder is completely self-service. You first need to create a valid configuration to enable the CBC Data Forwarder. See the Quick Setup in Postman & S3 Bucket Configuration guide for help.
Is there a way that I can scope down the Data Forwarder’s access to my S3 resources?
- Yes, you may configure your S3 policy in a variety of ways to ensure that the Data Forwarder only has permissions to certain resources. Please see the Data Forwarder and S3 Policy document for details.
Is there a way to choose what types of files the Data Forwarder forwards?
- No, only gzipped JSONL (
.jsonl.gz
) files are supported.
Is there a way to choose what specific types of events or alerts are forwarded?
- Yes, see the Data Filtering section on this page.
Are there other types supported besides “alert”, “endpoint.event”, or “watchlist.hit”?
- No, only alert, event, and watchlist hit forwarders are currently supported.
What does this field in my data mean?
- See the Carbon Black Cloud Data Forwarder Data Mapping document for more information about data schema and sample data.
I am migrating from EDR to the Carbon Black Cloud Data Forwarder. How do the data sent differ between the two?
- Refer to the Migration Guide for information on how EDR event data map to Carbon Black Cloud Data Forwarder event data. Some fields from the Migration Guide may be outdated, use the Data Forwarder Data Guide for the most up to date information.
Why am I getting an error?
- See the Troubleshooting Errors section above.
When will a new copy of the alert will be sent?
- When something changes on the backend.
Why don’t Events show up in the Forwarder at the same time as related Alerts?
- The Events stream is not annotated until after the Carbon Black Cloud backend has issued an Alert, so it takes time for the Events service to process the data that identifies Events associated with Alerts.