Quick Setup & S3 Bucket Configuration

Overview

This document outlines the steps for configuring a Carbon Black Cloud Data Forwarder and S3 bucket for storing Event, Alert, and Watchlist Hit data. Included is an example bucket policy, which is necessary for configuring your S3 bucket, and a Postman collection for configuring a Forwarder via API.

Requirements

Carbon Black Cloud Console
Carbon Black Cloud Data Forwarder
Amazon Simple Storage Service (Amazon S3)

Create a Forwarder

Option 1: Create a Forwarder in the Carbon Black Cloud Console

Recommended

To create a Data Forwarder in the console, go to Settings > Data Forwarders and select Add Forwarder from the upper-right corner. Further instructions are provided from the “Add Forwarder” form and the Carbon Black Cloud User Guide.

Option 2: Create a Forwarder via API

The following steps guide you through creating a Forwarder via the Data Forwarder API and setting up an AWS S3 bucket to receive the data:

Optional: Set up KMS Encryption
Create a bucket in your AWS Management Console
Configure an AWS S3 Bucket to allow the forwarder to write data
Create an AWS SQS Queue and configure the access policy
Create or choose an access level in the Carbon Black Cloud console
Create an API key in the Carbon Black Cloud console
Optional: Configure the API in Postman
Create a forwarder
Monitor the data flow to the S3 Bucket
Next steps

Optional: Setup KMS Encryption

If you require more security for your data at rest, we recommend that you use AWS’s built-in support, AWS KMS. This makes securing your data easy and provides Carbon Black the ability to write files without the ability to read them later. To enable KMS encryption, you will need a Customer Managed KMS Key.

Note: The Role Policy will also need modification to enable consumers to decrypt objects in the bucket using the KMS key.

Navigate to the AWS Key Management Service.
From the left side panel, choose Customer managed keys.
Create a key.
Leave the default selections for Symmetric keys, KMS key material origin, Single-region key.
Hit Next and fill in any Alias, Description or Tags you like, and any Key administrators, Key deletion or Key usage permissions you need to allocate.
Insert the following in the Key policy in the Statement section, using the appropriate principal id for your region. See the Carbon Black Cloud User Guide or further on this page for the principal ids for each region.

{
           "Sid": "KMS policy to allow CBC Data Forwarder",
           "Effect": "Allow",
           "Principal": {
               "AWS":
"arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder"
           },
           "Action": [
               "kms:GenerateDataKey",
               "kms:Decrypt"
           ],
           "Resource": "*"
       }

Create a bucket in your AWS Management Console

The Data Forwarder requires an AWS Simple Storage Service (S3) bucket to receive data about alerts or events. The S3 bucket lets you store any amount of data so you can fetch it or connect it to other products.

Sign in to the AWS Management Console.
From the top right corner of the page, use the dropdown to select the region for your bucket.

You must select the same region as your Carbon Black Cloud instance. It is possible to work around this requirement using S3 Cross-Region Replication (CRR).

Use the table below to determine the correct region:

Carbon Black Cloud Product URL	AWS Region Name	AWS Region
Prod01: https://dashboard.confer.net Prod 02: https://defense.conferdeploy.net Prod 05: https://defense-prod05.conferdeploy.net	US East (N. Virginia)	us-east-1
Prod 06: https://defense-eu.conferdeploy.net	Europe (Frankfurt)	eu-central-1
Prod NRT: https://defense-prodnrt.conferdeploy.net	Asia Pacific (Tokyo)	ap-northwest-1
Prod Syd: https://defense-prodsyd.conferdeploy.net/	Asia Pacific (Sydney)	ap-southeast-2
Prod UK: https://ew2.carbonblackcloud.vmware.com	Europe (London)	eu-west-2
AWS GovCloud (US): https://gprd1usgw1.carbonblack-us-gov.vmware.com	US Gov West 1	us-gov-west-1

Navigate to Services > S3 console.
Select Create bucket to open the Create Bucket wizard.
In Bucket name, enter a unique name for your bucket. The name may not contain uppercase letters or underscores. For additional guidance, see Amazon’s bucket naming restrictions.
Region should default to the region you selected in step 8. Ensure that the correct region is still selected.
Select Enable to Block all Public Access.
Note: Public access is not required for the S3 bucket to work with the Data Forwarder.
Select Create Bucket.

Configure an AWS S3 Bucket to allow the Forwarder to write data

Learn more about writing bucket policies for different use cases and configuring the bucket with varying levels of access here.

Once the bucket is created and the page is loaded with a success message, select the “Go to bucket details” button from the message, or click the name of the bucket you created from the list below.
The bucket policy gives the Forwarder permissions to write to your bucket. It is helpful to create a new folder, which will be the base folder where you will push all data. You can name this something like “carbon-black-cloud-forwarder”. You will use this for your “prefix-folder-name” in a later step.

From the Permissions tab, select Bucket Policy and configure it by copying the example below into the Bucket Policy Editor and adjusting the example placeholder text:

Update the “Id” value. This can be anything, such as “Policy04212020” (where 04212020 represents the date, in this case, April 21, 2020).
Update the “Sid” value. This can be anything, such as “Stmt04212020”.

Use this table to check that you are using the correct Principal value.

AWS Region	Principal ID
US East (N. Virginia) us-east-1	arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder
Europe (Frankfurt) eu-central-1	arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-eu-central-1-event-forwarder
Asia Pacific (Tokyo) ap-northwest-1	arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-ap-northeast-1-event-forwarder
Asia Pacific (Sydney) ap-southeast-2	arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-ap-southeast-2-event-forwarder
Europe (London) eu-west-2	arn:aws:iam::132308400445:role/mcs2-psc-data-forwarder-s3
US Gov West 1 us-gov-west-1	arn:aws-us-gov:iam::507058390320:role/mcs2-psc-data-forwarder-s3

Update the “Resource” value.
1. Change “carbon-black-customer-bucket-name” with the name you provided in step 11.
2. Replace “prefix-folder-name” with the folder name you created in step 16.
3. Example: “Resource”: “arn:aws:s3:::your-bucket-name/prefix-folder-name/*”
4. The Resource value must end with “/*” to allow the Forwarder to access all sub-folders.
5. Once you have replaced all three values, hit save, and you have configured your bucket policy.

Example Bucket Policy

{
    "Version": "2012-10-17",
    "Id": "Policy9999999981234",
    "Statement": [
        {
            "Sid": "Stmt1111111119376",
            "Effect": "Allow",
            "Principal": {
                "AWS":
"arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource":
"arn:aws:s3:::carbon-black-customer-bucket-name/prefix-folder-name/*"
        }
    ]
}

If adding KMS Encryption, select the Properties tab, scroll to Default Encryption and click Edit
1. Enable Server-side encryption.
2. For Encryption key type, choose AWS Key Management Service key (SSE-KMS).
3. For AWS KMS key, either Choose from your AWS KMS keys, or Enter AWS KMS key ARN.
  1. If you choose to Enter AWS KMS key ARN, copy the ARN of the KMS key .
  2. If you select Choose from your AWS KMS keys, select the Forwarder Key you created in steps 3-6.
4. Enable the Bucket Key.
5. Hit save to finalize KMS encryption for your bucket.
Note: Enabling the Bucket Key is NOT mandatory. AWS recommends using a Bucket Key for cost reasons and support of KMS with the Data Forwarder was validated using this recommendation.

If you choose not to enable Bucket Key, there are no known, negative impacts on Data Forwarder.

Create an SQS Queue

Create an SQS queue in your AWS Management Console.

Configure the Access policy. Replace the tokens with your own values.

    {
        "Version": "2008-10-17",
        "Id": "__default_policy_ID",
        "Statement": [
            {
                "Sid": "__sender_statement",
                "Effect": "Allow",
                "Principal": {
                    "Service": "s3.amazonaws.com"
                },
                "Action": "SQS:SendMessage",
                "Resource": "arn:aws:sqs:<aws-region>:<AWS Account Number>:<name-of-queue>",
                "Condition": {
                    "ForAllValues:ArnEquals": {
                        "aws:SourceArn": "arn:aws:s3:::<name-of-s3-bucket>"
                    }
                }
            }
        ]
    }

Configure the Event Notification in the AWS S3 bucket to use this queue. Navigate to Properties > Event Notifications and set the Destination SQS queue to the arn of the new queue.

Note: If you need to reload older events and are using SQS to pull buckets, the events will not be available in the queue once they are retrieved. To view historical events or reload data, copy the events to another prefix to copy it to the queue.

Create or choose an access level in the Carbon Black Cloud console

The data forwarder requires a “Custom” access level type. You can either follow the steps below to create a custom access level with the least privileges (recommended), or you can use the existing access levels Super Admin or System Admin when you create your API key.
For more information about Access Levels, see the Carbon Black Cloud Authentication Guide.

Log into Carbon Black Cloud Console.
Go to Settings > API Access.
Select Access Levels tab from the top-left.
Select “Add Access Level” and fill in the following information:
1. Give it a name (example: “CB_Cloud_Forwarder”); you will need this for a later step.
2. Give it a description (example: “Only used to forward Events to S3”).
3. Set “Copy Permissions from” to “None”.
From the table, scroll down to the category called Data Forwarding (.notation name event-forwarder.settings) and check the permissions for: Create, Read, Update, Delete.
Select Save and your access level is created.

Create an API key in the Carbon Black Cloud console

For more information about API Keys, see the Carbon Black Cloud Authentication Guide.

Select API Keys tab from the top-left.
Select “Add API Key” and provide the following information:
1. Give it a name.
2. For Access Level Type, choose “custom” and then choose the access level you created in the previous section.
3. Include a description as desired.
Select Save.
Now you will see your API Credentials. You will need these for a later step.

Optional: Configure the API in Postman

You can run API calls in code (cURL, HTTP) or use Postman, which is a platform that helps simplify the use of APIs, especially if you are running several API calls or using multiple Carbon Black APIs.

Download Postman from here and follow the prompts to install it.
Navigate to the Carbon Black Postman Collection and select Open in Postman from the upper-right corner, or fork the collection from the Carbon Black Postman Workspace.
In the Postman app, select a workspace for the collection.
Navigate to the Data Forwarder API folder from Carbon Black Cloud > Platform APIs.
Fill out your configuration by selecting the eyeball icon near the upper right corner next to the settings gear, and then select Edit under the eyeball icon.
1. Verify that the environment URL matches the URL of your Carbon Black Cloud console.
2. Add your API ID and API Secret Key from steps 26-28 under the Current Value column.
3. Add your Org Key, found in the Carbon Black Cloud console under Settings > API Access in the API Keys tab.
4. The cb_forwarder_id can be added later.
From the Collections panel on the left, select the “Get Configured Forwarders” route.
Hit Send to run the call. The result should be null since you have not created any Forwarders.

Create a Forwarder

Select the “Create Forwarder” call from the Postman Collection to configure the Forwarder.
Click on the Body tab and replace all the bracketed text with actual values, making sure to remove the <>.
1. Give the Forwarder a name.
2. Replace the bucket name with the name from step 11.
3. Replace the S3 prefix with your folder name from step 16. Optionally, you can append it with a unique sub-folder name (Example: “prefix-folder-name/events”).
4. Choose what type of data you wish to forward. Options include:
  - Alerts
  - Endpoint events
  - Watchlist hits
Hit Send.

Example Event Forwarder

{
    "name":"your-forwarder-name",
    "s3_bucket_name":"your-bucket-name",
    "s3_prefix":"prefix-folder-name/events",
    "type":"endpoint.event"
}

Example Alert Forwarder

{
    "name":"your-forwarder-name",
    "s3_bucket_name":"your-bucket-name",
    "s3_prefix":"prefix-folder-name/alerts",
    "type":"alert"
}

Run the “Get Configured Forwarders” call again to confirm the configuration is correct.
If your Forwarder was configured successfully, a “healthcheck.json” file is sent to your bucket in a folder named “healthcheck”.
Note: If you created a sub-folder in step 37.3, the healthcheck folder may be in the sub-folder (Ex: prefix-folder-name > events > healthcheck).
1. The healthcheck runs automatically when a Forwarder is created.
2. To check the Forwarder health manually, select the “Forwarder Healthcheck” call.

Example Success Message

{
 "id": "<Forwarder_ID>",
 "enabled": true,
 "update_time": "<YYYY-MM-ddTHH:mm:ssZ>",
 "status": "Valid Bucket Configuration for Bucket: <BucketName> with Prefix: <prefix>",
 "error": ""
}

Example Failure Messages

Invalid Bucket Configuration Error

If you receive this error, check that you used the correct bucket name.

{
 "id": "<Forwarder_ID>",
 "enabled": true,
 "update_time": "<YYYY-MM-ddTHH:mm:ssZ>",
 "status": "Invalid Bucket Configuration for Bucket: <BucketName> with Prefix: <prefix>",
 "error":"NoSuchBucket”
}

Access Denied Error

If you receive this error, there is an issue with your bucket policy. This can occur if the Resource field has the incorrect prefix route. Review steps 16, 17-d, and 36-c to ensure your prefix file path is correct, or see the Writing a Bucket Policy guide for troubleshooting this error.

{
 "id": "<Forwarder_ID>",
 "enabled": true,
 "update_time": "<YYYY-MM-ddTHH:mm:ssZ>",
 "status": "Invalid Bucket Configuration for Bucket: <BucketName> with Prefix: <prefix>",
 "error":"AccessDenied”
}

Monitor the data flow to the S3 bucket

Go back to the Amazon S3 console.
Go to the configured Bucket and Prefix for configured Forwarder(s).
Within the next 5-15 minutes, data should begin to appear in time-based sub-directories.
1. For example, data sent on 4/21/2020 at 11:01:54UTC using the example Forwarder configuration from the “Create a Forwarder” section above will appear in a folder with the following path: prefix-folder-name/events/org_key=ABCD123/year=2020/month=4/day=21/hour=11/minute=1/second=54/xxxfilename.jsonl.gz.

Next Steps

Once Forwarders are configured, you can fetch the data or connect other tools to process it as needed. For example, you can configure a SIEM to collect this data from the Amazon S3 bucket.

Refer to the Data Forwarder Fields guide for more information about your data.

View the Full Documentation

Data Forwarder API Documentation

Give Feedback

Use this form to give us feedback about this site or any of the documentation.

Last modified on April 12, 2023

Carbon Black Cloud

Page Contents