VMware Carbon Black Cloud Alert Export Best Practices

Posted on August 25, 2021

Note: See the Alert Bulk Export Guide which has been updated for the Alerts v7 API, released in June 2023.

Forward Alerts to an S3 Bucket

The Data Forwarder is the recommended export method for reliable and guaranteed delivery. This method works at scale to support any size customer or MSSP by writing jsonl zipped content to an S3 bucket. See the Quick Setup instructions for more details.

Exporting Alerts via the Alerts API

If the Data Forwarder doesn’t work for you then the following algorithm will allow you to fetch alerts with no duplicates using the Alerts API. You can run the logic on a cycle to continually fetch alerts or you could do a single time execution to fetch a time range of alerts.

If you need a bulk export, we recommend keeping the max time window to 30 days and repeat as needed to reduce the search space and impact on the backend. If you are repeating on a cycle, we recommend keeping a 30 second delay from current time to allow for asynchronous event processing to complete or if you want to wait for updates to stop for a CB Analytic alert, keep a delay of 15mins. A polling window should be no smaller than 60s and should use a start and end time range for either create_time or last_update_time to keep the search space to a minimum reducing load on the backend.

Note: Using create_time to have no alert duplicates will miss updates made to a CB Analytic alert which could include changes to reputation or additional context when more events are added to the alert. A CB Analytic alert can be updated up to 15mins after the create_time at which point it will be immutable. If you wish to receive the alert updates use last_update_time instead

1. Fetch Alerts for 5min window and ensure 30s delay from current time to allow for Carbon Black Cloud backend to make all Alerts available to API

   curl -s -H "X-Auth-Token: $ACCESS_TOKEN" -H "Content-Type: application/json" https://$CBD/appservices/v6/orgs/$ORG/alerts/_search \
   -d '{
     "criteria": {
       "create_time": {
         "start": "2021-06-29T12:00:00.000Z",
         "end": "2021-06-29T12:05:00.000Z"
       }
     },
     "rows": 10000,
     "start": 0,
     "sort": [
       {
         "field": "create_time",
         "order": "ASC"
       }
     ]
   }'
   

2. Verify if num_found <= 10000 in API response, if less than or equal to 10k results wait for next 5min window and repeat step 1

3. If there are more than 10k results then you will need additional API requests to fetch all Alerts for the 5min window.

   a. Use the last index of the results array to fetch the create_time timestamp that you will use as the start for the time range

   Example Response:

   {
     "num_found": "23568"
     "results": [
       ...,
       {
         ...
         "create_time": "2021-06-29T12:03:21.000Z"
         ...
       }
     ]
   }
   

   b. Add one millisecond to the timestamp to ensure the Alert is not duplicated.

   Note: This does risk missing an alert if there are multiple alerts with the same create_time. If you want to avoid missing any alerts then you can use the same create_time however you will need to add additional processing logic to remove the duplicate alert(s) which will be the first index(s) based on how many alerts were previously fetched with the same create_time.

   curl -s -H "X-Auth-Token: $ACCESS_TOKEN" -H "Content-Type: application/json" https://$CBD/appservices/v6/orgs/$ORG/alerts/_search \
   -d '{
     "criteria": {
       "create_time": {
         "start": "2021-06-29T12:03:21.001Z",
         "end": "2021-06-29T12:05:00.000Z"
       }
     },
     "rows": 10000,
     "start": 0,
     "sort": [
       {
         "field": "create_time",
         "order": "ASC"
       }
     ]
   }'
   

   c. Verify if num_found <= 10000 in API response, if less than or equal to 10k results wait for next 5min window and return to step 1 otherwise repeat from step 3a

If you want to use the Carbon Black Cloud SDK the following code will fetch the alerts and concatenate them into a single list for you to add custom logic or write the ouput to a file.

"""Algorithm to Export Alerts from Carbon Black Cloud"""

from cbc_sdk import CBCloudAPI
from cbc_sdk.platform import BaseAlert

from datetime import datetime, timedelta, timezone

cb = CBCloudAPI(profile="")

# Time field and format to use
time_field = "create_time"
time_format = "%Y-%m-%dT%H:%M:%S.%fZ"

# Time window to fetch using current time - 30s to allow for Carbon Black Cloud asynchronous event processing completion
end = datetime.now(timezone.utc) - timedelta(seconds=30)
start = end - timedelta(minutes=5)

# Fetch initial Alert batch
alerts = list(cb.select(BaseAlert)
                .set_time_range(time_field,
                                start=start.strftime(time_format),
                                end=end.strftime(time_format))
                .sort_by(time_field, "ASC"))

# Check if 10k limit was hit and iteritevly fetch remaining alerts by increasing start time to the last alert fetched
if len(alerts) >= 10000:
    last_alert = alerts[-1]
    while True:
        new_start = datetime.strptime(last_alert.create_time, time_format) + timedelta(milliseconds=1)
        overflow = list(cb.select(BaseAlert)
                          .set_time_range(time_field,
                                          start=new_start.strftime(time_format),
                                          end=end.strftime(time_format))
                          .sort_by(time_field, "ASC"))

        # Extend alert list with follow up alert batches
        alerts.extend(overflow)
        if len(overflow) >= 10000:
            last_alert = overflow[-1]
        else:
            break

print(f"Fetched {len(alerts)} alert(s) from {start.strftime(time_format)} to {end.strftime(time_format)}")