Carbon Black Cloud Threat Intelligence Connector

This is a python connector for ingesting and processing STIX Content from various third party sources, such as TAXII servers or directly from XML or JSON files. The current supported versions for STIX are 1.x, 2.0, 2.1, and it supports TAXII 1.0, 1.1, 2.0, 2.1.

Requirements

Enterprise EDR
Custom API Key

Carbon Black Cloud Configuration

Custom API Key

The connector requires an API key with the relevant custom permissions below:

To generate the specific permissions navigate to Carbon Black Cloud and then Settings > API Access > Access Level > Add Access Level. Then to create the API key navigate to Settings > API Access > API Keys.

Permissions	Operation
Custom Detections > Feeds <org.feeds>	Create, Read, Update, Delete
Custom Detections > Watchlists <org.watchlists>	Create, Read, Update, Delete

Credentials file

Create а credentials file following the guides here and here. Then create a profile in the credentials file using the Custom API Key that you have created in the previous step.

Installation

You can install the Carbon Black Cloud Threat Intelligence Connector using GitHub following the README.

Getting Started

The connector requires you to have an already setup Feed and a Watchlist in Carbon Black Cloud and to ease that process you can use the create-feed and create-watchlist commands.

How it works

The connector is a tool that sits between the CBC and your TAXII providers take a look at the following figure.

This figure shows how the connector is working.

The data that is being ingested is in JSON or XML format depending on the TAXII Server and the STIX version. STIX 2.x content is served in a JSON format and STIX 1.x is served in XML format.

When the script is being executed it will first get the configuration for the TAXII Servers and it will try to create python clients for them (those are python objects), then it will pass the client for each TAXII Server along with the user specifics: collections, api_roots, begin_date, end_date, added_after (the specifics are based on the server’s version) to the STIX Parsers.

The STIX Parsers are going to gather all of the STIX objects that we support, currently the supported objects are: Indicator (STIX 1.x, 2.x), ObservedData (STIX 1.x). And here is a table with the supported mappings.

Mappings for Indicator (STIX 1.x)

Field	CBC Field	Description
`AddressObj (ipv4-addr)`	`netconn_ipv4`	IPv4 address
`AddressObj (ipv6-addr)`	`netconn_ipv6`	IPv6 address
`DomainNameObj`	`netconn_domain`	Domain name (target FQDN)
`FileObj (hashes.SHA256)`	`process_hash`	The hash of the file
`FileObj (hashes.MD5)`	`process_hash`	The hash of the file
`FileObj (file_name)`	`process_name`	The name of the file
`URIObj`	`netconn_domain`	The URI address

Mappings for the Indicator pattern (STIX 2.x)

Field	CBC Field	Description
`ipv4-addr:value`	`netconn_ipv4`	IPv4 address
`ipv6-addr:value`	`netconn_ipv6`	IPv6 address
`file:hashes.'SHA-256'`	`process_hash`	The SHA256 hash of the file
`artifact:hashes.'SHA-256'`	`process_hash`	The SHA256 hash of the file
`file:hashes.'MD5'`	`process_hash`	The MD5 hash of the file
`artifact:hashes.'MD5'`	`process_hash`	The MD5 hash of the file
`url:value`	`netconn_domain`	The URL address
`domain-name:value`	`netconn_domain`	Domain name (target FQDN)

After the successful parsing of the data into IOCs (Indicator of Compromises), the results are being sent to the final stage which is the importing them into CBC. There the script goes into the logic where it decides the number of Reports that are going to be created along with the Feeds (In the most cases there is going to be one feed). This logic exists because of certain limitation within CBC.

The logic follows these limits: Max of 1,000 IOCs in a Report and Max of 10,000 Reports in a Feed. That estimates the maximum IOCs that can exists in a Feed are 10,000,000. In a case where those limitations are surpassed, the connector will create a new feed with the remaining results, the name will be the same as the first feed but the Part in the end will be 2 or more.

Creating a Feed

Usage: cbc-threat-intel create-feed [OPTIONS] [FEED_NAME] [PROVIDER_URL]
                                    [SUMMARY]

  Creates a feed in CBC

  Example usage:

      cbc-threat-intel create-feed STIXFeed http://test.test/ empty

      cbc-threat-intel create-feed STIXFeed http://test.test/ empty -ca STIX
      -c default -q

Arguments:
  [FEED_NAME]     The name for the feed that is going to be created
  [PROVIDER_URL]  The URL of the provider of the content
  [SUMMARY]       Summary of the feed

Options:
  -ca, --category TEXT    The category that the feed will have  [default:
                          STIX]
  -c, --cbc-profile TEXT  The CBC Profile set in the CBC Credentials
                          [default: default]
  -q, --quiet             This will only print the id of the created feed
  --help                  Show this message and exit.

From the example above you can see that there are couple of parameters that you should be aware of before creating the feed.

Option/Argument	Default	Description
FEED_NAME	String, Required	The name for the feed that is going to be created
PROVIDER_URL	String (URL), Required	The URL of the provider of the content.
SUMMARY	String, Required	The summary of the feed
`--category (-ca)`	String (“STIX”)	The category of the feed
`--cbc-profile (-c)`	String (“default”)	The CBC Profile set in the CBC Credentials
`--quiet (-q)`	Boolean (False)	This will only print the id of the created feed

In some cases you can use -q to pipe this command with the create-watchlist and automatically create feeds that are subscribed from a newly created watchlist.

Creating a Watchlist

Usage: cbc-threat-intel create-watchlist [OPTIONS] [FEED_ID] [WATCHLIST_NAME]

  Creates a Watchlist in CBC (from already created feed)

  Example usage:

      cbc-threat-intel create-watchlist 55IOVthAZgmQHgr8eRF9rA STIXWatchlist

      cbc-threat-intel create-watchlist 55IOVthAZgmQHgr8eRF9rA STIXWatchlist
      -d description -e -t -q

Arguments:
  [FEED_ID]         The watchlist will subscribe from that feed
  [WATCHLIST_NAME]  The name of the watchlist

Options:
  -d, --description TEXT  The description of the watchlist  [default: empty]
  -e, --enable-alerts     Whether alerts should be enabled for this watchlist
  -q, --quiet             This will only print the id of the created feed.
  -c, --cbc-profile TEXT  The CBC Profile set in the CBC Credentials
                          [default: default]
  --help                  Show this message and exit.

From the example above you can see that there are couple of parameters that you should be aware of before creating the feed.

Option/Argument	Default	Description
FEED_ID	String, Required	The watchlist will subscribe from that feed
WATCHLIST_NAME	String (URL), Required	The name of the watchlist
`--description (-d)`	String (“empty”)	The description of the watchlist
`--enable-alerts (-e)`	Boolean (False)	Whether alerts should be enabled for this watchlist
`--quiet (-q)`	Boolean (False)	This will only print the id of the created watchlist
`--cbc-profile (-c)`	String (“default”)	The CBC Profile set in the CBC Credentials

Parsing STIX Threat Intelligence file into Carbon Black Cloud

You can parse a file with the connector with the process-file command.

An example usage and description of that command can be found with:

$ cbc-threat-intel process-file --help
Usage: cbc-threat-intel process-file [OPTIONS] [STIX_FILE_PATH] [FEED_ID]

  Process and import a single STIX content file into CBC `Accepts *.json (STIX
  2.1/2.0) / *.xml (1.x)`

  Example usage:

      cbc-threat-intel process-file ./stix_content.xml 55IOVthAZgmQHgr8eRF9rA

      cbc-threat-intel process-file ./stix_content.xml 55IOVthAZgmQHgr8eRF9rA
      -s 5

      cbc-threat-intel process-file ./stix_content.xml 55IOVthAZgmQHgr8eRF9rA
      -c default

Arguments:
  [STIX_FILE_PATH]  The location of the STIX Content file.
  [FEED_ID]         The id of the feed

Options:
  -s, --severity INTEGER  The severity of the generated Reports  [default: 5]
  -r, --replace           Replacing the existing Reports in the Feed, if false
                          it will append the results
  -c, --cbc-profile TEXT  The CBC Profile set in the CBC Credentials
                          [default: default]
  --help                  Show this message and exit.

The connector will automatically figure out the STIX version for you and use its appropriate parsers (you can use json files too), all you need to do is to pass the file and the required parameters.

Default values for `process-file` command

Option/Argument	Default	Description
STIX_FILE_PATH	String, Required	The location of the STIX Content file.
FEED_ID	String, Required	The id of the feed
`--severity (-s)`	Integer, (5)	Replacing the existing Reports in the Feed, if false it will append the results
`--replace (-r)`	Boolean (False)	The end date of the STIX Content, The format should be ISO 8601 (YYYY-MM-DD)
`--cbc-profile (-c)`	String (“default”)	The CBC Profile set in the CBC Credentials.

Processing STIX content from a TAXII Server

You can use a TAXII server that is providing STIX Content and ingest the data from there using the process-server command.

An example usage and description of that command can be found with:

$ cbc-threat-intel process-server --help
Usage: cbc-threat-intel process-server [OPTIONS]

  Process and import a TAXII Server (2.0/2.1/1.x)

  Example usage:

      cbc-threat-intel process-server --config-file=./config.yml

Options:
  --config-file TEXT  The configuration of the servers  [default: ./config.yml]
  --help              Show this message and exit.

The default path for your config path is {CURRENT_DIR}/config.yml.

This command will get your config file and it will start to ingest STIX content that is served by those TAXII Servers. In the example.yml file you can find an example configuration that you can use to setup your STIX content providers. Alternatively if you have used our old connector you can use the cbc-threat-intel-wizard command to migrate your old configuration into the new one.

Using the configuration wizard

You can use the configuration wizard to easily manage the config.yml.

An example usage of the command:

$ cbc-threat-intel-wizard

This is going to provide a menu with the options:

migrate your current config
create new config file
add new site/feed information

If you were using the old config.yml and want to migrate it, copy the old config.yml in the root directory of your project and run the wizard using the first option. This is going to override the old config and convert it in the new format new config.

With the second option of the wizard you will be able to create a completely new config (if there was existing config.yml it will be deleted). The wizard will lead you through the configurations and values you need to provide to have a valid config. Please enable only the feeds that you would like to use.

The last option allows you to add one more site information to the existing ones. You need to have a valid config.yml in the new format with existing site/feed information in order to use it.

Creating the configuration manually

If you don’t want to use the wizard tool for creating a config interactively you can create the config by yourself following the example.yml and their descriptions inside.

Using the connector with a CRON job

If you want to run the connector with a CRON job, there is an example in the examples/docker_example folder. The CRON job is setup in the crontab_process_server file, by default it runs once monthly but you can adjust that to your preferences.

Make sure to fill the credentials in the examples/docker_example/credentials.cbc to access the Carbon Black Cloud. The TAXII Server configuration (config.yml) should be located at the root of the repository. You can change that by specifying a different configuration in the crontab_process_server file in the --config-file option.

Using the connector as a Python module

If you want to handle the process of polling/importing of the STIX data yourself and only use the “converting” process you can use only the converting parts of the connector. Make sure to initialize an authenticated CBCloudAPI instance and an authenticated TAXII Client instance.

For more information of how to setup an authenticated instance of CBCloudAPI check the documentation.

STIX Parsers

Using the File Parser functionality

from cbc_importer.stix_parsers.v1.parser import STIX1Parser
from cbc_importer.stix_parsers.v2.parser import STIX2Parser

from cbc_sdk import CBCloudAPI

cbcsdk = CBCloudAPI(profile="default")

# Initializing the STIX1 Parser
stix_parser_v1 = STIX1Parser(cbcsdk)
stix_parser_v1.parse_file("./stix_data.xml")

# Initializing the STIX2 Parser
stix_parser_v2 = STIX2Parser(cbcsdk)
stix_parser_v2.parse_file("./stix_data.json")

iocs_v1 = stix_parser_v1.iocs
iocs_v2 = stix_parser_v2.iocs

Using the TAXII Server Parser

You will need a configured instance for the TAXII server that you are going to use. The collections/gather_data variables follow the logic shown in the configuration file.

TAXII 1 Specifics

The collection_management_uri is being used whenever there is a specified collection management URI by the TAXII Server itself otherwise it uses the default one provided by the discovery endpoint of the TAXII Server.

If you specify certain collections within the collections variable it should look like this:

from cabby import Client10, Client11


collections = [
    "collection-a",
    "collection-b"
]

# Used for `begin_date`, `end_date`
kwargs = {}

taxii_client = Client10(...) # or Client11(...)
cbcsdk = CBCloudAPI(profile="default")
stix_parser = STIX1Parser(cbcsdk)

stix_parser.parse_taxii_server(
    client=taxii_client,
    collections=collections,
    collection_management_uri=None,
    **kwargs
)

iocs = stix_parser.iocs

You can specify collections="*" to pull every collection.

TAXII 2 Specifics

You can add multiple roots (dict elements of the list below ) in the gather_data variable. They should look like this:

from taxii2client.v20 import Server as Client20
from taxii2client.v21 import Server as Client21


gather_data = [
    {
        "title": "api_root_title",
        "collections": ["collection-a", "collection-b"]
    },
    {
        "title": "api_root_title2",
        "collections": "*"
    }
]

# Used for `added_after` date
kwargs = {}

taxii_client = Client20(...) # Or Client21(...)
cbcsdk = CBCloudAPI(profile="default")
stix_parser = STIX2Parser(cbcsdk)

stix_parser.parse_taxii_server(
    taxii_client,
    gather_data=gather_data,
    **kwargs
)

iocs = stix_parser.iocs

Or you can set the gather_data="*" to pull every collection out of every root.

Using the STIX 1 Indicator/Observable Parsers

The STIXPackage is usually created by from_xml class method. For more information check the documentation.

from cbc_importer.stix_parsers.v1.parser import STIX1Parser

from cbc_sdk import CBCloudAPI
from stix.core import STIXPackage


cbcsdk = CBCloudAPI(profile="default")
stix_parser = STIX1Parser(cbcsdk)
stix_package = STIXPackage(...).from_xml(...)

indicators = stix_package.indicators
observables = stix_package.observables

stix_parser._parse_stix_indicators(indicators)
stix_parser._parse_stix_observables(observables)

iocs = stix_parser.iocs

Using the STIX 2 Indicator Parser

For more information on Bundle object and how it gets created check the documentation

from cbc_importer.stix_parsers.v2.parser import STIX2Parser

from cbc_sdk import CBCloudAPI
from stix2 import parse as stix2parse

cbcsdk = CBCloudAPI(profile="default")
stix_parser = STIX2Parser(cbcsdk)
stix_data = None # This can be string, dict or file-like object

stix_content = stix2parse(stix_data, allow_custom=True, version="2.0")
iocs = stix_parser._parse_stix_objects(stix_content)

Building and Running

You can use a Docker container for running the application.

First clone the repository, then run the following command:

$ docker build -f ./examples/docker_example/Dockerfile .

After that you can run the container using its hash, keep in mind that your hash is going to be different.

$ docker run 61412f4b303cdccb...89d8490f6c3527

Known Problems

stix2generator is not working properly with the connector, the data generated from the lib is not valid and it cannot be validated by the stix2-validator. The lib is primarily used for testing and prototyping.
Sometimes some TAXII Servers are not accepting the date ranges params and they are throwing errors, if you experience this kind of issue try to remove the date params from the configuration.

Helpful Links

Last modified on January 23, 2023

Carbon Black Cloud

Page Contents