Carbon Black SDK release
Posted on January 1, 0001
CB SDK RELEASE
The Carbon Black SDK provides a framework for easilly creating arbitrary connectors and integrations with Carbon Black products. The cb-integration project provides python libraries for generic integrations, a specialized framework for binary analysis connectors. See the source code in the cb-integration repo for implementation details.
The CBSDK is cross platform, and should work on any environment that has docker 1.7+ and docker-compose.
At its core, the CBSDK provides a lightweight linux container, for connectors -
that can be pulled from dockerhub with: $ docker pull cbdevnetwork/cbsdk .
This container provides an environment with the appropriate python dependencies already installed - the CBAPI bindings for python, and the cb-integration code. The SDK provides a light wrapper around supervisor - which manages connectors where intictl did in the past. This allows developers to shunt a lot of responsibility over to supervisord: auto restarts, log rotation/compacting , monitoring, etc.
https://github.com/Supervisor/supervisor
Connectors are thereby simplified from stand-alone RPMs with binaries, to simple python scripts managed by supervisor.
This integration SDK can, in principle, manage the execution of arbitrary services and is language and technology agnostic. Developers already familiar with supervisor should have no problem adapting to this new framework.
The SDK provides a docker-compose configuration. This mounts a target volume directory containing subdirectories for each desired connector, internally providing relevant configuration files, etc.
This mounted directory becomes /vol within the CBSDK container, and
the supervisor daemon will manage any directories created within, so long as
they have their own supervisord.conf. The connector level supervisord.conf can control the startup
and shutdown of any dependent services the connector may rely on, for instance a database.
The CBSDK requires docker 1.71+ and docker-compose to be available on the target.
By default, a react based UI is available on port 5000 - this can be changed
in the docker-compose.yml . This is an extension of the basic supervisor UI - each connector
can provide an xmlprc interface extension that will be automatically loaded and integrated within the
UI to provide custom actions for each connector. Supervisor’s base RPC interface provides for logging,
log rotation, startup and shutdown, status etc. A custom connector level rpc interface can
extend this to provide distinct features like “Lookup a binary by md5”, “Rescan binary by md5” etc.
See the yaraconnector and supervisor’s xmlrpc interface documentation for details on how to do this.
Getting started - YARA Binary Analysis Example
The CBSDK project comes with the yara-connector loaded as a submodule. Checkout or clone the project with git or download the archive and extract to the working directory of your choice.
git clone git@github.com:carbonblack/cb-integration
Move into the cb-integration directory to begin.
The YARA binary analysis connector is provided as an example, intended to serve as a guide to developers and users.
The YARA connector is provided inline in the cb integration repo as a submodule.
Supervisord is configured to montior the mounted /vol directory.
The YARA connector resides in $WORKDIR/vol/yaraconnector.
$ docker-compose up will raise the CBSDK container.
```
vol:
yaraconnector:
yaraconnector.conf
supervisord.conf
db:
feed:
```
The feed directory stores a json formatted threat intel feed produced
from the results of YARA analysis of scanned binaries.
The db directory provides space to store SQLITE db files.
The supervisord.conf in the vol/yaraconnector directory:
[rpcinterface:yaraconnector]
supervisor.rpcinterface_factory = yaraconnector.supervisordrpcinterface:make_custom_rpcinterface
[program:redis]
command=/usr/bin/redis-server
user=root
autostart=true
autorestart=true
stdout_logfile=/vol/yaraconnector/redis.log
stderr_logfile=/vol/yaraconnector/redis.log
[program:yaraconnector]
directory=/vol/yaraconnector
stdout_logfile=/vol/yaraconnector/yaraconnector.log
stderr_logfile=/vol/yaraconnector/yaraconnector.log
command=python3 main.py
user=root
autostart=true
autorestart=true
priority=800
[program:yara_workers]
stdout_logfile=/vol/yaraconnector/yaraconnector_workers.log
stderr_logfile=/vol/yaraconnector/yaraconnector_workers.log
user=supervisor
directory=/vol/yaraconnector
command=celery -A tasks worker --autoscale=16,4 --loglevel=debug
autostart=true
autorestart=true
priority=900
Simple stated, this supervisord.conf controls the startup of the YARA connector
and its' dependent services. A redis DB and a celery queue (which automagically scales) for async scanning
of the desired binaries - and provides an extension to supervisorD’s xmlrpc
interface to allow yara-connector specific actions and flows. See the rpcinterface definition
in the yara-connector project for details.
The yaraconnector.conf has the configuration parameters for the YARA
connector, in .ini format. Be sure to specify the parameters for connecting to
the desired carbon black environment, etc.
Getting started (SDK development, Connector Development)
Development requires: Yarn (npm works too in a pinch), git, docker & docker-compose as well as NPM10+.
Clone or download this github repo:
git clone git@github.com:carbonblack/cb-integration
If your connector requires additionaln python dependencies, you can create a virtualenvironment
and add the dependencies within - and add the environment directive to the [program:<myconnector>] stanza in the
connector level supervisord.conf - making sure to set PATH=$YOURVIRTUALENV/bin. This will cause
supervisord to run your connector within the desired virtual environment.
You can modify the docker-compose to use the local dockerfile when building to inject additional dependencies by commenting out the ‘image’ directive and commenting-in the provided ‘build’ and ‘dockerfile’ directives which indicate to compose how to build the image locally.
A makefile is provided for building the project and UI.
The environment in the CB SDK image has python3.6,pip, and a very up to date toolchain. If you need to add addtional dependencies that cannot be installed in a virtualenv, extend the base CBSDK image using a ‘FROM ' directive in your own dockerfile and modify the docker-compose.yml to use this modified image instead.