Unified Binary Store REST API Reference

Carbon Black Cloud Enterprise EDR (Endpoint Detection and Response) is the new name for the product formerly called CB ThreatHunter.

Version: v1

Overview

The unified binary store (UBS) is a centralized service that is part of the Carbon Black Cloud. UBS is responsible for storing all binaries and corresponding metadata for those binaries. UBS comes packaged with Enterprise EDR. To obtain the maximum benefit of UBS, organizations must opt-in to binary uploads on the Policies page.

Enterprise EDR customers are able to utilize the UBS APIs to download binaries observed on their endpoints, as well as to retrieve the corresponding metadata. At this time, the data retention for UBS is infinite - so as long as an organization has ever observed execution of that binary on one or more endpoints, that organization will be able to retrieve the associated metadata (and optionally the binary itself, if binary uploads were opted in at the time of execution).

UBS obtains data on first execution of a binary for any device on which the Carbon Black Cloud sensor is running and Enterprise EDR has been subscribed - both metadata about the binary, and if uploads are enabled for the organization, the binary itself. This has two consequences:

  1. If the binary did not successfully execute, no metadata or binary will be uploaded to the Carbon Black Cloud, and UBS will not have any data related to that binary’s non-execution.
  2. If any metadata about the binary changes after initial execution (e.g. file path, signature state), UBS will not receive that updated binary metadata.

For binaries less than 25 MB, the sensor uploads the entire file. For binaries over 25 MB, the sensor uploads the first 25 MB of the binary file.

At this time UBS is supported on the Windows CBC sensor, so all binaries available in UBS are Windows PE files.

Usage

UBS is a RESTful service, utilizing HTTP as the transport protocol. All routes to the UBS take the following format:

<cbc-hostName>/ubs/<versionId>/orgs/{org_key}/…

The following table explains the components of the above route:

Variable Description Example
cbc-hostName The URL of the environment where your organization is hosted. https://defense-prod05.conferdeploy.net
versionId The version of the UBS API that is being targeted. This document specifically documents v1. v1
org_key The organization key. ABCD123

The org_key may be obtained from the API Access page under Settings in the Carbon Black Cloud console.

Overall Response Details

POST routes must have the Content-Type header set to application/json.

Success Responses

All successful API calls will result in returning an HTTP 200 status code. The result will be returned in the body of the response. The specific responses will be detailed in subsequent sections, for each API.

Error Responses

Whenever an HTTP status code not equal to 200 is returned, the response will be as follows:

{
    "message": <string>
    "error_code": <string>
}

error_code is an enum, with values that will be defined below. This field may be used for programmatically handling errors. message is a short explanation as to why an error occurred. This field is intended to be interpreted for debugging purposes, and not to be consumed programmatically.

The following table outlines the expected error HTTP status codes:

Status Code Reason error_code
400 Invalid user input. Ensure that any SHA-256s are valid. BAD_REQUEST
403 Authentication error. Ensure that both the API key andorg_keyare correct. Check the response message field for a more detailed error. FORBIDDEN
404 Requested resource not found.
500 Server error. This is typically a transient error and the request may be retried, ideally with an exponential backoff. INTERNAL_SERVER_ERROR

Additional Notes

Unless otherwise stated, all dates/timestamps will be strings, formatted as ISO 8601 UTC, i.e. YYYY-MM-DDThh:mm:ss.sssZ POST routes MUST have the conventional header Content-Type: application/json or the API will return a 500 error.

File APIs

These APIs are specifically for interacting with files stored within the UBS.

Download

This API provides a means to download files. The files are able to be downloaded via AWS S3 pre-signed URLs. The URLs enables the client to perform a GET on the provided URLs and download the files. The links will automatically expire after one hour, unless a different expiration time is requested.

CAUTION - Anyone who has these links will be able to download the files until the requested expiration time (default one hour after the request). Ensure that care is taken when utilizing this API.

RBAC Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud ubs.org.file READ Majority of environments
VMware Cloud Services Platform _API.Unified.Binary.Store:ubs.Org.File.read N/A - included in permission name Prod UK

Request

The following table describes the inputs required to utilize this API.

Description Value
Route <cbc-hostname>/ubs/<versionId>/orgs/{org_key}/file/_download
HTTP verb POST

The body for the request should be as follows:

{
    "sha256": [
        <string>,
        ...
    ],
    "expiration_seconds": <int>
}

The following table explains the above data:

Item Data Type Description Required
sha256 array of strings An array of SHA-256 hashes as ASCII hex strings, with a limit of 100. More than 100 items will result in an HTTP 400. Yes
expiration_seconds integer The number of seconds to make the download URLs available for. The default is 3600 (1 hour). No

Response

The body for the response is as follows:

{
    "found": [
        {
            "sha256": <string>,
            "url": <string>
        },
       ...
    ],
    "not_found": [
        <string>,
       ...
    ],
    "error": [
        <string>,
       ...
    ]
}

The following table explains the above response:

Item Data Type Description
found array of JSON object Details about the files that were found to be available for download
found[ix].sha256 string SHA-256 hash of file that is available to be downloaded
found[ix].url string An AWS S3 presigned URL for this file. Perform a GET on this URL to download the file.
not_found array of strings The SHA-256 hashes that were not found.
error array of strings The SHA-256 hashes that had an intermittent error. These hashes should be retried.

CURL Example - Download File by Hash

Set AUTHENTICATION to your APIKEY/ConnectorID and the HOST, ORGID and TARGETSHA variables appropriately.

curl -XPOST https://"$HOST"/ubs/v1/orgs/"$ORGID"/file/_download  -H 'Content-Type: application/json' -H "X-Auth-Token: $AUTHENTICATION"  -d '{"sha256": ["87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776"]}'

{"found":[
    {"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
    "url":"https://cdc-file-storage-staging-us-east-1.s3.amazonaws.com/87/97/6f/34/30/cc/99/bc/93/9e/06/94/24/7c/07/59/96/1a/49/83/2b/87/21/8f/43/13/d6/fc/0b/c3/a7/76/87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776.zip?AWSAccessKeyId=ASIAQZRW24JUWSSITSS4&Signature=FUVo3yW9DTj8iAZckynCCAXg10U%3D&x-amz-security-token=FQoGZXIvYXdzELn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDNnE2ZGKw4sKvDQgzCK3A27%2BmyKzV%2BQcYEdnqUhW8WJ6L9GbiIsDL0rv6uK46DCkdwvMPYDdBoWitDaSkb6KXzbOpItfPDnnpPAjswzdeh8nr4LHjkcaZUtYBBw2L1W33Ee0A%2F7RgY%2Fw75nWtgbnamxWWlNq9kapELBV6v1ohJzAiq%2FwRgZKCNeGSToT%2BR0RVjo5wJFFoKM7C4eVfZxIZyaKi2eA4gMdrmDhXvhBi%2Bphdk0ER1%2FkW8Q9XvtePjvypKCc%2BDwAXNA0BNa4pmeqmOQofUzzBD2FPdbuucR0W5LG7ajYCCrM1S5LcbatwTmqtZ8HbVB7ZNX%2BSxuyb4yMYNnBj22bldsjmUTuY6Ya%2FokZVuCRoTqi59PVB9D3PB%2FD4dk5RuRnQaCzz0TSnaYFTN9qznbWAhF%2FynUiyDC%2FozX2b%2BbpNcawixTSrnp76rEj8OLYm8Yrl9DqPs14Tw8WanULirzgtcOwvrMBioR0PBIYbInKNEhWhzq2D%2FdMkqyAxcasbR%2BFN7CVzXAGXjsFE3yVnyXz7CTQaOfowj1HNoj%2FA3dIdLZzSiaDD1nj6r5e81V7OphQFFfp%2BFB5Io82mwBuw%2FcH8jooj7nV4wU%3D&Expires=1551201116"}]
    ,"not_found":[],
    "error":[]
}

Metadata APIs

This data is specific to the binary and is not a function of organization data.

This API returns all of the metadata for the specified binary identified by the SHA-256 hash.

Retrieve Metadata

RBAC Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud ubs.org.sha256 READ Majority of environments
VMware Cloud Services Platform _API.Unified.Binary.Store:ubs.Org.Sha256.read N/A - included in permission name Prod UK

Request

The following table describes the inputs required to utilize this API.

Description Value
Route <cbc-hostname>/ubs/<version_id>/orgs/{org_key}/sha256/<sha256>/metadata
HTTP verb GET
sha256 The requested SHA-256 hash to obtain metadata information.

Response

The body for the response is as follows:

{
    "sha256": <string>,
    "md5": <string>,
    "file_available": <boolean>,
    "available_file_size": <long>,
    "file_size": <long>,
    "os_type": <string>,
    "architecture": [
        <string>
    ],
    "lang_id": <int>,
    "charset_id": <int>,
    "internal_name": <string>,
    "product_name": <string>,
    "company_name": <string>,
    "copyright": <string>,
    "trademark": <string>,
    "file_description": <string>,
    "file_version": <string>,
    "comments": <string>,
    "original_filename": <string>,
    "product_description": <string>,
    "product_version": <string>,
    "private_build": <string>,
    "special_build": <string>
}

The following table explains the above response:

Item Data Type Description
sha256 string The SHA-256 hash of the file.
md5 string The MD5 hash of the file.
file_available boolean If true, the file is available for download.
available_file_size long The size of the file, that is available for download. If the file is unavailable the size will be zero. In the case of the null file, then the size will be zero and file_available will be true. Note that files may be truncated, before upload, which is why this number may be less that file_size.
file_size long The size of the actual file. This is the size of the file represented by this hash.
os_type string The OS that this file is designed for. This may contain one or more of the following values: WINDOWS, ANDROID, MAC, IOS, LINUX, and OTHER.
architecture array of string The set of architectures that this file was compiled for. This may contain one or more of the following values: none, x86, amd64, and arm64.
lang_id unsigned int The Language ID value from the Windows VERSIONINFO resource. See this link on MSDN for more information.
charset_id unsigned int The Character set ID value from the Windows VERSIONINFO resource. See this link on MSDN for more information.
internal_name nullable string If present, Internal name from FileVersionInformation
product_name nullable string If present, Product name from FileVersionInformation
company_name nullable string If present, Company name from FileVersionInformation
copyright nullable string If present, Legal copyright from FileVersionInformation
trademark nullable string If present, Legal trademark from FileVersionInformation
file_description nullable string If present, File description from FileVersionInformation
file_version nullable string If present, File version from FileVersionInformation
comments nullable string If present, comments from FileVersionInformation
original_filename nullable string If present, Original filename from FileVersionInformation
product_description nullable string If present, Product description from FileVersionInformation
product_version nullable string If present, Product version from FileVersionInformation
private_build nullable string If present, Private build from FileVersionInformation
special_build nullable string If present, Special build from FileVersionInformation

CURL Example - Get File Metadata

Set the TARGETSHA, ORGID, and HOST envars appropriately.

curl -H "X-Auth-Token: $AUTHENTICATION" https://$HOST/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/metadata
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"architecture":["amd64"],
"available_file_size":96672,
"charset_id":1200,
"comments":null,
"company_name":null,
"copyright":"\u00a9 Microsoft Corporation. All rights reserved.",
"file_available":true,
"file_description":"Runtime Broker",
"file_size":96672,
"file_version":"10.0.16299.15 (WinBuild.160101.0800)",
"internal_name":"RuntimeBroker.exe",
"lang_id":1033,
"md5":"bd4401441a21bf1abce6404f4231db4d",
"original_filename":"RuntimeBroker.exe",
"os_type":"WINDOWS",
"private_build":null,
"product_description":null,
"product_name":"Microsoft\u00ae Windows\u00ae Operating System",
"product_version":"10.0.16299.15",
"special_build":null,
"trademark":null
}

Summary APIs

These APIs return summarized information about a given SHA-256 hash for the target organization.

Device

This API returns a summarized view of the device details. Use this API to get an overview of the devices that executed the file.

RBAC Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud ubs.org.sha256 READ Majority of environments
VMware Cloud Services Platform _API.Unified.Binary.Store:ubs.Org.Sha256.read N/A - included in permission name Prod UK

Request

The following table describes the inputs required to utilize this API.

Description Value
Route <cbc-hostname>/ubs/<versionId>/orgs/{org_key}/sha256/<sha256>/summary/device
HTTP verb GET
sha256 The requested SHA-256 hash to obtain information for.

Response

The body for the response is as follows:

{
    "sha256": <string>,
    "num_devices": <int>,
    "first_seen_device_timestamp": <string>,
    "first_seen_device_id": <int>,
    "first_seen_device_name": <string>,
    "last_seen_device_timestamp": <string>,
    "last_seen_device_id": <int>,
    "last_seen_device_name": <string>
}

The following table explains the above response:

Item Data Type Description
sha256 string The SHA-256 hash of the file.
num_devices long The total number of devices, for this organization, that have observed this file.
first_seen_device_timestamp string The time that this file was first seen, for this organization.
first_seen_device_id long The Device ID of the device that first saw this file.
first_seen_device_name string The name of the device that first saw this file.
last_seen_device_timestamp string The time that this file was most recently seen, for this organization.
last_seen_device_id long The Device ID of the device that most recently saw this file.
last_seen_device_name string The name of the device that last saw this file.

CURL Example - Get Devices for SHA256

Set AUTHENTICATION, HOST, ORGID and TARGETSHA variables appropriately.

curl -H "X-Auth-Token: $AUTHENTICATION" https://"$HOST"/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/summary/device

{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"num_devices":3,
"first_seen_device_timestamp":"2018-12-10T15:55:51.828771Z",
"first_seen_device_id":716428,
"first_seen_device_name":"CRAIG_WIN10_X64",
"last_seen_device_timestamp":"2018-12-28T16:49:30.185259Z",
"last_seen_device_id":787514,
"last_seen_device_name":"CRAIG_VM_WIN10_X64"}

Signature

This API will return a summary of the observed digital signature results for a given SHA-256 hash. The digital signature information for a binary may vary from one machine to another based on a variety of factors, including the presence of an up-to-date signature catalog on the host, system clock variations, ability to reach OCSP servers, custom root trust anchors, and more. Therefore, the results are ordered by prevalence, such that the most observed signatures will be returned first. The number of results are configurable, up to a max of 100 entries. Digital signatures can be recorded in a separate file (known as a “catalog” file) or embedded inside of the binary itself (an “embedded” signature). This signature API will capture the results of the endpoint’s verification of the digital signature associated with a given SHA-256 hash, stating whether that signature validation was based on a catalog file or an embedded signature. It is not uncommon for a catalog signature to be frequently updated, such that this API can return a wide variety of sign_timestamp values for a given SHA-256 hash.

RBAC Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud ubs.org.sha256 READ Majority of environments
VMware Cloud Services Platform _API.Unified.Binary.Store:ubs.Org.Sha256.read N/A - included in permission name Prod UK

Request

The following table describes the inputs required to utilize this API.

Description Value
Route <cbc-hostname>/ubs/<versionId>/orgs/{org_key}/sha256/<sha256>/summary/signature?rows=<rows>
HTTP verb GET
sha256 The requested SHA-256 hash to obtain information for.
rows The number of results to return. The default is 5 and the max is 100.

Response

The body for the response is as follows:

{
    "sha256": <string>,
    "total_signatures_count": <int>,
    "signatures_count": <int>,
    "signatures": [
        {
            "count": <int>,
            "is_catalog_signature": <boolean>,
            "publisher_name": <string>,
            "issuer_name": <string>,
            "sign_timestamp": <string>,
            "sign_states": [
                {
                    "device_count": <int>,
                    "sign_state": [
                        <string>
                    ],
                    "first_seen_timestamp": <string>
                },
                ...
            ]
        },
        ...
    ]
}

The following table explains the above response:

Item Data Type Description
sha256 string The SHA-256 hash of the file.
total_signatures_count long The total number of signatures that have been observed by this organization for this file. This is the sum, over all devices that have executed this file, of the count of signatures for that file observed by each device.
signatures_count long The total number of unique signatures that have been observed, by this organization, for this file. This field represents only unique occurrences, not total existence. Assuming that each device observed the same signature details for the file, then the following should be true: total_signatures_count = (number of devices that executed file) * signatures_count
signatures array of JSON objects The signature details. The length of the array is limited based off the provided input parameter “rows” with a maximum of 100.
signatures[ix].count long The total number of occurrences where this signature was observed, by this organization, for this file. This field is similar to total_signatures_count; however, it represents the count at a per signature-level instead of at the file-level. The sum of signatures[ix].count (over all signatures) should be equal to total_signatures_count.
signatures[ix].is_catalog_signature boolean If true, then this signature is a catalog signature.
signatures[ix].publisher_name nullable string The name of the company who published the file, as recorded in the digital certificate.
signatures[ix].issuer_name nullable string The name of the company who issued the signature. This is the company who signed the certificate. This may be an intermediate or the root certificate authority.
signatures[ix].sign_timestamp nullable string The date which the signature was issued.
signatures[ix].sign_states array of JSON object The signature states for the signatures. The length of this array is unbounded and will be as big as needed to return all available signature states.
signatures[ix].sign_states[ix2].device_count long The number of devices, for this organization, that observed this signature, for this file, with this specific signature result.
signatures[ix].sign_states[ix2].sign_state array of strings The human readable signature states. This may contain one or more of the following values: invalid, signed, verified, not_signed, unknown, chained, trusted, os, and catalog_signed. Some of those values are mutually exclusive, e.g. not_signed and signed will never both appear in the same array. Important note - a state of signed does not mean that the signature is valid. Both signed and valid must be within the array to indicated that it is valid. A value of unknown is used when signature verification errors occur and is also the initial state of all files. A value of chained means that the signature chains to a locally trusted root certificate. A value of os means that the file is a component of the operating system. A value of catalog_signed means that the signature was in a catalog file instead of within the file. A valid of trusted means that the local machine trusts the signing chain.
signatures[ix].sign_states[ix2].first_seen_timestamp string The date which a device first executed a file with this signature and signature state.

CURL Example - Get Signature

curl -H "X-Auth-Token: $AUTHENTICATION" https://"$HOST"/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/summary/signature
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"total_signatures_count":51,
"signatures_count":1,
"signatures":[
    {"count":51,
    "is_catalog_signature":false,
    "publisher_name":"Microsoft Windows",
    "issuer_name":"Microsoft Windows Production PCA 2011",
    "sign_timestamp":"2017-09-29T04:30:49.456000Z",
    "sign_states":[{"device_count":51,
                    "sign_state":["signed","verified","trusted","os"],
                    "first_seen_timestamp":"2018-12-11T01:40:04.827586Z"}]
    }]
 }

File Path

This API will return a summary of the observed file paths. The results are ordered by prevalence, such that the most observed file path will be returned first. The number of results are configurable, up to a max of 100 entries.

Input

The following table describes the inputs required to utilize this API.

Description Value
Route <cbc-hostname>/ubs/<versionId>/orgs/{org_key}/sha256/<sha256>/summary/file_path?rows=<rows>
HTTP verb GET
sha256 The requested SHA-256 hash to obtain information for.
rows The number of results to return. The default is 5 and the max is 100.

RBAC Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud ubs.org.sha256 READ Majority of environments
VMware Cloud Services Platform _API.Unified.Binary.Store:ubs.Org.Sha256.read N/A - included in permission name Prod UK

Response

The body for the response is as follows:

{
    "sha256": <string>,
    "total_file_path_count": <int>,
    "file_path_count": <int>,
    "file_paths": [
        {
            "count": <int>,
            "file_path": <string>,
            "first_seen_timestamp": <string>
        },
        ...
    ]
}

The following table explains the above response:

Item Data Type Description
sha256 string The SHA-256 hash of the file.
total_file_path_count long The total number of file paths that have been observed, by this organization, for this file.
file_path_count long The total number of unique file paths that have been observed, by this organization, for this file.
file_paths array of JSON objects The file path details. The length of the array is limited based off the provided input parameter.
file_paths[ix].count long The number of occurrences where this signature was observed, by this organization, for this file.
file_paths[ix].file_path string The file path. Note that only the file path for the first execution for a particular device is captured, as such each device will only ever have a single file path.
file_paths[ix].first_seen_timestamp string The time that this file path was first observed.

CURL Example - Get FilePath information

Set the AUTHENTICATION, HOST, ORGID, and TARGETSHA variables appropriately.

curl -H "X-Auth-Token: $AUTHENTICATION" https://"$HOST"/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/summary/file_path

{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"total_file_path_count":3,
"file_path_count":1,
"file_paths":[{"count":3,"file_path":"c:\\windows\\system32\\runtimebroker.exe",
"first_seen_timestamp":"2018-12-10T15:55:51.828771Z"}]}
Last modified on June 21, 2022