The unified binary store (UBS) is a centralized service that is part of the CB Predictive Security Cloud (PSC). The UBS is responsible for storing all binaries and corresponding metadata for those binaries. The UBS comes packaged in with CB ThreatHunter. To obtain the maximum benefit of the UBS, users must opt-into binary uploads.
CB ThreatHunter customers are able to utilize the UBS APIs to download binaries as well as retrieve the corresponding metadata. At this time, the UBS’ data retention is infinite, so as long as a user has access to a binary, that user will be able to retrieve the file and any associated metadata.
The UBS obtains data on first execution of a PE for a given device that CB ThreatHunter has visibility into. This means that even if some of the metadata around the file changes (such as file path or signature results) the UBS will still only know the original state. Usage The UBS is a RESTful service, utilizing HTTP as the transport protocol. All routes to the UBS take the following format:
<psc-hostName>/ubs/<versionId>/orgs/<org_key>/…
The following table explains the components of the above route:
Variable | Description | Example |
---|---|---|
hostName | The host name. This should target the appropriate backend gateway serving the request. | https://defense-eap01.conferdeploy.net |
versionId | The version of the UBS API that is being targeted. This document specifically covers values of v1. | v1 |
org_key | The organization key. | ABCD123 |
Authentication is handled by use of the X-Auth-Token header. The value for that header should be the user’s API key and connector id separated by a forward slash (/). For more information on generating the X-Auth-Token header, see the Developer Network website at https://developer.carbonblack.com/reference/cb-defense/authentication/. Each of the UBS APIs additionally require providing an appropriate org_key. The provided API key must have permission to access the data for that particular org_key. If the user’s API key is invalid for the specified org_key an HTTP 403 response code will be returned.
The org_key may be obtained from the connectors page in the console.
POST routes must have the Content-Type header set to application/json.
All successful API calls will result in returning an HTTP 200 status code. The result will be returned in the body of the response. The specific responses will be detailed in subsequent sections, for each API.
Whenever an HTTP status code not equal to 200 is returned, the response will be as follows:
{
"message": <string>
"error_code": <string>
}
error_code
is an enum, with values that will be defined below.
This field may be used for programmatically handling errors.
message
is a short explanation as to why an error occurred.
This field is intended to be interpreted for debugging purposes, and not to be consumed programmatically.
The following table outlines the expected error HTTP status codes:
Status Code | Reason | error_code |
---|---|---|
400 | Invalid user input. Ensure that any SHA-256s are valid. | BAD_REQUEST |
403 | Authentication error. Ensure that both the API key and org_key are correct. Check the response message field for a more detailed error. |
FORBIDDEN |
404 | Requested resource not found. | |
500 | Server error. This is typically a transient error and the request may be retried, ideally with an exponential backoff. | INTERNAL_SERVER_ERROR |
Unless otherwise stated, all dates/timestamps will be strings, formatted as ISO 8601, i.e. YYYY-MM-DDThh:mm:ss.sssZ POST routes MUST have the conventional header: Content-Type: application/json or they will return a 500 error.
These APIs are specifically for interacting with files stored within the UBS.
This API provides a means to download files. The files are able to be downloaded via AWS S3 pre-signed URLs. The URLs enables the client to perform a GET on the provided URLs and download the files. The links will automatically expire after one hour, unless a different expiration time is requested.
CAUTION - Anyone who has these links will be able to download the files until the requested expiration time (default one hour after the request). Ensure that care is taken when utilizing this API.
The following table describes the inputs required to utilize this API.
Description | Value |
---|---|
Route | <psc-hostname>/ubs/<versionId>/orgs/<org_key>/file/_download |
HTTP verb | POST |
The body for the request should be as follows:
{
"sha256": [
<string>,
...
],
"expiration_seconds": <int>
}
The following table explains the above data:
Item | Data Type | Description | Required |
---|---|---|---|
sha256 | array of strings | An array of SHA-256 hashes as ASCII hex strings, with a limit of 100. More than 100 items will result in an HTTP 400. | Yes |
expiration_seconds | integer | The number of seconds to make the download URLs available for. The default is 3600 (1 hour). | No |
The body for the response is as follows:
{
"found": [
{
"sha256": <string>,
"url": <string>
},
...
],
"not_found": [
<string>,
...
],
"error": [
<string>,
...
]
}
The following table explains the above response:
Item | Data Type | Description |
---|---|---|
found | array of JSON object | Details about the files that were found to be available for download |
found[ix].sha256 | string | SHA-256 hash of file that is available to be downloaded |
found[ix].url | string | An AWS S3 presigned URL for this file. Perform a GET on this URL to download the file. |
not_found | array of strings | The SHA-256 hashes that were not found. |
error | array of strings | The SHA-256 hashes that had an intermittent error. These hashes should be retried. |
Set AUTHENTICATION to your APIKEY/ConnectorID and the HOST, ORGID and TARGETSHA variables appropriately.
curl -XPOST https://"$HOST"/ubs/v1/orgs/"$ORGID"/file/_download -H 'Content-Type: application/json' -H "X-Auth-Token: $AUTHENTICATION" -d '{"sha256": ["87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776"]}'
{"found":[
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"url":"https://cdc-file-storage-staging-us-east-1.s3.amazonaws.com/87/97/6f/34/30/cc/99/bc/93/9e/06/94/24/7c/07/59/96/1a/49/83/2b/87/21/8f/43/13/d6/fc/0b/c3/a7/76/87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776.zip?AWSAccessKeyId=ASIAQZRW24JUWSSITSS4&Signature=FUVo3yW9DTj8iAZckynCCAXg10U%3D&x-amz-security-token=FQoGZXIvYXdzELn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDNnE2ZGKw4sKvDQgzCK3A27%2BmyKzV%2BQcYEdnqUhW8WJ6L9GbiIsDL0rv6uK46DCkdwvMPYDdBoWitDaSkb6KXzbOpItfPDnnpPAjswzdeh8nr4LHjkcaZUtYBBw2L1W33Ee0A%2F7RgY%2Fw75nWtgbnamxWWlNq9kapELBV6v1ohJzAiq%2FwRgZKCNeGSToT%2BR0RVjo5wJFFoKM7C4eVfZxIZyaKi2eA4gMdrmDhXvhBi%2Bphdk0ER1%2FkW8Q9XvtePjvypKCc%2BDwAXNA0BNa4pmeqmOQofUzzBD2FPdbuucR0W5LG7ajYCCrM1S5LcbatwTmqtZ8HbVB7ZNX%2BSxuyb4yMYNnBj22bldsjmUTuY6Ya%2FokZVuCRoTqi59PVB9D3PB%2FD4dk5RuRnQaCzz0TSnaYFTN9qznbWAhF%2FynUiyDC%2FozX2b%2BbpNcawixTSrnp76rEj8OLYm8Yrl9DqPs14Tw8WanULirzgtcOwvrMBioR0PBIYbInKNEhWhzq2D%2FdMkqyAxcasbR%2BFN7CVzXAGXjsFE3yVnyXz7CTQaOfowj1HNoj%2FA3dIdLZzSiaDD1nj6r5e81V7OphQFFfp%2BFB5Io82mwBuw%2FcH8jooj7nV4wU%3D&Expires=1551201116"}]
,"not_found":[],
"error":[]
}
This data is specific to the binary and is not a function of organization data. Retrieve Metadata for a Binary This API returns all of the metadata for the specified binary identified by the SHA-256 hash.
The following table describes the inputs required to utilize this API.
Description | Value |
---|---|
Route | <psc-hostname>/ubs/<version_id>/orgs/<org_key>/sha256/<sha256>/metadata |
HTTP verb | GET |
sha256 | The requested SHA-256 hash to obtain metadata information. |
The body for the response is as follows:
{
"sha256": <string>,
"md5": <string>,
"file_available": <boolean>,
"available_file_size": <long>,
"file_size": <long>,
"os_type": <string>,
"architecture": [
<string>
],
"lang_id": <int>,
"charset_id": <int>,
"internal_name": <string>,
"product_name": <string>,
"company_name": <string>,
"copyright": <string>,
"trademark": <string>,
"file_description": <string>,
"file_version": <string>,
"comments": <string>,
"original_filename": <string>,
"product_description": <string>,
"product_version": <string>,
"private_build": <string>,
"special_build": <string>
}
The following table explains the above response:
Item | Data Type | Description |
---|---|---|
sha256 | string | The SHA-256 hash of the file. |
md5 | string | The MD5 hash of the file. |
file_available | boolean | If true, the file is available for download. |
available_file_size | long | The size of the file, that is available for download. If the file is unavailable the size will be zero. In the case of the null file, then the size will be zero and file_available will be true. Note that files may be truncated, before upload, which is why this number may be less that file_size . |
file_size | long | The size of the actual file. This is the size of the file represented by this hash. |
os_type | string | The OS that this file is designed for. This may contain one or more of the following values: WINDOWS, ANDROID, MAC, IOS, LINUX, and OTHER. |
architecture | array of string | The set of architectures that this file was compiled for. This may contain one or more of the following values: none, x86, amd64, and arm64. |
lang_id | unsigned int | The Language ID value from the Windows VERSIONINFO resource. See this link on MSDN for more information. |
charset_id | unsigned int | The Character set ID value from the Windows VERSIONINFO resource. See this link on MSDN for more information. |
internal_name | nullable string | If present, Internal name from FileVersionInformation |
product_name | nullable string | If present, Product name from FileVersionInformation |
company_name | nullable string | If present, Company name from FileVersionInformation |
copyright | nullable string | If present, Legal copyright from FileVersionInformation |
trademark | nullable string | If present, Legal trademark from FileVersionInformation |
file_description | nullable string | If present, File description from FileVersionInformation |
file_version | nullable string | If present, File version from FileVersionInformation |
comments | nullable string | If present, comments from FileVersionInformation |
original_filename | nullable string | If present, Original filename from FileVersionInformation |
product_description | nullable string | If present, Product description from FileVersionInformation |
product_version | nullable string | If present, Product version from FileVersionInformation |
private_build | nullable string | If present, Private build from FileVersionInformation |
special_build | nullable string | If present, Special build from FileVersionInformation |
Set the TARGETSHA, ORGID, and HOST envars appropriately.
curl -H "X-Auth-Token: $AUTHENTICATION" https://$HOST/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/metadata
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"architecture":["amd64"],
"available_file_size":96672,
"charset_id":1200,
"comments":null,
"company_name":null,
"copyright":"\u00a9 Microsoft Corporation. All rights reserved.",
"file_available":true,
"file_description":"Runtime Broker",
"file_size":96672,
"file_version":"10.0.16299.15 (WinBuild.160101.0800)",
"internal_name":"RuntimeBroker.exe",
"lang_id":1033,
"md5":"bd4401441a21bf1abce6404f4231db4d",
"original_filename":"RuntimeBroker.exe",
"os_type":"WINDOWS",
"private_build":null,
"product_description":null,
"product_name":"Microsoft\u00ae Windows\u00ae Operating System",
"product_version":"10.0.16299.15",
"special_build":null,
"trademark":null
}
These APIs return summarized information about a given SHA-256 hash for the target organization.
This API returns a summarized view of the device details. Use this API to get an overview of the devices that executed the file.
The following table describes the inputs required to utilize this API.
Description | Value |
---|---|
Route | <psc-hostname>/ubs/<versionId>/orgs/<org_key>/sha256/<sha256>/summary/device |
HTTP verb | GET |
sha256 | The requested SHA-256 hash to obtain information for. |
The body for the response is as follows:
{
"sha256": <string>,
"num_devices": <int>,
"first_seen_device_timestamp": <string>,
"first_seen_device_id": <int>,
"first_seen_device_name": <string>,
"last_seen_device_timestamp": <string>,
"last_seen_device_id": <int>,
"last_seen_device_name": <string>
}
The following table explains the above response:
Item | Data Type | Description |
---|---|---|
sha256 | string | The SHA-256 hash of the file. |
num_devices | long | The total number of devices, for this organization, that have observed this file. |
first_seen_device_timestamp | string | The time that this file was first seen, for this organization. |
first_seen_device_id | long | The Device ID of the device that first saw this file. |
first_seen_device_name | string | The name of the device that first saw this file. |
last_seen_device_timestamp | string | The time that this file was most recently seen, for this organization. |
last_seen_device_id | long | The Device ID of the device that most recently saw this file. |
last_seen_device_name | string | The name of the device that last saw this file. |
Set AUTHENTICATION, HOST, ORGID and TARGETSHA variables appropriately.
curl -H "X-Auth-Token: $AUTHENTICATION" https://"$HOST"/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/summary/device
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"num_devices":3,
"first_seen_device_timestamp":"2018-12-10T15:55:51.828771Z",
"first_seen_device_id":716428,
"first_seen_device_name":"CRAIG_WIN10_X64",
"last_seen_device_timestamp":"2018-12-28T16:49:30.185259Z",
"last_seen_device_id":787514,
"last_seen_device_name":"CRAIG_VM_WIN10_X64"}
This API will return a summary of the observed digital signature results for a given SHA-256 hash. The digital signature information for a binary may vary from one machine to another based on a variety of factors, including the presence of an up-to-date signature catalog on the host, system clock variations, ability to reach OCSP servers, custom root trust anchors, and more. Therefore, the results are ordered by prevalence, such that the most observed signatures will be returned first. The number of results are configurable, up to a max of 100 entries. Digital signatures can be recorded in a separate file (known as a “catalog” file) or embedded inside of the binary itself (an “embedded” signature). This signature API will capture the results of the endpoint’s verification of the digital signature associated with a given SHA-256 hash, stating whether that signature validation was based on a catalog file or an embedded signature.
The following table describes the inputs required to utilize this API.
Description | Value |
---|---|
Route | <psc-hostname>/ubs/<versionId>/orgs/<org_key>/sha256/<sha256>/summary/signature?rows=<rows> |
HTTP verb | GET |
sha256 | The requested SHA-256 hash to obtain information for. |
rows | The number of results to return. The default is 5 and the max is 100. |
The body for the response is as follows:
{
"sha256": <string>,
"total_signatures_count": <int>,
"signatures_count": <int>,
"signatures": [
{
"count": <int>,
"is_catalog_signature": <boolean>,
"publisher_name": <string>,
"issuer_name": <string>,
"sign_timestamp": <string>,
"sign_states": [
{
"device_count": <int>,
"sign_state": [
<string>
],
"first_seen_timestamp": <string>
},
...
]
},
...
]
}
The following table explains the above response:
Item | Data Type | Description |
---|---|---|
sha256 | string | The SHA-256 hash of the file. |
total_signatures_count | long | The total number of signatures that have been observed by this organization for this file. This is the sum, over all devices that have executed this file, of the count of signatures for that file observed by each device. |
signatures_count | long | The total number of unique signatures that have been observed, by this organization, for this file. This field represents only unique occurrences, not total existence. Assuming that each device observed the same signature details for the file, then the following should be true: total_signatures_count = (number of devices that executed file) * signatures_count |
signatures | array of JSON objects | The signature details. The length of the array is limited based off the provided input parameter “rows” with a maximum of 100. |
signatures[ix].count | long | The total number of occurrences where this signature was observed, by this organization, for this file. This field is similar to total_signatures_count; however, it represents the count at a per signature-level instead of at the file-level. The sum of signatures[ix].count (over all signatures) should be equal to total_signatures_count. |
signatures[ix].is_catalog_signature | boolean | If true, then this signature is a catalog signature. |
signatures[ix].publisher_name | nullable string | The name of the company who published the file, as recorded in the digital certificate. |
signatures[ix].issuer_name | nullable string | The name of the company who issued the signature. This is the company who signed the certificate. This may be an intermediate or the root certificate authority. |
signatures[ix].sign_timestamp | nullable string | The date which the signature was issued. |
signatures[ix].sign_states | array of JSON object | The signature states for the signatures. The length of this array is unbounded and will be as big as needed to return all available signature states. |
signatures[ix].sign_states[ix2].device_count | long | The number of devices, for this organization, that observed this signature, for this file, with this specific signature result. |
signatures[ix].sign_states[ix2].sign_state | array of strings | The human readable signature states. This may contain one or more of the following values: invalid, signed, verified, not_signed, unknown, chained, trusted, os, and catalog_signed. Some of those values are mutually exclusive, e.g. not_signed and signed will never both appear in the same array. Important note - a state of signed does not mean that the signature is valid. Both signed and valid must be within the array to indicated that it is valid. A value of unknown is used when signature verification errors occur and is also the initial state of all files. A value of chained means that the signature chains to a locally trusted root certificate. A value of os means that the file is a component of the operating system. A value of catalog_signed means that the signature was in a catalog file instead of within the file. A valid of trusted means that the local machine trusts the signing chain. |
signatures[ix].sign_states[ix2].first_seen_timestamp | string | The date which a device first executed a file with this signature and signature state. |
curl -H "X-Auth-Token: $AUTHENTICATION" https://"$HOST"/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/summary/signature
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"total_signatures_count":51,
"signatures_count":1,
"signatures":[
{"count":51,
"is_catalog_signature":false,
"publisher_name":"Microsoft Windows",
"issuer_name":"Microsoft Windows Production PCA 2011",
"sign_timestamp":"2017-09-29T04:30:49.456000Z",
"sign_states":[{"device_count":51,
"sign_state":["signed","verified","trusted","os"],
"first_seen_timestamp":"2018-12-11T01:40:04.827586Z"}]
}]
}
This API will return a summary of the observed file paths. The results are ordered by prevalence, such that the most observed file path will be returned first. The number of results are configurable, up to a max of 100 entries.
The following table describes the inputs required to utilize this API.
Description | Value |
---|---|
Route | |
HTTP verb | GET |
sha256 | The requested SHA-256 hash to obtain information for. |
rows | The number of results to return. The default is 5 and the max is 100. |
The body for the response is as follows:
{
"sha256": <string>,
"total_file_path_count": <int>,
"file_path_count": <int>,
"file_paths": [
{
"count": <int>,
"file_path": <string>,
"first_seen_timestamp": <string>
},
...
]
}
The following table explains the above response:
Item | Data Type | Description |
---|---|---|
sha256 | string | The SHA-256 hash of the file. |
total_file_path_count | long | The total number of file paths that have been observed, by this organization, for this file. |
file_path_count | long | The total number of unique file paths that have been observed, by this organization, for this file. |
file_paths | array of JSON objects | The file path details. The length of the array is limited based off the provided input parameter. |
file_paths[ix].count | long | The number of occurrences where this signature was observed, by this organization, for this file. |
file_paths[ix].file_path | string | The file path. Note that only the file path for the first execution for a particular device is captured, as such each device will only ever have a single file path. |
file_paths[ix].first_seen_timestamp | string | The time that this file path was first observed. |
Set the AUTHENTICATION, HOST, ORGID, and TARGETSHA variables appropriately.
curl -H "X-Auth-Token: $AUTHENTICATION" https://"$HOST"/ubs/v1/orgs/"$ORGID"/sha256/"$TARGETSHA"/summary/file_path
{"sha256":"87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776",
"total_file_path_count":3,
"file_path_count":1,
"file_paths":[{"count":3,"file_path":"c:\\windows\\system32\\runtimebroker.exe",
"first_seen_timestamp":"2018-12-10T15:55:51.828771Z"}]}