Process Analysis API Reference v7.5+

Carbon Black EDR (Endpoint Detection and Response) is the new name for the product formerly called CB Response.

The 7.5.0 release of VMware Carbon Black EDR introduces a new set of API endpoints (v6/process) to assist in the analysis of processes and their events. The endpoints share a new strategy to load, process, and return small portions of process data at one time. The strategy allows greater efficiency. On average the endpoints are faster and more predictable. There is a tradeoff in API usage. The new endpoints work differently as a whole compared to process/v5. Your code will need to adapt to the new conventions.

Stream API

Process events are now viewed using the metaphor of a stream, much like streaming video online. The whole timeline of events exists on the server, not all events are viewed at once. A stream tracks your position as you view individual pages of events. You can seek forward and backward by any number of events. You can also jump forward and backward by any number of process segments. If you know a particular segment id, you can jump directly to it.

Streams are stateful. They must be created, used, then deleted. Streams are referenced by their STREAM_ID.

Create a Stream

To create a stream, you need a PROCESS_ID and SEGMENT_ID. By default, the new stream will expire after StreamTTLSecs (a cb.conf option). You may specify persistent: true to prevent the stream from expiring automatically. Streams do not store much state - only a handful of fields are stored to track stream progress. Nevertheless, it is a good idea to delete streams when you are done with them.

Request

curl -X POST "$SERVER/api/v6/process/stream" \
  -H "X-Auth-Token: $TOKEN"  \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d"
{
  \"process_id\": \"$PROCESS_ID\",
  \"segment_id\": \"$SEGMENT_ID\",
  \"persistent\": true  # or false
}"

Response

{
  "stream_id": "727cdaed-fc91-4f48-a00b-08dfe6aec4da",
  "status_fw": "ready",
  "status_bw": "ready",
  "ref_segment_id": 1621882944903,
  "cur_segment_id": 1621882944903,
  "head_as_ref_segment_offset": 0,
  "head_as_cur_segment_offset": 0,
  "events": []
}

Use a Stream

Once you have obtained a STREAM_ID, you can perform several operations. First, consider your current position. When you create a stream, the initial position will be the first event of the segment you provided. If you request ten rows, you’ll get back ten rows starting from your current position. The position does not move unless you perform a seek or jump operation. If you seek by 10 rows and request 10 rows in one call, you’ll move the stream position forward by ten rows, and then get back ten rows starting from your new position.

Several status fields are returned after each operation. status_fw/status_bw are of particular interest. For instance, status_fw will be “ready” if you could perform the same operation again without reaching the end of the process. It will be “current” if you have reached the end of the process. It will be “complete” if you have reached the end of the process, and the process is terminated (there will never be more events). In the latter two cases you may retrieve fewer events than you requested. The status will be “void” if the operation moved the position out of bounds, e.g. there are only 100 events but you tried to seek forward by one thousand.

Requests

# View ten rows
curl -X GET "$SERVER/api/v6/process/stream/$STREAM_ID?rows=10"\
  -H "X-Auth-Token: $TOKEN"

# Seek forward by ten rows
curl -X GET "$SERVER/api/v6/process/stream/$STREAM_ID?seek&by=10&rows=10" \
  -H "X-Auth-Token: $TOKEN"

# Seek backward by ten rows
curl -X GET "$SERVER/api/v6/process/stream/$STREAM_ID?seek&by=-10&rows=10" \
  -H "X-Auth-Token: $TOKEN"

# Jump forward by one segment
curl -X GET "$SERVER/api/v6/process/stream/$STREAM_ID?jump&by=1&rows=10" \
  -H "X-Auth-Token: $TOKEN"

# Jump backward by one segment
curl -X GET "$SERVER/api/v6/process/stream/$STREAM_ID?jump&by=-1&rows=10" \
  -H "X-Auth-Token: $TOKEN"

# Jump to a specific segment
curl -X GET "$SERVER/api/v6/process/stream/$STREAM_ID?jump&to=$SEGMENT_ID&rows=10" \
  -H "X-Auth-Token: $TOKEN"

Sometimes you want to view events of a certain type, or events that mach certain criteria. You may use the following PATCH request to modify the criteria that are used to filter events in the stream. The criteria structure may specify fields mentioned in Facet Fields below. Multiple values per field are allowed.

# Filter a Stream
curl -X PATCH "$SERVER/api/v6/process/stream/$STREAM_ID?&rows=10" \
-H "Content-Type: application/json" \
-H "Accept: application/json"  \
-d '
{
  "criteria": {
    "event_type": [
      "filemod"
    ],
    "directory": [
      "/var/run",
      "/tmp"
    ]
  }
}'

The query argument end_time may be used with the above requests to omit all events with a timestamp occurring after the specified time.

end_time=YYYY-MM-DDThh:mm:ss.sssZ

Data will be returned in the following format. Events may contain additional binary info, alliance hit information, and on demand hit information if threat intelligence has been configured appropriately.

Response

{
  "stream_id": "508c8e5b-5ffd-40a0-97cc-b31414298667",
  "status_fw": "ready",
  "status_bw": "ready",
  "ref_segment_id": 1621882944903,
  "cur_segment_id": 1621882944903,
  "head_as_ref_segment_offset": 0,
  "head_as_cur_segment_offset": 0,
  "events": [
    {
      "type": "childproc",
      "time": "2021-05-24 19:01:56.564000",
      "data": {
        "md5": "b6fe756bf9828f05bd6c4d300f93ab91",
        "path": "/usr/libexec/nm-dispatcher",
        "commandLine": "",
        "userName": "",
        "sha256": "",
        "pid": 26885,
        "processId": "00000001-0000-6905-01d7-50cf4455dc1e-000000000001",
        "is_tampered": false,
        "type": "start",
        "start": "2021-05-24T19:01:56.564000Z"
      },
      "segment_id": 1621882944903,
      "binary_info": {
        "digsig_result": null,
        "digsig_publisher": null
      },
      "alliance_hits": null,
      "ondemand_feeds": null,
      "ondemand_classification": null
    }
  ]
}

Delete a Stream

You should take care to delete persistent streams you create. You may also delete transient streams (those that expire due to being created with “persistent”: false).

Request

curl -X POST "$SERVER/api/v6/process/stream/$STREAM_ID" \
  -H "X-Auth-Token: $TOKEN" \
  -H "Accept: application/json"

Response

{
  "result": "success"
}

Summary Stream API

The summary stream API can be used to obtain an overview of the event types represented in the whole process. It works similarly to the v5 route of the same name with one difference. Batches of segment summary info are streamed from the server in a single HTTP request using the ndjson format. You can handle each incoming batch while the request is ongoing. (If you are using python, the requests library can be configured using streaming requests).

Request

# Retrieve batches of summary data pertaining to the segments of a process
curl -X GET "$SERVER/api/v6/process/$PROCESS_ID/$SEGMENT_ID/segment" \
-X "Accept: application/x-ndjson"

Response

Notice each line is a JSON document with its own batch of segment summary data.

{"elapsed":2.1457672119140625e-06,"process":{"segments":[{"unique_id":"00000001-0000-0001-01d6-eea15d60d4c4-0179b1c541a3","segment_id":1622185165219,"last_update":"2021-05-28T06:59:12.748Z","last_server_update":"2021-05-28T06:59:25.288Z","event_counts":{"filemod":7}},{"unique_id":"00000001-0000-0001-01d6-eea15d60d4c4-0179b1c62cce","segment_id":1622185225422,"last_update":"2021-05-28T07:00:01.775Z","last_server_update":"2021-05-28T07:00:25.492Z","event_counts":{"filemod":35}}]}}
{"elapsed":2.6226043701171875e-06,"process":{"segments":[{"unique_id":"00000001-0000-0001-01d6-eea15d60d4c4-0179b257a6ac","segment_id":1622194759340,"last_update":"2021-05-28T09:38:59.097Z","last_server_update":"2021-05-28T09:39:19.424Z","event_counts":{"childproc":1}}]}}

Bulk API

The bulk API provides information about large portions of a process. It consolidates two previous features: alliance feed hits and on demand feed hits. It introduces one new feature: event faceting across all events. Some limitations have been introduced to ensure these features can be used reliably.

  • Processing begins at the specified SEGMENT_ID
  • Processing ends after a configurable timeout (BulkDataTimeoutSecs in cb.conf)
  • Processing ends if an end_time is provided (see end_time format specification above)

The API response will indicate if the result is partial due to timeout. Reaching the specified end_time is not considered a timeout. The API may handle a process with few segments in one call. A larger process with thousands of segments may need to be processed in chunks. In fact the EDR Console uses a combination of SEGMENT_ID and END_TIME to limit processing to the timeline window selection.

Request

curl -X GET "$SERVER/api/v6/process/$PROCESS_ID/$SEGMENT_ID/bulk?end_time=$END_TIME" \
  -H "X-Auth-Token: $TOKEN" \
  -H "Accept: application/json"

Response

{
  "alliance_hits": {...},
  "on_demand_hits": {
    "threatintel": {...}
  },
  "facets": {
    "terms": [...],
    "num_found": n,
    "processed_segments": n
  },
  "partial_results": false
}

The following is an example of what can be expected in facets:

{
  "terms": [
    {
      "field": "event_type",
      "values": [
        {
          "id": "modload",
          "name": "modload",
          "total": 46
        },
        {
          "id": "crossproc",
          "name": "crossproc",
          "total": 2
        }
      ]
    },
    {
      "field": "directory",
      "values": [
        {
          "id": "c:\\windows\\syswow64",
          "name": "c:\\windows\\syswow64",
          "total": 40
        },
        {
          "id": "c:\\windows\\system32",
          "name": "c:\\windows\\system32",
          "total": 6
        },
        {
          "id": "c:\\users\\user1\\downloads\\dir_1\\file_x",
          "name": "c:\\users\\user1\\downloads\\dir_1\\file_x",
          "total": 1
        },
        {
          "id": "c:\\windows\\dir_2\\file_y",
          "name": "c:\\windows\\dir_2\\file_y",
          "total": 1
        }
      ]
    }
  ],
  "num_found": 48,
  "processed_segments": 1
}

There are two properties in addition to terms: num_found indicates the number of filterable items found and processed_segments is the number of segments that were searched to find all the items.

Facet Fields

The following are valid values for the event_type field: filemod, childproc, netconn, regmod, processblock. Note that childproc includes events of types exec, fork, and other_exec as well as childproc.

The following fields are possible for each event type:

  • filemod
    • directory
    • filemod_action
    • filemod_filetype
  • childproc
    • directory
    • childproc_filepath
    • childproc_md5
    • childproc_sha256
  • netconn
    • ipv4_address
    • ipv6_address
    • domain
    • ja3_fingerprint
    • ja3s_fingerprint
  • regmod
    • regmod_action
    • regmod_hive
  • processblock
    • directory
  • crossproc
    • directory
  • modload
    • directory

Metadata API

The metadata API will return all non-event fields of the process.

Request

curl -X GET "$SERVER/api/v6/process/$PROCESS_ID/metadata" \
  -H "X-Auth-Token: $TOKEN" \
  -H "Accept: application/json"

Response

{
  "process_metadata": {
    "unique_id": "00000008-0000-1778-01d7-531d279b5c07-0179aedb558a",
    "sensor_id": 8,
    "hostname": "tl-750-rhel83-w",
    "group": "default group",
    "os_type": "windows",
    "host_type": "workstation",
    "interface_ip": 181100788,
    "comms_ip": 181100788,
    "parent_unique_id": "00000008-0000-04b0-01d7-4ccd83f8de79-000000000001",
    "start": "2021-05-27T17:24:31.436Z",
    "day_of_week": 4,
    "hour_of_day": 13,
    "process_pid": [
      6008
    ],
    "parent_pid": 1200,
    "process_md5": "ccee2e1a4672697747d199fb7e0caa37",
    "process_sha256": "f75345665bf69e4a5beb0525b80b714f1b2d8b25a7ece8c88f48607b46f5d7f9",
    "cmdline": [
      "\"C:\\Users\\user_1\\Downloads\\dir_1\\dir_2\\file_x.exe\" "
    ],
    "uid": "s-1-5-21-601375543-4271196527-4255210300-1001",
    "username": [
      "TL-750-RHEL83-W\\user_1"
    ],
    "parent_name": "explorer.exe",
    "path": "c:\\users\\user_1\\downloads\\dir_1\\dir_2\\file_x.exe",
    "process_name": "file_x.exe",
    "logon_type": 2,
    "creation_observed": false,
    "alliance_score_srsthreat": 100,
    "alliance_link_srsthreat": "https://services.bit9.com/Services/extinfo.aspx?ak=b8b4e631d4884ad1c56f50e4a5ee9279&sg=0313e1735f6cec221b1d686bd4de23ee&md5=ccee2e1a4672697747d199fb7e0caa37",
    "alliance_data_srsthreat": [
      "ccee2e1a4672697747d199fb7e0caa37"
    ],
    "alliance_updated_srsthreat": "2021-05-26T23:19:19Z",
    "facet_id": 201654,
    "last_update": "2021-05-27T17:24:31.518Z",
    "last_server_update": "2021-05-27T17:24:40.553Z",
    "terminated": false,
    "server_added_timestamp": "2021-05-27T17:24:40.553Z",
    "id": "00000008-0000-1778-01d7-531d279b5c07",
    "parent_id": "00000008-0000-04b0-01d7-4ccd83f8de79",
    "segment_id": 1622136280458,
    "filtering_known_dlls": false,
    "tampered": false,
    "has_emet_event": false,
    "has_emet_config": false
  }
}

Segment Verification API

Several v6 API endpoints require valid segment IDs. This API may be used to verify the existence of those segment IDs. If the provided SEGMENT_ID exists, it will be returned unchanged.

Request

curl -X GET "$SERVER/api/v6/process/$PROCESS_ID/$SEGMENT_ID/verify" \
  -H "X-Auth-Token: $TOKEN" \
  -H "Accept: application/json"

Response

{
  "segment_id": $SEGMENT_ID
}

If the provided SEGMENT_ID does not exist, the id of the earliest available segment will be returned instead.

Request

curl -X GET "$SERVER/api/v6/process/$PROCESS_ID/$INVALID_SEGMENT_ID/verify" \
  -H "X-Auth-Token: $TOKEN" \
  -H "Accept: application/json"

Response

{
  "segment_id": $EARLIEST_SEGMENT_ID
}

Event Suppression Status API

You may encounter a childproc event while browsing an event stream. To determine if the childproc is suppressed (or not), use this API endpoint. This operation will timeout after SuppressionStatusCheckTimeoutSecs (a cb.conf option).

Request

curl -X GET "$SERVER/api/v6/process/suppression_status?parent_guid=$PARENT_GUID&childproc_guid=$CHILDPROC_GUID&childproc_md5=$CHILDPROC_MD5" \
  -H "X-Auth-Token: $TOKEN" \
  -H "Accept: application/json"

Response

{
  "is_suppressed": false,
  "timeout": false
}

Last modified on June 14, 2021