Job Service API

Introduction

The Job Service API tracks the execution of long running tasks making it easier to monitor progress of asynchronous jobs and retrieve the jobs’ output once completed and prevents important tasks from timing out mid execution. It can be used with other APIs to asynchronously export large amounts of data including

More about this API:

  • Long running or slow jobs tracked through the jobs service will complete without timing out
  • Downloads can be repeated without having to run the underlying job multiple times
  • Jobs can only be seen by the connector or user who created them
  • Jobs are saved for 30 days

Requirements

  • Any of the Carbon Black Cloud products
  • All API calls require an API key with appropriate permissions, see Authentication for details
  • Must have a Job Investigate Page Export or Live Query to get the status of other asynchronous jobs

Resources

Authentication

Determine whether you use Carbon Black Cloud or VMware Cloud Services Platform to manage identity and authorization, or see the Carbon Black Cloud API Access Guide for complete instructions.


Carbon Black Cloud Managed Identity and Authentication
Customize your access to the Carbon Black Cloud APIs with Role-Based Access Control; All APIs and Services authenticate via API Keys. To access the data in Carbon Black Cloud via API, you must set up a key with the correct permissions for the calls you want to make and pass it in the HTTP Headers.

Environment
Available on majority of environments; Use the Carbon Black Cloud Console URL, as described here.

API Route
Replace the {cbc-hostname} and {org_key} with the URL of your Environment and the org_key for your specific Org.
  • Jobs: {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs/

Access Level
Before you create your API Key, you need to create a "Custom" Access Level including each category:
  • Background Tasks > Status > jobs.status, allow permission to READ
  • Search > Events > org.search.events, allow permission to CREATE

API Key
When creating your API Key, use the Access Level Type of "Custom" and select the Access Level you created. Details on constructing and passing the API Key in your requests are available here.


Cloud Services Platform Managed Identity and Authentication
Customize your access to the Carbon Black Cloud APIs with OAuth Access Control; API access is controlled using OAuth apps or User API Tokens. This is currently limited to the UK Point of Presence and AWS GovCloud (US).

Environment
Available on Prod UK and AWS GovCloud (US). Full list of environments is available here; Use the Carbon Black Cloud Console URL from Cloud Services Platform, as described here.

API Route
Replace the {cbc-hostname} and {org_key} with the URL of your Environment and the org_key for your specific Org.
  • Jobs: {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs/

Access Level
Before you create your OAuth App, you need to create a custom Role with the following permissions under IDENTITY & ACCESS MANAGEMENT > Roles > VMware Carbon Black Cloud:
  • _API.Background.Tasks:jobs.Status, allow permission to READ
  • _API.Search:org.events, allow permission to CREATE

API Authentication
The Cloud Services Platform supports several authentication options, Access Token, API Token, and for backward compatibility, X-Auth-Token. To learn about the differences or how to use the authentication methods see the Authentication Guide.


Quick Setup

Typical use of the API follows this sequence:

  1. Call a route in a service that utilizes the job service (eg. /jobs/v1/orgs/{org_key}/jobs/start/event_export or /livequery/v1/orgs/{org_key}/runs/{id}/results/_search?format=csv&async=true)
  2. Receive the job ID
  3. Call /jobs/v1/orgs/{org_key}/jobs/{job_id}/progress route to get job progress
  4. Repeat 3 until the status shows COMPLETED (or FAILED)
  5. If successful, call the /jobs/v1/orgs/{org_key}/jobs/{job_id}/download route to retrieve the output of the job

Helpful tips:

  • Use the /jobs/{job_id}/progress route to check the progress of long running exports. Shorter jobs may not update the progress before they complete.
  • You cannot call GET /jobs/{job_id}/status. You can retrieve status from the GET /jobs/{job_id} route
  • All time stamps are formatted as ISO 8601 UTC, i.e. YYYY-MM-DDThh:mm:ss.sssZ. This includes create_time, last_update_time, and time_range.

Start an Event Export Job

Used with investigate searches - Processes, Process Events, Observations, Auth Events and Enriched Events - to start a search job.

Then use Get Job Progress to determine when the job is complete followed by Download Job Output to get the results in a zipped csv file.


API Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud jobs.status READ Majority of environments
Carbon Black Cloud org.search.events CREATE Majority of environments
VMware Cloud Services Platform _API.Background.Tasks:jobs.Status.READ N/A - included in permission name Prod UK and AWS GovCloud (US)
VMware Cloud Services Platform _API.Search:org.events:CREATE N/A - included in permission name Prod UK and AWS GovCloud (US)


Request 
POST {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs/start/event_export


Request Body - application/json 

{
    "api_resource": "<string>",
    "version": "v2",
    "query": {
        "criteria": { "<key>": ["<value>"] },
        "exclusions": { "<key>": ["<value>"] },
        "fields": ["<string>", "<string>"],
        "query": "<string>",
        "rows": <long>,
        "sort": [
            {
                "field": "<string>",
                "order": "<string>"
            },
            {
                "field": "<string>",
                "order": "<string>"
            }
        ],
        "start": <long>,
        "time_range": {
            "end": "<string>",
            "start": "<string>",
            "window": "<string>"
    },
    "process_guid": "<string>",
    "search_id": "<string>"
}


Body Schema

The fields that can be used in criteria, exclusions, fields and sort are defined on Platform Search Fields and match the api_resource specified in the request.

Field Definition Data Type Values
api_resource REQUIRED Specify the type of investigation object to search for String ENRICHED_EVENTS, PROCESSES, PROCESS_EVENTS, AUTH_EVENTS, OBSERVATIONS
version Version of the response schema. Only v2 is valid. String v2
search_id Only with api_resource of PROCESSES. If a search job has been started then the export job can search within those results by specifying the job_id of that search. String e.g. 291778a0-62cf-42ad-82bc-f7465139598e-sqs
process_guid Used to get the process analysis data for a specific process. If process_guid is non-null, then the search_id should be null and api_resource will be ignored; results are always PROCESS_EVENTS. String e.g. ABCD1234-0085076f-00000bac-00000000-1d8333e8bc8a17f
criteria Criteria is an object that represents values that must be in the results. Either query or criteria/exclusion must be included. Object 
{
  "process_name": [
    "chrome.exe"
  ]
}
Additional fields at Platform Search Fields
exclusions Exclusions is a map that represents values that must not be in the results. Either query or criteria/exclusion must be included. Object 
{
  "process_name": [
    "chrome.exe"
  ]
}
Additional fields at Platform Search Fields
fields A list of fields to include in the results, specify * to return all the default fields and add additional fields that are not returned by default String 
[ "*", "process_start_time" ]
Default: ["*"]
query Query in lucene syntax and/or including value searches. Either query or criteria/exclusion must be included. String N/A
rows Number of rows to request, can be paginated Long Default: 500
Max: 10k
sort Sort is a collection of sort parameters that specify a field and order to sort the results. Array 
[{
  "field": "device_timestamp",
  "order": "ASC"
}]
order supports ASC or DESC
start First row to use for pagination Long Default: 0
time_range Describes a time window to restrict the search to match using device_timestamp as the reference. Window will take priority over start and end if provided. Object 
{
  "end": "2020-01-21T18:34:04Z",
  "start": "2020-01-18T18:34:04Z",
  "window": "-2w"
}
window: “-2w” where y=year, w=week, d=day, h=hour, m=minute, s=second

start: ISO 8601 UTC timestamp

end: ISO 8601 UTC timestamp


Response Codes

Code Description Content-Type Content
200 OK application/json Example response below
400 Bad Request N/A N/A
401 Unauthorized N/A N/A
403 Forbidden N/A N/A
404 Not Found N/A N/A
500 Internal Server Error N/A N/A


Examples

Request 
POST https://defense.conferdeploy.net/jobs/v1/orgs/ABCD1234/jobs/start/event_export
Request Headers 
X-AUTH-TOKEN: "ABCDEFGHIJKLMNO123456789/ABCD123456"
Content-Type: "application/json"
Request Body 
{
    "api_resource": "PROCESSES",
    "version": "v2",
    "query": {
        "criteria": {},
        "exclusions": {},
        "query": "*:*",
        "time_range": {
            "start": "2023-03-26T02:00:00.000Z",
            "end": "2023-03-29T02:06:20.864Z"
        },
        "rows": 10000,
        "fields": [
            "*"
        ],
        "sort": [
            {
                "field": "device_timestamp",
                "order": "DESC"
            }
        ]
    }
}
Response Body 
{
    "id": 5731487,
    "type": "event_export",
    "job_parameters": {
        "job_parameters": {
            "query": {
                "criteria": {},
                "exclusions": {},
                "query": "*:*",
                "time_range": {
                    "start": "2023-03-26T02:00:00.000Z",
                    "end": "2023-03-29T02:06:20.864Z"
                },
                "rows": 10000,
                "fields": [
                    "*"
                ],
                "sort": [
                    {
                        "field": "device_timestamp",
                        "order": "DESC"
                    }
                ]
            }
        },
        "process_guid": null,
        "api_resource": "PROCESSES",
        "version": "v2",
        "search_id": null
    },
    "connector_id": "12345ABCD",
    "org_key": "ABCD1234",
    "status": "CREATED",
    "create_time": "2023-03-29T03:19:56.752Z",
    "last_update_time": "2023-03-29T03:19:56.753Z"
}
To download or review the Carbon Black Cloud Postman collection, click here.

List All Available Jobs

Lists all available jobs. A user or connector can only see jobs assigned to themselves.

API Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud jobs.status READ Majority of environments
VMware Cloud Services Platform _API.Background.Tasks:jobs.Status.read N/A - included in permission name Prod UK and AWS GovCloud (US)

Request

GET {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs

Response

Code Description Content-Type Content
200 Successful Request application/json View example response below
400 The JSON body was malformed, or some part of the JSON body included an invalid value N/A N/A
401 Unauthorized N/A N/A
403 Forbidden N/A N/A
404 Not found N/A N/A
500 Internal Server Error N/A N/A

Response Body 

{
  "num_found": "<integer>",
  "results": [
    {
      "connector_id": "<string>",
      "create_time": "<string>",
      "errors": "<string>",
      "id": "<integer>",
      "job_parameters": {},
      "last_update_time": "<string>",
      "org_key": "<string>",
      "owner_id": "<integer>",
      "progress": {
        "num_total": "<integer>",
        "num_completed": "<integer>",
        "message": "<string>"
      },
      "status": "<string>",
      "type": "<string>"
    }
  ]
}

Additional response parameter details

Parameter Possible Values
["progress"]["message"] QUEUED, IN_PROGRESS, FINISHED
["status"] CREATED, FAILED, COMPLETED
["type"] event_export, livequery_export

If a parameter in the response is Null, it will not appear in the response. For example, if a job does not have any errors, the errors parameter will not appear in the response, as shown in the example below.

Example

Request

GET https://defense.conferdeploy.net/jobs/v1/orgs/ABCDEF1/jobs

Response 

{
  "num_found": 1,
  "results": [
    {
      "id": 1,
      "type": "event_export",
      "job_parameters": {
        "job_parameters": {
          "query": {
            "query": "(TTP:SUSPICIOUS_BEHAVIOR OR TTP:PACKED_CALL)",
            "time_range": {
              "start": "2020-03-16T13:06:06-04:00"
            },
            "rows": 10000,
            "fields": [
              "document_guid"
            ],
            "sort": [
              {
                "field": "device_timestamp",
                "order": "DESC"
              }
            ]
          }
        },
        "api_resource": "ENRICHED_EVENTS",
        "version": "v2"
      },
      "connector_id": "2SP9TZA1LL",
      "org_key": "ABCDEF1",
      "status": "COMPLETED",
      "progress": {
        "num_total": 0,
        "num_completed": 0,
        "message": "FINISHED"
      },
      "create_time": "2020-05-19T18:08:00.690Z",
      "last_update_time": "2020-05-19T18:08:12.183Z"
    }
  ]
}

Get Job Details

Get the details of a specific job.

API Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud jobs.status READ Majority of environments
VMware Cloud Services Platform _API.Background.Tasks:jobs.Status.read N/A - included in permission name Prod UK and AWS GovCloud (US)

Request

GET {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs/{job_id}

Response

Code Description Content-Type Content
200 Successful Request application/json View example response below
400 The JSON body was malformed, or some part of the JSON body included an invalid value N/A N/A
401 Unauthorized N/A N/A
403 Forbidden N/A N/A
404 Not found N/A N/A
500 Internal Server Error N/A N/A

Response Body 

{
  "connector_id": "<string>",
  "create_time": "<string>",
  "errors": "<string>",
  "id": "<integer>",
  "job_parameters": {},
  "last_update_time": "<string>",
  "org_key": "<string>",
  "owner_id": "<integer>",
  "progress": {
    "num_total": "<integer>",
    "num_completed": "<integer>",
    "message": "<string>"
  },
  "status": "<string>",
  "type": "<string>"
}

Additional response parameter details

Parameter Possible Values
["progress"]["message"] QUEUED, IN_PROGRESS, FINISHED
["status"] CREATED, FAILED, COMPLETED
["type"] event_export, livequery_export

Example

Request

GET https://defense.conferdeploy.net/jobs/v1/orgs/ABCDEF1/jobs/1

Response 

{
  "id": 3039,
  "type": "event_export",
  "job_parameters": {
    "job_parameters": {
      "query": {
        "query": "(TTP:ADAPTIVE_WHITE_APP OR TTP:UNKNOWN_APP OR TTP:DETECTED_SUSPECT_APP OR TTP:DETECTED_PUP_APP OR TTP:DETECTED_BLACKLIST_APP OR TTP:DETECTED_MALWARE_APP)",
        "time_range": {
          "start": "2020-03-16T13:06:06-04:00"
        },
        "rows": 10000,
        "fields": [
          "*"
        ],
        "sort": [
          {
            "field": "device_timestamp",
            "order": "DESC"
          }
        ]
      }
    },
    "api_resource": "ENRICHED_EVENTS",
    "version": "v2"
  },
  "connector_id": "2SP9TZA1LL",
  "org_key": "ABCDEF1",
  "status": "COMPLETED",
  "progress": {
    "num_total": 10000,
    "num_completed": 10000,
    "message": "FINISHED"
  },
  "create_time": "2020-05-19T18:08:46.490Z",
  "last_update_time": "2020-05-19T18:08:46.490Z"
}

Download Job Output

Provides a redirect with a pre-signed S3 URL to download the output of the job, in a zipped csv file.

If the job does not exist or the output is not yet available, this route will return 404.

API Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud jobs.status READ Majority of environments
VMware Cloud Services Platform _API.Background.Tasks:jobs.Status.read N/A - included in permission name Prod UK and AWS GovCloud (US)

Request

GET {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs/{job_id}/download

Response

Code Description Content-Type Content
200 Successful Request Varies by job Output of job
400 The JSON body was malformed, or some part of the JSON body included an invalid value N/A N/A
401 Unauthorized N/A N/A
403 Forbidden N/A N/A
404 Not found N/A N/A
500 Internal Server Error N/A N/A

Example

Request

GET https://defense.conferdeploy.net/jobs/v1/orgs/ABCDEF1/jobs/1/download

Get Job Progress

This route only returns the progress of a specific job. This route is preferred over listing the entire job for checking the progress.

API Permissions Required

Identity Manager Permission (.notation name) Operation(s) Environment
Carbon Black Cloud jobs.status READ Majority of environments
VMware Cloud Services Platform _API.Background.Tasks:jobs.Status.read N/A - included in permission name Prod UK and AWS GovCloud (US)

Request

GET {cbc-hostname}/jobs/v1/orgs/{org_key}/jobs/{job_id}/progress

Response

Code Description Content-Type Content
200 Successful Request application/json View example response below
400 The JSON body was malformed, or some part of the JSON body included an invalid value N/A N/A
401 Unauthorized N/A N/A
403 Forbidden N/A N/A
404 Not found N/A N/A
500 Internal Server Error N/A N/A

Response Body 

{
    "num_total": "<integer>",
    "num_completed": "<integer>",
    "message": "<string>"
}

Additional response parameter details

Parameter Possible Values
message QUEUED, IN_PROGRESS, FINISHED

Example

Request

GET https://defense.conferdeploy.net/jobs/v1/orgs/ABCDEF1/jobs/1

Response 

{
    "num_total": 3142,
    "num_completed": 3142,
    "message": "FINISHED"
}

Last modified on April 3, 2023