Back to Blogs

Using the new Jobs Service API

Posted on June 16, 2020

First, who should use the Job Service API?

In May we released the Job Service API, an API that helps manage long-running tasks. This API is most useful for users managing large data sets where there is risk of an API request timing out before the task completes. The Job Service API enables asynchronous task execution so that jobs don’t time out, thus preventing data loss.

For those managing smaller data sets, this API is less useful, and you can use regular API calls instead of using asynchronous API routes.

How do you use it?

Generally, an asynchronous query can be performed in five steps:

  1. Call an API route that utilizes the job service, like POST {psc-hostname}/jobs/v1/orgs/{org_key}/jobs/start/event_export
  2. Receive the Job ID to track job progress
  3. Monitor job progress with the /jobs/v1/orgs/{org_key}/jobs/{job_id}/progress route
  4. Repeat Step 3 until job progress shows FINISHED
  5. Download job output with the /jobs/v1/orgs/{org_key}/jobs/{job_id}/download route

Different ways to use the API

Let’s explore in depth two different ways of using the API:

  1. Asynchronously Export Events where an actor tries to invoke a command interpreter
  2. Asynchronously Export Events where an actor tries to inject code or modify memory of another process

These example queries were written in the Carbon Black Cloud console. From the Investigate page of your console, click “Search guide” in the top right corner to review the syntax and field names, and then create some queries of your own!

The following API calls were executed in Postman, but the API calls could also be done on the command line with cURL.

How to asynchronously export events where an actor tries to invoke a command interpreter (Mac)

When a query results in very large datasets, asynchronous downloads allow for uninterrupted retrieval of results. A query checking for when an actor invokes a command interpreter is an example of a query that could yield a large result set.

1. Submit the Query

osquery:

target.app name:sh OR target.app name:zsh OR target.app name:csh OR target.app name:bash OR target.app name:tcsh OR target.app name:python

Request Body:

{
  "query": {
    "query": "target.app name:sh OR target.app name:zsh OR target.app name:csh OR target.app name:bash OR target.app name:tcsh OR target.app name:python",
    "time_range": {
      "start": "2020-05-01T00:00:00-00:00"
    },
    "rows": 100000
  },
  "api_resource": "enriched_events",
  "version": "v2"
}

The above request restricts results to any event after 5/1/2020, and limits the results to 100,000 rows. Keep in mind that events are only stored for 30 days.

Request URL to Begin Exporting Events:

POST {psc-hostname}/jobs/v1/orgs/{org_key}/jobs/start/event_export
2. Receive the Job ID

This request returns details about the Event query being performed, including a Job ID.

Abbreviated Response Body:

{
    "id": 4522,
    "type": "event_export",
    "status": "CREATED",
    [...]
}

The Job ID will be used to track progress and download the job output.

3. Monitor Job Progress

Use the Job ID to monitor the query until its progress message returns FINISHED.

Request URL to Monitor Job Status:

GET {psc-hostname}/jobs/v1/orgs/{org_key}/jobs/{job_id}/progress

The Get Job Progress route of the Job Service API returns the current progress:

{
    "num_total": 100000,
    "num_completed": 100000,
    "message": "FINISHED"
}

The message can be QUEUED, IN_PROGRESS, or FINISHED.

4. Download Job Output

The job has FINISHED status, so the results can now be downloaded via the Download Job Output route of the Job Service API:

GET {psc-hostname}/jobs/v1/orgs/{org_key}/jobs/{job_id}/download

The Job Service API responds with a redirect with a pre-signed AWS S3 URL to download the output of the job. If the job output is not yet available, this route will return 404 Not Found.

How to asynchronously export events where an actor tries to inject code or modify memory of another process

A query checking for when an actor injects code or modifies memory is another example of a query that could yield a large result set. These instructions are the same as the above example, with a different query in the original request.

1. Submit the Query

osquery:

TTP:INJECT_CODE OR TTP:HAS_INJECTED_CODE OR TTP:COMPROMISED_PROCESS OR TTP:PROCESS_IMAGE_REPLACED OR TTP:MODIFY_PROCESS OR TTP:HOLLOW_PROCESS

Request Body:

{
  "query": {
    "query": "TTP:INJECT_CODE OR TTP:HAS_INJECTED_CODE OR TTP:COMPROMISED_PROCESS OR TTP:PROCESS_IMAGE_REPLACED OR TTP:MODIFY_PROCESS OR TTP:HOLLOW_PROCESS",
    "time_range": {
      "start": "2020-05-01T00:00:00-00:00"
    },
    "rows": 100000
  },
  "api_resource": "enriched_events",
  "version": "v2"
}

The above request restricts results to any event after 5/1/2020, and limits the results to 100,000 rows. Keep in mind that events are only stored for 30 days.

Request URL to Begin Exporting Events:

POST {psc-hostname}/jobs/v1/orgs/{org_key}/jobs/start/event_export

After submitting the new query, the steps to retrieve job output are the same as in the first example. Follow steps 2 - 4 above to download the output, this time using the new Job ID you receive from your query request.