AWS S3: Writing an S3 Bucket Policy

Overview

This document provides details on how to build a Data Forwarder-compatible bucket policy and also provides troubleshooting guidance for errors related to incorrectly configured policies.

This guide covers:

Requirements

  • Carbon Black Cloud Console
  • Carbon Black Cloud Data Forwarder
  • Amazon Simple Storage Service (Amazon S3)

What is a policy and how are policies used by the Forwarder?

Policies are AWS objects used to manage access to specific resources by defining the resource’s permissions. Permissions in the policies determine whether a principal (a user or a role) making a request is allowed to perform the action in the request.

The Data Forwarder requires you to create an S3 bucket with a policy that grants the necessary permissions to the Principal role used by the Data Forwarder. This policy is a resource-based policy.

What configuration do I need in my AWS policy so the Forwarder can upload to my bucket?

The policy grants specific permissions to the Forwarder’s Principal ARN Role that are necessary for writing data to your S3 Bucket. Most fields in the policy will be the same across any policy definition except for the Principal and Resource fields.

If you use the AWS Policy Generator, you will fill out the following fields:

  • Policy Type - This defines the type of “container” to use for the permissions allowed. There are a number of different types of policies that can be created, but you will need a policy that grants permissions to the principal, enabling it to perform certain actions on a specified S3 bucket. Use the value S3 Bucket Policy.
  • Effect - This defines whether the policy is additive (it grants permissions to allow the principal to perform certain actions) or reductive (it defines actions that a principal is prohibited from performing). Use the value Allow.
  • Principal - This is an ARN for the user or role for which the policy is defining permissions. The principal defined in the policy is the role that the Forwarder assumes, so it can access and perform actions on the customer’s S3 bucket. This value is determined by what region your S3 Bucket is in. See table below to find the Principal ID for your bucket’s region.

Note: The S3 Bucket must be created in the same region as the tenant organization from which you’ll forward data, or it will fail. It is possible to work around this requirement using S3 Cross-Region Replication (CRR).

AWS Region Principal ID
US East (N. Virginia) us-east-1 arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder
Europe (Frankfurt) eu-central-1 arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-eu-central-1-event-forwarder
Asia Pacific (Tokyo) ap-northwest-1 arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-ap-northeast-1-event-forwarder
Asia Pacific (Sydney) ap-southeast-2 arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-ap-southeast-2-event-forwarder
Europe (London) eu-west-2 arn:aws:iam::132308400445:role/mcs2-psc-data-forwarder-s3
US Gov West 1 us-gov-west-1 arn:aws-us-gov:iam::507058390320:role/mcs2-psc-data-forwarder-s3
  • AWS Service - This is the service that the permissions are attached to. Use the value Amazon S3.
    • Actions - This defines the specific actions that the policy is granting permission for the principal to take. The required actions are:
      • PutObject
      • PutObjectAcl
  • Amazon Resource Name (ARN) - Defines one or many resource(s) that the principal is granted permission to take the defined actions on.
  • An individual ARN can be as general or granular depending on your use case. Within the value, you will specify which folder in the bucket you want to grant the API permission to access. You can choose to grant access to all objects within a bucket, a top-level folder in your bucket, or any sub-folder.
Note: The Resource value(s) must end in '/*', so the Data Forwarder can create and write to the folder and sub-folders. This lets you configure multiple forwarders in the same bucket or folder while keeping the data organized.

How should I define the resource(s) in my policy?

The Resource field defines what resources the principal has permission to perform action on. If you want to grant stricter permissions, this impacts the values you provide when creating a new forwarder-compatible bucket policy. The following are suggestions for how to configure your bucket policy based on different use cases.

Case 1: Single bucket with no sub-folders

I have one bucket and want to partition my data in separate sub-folders, one each for event forwarding and alert forwarding, and I have no other sub-folders within the bucket. If the S3 bucket is only used by the Forwarder, you can define your policy such that it grants the Forwarder access to any object within the bucket. This will allow the Data Forwarder to upload each configured forwarder’s data to their own sub-folders, which partitions the data without you needing to specify permissions for each forwarder in the policy.

In this case, the Resource field in your policy can be a single item, the bucket itself:

"Resource": "arn:aws:s3:::super-awesome-bucket/*"

Because you are granting the Forwarder permission to any object in the bucket, you won’t need to manually create any sub-folders. When you create a new forwarder instance in Carbon Black Cloud, the Forwarder will automatically create a sub-folder for each forwarder. The sub-folder is named after the “s3_prefix” value you provide.

For example, using the “super-awesome-bucket” defined in the policy above and an example org key of “ABCD123”, you can create two forwarders (one for events and one for alerts) by sending the following payloads in your configuration service API requests:

Event Forwarder:

{
    "name":"example-event-forwarder",
    "s3_bucket_name":"super-awesome-bucket",
    "s3_prefix":"events",
    "type":"endpoint.event"
}

Alert Forwarder:

{
    "name":"example-alert-forwarder",
    "s3_bucket_name":"super-awesome-bucket",
    "s3_prefix":"alerts",
    "type":"alert"
}

By doing this, you should expect both forwarders to send their events to the same bucket, but under different sub-folders created by the forwarder itself.

Events sent on 4/21/2020 at 11:01:54 will end up in the following structure:

super-awesome-bucket/events/org_key=ABCD123/year=2020/month=04/day=21/hour=11/minute=1/second=54/xxxxxxxxfilename.jsonl.gz

Alerts sent on the same date will end up in the following structure:

super-awesome-bucket/alerts/org_key=ABCD123/year=2020/month=04/day=21/hour=11/minute=1/second=54/xxxxxxxxfilename.jsonl.gz

Case 1 Example Policy:

{
  "Id": "Policy1587586847422",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1587586830454",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::super-awesome-bucket/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder"
        ]
      }
    }
  ]
}

Case 2: Single bucket with layered sub-folders

I have one bucket with multiple sub-folders, and within one of those sub-folders, I want to create additional sub-folders each for event forwarding and alert forwarding, ensuring I do not grant Forwarder permission to access any of the other top-level sub-folders. This example is similar to the first example in that you want to grant the Forwarder permission to access any object within the top-level folder. In this case, as long as additional forwarders are created with the correct folder name included in their prefix, you can use the same bucket policy for each new forwarder.

In this case, the Resource field in your policy can be a single item specifying both the bucket and folder that the Forwarder will access:

“Resource”: “arn:aws:s3:::super-awesome-bucket/cbc-forwarders/*”

Because you are granting the Forwarder permission to any object below the folder “cbc-forwarders”, there is no need to manually create any additional sub-folders within the bucket. When you create a new forwarder instance in Carbon Black Cloud, the Forwarder will automatically create a sub-folder for each forwarder. The sub-folder is named after the “s3_prefix” value you provide.

For example, using the “super-awesome-bucket” defined in the policy above, the sub-folder “cbc-forwarders” and an example org key of “ABCD123”, you, as a customer can create two forwarders, one for events and one for alerts, by sending the following payloads in your configuration service API requests:

Event Forwarder:

{
    "name":"example-event-forwarder",
    "s3_bucket_name":"super-awesome-bucket",
    "s3_prefix":"cbc-forwarders/events",
    "type":"endpoint.event"
}

Alert Forwarder:

{
    "name":"example-alert-forwarder",
    "s3_bucket_name":"super-awesome-bucket",
    "s3_prefix":"cbc-forwarders/alerts",
    "type":"alert"
}

By doing this, you should expect both forwarders to send their events to the same bucket and top-level folder, but under different sub-folders created by the forwarder.

Events sent on 4/21/2020 at 11:01:54 will end up in the following structure:

super-awesome-bucket/cbc-forwarders/events/org_key=ABCD123/year=2020/month=04/day=21/hour=11/minute=1/second=54/xxxxxxxxfilename.jsonl.gz

While Alerts sent on the same date will end up in the following structure:

super-awesome-bucket/cbc-forwarders/alerts/org_key=ABCD123/year=2020/month=04/day=21/hour=11/minute=1/second=54/xxxxxxxxfilename.jsonl.gz

The policy is essentially saying, “grant the Forwarder permission to access any object below “/cbc-forwarders” and nowhere else," so you must configure the “s3_prefix” value to match the folder structure in your S3 bucket .

Case 2 Example Policy:

{
  "Id": "Policy1587586847422",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1587586830454",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::super-awesome-bucket/cbc-forwarders/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder"
        ]
      }
    }
  ]
}

Case 3: Single bucket with restricted sub-folders

I have one bucket and want different sub-folders, one each for event forwarding and alert forwarding, but I also have other sub-folders on the same level that I don’t want the Forwarder to access. Because you have multiple objects within the same bucket folder that the Forwarder needs to access, you can define your policy such that it grants the Forwarder access to only the sub-folders you specify.

In this case, the Resource field in your policy will be a list of items specifying the exact folders you wish to grant access to:

“Resource”: [

    "arn:aws:s3:::super-awesome-bucket/events/*",

    "arn:aws:s3:::super-awesome-bucket/alerts/*"

  ]

A limitation of this approach is that it requires you to make manual changes to the policy whenever you create a new forwarder since the “s3-prefix” value in the forwarder must be listed as a resource in your policy in order for the Forwarder to access them.

You do not need to manually create sub-folders for new forwarders in your bucket as long as the sub-folder names defined in the policy match “s3_prefix” value you configure in the new forwarders.

For example, using the “super-awesome-bucket” defined in the policy above with the sub-directories “/events” and “/alerts” defined in the resource field in the policy and an example org key of “ABCD123”, you can create two forwarders (one for events and one for alerts) by sending the following payloads:

Event Forwarder:

{
    "name":"example-event-forwarder",
    "s3_bucket_name":"super-awesome-bucket",
    "s3_prefix":"events",
    "type":"endpoint.event"
}

Alert Forwarder:

{
    "name":"example-alert-forwarder",
    "s3_bucket_name":"super-awesome-bucket",
    "s3_prefix":"alerts",
    "type":"alert"
}

By doing this, you should expect the forwarders to send their events to the same bucket and organized into the sub-folders that were created by the forwarder.

Events sent on 4/21/2020 at 11:01:54 will end up in the following structure:

super-awesome-bucket/events/org_key=ABCD123/year=2020/month=04/day=21/hour=11/minute=1/second=54/xxxxxxxxfilename.jsonl.gz

While Alerts sent on the same date will end up in the following structure:

super-awesome-bucket/alerts/org_key=ABCD123/year=2020/month=04/day=21/hour=11/minute=1/second=54/xxxxxxxxfilename.jsonl.gz

The policy is essentially saying, “Grant the Forwarder permission to access any object below ‘/events’ or ‘/alerts’ but nowhere else." Since the bucket policy is written to only grant the Forwarder access to these two sub-folders, you must configure the forwarder to use one of these two values for the “s3_prefix” or the request will fail.

Case 3 Example Policy:

{
"Id": "Policy1587586974250",
"Version": "2012-10-17",
"Statement": [
    {
"Sid": "Stmt1587586972918",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
      ],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::super-awesome-bucket/events/*",
"arn:aws:s3:::super-awesome-bucket/alerts/*"
      ],
"Principal": {
"AWS": [
"arn:aws:iam::132308400445:role/mcs-psc-prod-event-forwarder-us-east-1-event-forwarder"
        ]
      }
    }
  ]
}

Case 4: Multiple buckets

I have separate buckets for different forwarders, each with their own folder structure. Because you wish to grant the Forwarder access to multiple buckets, you have two options:

You can follow the examples in Case 1 or 2 (depending on your bucket folder structure) and create separate policies for each bucket, allowing you to manage each separately. You can create a single policy that includes both buckets as the multiple resources like in Case 3, and apply the same policy to both buckets.

Troubleshooting errors

I am getting “Error: This policy contains invalid Json” when trying to edit my bucket policy in S3.

  • Try using the AWS Policy Generator to automatically generate the JSON to include in your bucket’s policy.

I have created a bucket with a policy, but when I try to create a new forwarder, I get an error with the status “Invalid Bucket Configuration for Bucket: with Prefix: ” and error “AccessDenied”.

  • When attempting to create a new forwarder, the Data Forwarder will automatically check the configuration to ensure that it has sufficient write access to the specified resource. This error suggests the policy was configured incorrectly. Please ensure the following:
    • Your bucket is created in the same region as the tenant organization from which you’ll forward data, and you are using the correct Principal for that region
    • The resource(s) listed in your policy include the target S3 bucket and sub-folders (where relevant) defined in the payload to create a new forwarder
    • Each resource listed in your policy is followed by “/*”

Is there a way that I can check that an existing configuration is valid and able to access my bucket?

  • Yes, the Data Forwarder API includes a healthcheck route that enables a customer to manually check that the Forwarder is able to write to the specified S3 Bucket using the existing configuration. Please refer to the Data Forwarder API documentation for details on healthcheck requests and responses.
Note: This API endpoint is exposed as a 'Test Bucket' feature on the Data Forwarders page of the Carbon Black Cloud UI.

View the Full Documentation

Data Forwarder API Documentation

More Related Content


Give Feedback

New survey coming soon!


Last modified on February 9, 2023