Back to Blogs

Semantic Versioning Support in Carbon Black Cloud Data Forwarder

Posted on July 10, 2023


The Carbon Black Cloud Data Forwarder is being updated with Semantic Versioning to enable improvements to be made to the Data Forwarder while also giving customers control to select which updates to the forwarded data they wish to add to their existing forwarders and whether to automatically or manually adopt such data changes.

What is Semantic Versioning?

Semantic Versioning is when the structure of the version number conveys information about the type of change that occurred.

In the case of the Carbon Black Cloud Data Forwarder Schema, a three part version has been adopted in the form MAJOR.MINOR.PATCH.

  • When the PATCH version changes, it indicates bug fixes. e.g. 2.0.1 to 2.0.2
  • MINOR indicates non-breaking changes such as new fields or increased data volume. e.g. 2.0.0 to 2.1.0
  • MAJOR indicates a substantial change to the schema that is expected to require consumers to refactor their code before using the new schema. For example the change in July 2023 from Alert v1.0.0 to Alert v2.0.0..

See What kinds of changes come with each level of wildcarding? for more detail.

Why did Carbon Black Cloud Data Forwarder add support for semantic versioning?

Carbon Black Cloud Data Forwarder will always strive to offer parity with the latest and greatest data from the associated Carbon Black Cloud API. Carbon Black Cloud’s data continues to evolve as we add support for new kinds of findings, and as we periodically re-align the data models across the platform - new fields are occasionally added, new field values can be added, and semantics about how those fields are generated can vary.

In the past the Carbon Black Cloud Data Forwarder has made updates to the schema of each Forwarder type (alert, endpoint.event, watchlist.hit) - we have added data fields occasionally, as well as making MINOR bug fixes - without giving notice and control to our Forwarder consumers. As we evolve the underlying detections and telemetry for significant data entities such as Alerts, Observations and Events, our ecosystem faces significant breaking changes to existing Forwarder types. We have determined it’s time to offer versioning support as well as consumer choice about whether to automatically or manually accept updates to existing Forwarders.

Carbon Black Cloud Data Forwarder consumers have increasingly articulated the need to have visibility and control over the data emitted by the Data Forwarder, to ensure stability of the applications and code that are integrated with the Data Forwarder. If we didn’t offer versioning within our Forwarders, it would require us to spawn an endless list of new “Forwarder types”, which is needless complexity and disorganization. Once we decided to offer versioning support, we decided to ensure that customers had intuitive visibility into what kinds of changes are made available, and ability to automatically adopt certain types of changes - or manually prevent updates without consumer configuration.

We are adopting Semantic Versioning in the Carbon Black Cloud Data Forwarder over the coming months, and we will as much as possible make our implementation comparable to the approach outlined at https://semver.org. This will give customers the ability to specify (a) which data schema version they wish to consume from the Data Forwarder and (b) what level of schema changes the Carbon Black Cloud can automatically make to their Forwarder configuration


Where is Semantic Versioning support implemented (now and future)?

As of July 2023 launch of the 2.0.0 Data Forwarder for Alerts (aka “2.0.0 Alert Forwarder”), we offer a preview of our Semantic Versioning (semver) support by making available the 2.0.0 Alert Forwarder alongside the now-retroactively-dubbed 1.0.0 Alert Forwarder.

Later this year we will provide full-blown support for versioning of the Endpoint Event and Alert Forwarders, including the ability to set wildcard-based semantic versioning on those Forwarder types. This will coincide with the release of a version 1.1.0 Endpoint Event Forwarder that will include support for the significant new netconn data that comes with a subscription to Carbon Black Cloud XDR.

All new Forwarder types from then on will include semver support.


How does Data Forwarder configuration work now?

As of July 2023 release of 2.0.0 Alert Forwarder:

  • Customers creating a new Alert Forwarder in the Carbon Black Cloud console will be presented with a new Schema choice
    • Two options are available: 1.0.0 and 2.0.0
    • The Carbon Black Cloud console will default this selection to 2.0.0
    • You are free to select 1.0.0 before or after saving that Forwarder configuration
    • Whichever version you elect, a GET request on the Data Forwarder Config API will reflect the current version of the data that will be generated by that Forwarder instance, as a returned parameter of that Forwarder Config known as current_version
  • Customers editing an existing Alert Forwarder in the Carbon Black Cloud console will also be presented with the Schema choice
    • During edits, the Schema option will also default to 2.0.0 for existing Alert Forwarders, whether they were created before or after the launch of the 2.0.0 Alert Forwarder
    • Within approximately 10 minutes, the data emitted for that Forwarder instance will reflect your current choice. That is, if you elect to update an existing (or 1.0.0) Alert Forwarder to 2.0.0, there will be a slight delay until new Alerts start emitting with the version field set to 2.0.0

What changes when full server support is added?

When full semver support is added to each Forwarder type, all semver-enabled Forwarders will support an optional input parameter version_constraint that allows the user to specify the version or range of data schema versions you would like Carbon Black Cloud to use in this specific Forwarder.

Here are some examples, assuming the Alert Forwarder supports 1.0.0, 2.0.0 and 2.1.0 schema versions when you configure your Alert Forwarder

  • If you select 2.0.0 as your version_constraint, this means that your Forwarder will emit Alerts using the 2.0.0 schema and will never update to future versions automatically
    • In many circles this is known as a PINNED version
  • If you select 2.*.* as your version_constraint (notice the “*” asterisk characters), this means that your Forwarder will immediately start out emitting Alerts using the 2.1.0 Alert schema, and will automatically upgrade to 2.2.0 or later 2.x.y schema versions as and when they are made available by the Carbon Black Cloud Data Forwarder
    • In semver terms this is known as a MINOR increment
  • If you select 2.1.* as your version_constraint (again, notice the “*” asterisk character), this means that your Forwarder will start emitting Alerts using the 2.1.0 Alerts schema, and will only be automatically upgraded to any 2.1.1 or later “PATCH” (bug fix) versions of the 2.1 Alert schema. When a 2.2.0 (and later) version is made available, your Forwarder instance will not be automatically updated and it will be up to you to reconfigure your Forwarder to use the newest schema version(s).
    • In semver terms this is known as a PATCH increment
  • If you select 2.0.* as your version_constraint (again, notice the “*” asterisk character), this means that your Forwarder will stay on the 2.0.0 Alerts schema, because there are no (and will likely never be) any 2.0.1-2.0.n Alerts schema versions.

What happens if I don’t specify a version constraint?

After the July 2023 release of 2.0.0 Alert Forwarder:

  • Existing Alert Forwarders will be pinned to schema version 1.0.0
  • New Alert Forwarders will default to schema version 2.0.0


What happens to my current Forwarder instance when a new Forwarder data schema is released?

Between now and the release of full semver support later this year, all 2.0.0 Alert Forwarder instances will always report their version_constraint as 2.0.0, and all Alerts forwarded will assert version as 2.0.0.

During this period, we will make every effort to keep the v2 Alert Forwarder schema stable until such time as full semver support is available; the only exception will be any necessary PATCH updates. This may necessitate delaying availability of new fields that are added to v7 Alerts API.

Once full semver support is released later this year for the Alert Forwarder:

  • all Alert Forwarders will remain pinned at their current version_constraint; No automatic upgrades will be performed
    • all 2.0.0 Alert Forwarder instances will have their version_constraint left unchanged, PINNED to 2.0.0
    • all 1.0.0 Alert Forwarder instances will have their version_constraint left unchanged, PINNED to 1.0.0
    • If you wish to have your Alert Forwarder automatically receive new schema versions, you will be able to reconfigure the version_constraint from 2.x.y to 2.x.* or 2.*.*.

As new versions of every Forwarder’s schema are released, the Developer Network documentation will be updated to reflect the new and changed schema fields and values, which fields are available in each schema version, and what changes (deltas) occurred between each Forwarder schema version.


What are my choices, and what effect do they have?

As of July 2023 release of 2.0.0 Alert Forwarder Schema, your only choice is between 1.0.0 and 2.0.0 versions of Alert schema.

When full semver support is added to the 2.0.0 Alert Forwarder, there will immediately be four levels of schema configuration available to apply to your Alert Forwarder:

  • 2.0.0 - Alert Forwarders will never automatically upgrade to any subsequently-released versions.
  • 2.0.* - Alert Forwarders will automatically upgrade to any 2.0.1 through 2.0.n versions (all bug fixes made to the v2 Alert Forwarder schema)
  • 2.*.* - Alert Forwarders will automatically upgrade to any 2.0.1 through 2.n.m versions (e.g. 2.1.2, 2.5.0, 2.8.5) as and when they’re released to production
  • 1.0.0 - The deprecated v1 Alert Schema which will not have any new versions available. You should plan to update this to a v2 Alert Forwarder Schema at your earliest convenience

Similar “PINNED”, “PATCH” and “MINOR” version constraints will eventually be available on every Forwarder type - we’ll introduce full semver support first for Alert and Endpoint Event forwarder types, then for the other Forwarders as and when they are made available.


What kinds of changes come with each level of wildcarding?

  • PINNED: a version_constraint of 2.0.0 will never allow any changes to the data schema for that Forwarder instance - not even bug fixes.
  • PATCH: a version_constraint of 2.0.* will include bug fixes, such as:
    • improper implementation of a data type
    • adding new enum values to an enum field (e.g. adding a new value to device_os field alongside WINDOWS, MAC, LINUX)
  • MINOR: a version_constraint of 2.*.* will include non-breaking changes to the existing schema such as
    • adding new data fields to the schema
    • significant increase in estimated data volume due to allowing newly-instrumented data types - e.g. the increased event volume coming with the 1.1.0 Endpoint Event Forwarder that will support Carbon Black Cloud XDR
    • new alert types (e.g. adding support for new values alongside Intrusion)
  • MAJOR: no such version_constraint will be available - theoretically it would be expressed as *.*.*. We expect such MAJOR breaking changes in the data schema will always require updating your applications and code that consume from this data, and should never be automatically upgraded outside your direct action.

Further examples will emerge as and when we start making new Forwarder data schema versions available. We will keep you updated through the Developer Network blog and documentation.


When will full semver support be added to Alert, Event and Watchlist Hit Forwarders?

Any projections are subject to change, but as of July 2023, we hope to add semver support to each existing Data Forwarder in the following order:

  • Endpoint Event and Alert Forwarder later in 2023
  • Watchlist Hit Forwarder in early 2024

Other FAQs

  • If one of the Forwarders has released a MINOR version update, will Carbon Black Cloud ever make a PATCH version available for previous versions?
    • We expect at this time that all Forwarder changes will be forward-compatible and will only be made on top of the very latest version of each Forwarder’s schema
    • For example, let’s say we’re at a place where the Alert Forwarder supports four versions: 1.0.0, 2.0.0, 2.0.1 and 2.1.0
    • Let’s say a PATCH has to be made to the Alert Forwarder
    • We expect to only add a 2.1.1 version to the Alert Forwarder, but not a 2.0.2 version - and absolutely never a 1.0.1 version
  • If I decide to change a Forwarder’s version_constraint that matches a later or earlier version of the schema than the one I’m currently using, will the Forwarder automatically start sending that newer (or older) data schema in new emitted records?
    • For example, let’s say the Alert Forwarder currently supports 1.0.0, 2.0.0, 2.0.1 and 2.1.0 versions
    • Let’s say your Alert Forwarder is currently at 2.1.0 version_constraint and you then configured your Alert Forwarder to use a 2.0.* version_constraint
    • The Data Forwarder will start forwarding Alerts using the 2.0.1 version
  • Will MAJOR version upgrades ever be undertaken in the future?
    • It’s safe to assume that every few years, data structures will change to maintain security efficacy in the Carbon Black Cloud. This will necessitate breaking changes in APIs and Data Forwarder schema.
    • We have undertaken this with Alerts in 2023 with the update from API version 6 to 7, and Alert Forwarder version from 1.0.0 to 2.0.0.
  • Is this replacing the Data Forwarder Configuration API version concept (e.g. /data_forwarder/v2/orgs/{org_key}/configs)?
    • No, the /v2/ portion of the API route does not relate in any way to the data schema versions (version) emitted by the Carbon Black Cloud Data Forwarder
    • The v2 Data Forwarder Config API remains backward compatible and adds support for version_constraint.
    • The v1 Data Forwarder Config API is deprecated. A migration guide and the decommission schedule for the v1 Data Forwarder Config API will be announced shortly on the Developer Network blog.

More Information

Have questions or feedback?

  • Subscribe to the Developer Network Newsletter