App for IBM QRadar - Troubleshooting v2.3.0

Frequently Asked Questions

• The "Last Contact" field under Settings > Data should contain a current timestamp within the span of the configured "Polling Interval". In this example, the timestamp should be updated every 60 seconds. • If you go to Settings > Configuration requests are triggered to check the validity. If there is something wrong with the credentials, or the Device API or Alerts API at the current moment, validation errors will be shown.
• Check that the API key is of the correct Access Level type.
• Check that the "Custom" Access Level Type has the necessary permissions.
• Check if Polling under Settings > Data sub-tab is enabled.
• Make sure the respective Alerts type(s) (CB Analytics Alerts, Container Runtime Alerts, Device Control Alerts, Host Based Firewall Alerts, Intrusion Detection System Alerts, Watchlist Alerts) under Settings > Data sub-tab are enabled.
• If you use the Built-in input, make sure "Minimum Successful Events for Autodetection" in the Log Source Type configuration is set low enough. Details on how to set it up are available in step 4. of the Installation & User Guide > Log Source Type Configuration.
• Once the app makes contact with the Carbon Black Cloud, it will start polling data. It might take a few minutes until QRadar starts recognising the incoming records as Carbon Black Cloud data. All data polled in the interim will be displayed in the Log Activity page as "Unknown log event" collected by "SIM Generic Log DSM-7".
• If you go to Settings > Configuration requests are triggered to check the validity. If there is something wrong with the credentials, or the Device API or Alerts API at the current moment, validation errors will be shown.
• Check if Polling is Enabled under Settings > Data sub-tab. • If the above did not resolve the problem, check the network on the QRadar host to confirm connectivity to Carbon Black Cloud.
• To turn on Read Only access, configure your access level with only the "Read" permissions. This will result in certain functions becoming unavailable in the app.
1. Click on the menu icon with three horizontal lines.
2. Click on the star icon to add Carbon Black Cloud to the menu bar.
• No, all of the Alerts, Audit Logs, and Events will be pulled in once the application is working again. However, depending on the downtime period, it might take some time for the app to catch-up.
• Yes, multiple Data Forwarders can push to the same S3 bucket. The Data Forwarder is creating a directory structure for each Org Key.
• Yes, 2.1.0. onwards supports multi-tenancy. More information is available here.
• Yes, the app lives in a docker container, and has its own logs separate from the QRadar logs. You can find more details about logging and troubleshooting them in the IBM's page for App Troubleshooting.
• Use the full dashboard URL of your Carbon Black Cloud Console. Full detail on the URLs for each environment are available here.
• The Data Forwarder is the recommended approach for ingesting Alerts and Endpoint Events into QRadar due to its reliability, scale, and low latency. This approach is only required to ingest Endpoint Event data.
• QRadar 7.5.0 UP3+
• Yes, 2500.
• Use the Developer Community Forum to discuss issues and get answers from other API developers in the Carbon Black Community.
• Report bugs and change requests to Carbon Black Support.
• View all API and integration offerings on the Developer Network along with reference documentation, video tutorials, and how-to guides.
• To view the full set of new features, visit the Release Notes section.
• Detailed information on app upgrades, requirements, and configuration changes is available here.
Yes, 10,000. If your organization has more than 10,000 alerts each polling interval, you can:
• Tune CB Analytics alerts that are known-good in your environment using the Dismiss all future alerts functionality.
• Follow recommendations from our Threat Research team here.
• Modify the configured Alert Input and increase the Minimum Alert Severity.
• Change the polling interval from the default of 180 seconds to 120 or 60 seconds.
• Switch to ingesting Alerts via the Data Forwarder input.
• No, but to be able to use the full set of features of the app, like assigning Policies, performing Right-click Actions, viewing Device information and more, we recommend adding them. This includes Product URL, Org Key, "Custom" Type credentials.
• Go to Devices Tab and filter by status:QUARANTINE.
• System Time is the current time of the QRadar console in the local time zone and Last Contact is in UTC time zone.
• When using the "Built-in" data input method, delays between the incoming data intervals can signify memory overload on the container. A combination of high bursts of Alerts for extended periods and low physical memory on the app container can cause a memory overload. As discussed in this thread, the app's memory is limited to 10% of the system's physical memory, and this can cause delays in Alert and general data processing. If you experience such symptoms, consider using the "Data Forwarder" input.
• Check whether Coalescing Events option for your syslog log source is enabled and the Event Count for Alerts is larger than 1.
• You may be hitting the default 4096kb TCP Syslog max payload size. To remediate this, increase the payload as some alerts exceed 4k, which prevents them from being logged correctly in QRadar. A step-by-step guide is available here.
• Check if you are hitting your QRadar Event Processor System (EPS) licensed limit. Detailed information can be found on the IBM support page.
• Update your app. A known issue in previous versions was causing a small percentage of Alerts not to be logged. This issue was resolved in v.2.2.0 of the app.
After v2.3.0 onwards Audit Logs use Custom Key instead of API Key, so a new permission needs to be added to the Custom Key:
Audit Logs (org.audits) - READ.
After v2.3.0 onwards dismiss alert action requires two permissions:
Alerts (org.alerts.close) - EXECUTE
Background Tasks (jobs.status) - READ (new)
It is highly recommended to use version v2.2 or later of the app on QRadar 7.5 installations.
Under high load (high ammount of alerts and audit logs per minute), the Carbon Black Cloud QRadar App may stop forwarding messages to QRadar due to hitting a memory limitation which leads to app restarts.
The default memory limit for the QRadar app is set to 500MB. We highly recommend using the Data Forwarder as opposed to REST API polling for data ingestion mechanism as it is scalable and capable of handling high loads. If you are using data polling instead, it is possible that on some occasions under high load you might notice that no new CBC alerts are being fed to QRadar. The application logs may indicate application restarts. Have a look at poll.log and check if it indicates continous service restarts such as in this example:

2023-06-12 13:12:16 INFO Recreating SyslogOutput to use custom ip:
2023-06-12 13:12:18 INFO Starting poller in production mode - syslog output to QRadar console at IP address
2023-06-12 13:12:18 INFO Log Source Identifier to be used localhost
2023-06-12 13:12:20 INFO Sending 2500 Audit Log Events.
2023-06-12 13:12:31 INFO Recreating SyslogOutput to use custom ip:
2023-06-12 13:12:31 INFO Starting poller in production mode - syslog output to QRadar console at IP address
2023-06-12 13:12:31 INFO Log Source Identifier to be used localhost
2023-06-12 13:12:32 INFO Sending 2500 Audit Log Events.
2023-06-12 13:12:43 INFO Recreating SyslogOutput to use custom ip:

If you have root access to the QRadar Console (or the Apphost in case the app runs on a dedicated Apphost), run the ``dmesg`` command (as root) to inspect the kernel log for OOM kills such as the following:

slab_out_of_memory: 72 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0xdc0(GFP_KERNEL|__GFP_ZERO)
cache: signal_cache(632851:3a166877f91031145f6dc491df3c915c57f1439f4b3bcea4c6a37d1885ea27ea), object size: 1056, buffer size: 1088, default order: 3, min order: 0
node 0: slabs: 31, objs: 930, free: 0
oom_kill_process: 70 callbacks suppressed
python invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=0
CPU: 25 PID: 3143648 Comm: python Not tainted 5.4.0-29-generic

If this is the case, you might need to increase the memory limit for the Carbon Black Cloud app to a higher value (above 300MB, 500MB recommended).

Increasing the memory limit:

In order to increase the memory limit, you need to find the application ID of the CBC App. In order to do that you need to login (via ssh) to the QRadar Console and run /opt/qradar/support/recon ps. The output may look like the following:

App-ID Name Managed Host ID Workload ID Service Name Container Name Port
1055 QRadar Assistant 53 apps qapp-1055 - 0
1051 QRadar Log Source Management 53 apps qapp-1051 - 0
1056 QRadar Use Case Manager 53 apps qapp-1056 - 0
1057 VMware Carbon Black Cloud 53 apps qapp-1057 - 0

In the first column, there are the App IDs and in the second column there are the App names. Find the "Vmware Carbon Black Cloud" row and copy the corresponding App ID.
Next, open the following URL in your web browser: https://QRADAR_CONSOLE_IP/api_doc#version=19.0&api=%2Fgui_app_framework%2Fapplications%2F%7Bapplication_id%7D&method=POST

Replace QRADAR_CONSOLE_IP with the IP address of your QRadar console. You need to login with an account with administrative privileges. The resulting api_doc page should look like the following:

Input the App ID in the "application_id" input textbox (in the Value column) and 500 in the memory input textbox, they click the Try it Out! button.

The Carbon Black Cloud QRadar app will get stopped and then restarted (this should take about a minute or two).

To confirm the memory limit has been successfully changed, open the following URL in your browser: https://QRADAR_CONSOLE_IP/api_doc#version=19.0&api=%2Fgui_app_framework%2Fapplications%2F%7Bapplication_id%7D&method=GET

Type in the App ID in the application_id input box and click the Try it Out! button. Have a look at the Response Body section of the page. If everything was correct, it should contain the following string:

"memory": 500,

Once the memory limit is increased and the app is restarted, it should be seamlessly forwarding all the CBC alerts and audit logs.

App Errors

• Go to Carbon Black Cloud > Settings > Actions to configure a watchlist.
• Enter valid query or leave field empty to bulk search.
• Fill out information under Settings > Configuration.
• "HTTPS" is the secure version of "HTTP", which is the primary protocol used to send data between a web browser and a website. If you need to configure a "Product URL" for your app, it is a requirement to use the full address.
• Polling Interval determines how frequently to pull data from the Carbon Black Cloud, in seconds. The minimum value is 60 seconds, and the maximum value is 600. Either choose a value within those boundaries, or click "Close" to revert the changes.

Last modified on June 5, 2024