QRadar App v2.2.0 OOM issue
Posted on August 1, 2023
Carbon Black Cloud QRadar App stops forwarding alerts and/or audit logs due to memory limitations
Issue Description
Under high load (high ammount of alerts and audit logs per minute), the Carbon Black Cloud QRadar App may stop forwarding messages to QRadar due to hitting a memory limitation which leads to app restarts.
The default memory limit for the QRadar app is set to 200MB. We highly recommend using the Data Forwarder as opposed to REST API polling for data ingestion mechanism as it is scalable and capable of handling high loads. If you are using data polling instead, it is possible that on some occasions under high load you might notice that no new CBC alerts are being fed to QRadar. The application logs may indicate application restarts. Have a look at poll.log and check if it indicates continous service restarts such as in this example:
2023-06-12 13:12:16 INFO Recreating SyslogOutput to use custom ip: 1.2.3.4
2023-06-12 13:12:18 INFO Starting poller in production mode - syslog output to QRadar console at IP address 1.2.3.4
2023-06-12 13:12:18 INFO Log Source Identifier to be used localhost
2023-06-12 13:12:20 INFO Sending 2500 Audit Log Events.
2023-06-12 13:12:31 INFO Recreating SyslogOutput to use custom ip: 1.2.3.4
2023-06-12 13:12:31 INFO Starting poller in production mode - syslog output to QRadar console at IP address 1.2.3.4
2023-06-12 13:12:31 INFO Log Source Identifier to be used localhost
2023-06-12 13:12:32 INFO Sending 2500 Audit Log Events.
2023-06-12 13:12:43 INFO Recreating SyslogOutput to use custom ip: 1.2.3.4
If you have root access to the QRadar Console (or the Apphost in case the app runs on a dedicated Apphost), run the dmesg command (as root) to inspect the kernel log for OOM kills such as the following:
slab_out_of_memory: 72 callbacks suppressed
SLUB: Unable to allocate memory on node -1, gfp=0xdc0(GFP_KERNEL|__GFP_ZERO)
cache: signal_cache(632851:3a166877f91031145f6dc491df3c915c57f1439f4b3bcea4c6a37d1885ea27ea), object size: 1056, buffer size: 1088, default order: 3, min order: 0
node 0: slabs: 31, objs: 930, free: 0
oom_kill_process: 70 callbacks suppressed
python invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=0
CPU: 25 PID: 3143648 Comm: python Not tainted 5.4.0-29-generic
If this is the case, you might need to increase the memory limit for the Carbon Black Cloud app to a higher value (above 300MB, 500MB recommended).
Increasing the memory limit
In order to increase the memory limit, you need to find the application ID of the CBC App. In order to do that you need to login (via ssh) to the QRadar Console and run /opt/qradar/support/recon ps. The output may look like the following:
App-ID Name Managed Host ID Workload ID Service Name Container Name Port
1055 QRadar Assistant 53 apps qapp-1055 - 0
1051 QRadar Log Source Management 53 apps qapp-1051 - 0
1056 QRadar Use Case Manager 53 apps qapp-1056 - 0
1057 VMware Carbon Black Cloud 53 apps qapp-1057 - 0
In the first column, there are the App IDs and in the second column there are the App names. Find the “Vmware Carbon Black Cloud” row and copy the corresponding App ID.
Next, open the following URL in your web browser: https://QRADAR_CONSOLE_IP/api_doc#version=19.0&api=%2Fgui_app_framework%2Fapplications%2F%7Bapplication_id%7D&method=POST
Replace QRADAR_CONSOLE_IP with the IP address of your QRadar console. You need to login with an account with administrative privileges. The resulting api_doc page should look like the following:
Input the App ID in the “application_id” input textbox (in the Value column) and 500 in the memory input textbox, they click the Try it Out! button.
The Carbon Black Cloud QRadar app will get stopped and then restarted (this should take about a minute or two).
To confirm the memory limit has been successfully changed, open the following URL in your browser: https://QRADAR_CONSOLE_IP/api_doc#version=19.0&api=%2Fgui_app_framework%2Fapplications%2F%7Bapplication_id%7D&method=GET
Type in the App ID in the application_id input box and click the Try it Out! button. Have a look at the Response Body section of the page. If everything was correct, it should contain the following string:
"memory": 500,
Once the memory limit is increased and the app is restarted, it should be seamlessly forwarding all the CBC alerts and audit logs.
Have questions or feedback?
- Use the Developer Community Forum to discuss issues and get answers from other API developers in the Carbon Black Community
- Report bugs and change requests to Carbon Black Support
- Subscribe to the Developer Network Newsletter