Guides

Installing Open-appsec for Linux on Nginx

A practical walkthrough for deploying open-appsec on NGINX, plus why the usual "ModSecurity is dead" framing is wrong , and what the real case for an ML-based WAF is.

Open-appsec is a machine-learning Web Application & API Security engine from Check Point, released as open source. It is usually described as a replacement for ModSecurity, and that pitch normally opens with ModSecurity's death, which is worth correcting before the rest of this guide makes sense.

ModSecurity is not end of life. In January 2024 Trustwave transferred the project to the OWASP Foundation and discontinued its own commercial products and rule feeds. The open-source engine continued under community governance: it shipped a release on 2 July 2026, the current v3 line is 3.0.16 with 3.0.17 fixing CVE-2026-73857, and the OWASP Core Rule Set that most deployments actually rely on is still actively maintained.

What has changed is the operating burden. libmodsecurity3 has had a rough 2026. CVE-2026-30923 and CVE-2026-42268 in April, CVE-2026-52747 and CVE-2026-52761 in June, and a September batch covering multipart request parsing and the XML request-body processor, several of them high severity. Signature-based WAFs also need continuous rule tuning to stay both effective and quiet, and that work lands on whoever owns the deployment. That is the honest case for weighing an ML-based alternative: not that ModSecurity is gone, but that keeping a rule-based WAF sharp is ongoing labour, and the engine has had a difficult year.

Check Point's pitch is that open-appsec replaces static signatures with continuous traffic analysis, catching known and zero-day threats without the rule-tuning a signature engine needs. The architectural difference is real (there are no signatures to tune) but the performance claim is a vendor claim, and the honest version is that whether it catches more than a well-tuned ModSecurity plus OWASP CRS depends on your traffic and the effort you put into tuning. This is the comparison Check Point publishes between NGINX App Protect, ModSecurity WAF and open-appsec:

Open-appsec can be deployed as an add-on to common platforms. NGINX, the NGINX Ingress Controller, Kong API Gateway, APISIX, Envoy and Envoy Gateway, Istio (in beta), and NGINX Proxy Manager, among others. The current agent line is 1.1.36 (24 August 2026), distributed as containers and through Helm charts at charts.openappsec.io, with Prometheus metrics export in beta. Open-appsec's pitch is simpler maintenance than signature-based WAFs through automated learning and exception handling, no manual rule tuning, and security resources freed up. That is marketing, but the underlying point is the one worth testing, because the maintenance burden of a signature WAF is real and it is the main reason people go looking for alternatives.

In this blog post, we are going to deploy open-appsec for Linux on Nginx and test its capabilities to see how well it can protect our web applications.

Prerequisites

To install open-appsec, we are going to need the following:

Linux machine with:
- A supported OS and NGINX or Kong version. A list of all supported/pre-compiled attachments for NGINX and Kong per supported OS versions is available here.
- In case your versions are not supported yet, you can also build the code yourself, see here.
- Root permissions
- A working NGINX install, the first 24 hours on a new VPS covers getting a server to that point if you are starting from nothing
wget command-line tool installed on your Linux machine

Installation

Download the installer for Linux using these commands:

wget https://downloads.openappsec.io/open-appsec-install && chmod +x open-appsec-install

You can show the installer version and available options by running the following command to show the help info:

./open-appsec-install -h

This interactive installer provides 2 alternative modes for automatic vs. manual installation:

Mode 1: Automatic installation of Open-appsec and adding attachment (plugin) to NGINX

In this mode, open-appsec will automatically be installed with all required components and the attachment will be added and activated in the existing configuration for NGINX/Kong.

./open-appsec-install --auto

Optional open-appsec installer parameters

--token allows connecting directly to SaaS management. We will be adding the agent later to the management portal, and so you can skip this.
--prevent will set the default rule in the default policy file to prevent-learn instead of detect-learn, but the recommendation is to keep detect-learn as the default rule.

Mode 2: Download of software components and presenting manual installation instructions

In this mode all required components based on your NGINX version, OS version, Platform will be downloaded to your machine and instructions are presented for manual installation.

./open-appsec-install --download

Optionally, you can add a --tmpdir <path> option to specify an alternative path for the downloaded software components (default path is /tmp/open-appsec/)

Once the download has finished, follow these steps for manual installation:

Step 1: Deploying the attachment on an existing alpine NGINX/Kong server

Copy the associated libraries as shown in the output for Step 1 with commands similar to this:

cp /tmp/open-appsec/[version specific dir]/libosrc_shmem_ipc.so /usr/lib/libosrc_shmem_ipc.so
cp /tmp/open-appsec/[version specific dir]/libosrc_compression_utils.so /usr/lib/libosrc_compression_utils.so
cp /tmp/open-appsec/[version specific dir]/libosrc_nginx_attachment_util.so /usr/lib/libosrc_nginx_attachment_util.so

Copy the nginx attachment file as shown in the output for Step 1 with command similar to this:

cp /tmp/open-appsec/[version specific dir]/ngx_cp_attachment_module.so /usr/lib/nginx/modules/ngx_cp_attachment_module.so

Load the attachment on your NGINX by adding the following line to your nginx.conf, usually located here: /etc/nginx/

load_module /usr/lib/nginx/modules/ngx_cp_attachment_module.so;

Step 2: Installing open-appsec agent

Run the following commands as shown in the output

/tmp/open-appsec/openappsec/install-cp-nano-agent.sh --install --hybrid_mode --server 'NGINX Server'
/tmp/open-appsec/openappsec/install-cp-nano-service-http-transaction-handler.sh --install

Step 3 Validate configuration

nginx -t
service nginx restart

How to use the open-appsec-ctl Tool

The interactive CLI tool open-appsec-ctl allows you to perform various tasks related to your open-appsec for NGINX/Kong installation. The tool will be automatically installed with the agent. To check the available options that can be used with the tool, run:

open-appsec-ctl -h

To list available policies, run open-appsec-ctl --list-policies or open-appsec-ctl -lp. Currently, only a single configuration file is supported, support for multiple configuration files will be added soon.

You can view, edit, apply policies and event manage the open-appsec agent using the open-appsec-ctl tool. For a full list of available options, check out their official documentation here.

Configuring open-appsec using Local Policy File

When using open-appsec for NGINX on Linux, the configuration can be done in a declarative way using a single YAML file which holds all the relevant configuration objects. This way it is fully compatible for integration in GitOps CD-based processes. Alternatively, management can also soon be done centrally via the open-appsec Web UI.

The default location of the declarative configuration file is /etc/cp/conf/open-appsec.yaml. You can show and edit the full default declarative configuration file using your default text editor like nano or vim or by using the interactive CLI tool open-appsec-ctl --edit-policy [policy-file].

The default policy within the default configuration file, which is created during the installation, contains the following setting which sets the mode to detect-learn for all web resources provided by the NGINX:

policies.default.mode: detect-learn

If you want attacks instead to be prevented by default for all web resources, you can change this as follows:

policies.default.mode: prevent-learn

The section policies.specific-rules allows you to create specific rule entries for specific hostnames, hostname-path combinations or paths which override the default policy.

It is recommended that once sufficient confidence was gained in detect-learn mode that you add specific rules in the policies.specific-rules section for those hostnames, hostname-path combinations or paths for which you want to switch to prevent-learn mode

You find the full specification for the different configuration sections and elements that can be part of the declarative configuration file, including an example for each, in the Advanced policy configuration documentation

Monitoring events

To monitor events, you can use the following command:

open-appsec-ctl --view-logs

By default, logs are stored here: /var/log/nano_agent/cp-nano-http-transaction-handler.log<number>

You can configure flexible logging according to your requirements by using a custom log-trigger. You can find details of how to configure the log trigger when using declarative configuration file in the Advanced policy configuration documentation

You can also view the logs on the web UI, as discussed below.

Using open-appsec with the Web UI

Open-appsec provides a cloud-hosted central management for assets and policies, cloud logging, graphical dashboards, events analysis and ability to manage multiple deployments/clusters in a scalable way.

You can find full details on how to use the Web UI on the official documentation

How Good is Open-appsec WAF?

There is no point of setting up a WAF if it does not do what it is supposed to do - protect the web resources. To test open-appsec strength, I used the waf-bypass tool by nemesida. I ran this tool against a website running WordPress 6.4.1 on Nginx. That version is old now, and the results below are screenshots from that run, treat them as directional rather than a current benchmark.

WAF bypass Tool is an open source tool to analyze the security of any WAF for False Positives and False Negatives using predefined and customizable payloads. Check your WAF before an attacker does. WAF Bypass Tool is developed by Nemesida WAF team with the participation of community.

💡

Note: - This is by NO means a comprehensive test on how good/bad open-appsec actually is. However, testing it side by side with other well known WAFs gives us a good indicator on how it might perform against its competitors

Running the waf-bypass tool while having open-appsec installed, gave the following result:

Running the same waf-bypass tool on the same website but with Cloudflare (free version) as the WAF, gave the following results:

To give as a different perspective of how well Open-appsec performs against other WAFs, the open-appsec team also ran some results, comparing the different WAFS in the market these results are from:

These results come from open-appsec's own team, so they are marketing material as much as measurement. Treat them as an indication of intent, not independent verification. Here is the link to the full comparison.

We will be doing a full test soon of this and many other WAFs

Conclusion

A WAF is one layer, and it is worth pairing with solid server configuration, how to harden NGINX covers the TLS and header side that sits underneath it.

Open-appsec is a solid option, especially for people looking to move away from ModSecurity but still want a flexible and open-source WAF. What I can say from running it is narrower than that: deployment was straightforward and the learning mode is genuinely low-touch. Whether the ML engine beats a well-tuned signature WAF is not something one test on one site can establish, and the comparison below is vendor-run. Treat the zero-day blocking claim as plausible and unproven rather than demonstrated. With a good backing of Checkpoint, a leader in the IT security space, and community contributions, I believe open-appsec is a good WAF. Feel free to check it out and leave your comments below.