Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -150,19 +150,30 @@ The table below shows the average latency of 2-10% benchmarked using the Redis b

The benchmark above shows average TPS of Protect mode versus Monitor mode, and the latency added for Protect mode for several tests in the benchmark. The main way to lower the actual latency (microseconds) in Protect mode is to run on a system with a faster CPU. You can find more details on this open source Redis benchmark tool at https://redis.io/topics/benchmarks.

### Adding Scaling Constraints for Large Workload Environments
### Adjusting System Limits for Large Workload Environments

During NeuVector installation, if your host operating system has a large amount of workloads then the NeuVector Enforcer pods can fail to spin up when trying to open the large volume of files due to the pods host monitoring. This can also cause RKE2-server failures because of the large amounts of open files.
In environments with a high volume of workloads or containers, NeuVector Enforcer pods may fail to start or operate properly due to host monitoring opening a large number of file descriptors and inotify watches. This constraint can also cause control plane services (such as `rke2-server`) to fail.

As a workaround for large workload environments, you need to create a file such as `example-fs-max.conf` in the location `/etc/sysctl.d/` and add scaling constraints with the following configuration:
To accommodate larger workloads, you can increase the inotify and file handle limits by creating a configuration file (for example `example-fs-max.conf` in the location `/etc/sysctl.d/`) on your host nodes.

:::note
Different Linux distributions and kernel versions configure different default values for these limits. Review your system's current defaults using `sysctl <parameter_name>` and set values appropriate for your environment's scale.
:::

An example configuration:

```shell
# Increase inotify instances and watches to prevent monitoring failures
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288
fs.filemax=5000

# Adjust global system file handle limits if your OS default is too low.
# Note: Modern distributions often set fs.file-max to a high default or unlimited.
# Only adjust fs.file-max if your system default is restrictive.
# fs.file-max=2097152
```

Then ensure the configuration is applied with a restart via the following command:
Apply the updated configuration by restarting the `systemd-sysctl` service:

```shell
systemctl restart systemd-sysctl
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,19 +150,30 @@ The table below shows the average latency of 2-10% benchmarked using the Redis b

The benchmark above shows average TPS of Protect mode versus Monitor mode, and the latency added for Protect mode for several tests in the benchmark. The main way to lower the actual latency (microseconds) in Protect mode is to run on a system with a faster CPU. You can find more details on this open source Redis benchmark tool at https://redis.io/topics/benchmarks.

### Adding Scaling Constraints for Large Workload Environments
### Adjusting System Limits for Large Workload Environments

During NeuVector installation, if your host operating system has a large amount of workloads then the NeuVector Enforcer pods can fail to spin up when trying to open the large volume of files due to the pods host monitoring. This can also cause RKE2-server failures because of the large amounts of open files.
In environments with a high volume of workloads or containers, NeuVector Enforcer pods may fail to start or operate properly due to host monitoring opening a large number of file descriptors and inotify watches. This constraint can also cause control plane services (such as `rke2-server`) to fail.

As a workaround for large workload environments, you need to create a file such as `example-fs-max.conf` in the location `/etc/sysctl.d/` and add scaling constraints with the following configuration:
To accommodate larger workloads, you can increase the inotify and file handle limits by creating a configuration file (for example `example-fs-max.conf` in the location `/etc/sysctl.d/`) on your host nodes.

:::note
Different Linux distributions and kernel versions configure different default values for these limits. Review your system's current defaults using `sysctl <parameter_name>` and set values appropriate for your environment's scale.
:::

An example configuration:

```shell
# Increase inotify instances and watches to prevent monitoring failures
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288
fs.filemax=5000

# Adjust global system file handle limits if your OS default is too low.
# Note: Modern distributions often set fs.file-max to a high default or unlimited.
# Only adjust fs.file-max if your system default is restrictive.
# fs.file-max=2097152
```

Then ensure the configuration is applied with a restart via the following command:
Apply the updated configuration by restarting the `systemd-sysctl` service:

```shell
systemctl restart systemd-sysctl
Expand Down
Loading