modified: README.md
This commit is contained in:
1 parent
9168ab5581
commit
e4db94be2e
1 file changed
+39
-2
@@ -1227,6 +1227,29 @@ If the Secret no longer appears, it is now unmanaged.
|
||||
|
||||
**Error: failed to create fsnotify watcher: too many open files.**
|
||||
|
||||
When Alloy configured to collect logs (on_logs mode), the node can start to hit Linux inotify limits. Inotify is the kernel subsystem responsible for watching file changes, and Alloy relies heavily on it to monitor:
|
||||
|
||||
- container log files
|
||||
- rotated log files
|
||||
- symlinks under /var/log/containers
|
||||
- pod log directories under /var/log/pods
|
||||
|
||||
In clusters with many pods, sidecars, and continuous log rotation, Alloy can easily require hundreds of thousands of inotify watches.
|
||||
|
||||
The default Linux limits are extremely low for modern Kubernetes workloads. Once the node reached these limits, several components started failing to create watchers:
|
||||
|
||||
- Alloy log pipeline
|
||||
- K3s API server (which also uses file watchers internally)
|
||||
- containerd and kubelet log watchers
|
||||
- These failures produced high‑volume error logs such as:
|
||||
|
||||
``` bash
|
||||
fsnotify watcher: too many open files
|
||||
http: superfluous response.WriteHeader call
|
||||
```
|
||||
|
||||
Increasing the kernel limits resolved the issue immediately.
|
||||
|
||||
``` bash
|
||||
cat /proc/sys/fs/inotify/max_user_watches
|
||||
cat /proc/sys/fs/inotify/max_user_instances
|
||||
@@ -1235,7 +1258,21 @@ cat /proc/sys/fs/inotify/max_user_instances
|
||||
**Increase Values example:**
|
||||
|
||||
``` bash
|
||||
echo "fs.inotify.max_user_watches=1048576" | sudo tee -a /etc/sysctl.conf
|
||||
echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.conf
|
||||
# /etc/sysctl.conf
|
||||
fs.inotify.max_user_watches=1048576"
|
||||
fs.inotify.max_user_instances=1024"
|
||||
|
||||
```
|
||||
|
||||
``` bash
|
||||
sudo sysctl -p
|
||||
```
|
||||
|
||||
**If error continues try with this values:**
|
||||
|
||||
``` bash
|
||||
# /etc/sysctl.conf
|
||||
fs.inotify.max_user_watches=2097152
|
||||
fs.inotify.max_user_instances=8192
|
||||
fs.inotify.max_queued_events=65536
|
||||
```
|
||||
Reference in new issue
Block a user