modified: README.md
This commit is contained in:
1 parent
9168ab5581
commit
e4db94be2e
1 file changed
+39
-2
@@ -1227,6 +1227,29 @@ If the Secret no longer appears, it is now unmanaged.
|
|||||||
|
|
||||||
**Error: failed to create fsnotify watcher: too many open files.**
|
**Error: failed to create fsnotify watcher: too many open files.**
|
||||||
|
|
||||||
|
When Alloy configured to collect logs (on_logs mode), the node can start to hit Linux inotify limits. Inotify is the kernel subsystem responsible for watching file changes, and Alloy relies heavily on it to monitor:
|
||||||
|
|
||||||
|
- container log files
|
||||||
|
- rotated log files
|
||||||
|
- symlinks under /var/log/containers
|
||||||
|
- pod log directories under /var/log/pods
|
||||||
|
|
||||||
|
In clusters with many pods, sidecars, and continuous log rotation, Alloy can easily require hundreds of thousands of inotify watches.
|
||||||
|
|
||||||
|
The default Linux limits are extremely low for modern Kubernetes workloads. Once the node reached these limits, several components started failing to create watchers:
|
||||||
|
|
||||||
|
- Alloy log pipeline
|
||||||
|
- K3s API server (which also uses file watchers internally)
|
||||||
|
- containerd and kubelet log watchers
|
||||||
|
- These failures produced high‑volume error logs such as:
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
fsnotify watcher: too many open files
|
||||||
|
http: superfluous response.WriteHeader call
|
||||||
|
```
|
||||||
|
|
||||||
|
Increasing the kernel limits resolved the issue immediately.
|
||||||
|
|
||||||
``` bash
|
``` bash
|
||||||
cat /proc/sys/fs/inotify/max_user_watches
|
cat /proc/sys/fs/inotify/max_user_watches
|
||||||
cat /proc/sys/fs/inotify/max_user_instances
|
cat /proc/sys/fs/inotify/max_user_instances
|
||||||
@@ -1235,7 +1258,21 @@ cat /proc/sys/fs/inotify/max_user_instances
|
|||||||
**Increase Values example:**
|
**Increase Values example:**
|
||||||
|
|
||||||
``` bash
|
``` bash
|
||||||
echo "fs.inotify.max_user_watches=1048576" | sudo tee -a /etc/sysctl.conf
|
# /etc/sysctl.conf
|
||||||
echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.conf
|
fs.inotify.max_user_watches=1048576"
|
||||||
|
fs.inotify.max_user_instances=1024"
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
``` bash
|
||||||
sudo sysctl -p
|
sudo sysctl -p
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**If error continues try with this values:**
|
||||||
|
|
||||||
|
``` bash
|
||||||
|
# /etc/sysctl.conf
|
||||||
|
fs.inotify.max_user_watches=2097152
|
||||||
|
fs.inotify.max_user_instances=8192
|
||||||
|
fs.inotify.max_queued_events=65536
|
||||||
|
```
|
||||||
Reference in new issue
Block a user