diff --git a/README.md b/README.md index 3e5b3f2..3460929 100644 --- a/README.md +++ b/README.md @@ -1227,6 +1227,29 @@ If the Secret no longer appears, it is now unmanaged. **Error: failed to create fsnotify watcher: too many open files.** +When Alloy configured to collect logs (on_logs mode), the node can start to hit Linux inotify limits. Inotify is the kernel subsystem responsible for watching file changes, and Alloy relies heavily on it to monitor: + +- container log files +- rotated log files +- symlinks under /var/log/containers +- pod log directories under /var/log/pods + +In clusters with many pods, sidecars, and continuous log rotation, Alloy can easily require hundreds of thousands of inotify watches. + +The default Linux limits are extremely low for modern Kubernetes workloads. Once the node reached these limits, several components started failing to create watchers: + +- Alloy log pipeline +- K3s API server (which also uses file watchers internally) +- containerd and kubelet log watchers +- These failures produced high‑volume error logs such as: + +``` bash +fsnotify watcher: too many open files +http: superfluous response.WriteHeader call +``` + +Increasing the kernel limits resolved the issue immediately. + ``` bash cat /proc/sys/fs/inotify/max_user_watches cat /proc/sys/fs/inotify/max_user_instances @@ -1235,7 +1258,21 @@ cat /proc/sys/fs/inotify/max_user_instances **Increase Values example:** ``` bash -echo "fs.inotify.max_user_watches=1048576" | sudo tee -a /etc/sysctl.conf -echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.conf +# /etc/sysctl.conf +fs.inotify.max_user_watches=1048576" +fs.inotify.max_user_instances=1024" + +``` + +``` bash sudo sysctl -p ``` + +**If error continues try with this values:** + +``` bash +# /etc/sysctl.conf +fs.inotify.max_user_watches=2097152 +fs.inotify.max_user_instances=8192 +fs.inotify.max_queued_events=65536 +```