Gather metrics for Docker img filling up
Nearly everyday my docker image size (20gb total) goes from ~60% usage to 70%+ usage and the alarms start going off (70%). The image size fluctuates over the next few hours (69%, 72%, etc.) so a ton of alarms go off. I believe the "Container" and "Writeable" are growing quickly due to Plex traffic but am not sure that is the only problem and want to understand all my applications a bit more. I am not interested in increasing the docker image size or changing when the alarms go off, I want to understand which applications are consuming the space and why.
I have analyzed all my active docker containers and added a max log setting where applicable, when the log became large (>50mb), so I don't believe runaway logs are the problem:
--log-opt max-size=50m --log-opt max-file=1
I know I can get a current snapshot of my docker usage by clicking the "Container Size" button on the /Docker page but that is only a current snapshot and it isn't fast enough or historical. I was thinking about collecting this data and placing it into a time series DB (InfluxDB or Prometheus) and graphing it (Grafana).
I am most familiar with Prometheus but am not surer how to collect the data from:
/mnt/user/system/docker/docker.img
Any tips? Is this the best way to get this information/solve this problem? Has this been done before?
Thanks!
