I guess the question is why your observability stack isn't exposing proc and sys for you.

Mine (prometheus) doesn’t because there are a lot of high-dimensional values to track in /proc and /sys that would blow out storage on a time-series database. Even if they did though, they could not let me actively inject changes to a cgroup. What do you suggest I try that does?

▲

jeffbee 6 hours ago | parent [-]

Experience from another company where I (and you) worked suggests that having the endpoints to expose the system metrics, without actually collecting and storing them, is the way to go.

▲

crawshaw 6 hours ago | parent [-]

Years of debugging in that company’s restricted environments solidified my desire for shell access to production environments. I was there a month before I was hunting for breadcrumbs in a BINARY_INFO log that I had five minutes to grab before it was deleted.

	▲	jeffbee 6 hours ago \| parent [-]
		Well that's funny you mentioned it because one of my projects was a service that let users temporarily install binary info logs collectors triggered by predicates, remotely, which at least I thought was a better model than ssh into the host or, for the advanced caveman, pdsh into many hosts. I don't really see a reason why I can't do that for gRPC, either ... But, anyway, remote command and control of observability really is a thing in the industry, not just at one company.