Remix clone Hacker News

Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)

We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.

After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.

All of this is fully automatic, of course.

▲

Arnavion 3 days ago | parent | next [-]

This will not work with HPKP but hopefully nothing is using that any more. ( https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )

It won't work with programs that defensively validate the cert chain but those are rare.

It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.

But yes for the vast majority of programs it should work fine.

▲

adtac 3 days ago | parent [-]

Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)

We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).

[1] https://news.ycombinator.com/item?id=42923998

▲

chatmasta 2 days ago | parent | next [-]

No general solution? :) If you came up with this hack I’m sure you can extend it…

Cert pinning has to read a public cert from memory, right? And a public cert has a well-known shape… and you have bpf and access to the memory…

	▲	adtac 2 days ago \| parent [-]
		in a different context, maybe: https://news.ycombinator.com/item?id=38044786 :)

▲

smw 2 days ago | parent | prev [-]

kubeshark [0] is using ebpf to catch calls to openssl/go's tls lib and thus no need to juggle certs. Has pros and cons compared to your method, but an interesting comparison.

[0] https://www.kubeshark.co/

	▲	ddelnano 2 days ago \| parent [-]
		The approach you describe above is common for similar projects: - Pixie (https://px.dev) -- which I contribute to - Beyla (https://github.com/grafana/beyla) - Coroot (https://github.com/coroot/coroot) If you are interested in the details and how the strategy for this tracing has evolved, you can learn more in this blog (https://blog.px.dev/ebpf-tls-tracing-past-present-future/).

▲

TachyonicBytes 3 days ago | parent | prev | next [-]

Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.

[1] https://news.ycombinator.com/item?id=42919909

	▲	adtac 3 days ago \| parent [-]
		httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.

▲

memhole 3 days ago | parent | prev | next [-]

Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?

Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.

▲

adtac 3 days ago | parent [-]

Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.

Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)

	▲	memhole 3 days ago \| parent [-]
		That's helpful! Thanks for clarifying. I'll have to check it out.

▲

cyberax 3 days ago | parent | prev | next [-]

I can't decide if I'm horrified or amazed by this :)

	▲	1oooqooq 2 days ago \| parent [-]
		as with anything too clever and undocumented, you will be amazed until you waste a few weeks debugging an issue caused by it.

▲

hassleblad23 3 days ago | parent | prev | next [-]

Wow. I think more network inspection tools should adopt an approach like this. Great job.

▲

chatmasta 2 days ago | parent | prev [-]

This is a great hack. It won’t work 100% of the time but it’ll be close, and damn it’s just so clean. Proxies and intercepts all the way down… I love this.