|Nel mezzo del cammin di nostra vita|
mi ritrovai per una selva oscura,
ché la diritta via era smarrita.
|Midway upon the journey of our life|
I found myself within a forest dark,
For the straightforward pathway had been lost.
Divina Commedia, Inferno, Canto I
I had the chance to finally sit down and find a way to inspect TLS traffic flowing out of an application running on my machine. Although I did not invent anything, I needed to put together a lot of different tricks in order to succeed, and the documentation I could find online regarding this process is scattered, at best. So, here we are with a guide on “how to inspect TLS encrypted traffic without going nuts”. Hope you enjoy!
The opportunity to learn how to inspect TLS encrypted traffic came out of the need to tweak Vector to use it to create the Koyeb log exporter. Since I had not yet mastered the whole codebase, I was not sure whether the code I added was producing the correct requests. Hooking into the code to show what it was sending was way too complicated. On the other end, the endpoint was a production system and I had no chance of verifying that the request was properly formatted.
With these limitations being clear, and with no possibility to downgrade the traffic to an unencrypted form anywhere, I realized my only option was to try to inspect the live stream of data, provided I could find a way to decipher it.
Let me set the stage for this show. I am on Arch Linux, but the following procedure will work on any distribution that includes
systemd. It probably could also be ported to Docker containers, but I didn’t try. I will guide you through this journey using the following software (if you want to follow along, make sure to grab them with your favorite package manager):
I will prepare a container to run the application. The container will route all the traffic to
mitmdump (one of the programs of the
mitmproxy suite) running on the host OS. The link between the container and
mitmdump will use a
wireguard tunnel. On the host, we will use
wireshark to live-inspect and record the traffic of interest.
As outlined above, I will run the software whose traffic I want to inspect in a container. This is not a usual Docker container, but rather a
systemd-nspawn managed container (after all, containers are an abstraction that rely on some Linux kernel features). The reason for this is twofold:
systemd-nspawnthat creates an ephemeral container using my whole OS host as a blueprint.
Let me cut this short and go directly to the command line I used. I ran this from the root of the source code of the project I am working on:
sudo systemd-nspawn \ --boot \ --ephemeral \ --hostname=virgil \ --machine=virgil \ --network-veth \ --bind-ro=$PWD:/workdir \ --directory /
systemd-nspawn to start (
--boot) an ephemeral (
--ephemeral) container, with hostname (
--hostname) and named as (
--machine) virgil. It creates a virtual ethernet interface between the container and the host (
--network-veth), mounting the current directory to
--bind-ro=$PWD:/workdir), using my root directory as blueprint (
--directory /). This last bit is zero cost on my system because I use btrfs as filesystem for my root partition with CoW enabled. Booting the container like this is a zero-cost operation.
After this, from inside the container, I take notice of the IP of the gateway with:
ip route|grep default
This information will be used in the next step to properly configure the transparent proxying. My current value is
Now, back on the host OS, I start
mitmdump to start the intercepting side:
SSLKEYLOGFILE="$HOME/.mitmproxy/sslkeylogfile.txt" \ mitmdump --mode firstname.lastname@example.org:12345
mitmdump to save the TLS session key in a file at the specified path. This will be needed by
wireshark to live-inspect the (otherwise encrypted) TLS traffic. I am starting the program specifying it to use the
wireguard mode (
--mode email@example.com:12345), binding to the IP I retrieved before (the one used as the default gateway by the container).
mitmdump outputs something like the following:
[15:55:08.494] WireGuard server listening at 192.168.184.145:12345. ------------------------------------------------------------ [Interface] PrivateKey = aFXQvmS1avL+n5zxSoN2a5VAS98+j3yGQp1V8H7Tt2U= Address = 10.0.0.1/32 DNS = 10.0.0.53 [Peer] PublicKey = 1dtIcVOQmjdaTboceokPMMSXTRcDIn70LAXb+fieOU8= AllowedIPs = 0.0.0.0/0 Endpoint = 192.168.1.134:51280 ------------------------------------------------------------
The lines between the dashes are an (almost) valid configuration for the container to connect to
wireguard. I wrote almost because there is a bug (or is it a feature?) in the
mitmproxy code that prevents it from outputting this configuration with the address specified in the command line. Instead, it always uses the address of the internet-facing interface (
192.168.1.134, in this case). This has to be replaced with the address specified above:
[Interface] PrivateKey = aFXQvmS1avL+n5zxSoN2a5VAS98+j3yGQp1V8H7Tt2U= Address = 10.0.0.1/32 DNS = 10.0.0.53 [Peer] PublicKey = 1dtIcVOQmjdaTboceokPMMSXTRcDIn70LAXb+fieOU8= AllowedIPs = 0.0.0.0/0 Endpoint = 192.168.184.145:12345
I put this content in a file at
/root/mitm.conf inside the container. I also copy the
mitmdump CA that is used to perform the MITM attack from the host (located at
~/.mitmproxy/mitmproxy-ca-cert.pem) to the container (in
/root/mitm-ca.pem). Now, I have to add it to the trust store of the container:
trust anchor --store /root/mitm-ca.pem
Then, connect to
mitmdump, again from inside the container:
wg-quick up /root/mitm.conf
Now we are ready to decrypt the TLS encrypted traffic.
I now have all the pieces in place to start inspecting the traffic with
wireshark. Back on the host, start
wireshark. To tell it how to decrypt the TLS encrypted traffic, just go to
Edit -> Preferences -> Protocols -> TLS -> (Pre)-Master-Secret log filename and input the path specified in the
SSLKEYLOGFILE environment variable above.
wireshark to listen to the traffic on the host OS public interface (the one where the default route is. For example the wireless interface if you are connected through WiFi). Note that both
wireshark listen to the same traffic, but while the former alters it to perform the MitM, the latter just displays it.
We can now follow the decrypted streams!
It was challenging, but now I am able to live-inspect the traffic and validate the changes I make to the code in real time!
This is really useful in cases like the one I faced, where I could not be sure that the requests I was sending were properly formatted. Although the setup is admittedly not straightforward, I hope this information is useful to you should you ever embark on your own journey to
inferno inspect TLS encrypted traffic.
If you’d like to reach out about this article, write to us on the Koyeb Community.
Koyeb is a developer-friendly serverless platform to deploy any apps globally.Start for free
Deploy 2 services for free and enjoy our predictable pricing as you grow
Get up and running in 5 minutes