Remix.run Logo
alexpotato 3 days ago

At a past job (hedge fund), my role was to co-ordinate investigations into why latency may have changed when sending orders.

A couple of quants had built a random forest regression model that could take inputs like time of day, exchange, order volume etc and spit out an interval of what latency had historically been in that range.

If the latency moved outside that range, an alert would fire and then I would co-ordinate a response with the a variety of teams e.g. trading, networking, Linux etc

If we excluded changes on our side as the culprit, we would reach out to the exchange and talk to our sales rep there would might also pull in networking etc.

Some exchanges, EUREX comes to mind, were phenomenal at helping us identify issues. e.g. they once swapped out a cable that was a few feet longer than the older cable and that's why the latency increased.

One day, it's IEX, of Flash Boys fame, that triggers an alert. Nothing changed on our side so we call them. We are going back and forth with the networking engineer and then the sales rep says, in almost hushed tones:

"Look, I've worked at other exchange so I get where you are coming from in asking these questions. Problem is, b/c of our founding ethos, we are actually not allowed to track our own internal latency so we really can't help you identify the root cause. I REALLY wish it was different."

I love this story b/c HN, as a technology focused site, often thinks all problems have technical solutions but sometimes it's actually a people or process solution.

Also, incentives and "philosophy of the founders" matter a lot too.

philipov 3 days ago | parent | next [-]

What kind of founding ethos doesn't allow tracking internal latency? Is their founding ethos "Never Admit Responsibility?"; "Never Leave A Paper Trail?"

This company's official ethical foundation is "Don't Get Caught."

WJW 3 days ago | parent | next [-]

From the wiki about IEX: "It was founded in 2012 in order to mitigate the effects of high-frequency trading." I can see how they don't want to track internal latency as part of that, or at least not share those numbers with outsiders. That just encourages high frequency traders again.

pants2 3 days ago | parent [-]

One would hope for a more technical solution to HFT than willful ignorance lol. For example, they could batch up orders every second and randomize them.

atomicnumber3 3 days ago | parent | next [-]

I worked in HFT. (Though am now completely out of fintech and have no skin in the game). "Flash Boys" traditional HFT is dead already, the trade collapsed in 2016-2018 when both larger institutions got less dumb with order execution, and also several HFTs "switched sides" and basically offered "non-dumb order execution" as a service to any institutions who were unable to play the speed game themselves. Look at how Virtu's revenue changed from mostly trading to mostly order execution services over that time period.

Flash Boys was always poorly researched and largely ignorant of actual market microstructure and who the relevant market participants were, but it also aged quite poorly as all of their "activism" was useless because the market participants just all smartened up purely profit-driven.

If you want to be activist about something, the best bet for 2026 is probably that so much volume is moving off the lit exchanges into internal matching and it degrades the quality of price discovery happening. But honestly even that's a hard sell because much of that flow is "dumb money" just wanting to transact at the NBBO.

Actually, here's the best thing to be upset about: apps gamifying stock trading / investing into basically SEC-regulated gambling.

Workaccount2 3 days ago | parent | prev [-]

This is what should happen, because what the game actually being played is to profit off those who cannot react fast enough to news event, rather than profit off those who mispriced their order.

Or leave things in place, but put a 1 minute transaction freeze during binary events, and fill the order book during that time with no regard for when an order was placed, just random allocation of order fills coming out of the 1 minute pause.

These funds would lose their shit if they had to go back to knowledge being the only edge rather than speed and knowledge.

neonbrain 2 days ago | parent [-]

This isn't a good approach because it assumes there are no market makers on trading venues, and that they (as well as exchanges) do not compete for order flow. Also, maybe you haven't noticed, but stocks are often frozen during news announcements by regulatory request, so such pauses are already in place and are designed to maintain market integrity, not disrupt it with arbitrary fills.

stuxnet79 2 days ago | parent | prev [-]

The founder Brad Katsuyama talks about his background and motivation for starting the company here - https://www.youtube.com/watch?v=N9hoqFpDjVs

It might add a bit of color to this conversation.

lopatin 2 days ago | parent | prev | next [-]

Can you talk a bit more about the incentives to trade latency sensitive strategies on IEX in the first place? Is it still lucrative for its liquidity despite them artificially slowing down orders? Does a meta game evolve with HFTs all working around their system, essentially making it still a HFT playground but with extra steps? Do you think their unexpected latency increase for you guys was intentional, to free the water from sharks?

noitpmeder 3 days ago | parent | prev | next [-]

Curious what your actual role was -- sounds very interesting! Project manager? Dev? Operations specialist? E.g. were you hired into this role, and what were the requisites?

alexpotato 3 days ago | parent [-]

I was what was called "Trade Desk".

Many firms have them and they are a hybrid of:

- DevOps (e.g. we help, or own, deployments to production)

- SRE (e.g. we own the dashboards that monitored trading and would manage outages etc)

- Trading Operations (e.g. we would work with exchanges to set up connections, cancel orders etc)

My background is:

- CompSci/Economics BA

- MBA

- ~20 years of basically doing the above roles. I started supporting an in house Order Management System at a large bank and then went from there.

For more detail, here is my LinkedIn: https://www.linkedin.com/in/alex-elliott-3210352/

I also have a thread about the types of outages you see in this line of work here: https://x.com/alexpotato/status/1215876962809339904?s=20

(I have a lot of other trading/SRE related threads here: https://x.com/alexpotato/status/1212223167944478720?s=20)

noitpmeder 3 days ago | parent | next [-]

Thanks for all the info!

I'm a front office engineer at a prop firm -- always interesting to get insight into how others do it.

We have fairly similar parallels, maybe with the exception of throwing new exchange connections to the dedicated networking group.

Always love watching their incident responses from afar (usually while getting impacted desks to put away the pitchforks). Great examples of crisis management, effectiveness and prioritization under pressure, ... All while being extremely pragmatic about actual vs perceived risk.

(I'm sure joining KCG in August of 2012 was a wild time...)

alexpotato 3 days ago | parent [-]

You are very welcome!

It's definitely a job that you don't hear much about but has a lot of interesting positives for people who like technology and trading. Especially if you prefer shorter term, high intensity work vs long term projects (e.g. like developers).

> Always love watching their incident responses from afar

I actually have a thread on that too: https://x.com/alexpotato/status/1227335960788160513?s=20

> (I'm sure joining KCG in August of 2012 was a wild time...)

And not surprisingly, a thread on that as well: https://x.com/alexpotato/status/1501174282969305093?s=20

8cvor6j844qw_d6 3 days ago | parent | prev [-]

May I know if someone with no trading knowledge can get into this field? Or do new hires that you've seen generally have some background knowledge on related to trading, etc.?

I did consider applying for a role in a very similar field, but figured I'll be fighting an uphill battle with no knowledge in trading/stock market/etc.

alexpotato 3 days ago | parent | next [-]

So I wrote up how I ended there: https://x.com/alexpotato/status/1663668616233885699?s=20

but that story is not the most efficient way (although I do talk about a better approach at the end).

To summarize:

A LOT of hedge funds hire non finance people for specific roles e.g. cloud, Linux tuning, networking etc

The smarter ones have realized that there are great people everywhere e.g. Gaming company SREs have a lot of relevant experience due to high traffic load, short SLAs and lots of financial risk due to outages. Applying for a role in one of those departments is a lot easier than trying to jump directly into a trading desk/operations role.

Finally, knowing someone on the inside also helps a lot which is made MUCH easier by LinkedIn, Twitter etc

TheJoeMan 3 days ago | parent | prev [-]

You know how coders are expected to grind out leetcode interviews? For the finance fields, a common interview topic is what you read in “The Journal” (WSJ). So just stay on top of it for a few weeks, see some trends, etc.

kasey_junk 2 days ago | parent [-]

I’ve worked in finance for 25 years and never even heard of this coming up in an interview.

noitpmeder 2 days ago | parent [-]

Imo it's not out of place in context if one is trying to determine if the candidate has an interest in finance.

Now, I'm not disqualifying them if they dont read the journal. But if they can't demonstrate any proactive interest in finance, or tell me about some happenings/events/stories they personally find interesting (theres a ton of interesting stuff happening) its definitely an amber flag.

reactordev 3 days ago | parent | prev | next [-]

All technical problems are people problems

dboreham 3 days ago | parent | prev [-]

> they once swapped out a cable that was a few feet longer than the older cable and that's why the latency increased

That was not why. Possibly the cable made a difference (had an open circuit that made the NICs back down to a lower speed; noisy leading to retransmissions) but it wasn't the length per se.

thadt 2 days ago | parent | next [-]

Well, it depends on the granularity of the time scale right? When you're measuring milliseconds, then the cable length probably isn't a thing factoring into your latency calculation.

When we're measuring time on the scale of nanoseconds then, yes, cable length is definitely something we care about and will reliably show up in measurements. In some situations, we not only care about the cable length, but also its temperature.

thijson 3 days ago | parent | prev [-]

It can be the length, there's about 1ns latency per foot.