BLURB
Andrew Hunter likes making code go fast. Before joining Jane Street,
he worked for seven years at Google on multithreaded architecture, and
was a tech lead for tcmalloc, Google’s world-class scalable malloc
implementation. In this episode, Andrew and Ron discuss how,
paradoxically, in some ways it’s easier to optimize systems at
hyperscale because of the impact that even miniscule changes can
have. Finding performance wins in trading systems, which operate at a
smaller scale, but which have bursty, low-latency workloads, is often
trickier. Andrew explains how he approaches the problem, including his
favorite profiling techniques and visualization tools; the unique
challenges of optimizing OCaml versus C++; and when you should and
shouldn’t care about nanoseconds. They also touch on the joys of
musical theater, and how to pass an interview when you’re
sleep-deprived.
SUMMARY
Andrew Hunter likes making code go fast. Before joining Jane Street,
he worked for seven years at Google on multithreaded architecture, and
was a tech lead for tcmalloc, Google’s world-class scalable malloc
implementation. In this episode, Andrew and Ron discuss how,
paradoxically, in some ways it’s easier to optimize systems at
hyperscale because of the impact that even miniscule changes can
have. Finding performance wins in trading systems, which operate at a
smaller scale, but which have bursty, low-latency workloads, is often
trickier. Andrew explains how he approaches the problem, including his
favorite profiling techniques and visualization tools; the unique
challenges of optimizing OCaml versus C++; and when you should and
shouldn’t care about nanoseconds. They also touch on the joys of
musical theater, and how to pass an interview when you’re
sleep-deprived.
Some links to topics that came up in the discussion:
TRANSCRIPT
00:03
Ron
Welcome to Signals and Threads in-depth conversations about every
layer of the tech stack from Jane Street. I’m Ron Minsky. It’s my
pleasure to introduce Andrew Hunter. Andrew is a software engineer
who’s worked here for the last five years and he’s currently working
on our market data team. Andrew is, by inclination and background, an
expert performance engineer and performance is really what we’re going
to talk about today. Andrew, just to start off, can you tell me a
little bit about how you got into performance engineering and a little
bit more kind of about what your path was to getting into software
engineering in general?
00:36
Andrew
I can, but I’m just going to be lying to you because the problem is I
can give you all sorts of reasons why I do this, but the real reason
is just I find it really addictive. It’s just hard for me not to get
excited about how systems work. There’s all sorts of reasons why it’s
cool or complicated or fun, but I just get this electric high when I
make something faster. I guess it’s not a path, right?
00:54
Ron
Right. So I kind of know why you like it, but I’m curious how you got
here. One thing that I think of as characteristic of performance
engineering and actually lots of different software engineering
disciplines is part of how you get really good at them is getting
really interested in stuff that’s objectively kind of boring. You get
super psyched about the details of how CPUs work and interconnects and
compilers and just all sorts of these little pieces and there’s just a
lot of knowledge you need to build up over time to be really good at
it. And I’m always interested of what were people’s paths where they
built up that kind of knowledge and understanding and background that
lets them really dive in deeply in the way that’s necessary.
01:27
Andrew
Well, I think that’s exactly right. I just have to care deeply about
all the parts of the board. And the way I got there: here was a couple
of interesting times when in college, for example, I was taking an
operating systems class and I realized that the best way to study this
and to learn it well was to just go into the details of the code. And
it’s like whenever we’re talking about a topic about virtual memory or
whatever, I would go look at Linux’s virtual memory implementation,
I’d see what it looked like and I’d have more questions. And I just
kept asking these questions and I never said, well, that’s out of
scope. And I just kept finding these things interesting. And from then
I just realized that like you say, all of these little details matter
and if you keep caring about them and you just don’t accept no for an
answer, you get pushed towards the places where people really do care
about this, which often means performance. And then once you start
doing performance to get that little high that I’ve talked about.
02:15
Ron
So in what context did you first experience working in a kind of
serious way on performance sensitive systems?
02:22
Andrew
Grad school at the very least, where one of the projects I worked on
was this big graph traversal system that was trying to replicate some
really weird complicated hardware and do it in software but maintain
reasonable levels of performance. And we just had to think really
carefully about, okay, how wide is this memory controller? How many
accesses can it do in parallel? What happens when one of them stalls?
Wait, what does this even mean to have parallel memory accesses? How
many cycles do these things take? Because we were roughly trying to
replicate this really complicated chip in software, which meant you
had to know exactly how would the original hardware have worked and
how did all the parts of it that you can replicate in software work
and you end up looking up all these bizarre details and you learn so
much about it.
02:59
Ron
Is that work that you went in thinking you would spend your time
thinking about fancy math-y graph algorithms and ended up spending
most of your time thinking about gritty operating system and hardware
details?
03:08
Andrew
A little bit. I definitely thought there was going to be a little bit
more algorithmic content, but I really rapidly realized that the hard
and interesting part here was in fact just, oh God, how do you keep
this much stuff in flight? And the hardware has actually gotten way
better or more aggressive about these sort of things over time. So I’m
glad I learned that.
03:23
Ron
So how did that work in grad school end up leading to you working
professionally in this area?
03:28
Andrew
Well, I was interning at Google at the time for the summers and I kind
of realized that I could do the same large-scale systems research that
I was doing in grad school in a place that just had a lot more scale
and a lot more usage of it. A lot of grad school research is done for
the point of doing the research, whereas the proper industrial
research, the coolest part is it just shows up in production and
suddenly people care about it.
03:48
Ron
And this is a challenge for lots of different kinds of academic work
where there’s some part of it that really only connects and makes
sense at certain scales and a lot of those scales are just only
available inside of these very large organizations that are building
enormous systems.
04:01
Andrew
Well, that’s true, but I think even more than just the scale issue,
it’s the issue of what happens when this actually meets real data. And
I think this isn’t just true about performance. One thing I will tell,
for example, the most common question I get towards the end of an
internship is like what’s going to be different when I come back as a
full-timer? And what I tell them is that I was shocked the first time
that I had a professional project in my first real job and I finished
it and we submitted it to the code base and it rolled into production
and everyone was using it. And then a couple weeks later I got an IM
from somebody in some other group saying, Hey, we’re using your system
and it’s behaving weirdly in this way. What do you think about it? And
my mental reaction to this was like, “What do you mean? I turned it
in, I got an A!” You have to actually keep this going until you can
hand it off to someone else or quit, right? Which sounds depressing,
but at the same time means you actually see what it does in reality
and under fire and then you learn how to make it even better and you
get to do something even more complicated and just this virtuous cycle
of optimizations that you get, or features, depending what you’re
working on. Right?
04:58
Ron
Yeah. This always strikes me when I go off and give lectures at
various universities of, it’s really hard to teach software
engineering in an academic context because there is a weird thing that
all of the work writing software in a university context, this weird
kind of performance art where you’re like, you create this piece of
software and then it gets created and poof, it vanishes like a puff of
smoke and this is just not what the real world is like. Alright, so
let’s get back to performance engineering. One thing I’m curious about
is you have a bunch of experience at Google thinking about the kind of
performance engineering problems you ran into there and also thinking
about it here in a number of different spots, and I’m curious how
those two different problems feel different to you.
05:36
Andrew
The difference between performance engineering at Google and
performance engineering at Jane Street to me is fundamentally one of
leverage. The easy thing about performance engineering at a place like
Google or any of the other hyperscalers is that they operate with so
many machines consuming so many cycles, doing so many things that any
optimization that moves the needle on how much CPU you consume is
really valuable, which means that it’s very easy to find good targets,
it’s very easy to find things that are worth doing and it may be very
difficult to fix those problems. You have to think really carefully
and you have to understand these systems, but the return on that
investment means you can really support a lot of people who just sit
there doing nothing else other than how do I make memory allocation
faster? How do I make serialization faster? What can I do in my
compiler to just optimize code generation and compression and all
these things?
There’s actually a really interesting paper. It’s called Profiling a
Warehouse Scale Computer, which looked at, okay, if you just look at
all the things that a data center does for one of these hyperscalers,
the business logic is really, really, really diverse. Some things are
serving videos of cats and some things are doing searches or social
networking or whatever. And all of this does different stuff, but it
all uses the same infrastructure. And it turns out that the
infrastructure is a huge percentage. They coined the term that I like
a lot, the data center tax and the 10, 15, 20% of your cycles that you
spend on low level infrastructure that everything uses. And it’s not
even that that infrastructure is bad or slow, it’s just that’s the
common link that scales. Whereas fixing business logic kind of
doesn’t.
07:14
Ron
Right? It takes individual engineer effort on each individual piece of
business logic, but everyone’s using the same compiler or one of a
small set of compilers. Everyone’s using just a small set of operating
systems. And so you grab those lower levels of the infrastructure and
you optimize them and you can just improve everyone.
07:31
Andrew
Yeah, that’s exactly right. And you can improve it enough by just
moving things by half percent. Making logging cheaper just pays for
itself way, way more easily than it does if you’re not operating at
that level of scale, which means that you get this nice cycle where
you hit one hotspot and you point another profile profiler at the
system as a whole, you see the next hotspot and you just get better
and better at this just by doing the really obvious thing that sits in
front of your face. So I like to think of this as easy mode, not
because the optimizations are easy or because the work is easy or
because it doesn’t take skill, but just because it’s really clear what
to do.
08:05
Ron
It is a target rich environment.
08:06
Andrew
It’s a really target rich environment. There’s money falling from the
sky if you make something faster.
08:10
Ron
Right? In some sense, this has to do with cost structure. This works
best in an organization where a large amount of the costs are going to
the actual physical hardware that you’re deploying.
08:19
Andrew
Right. When we say that the business logic doesn’t matter, what it
really means is we just don’t really care what you’re working on. You
can be serving videos of cats, you can be doing mail, you can do
whatever you want. We don’t really have to care. As long as you make
something that everyone uses a little bit faster, it’ll pay for itself
because the only thing you care about is the number of CPUs you buy,
the amount of power you buy and the amount of user queries you
service. That’s the only three things that matter.
08:43
Ron
It’s not that the business logic doesn’t matter, and in fact,
optimizing the business logic might be the single most impactful thing
you can do to improve a given application.
08:51
Andrew
But it’s harder.
08:51
Ron
But the business logic doesn’t matter to you because you are working
in the bowels of the ship fixing the things that affect everyone, and
it’s kind of impossible for you to at scale improve the business
logic. So you are focused on improving the things you can improve.
09:04
Andrew
That’s exactly right.
09:06
Ron
Great. So how’s it different here?
09:08
Andrew
We don’t have that scale. The amount of total compute that we spend is
a fair bit of money, but it’s not enough that making things 1% faster
matters, which means that the average CPU cycle we spend is just not
very important. It’s kind of worthless. If you make logging faster,
everyone’s kind of gonna shrug and say like, okay, but that doesn’t
change the dial. In fact, a surprising thing is that most of our
systems spend most of their CPU time intentionally doing nothing. It
is just table stakes in a trading environment that you do user space
polling IO, you just sit there spinning in a hard loop on the CPU
waiting for a packet to arrive, and most of the time there’s nothing
there. So if you point a profiler at a trading system, it’s going to
tell you it’s been 95, 99% of its time doing nothing.
09:49
Ron
And actually at this point, I want to push back a little bit on this
narrative because when you say most of the systems are doing nothing,
it’s not actually most of our systems, we actually have a really
diverse set of different programs doing all sorts of different
pieces. But a lot of the work that you’re doing thinking about
performance optimization is focused on specifically trading
systems. And trading systems have exactly the character of what you’re
describing, which is they’re sitting there consuming market data. The
whole idea, which you hear about a lot in large scale, traditional web
and tech companies of trying to build systems that have high
utilization is totally crazy from our perspective. We’re trying to get
systems that have low latencies, they need to be able to quickly
respond to things and also to be able to perform well when there are
bursts of activity. So that means that most of the time when things
are quiet, they need to be mostly idle.
10:33
Andrew
Yeah, it’s definitely true that we have easy-mode targets that we care
a lot about at Jane Street. A really good example is historical
research. If you are trying to run simulations of what some strategy
might’ve done over the last 10 years, turns out that’s just a question
of throughput. You can pile as much input data as you can on the
hopper and you see how many trades fall out the other side in the next
10 seconds, and the faster that gets the happier we are. You can just
do the same easy-mode tactics that you would on hyperscalers.
10:58
Ron
But even there, the size of changes we chase there is considerably
larger.
11:03
Andrew
Yeah, you don’t care about 1% because it’s not the CPU you care about
here, it’s the user latency in some sense, it’s whether or not the
user gets a result in an hour or a day or a week.
11:12
Ron
Just to speak on the compilers team. We totally care about a 1%
improvement in code generation, but you don’t care about it on its
own. You care about it in combination with a bunch of other changes
because you want to build up bigger changes out of smaller ones. And
if you look at a much larger organization, people are looking for
compiler-level improvements that are in order of magnitude smaller
than that.
11:31
Andrew
I sometimes have to push people about this. In fact, I sometimes have
to say, oh, no, no, no. It’s not that this optimization matters on its
own. It’s not that this thing that I did that removes a cache line
from each data structure is going to make our trading exponentially
faster. It’s that I’m in a long process of pulling out the slack from
a system and every time I do this, it gets a little bit better and
everything is going to slowly converge to a good state. But it’s hard
to get the statistical power to say any of these small changes
matter. Sometimes I get pushback from people saying, well, did you
measure this? I’m like, no, I didn’t bother. I know it’s below the
noise floor, but I also know it’s right. That sort of small change
incrementally applied over time is really good. But the hard part
about it is you just have to have faith that you’re going to get there
and you have to have faith that you know that this is making an
improvement or find ways you can test it in isolation. Whereas if you
operate at the huge scale that some other people do, you can just look
at a change. It can be five bips and you can know like, oh no, that’s
really real and it’s actually worth a lot of money on its own.
12:24
Ron
I think, yeah, this is a problem that hits people who are thinking
about performance almost everywhere.
12:28
Andrew
It’s kind of funny to me in that a common line of pushback I get from
people who are not performance-focused people is like, well, I
remember in undergrad when my professor said, well, you should never
make a performance change without profiling it, and knowing that it
matters. And like, no, no, I actually think that’s wrong. If you know
you are working on systems where this is important you to have a
certain amount of self-discipline and, you know, not where it’s too
costly, it’s going to make the system more dangerous or riskier or
make your life worse, but make efficient choices as a matter of
defaults in your brain.
12:54
Ron
And this is one of the reasons why I think performance engineering
depends a lot on intuition and background and experience.
13:00
Andrew
And mechanical sympathy. Knowing that you know deep down what the CPU
is actually doing when you compile the code that you’ve got.
13:07
Ron
So actually stop for a second on that word, mechanical sympathy, which
is a phrase I really like. Tell me what that phrase means to you.
13:12
Andrew
What that phrase means to me—I think a race car driver invented it
actually— is just having an innate knowledge
13:19
Ron
Or maybe not innate, you probably weren’t born with it.
13:23
Andrew
Some people seem to come out of infancy just knowing these things. Did
you not read books about CPU architecture to your children?
13:28
Ron
I did not.
13:29
Andrew
Whatcha are even doing
13:30
Ron
Lambda calculus?
13:31
Andrew
Oh, that tracks. I think that it’s not innate, but this really
unconscious knowledge of just when you look at code that this is how
it structures on real systems because different languages have very
different models of how reality works, but reality only has one
model. As much as I love Lisp, the world and computers are not made of
cons cells, they’re made of big arrays—one big array—and some integer
arithmetic that loads things from arrays. That’s all a computer
actually does, right? And you have to understand what is the model by
which we get from that low level thing to my high level types with
structure in them. And you have to understand what layouts mean and
how this branching structure gets compiled into something that a CPU
actually knows how to operate. And you can’t just construct this from
scratch every time you do it. You have to develop an intuition towards
looking at something and knowing what that’s going to be.
14:19
Ron
So I asked you what the difference was between the easy-mode
performance optimization that you experienced at Google and this kind
of harder-to-figure-out version of the story where you don’t have the
same kind of scale. I’d love to hear a little bit more about what is
the texture of these problems? What kind of problems do you run into?
What is interesting and hard about the version of the performance
optimization problem you see here.
14:39
Andrew
Hard mode performance optimization typically, but not always is a
question of latency. And latency is a question of what is something
doing at a really important time. Not what is something doing in
general, not what it does usually, but what it does when you care
about it, which means it’s fundamentally a measurement problem because
to measure general performance, what your system is doing, you point a
profiler at it, you get an idea of it’s spending 20% of its time here
and 10% of its time here and 5% of its time here. I don’t care about
any of those percents. I care about what was it doing for the
nanosecond, the millisecond, the microsecond sometimes (or some of our
worst systems, the second) that something interesting was happening. I
care about what happens when it was sending an order or analyzing
market data. I care only about that and I don’t care about anything
else. So how do I even know what it’s doing at that point in time? How
do I measure this is really the key question.
15:28
Ron
Got it. And maybe somewhat inherently, that puts you in the opposite
situation that you’re in when you’re looking at a very big
organization where you’re thinking about the low levels of the
infrastructure and how to make them as fast as possible. Because if
you want to know what’s happening at a given point in time, you’re
somewhat unavoidably tied up in the business logic. You care about
what is the thing that happens when the important decision is made and
what are the latencies that occur and what are the code paths that
drive that latency? Is that a fair description?
15:54
Andrew
Yeah, it’s not universally true. There’s some really interesting cases
where the infrastructure rears its ugly head in the middle of stuff
you want to be doing otherwise, right? But it is generally a large
part of it is in fact just the business logic of how is this trading
system making a decision? And you have to look at that and that’s
what’s happening at the interesting point of time, sort of by
definition.
16:11
Ron
So you talked about sampling profilers as one common tool. Can you
actually just go in a little more detail of what is a sampling
profiler and how does it actually work at a low level?
16:21
Andrew
So there’s a lot of different implementations of this, but the general
shape of it is you take a system and you point a tool at it that stops
the world every so often. Let’s say every hundred microseconds
maybe. And it stops the world and asks, where are you right now? And
it looks at where the instruction pointer is, what are we currently
executing? And it generally looks at the stack trace, how did you get
here? And then it writes this down and it lets the program keep
going. And profilers only really differ on how do they stop the world
and how do they write this down. My favorite is the Linux kernel
profiler. It’s called perf, and it just uses a bunch of hardware
features to get an interrupt at exactly the right moment in time. And
then it just very quickly writes down the stack trace in this
compressed format. It’s very optimized. And then you take all these
stack traces. The profile is really just a list of stack traces and
sometimes a little bit of augmented information, but that’s
fundamentally the core idea. And then you present it to the user in
some way that adds them up. And like I say, the key thing is it tells
you, okay, 30% of the stack traces ended in the function foo. That’s a
hotspot. You’re spending 30% of your time there.
17:19
Ron
But there’s all these different kernel counters that you can use for
driving when you’re doing the sampling. How does the choice of kernel
counter affect the nature of the information you’re getting out of the
profiler?
17:28
Andrew
Yeah, people tend to think about people sampling profiles in time,
where the counter is just the number of cycles that’s elapsed. But one
of the cool things about it is it lets you cycle on L2 cache misses or
branch prediction misses or any of these weird architectural
events. And so you get a profile of when did these interesting things
happen? You know each of them is costly and they probably have some
cost in cycles. You can get much more precise measurements. And in
particular, the nice thing about it is that 10% let’s say of your
program is slowed down by branch prediction misses. But if you just
look at the cycles, you’re just going to see like, well, it’s
somewhere in this function. If you profile on branch misses, you will
see the branch that is hard to predict and you can actually do
something about that branch.
18:07
Ron
Got it. So branch missed predictions is one. What’s the next most
interesting counter that you might use?
18:12
Andrew
Actually the most interesting thing isn’t even branch prediction or it
isn’t even a hardware counter. The next most interesting thing to
profile on is the unit of memory allocation. A lot of allocators, in
fact the one we have in OCaml, but also various C++ ones, will let you
get a profile – not out of perf but kind of done in software – that
tells you where you were allocating memory the most. Cause that’s just
a common thing that’s very costly and reducing that it can really
improve the performance of a system.
18:36
Ron
And this comes down to something that we see in OCaml a lot, which is
when we write really, really high performance systems, we often try to
bring the amount of heap allocation we do all the way down to zero.
18:45
Andrew
We try.
18:46
Ron
Right, it’s hard to get it all the way down to zero. Something’s
misbehaving in a system performance=wise. A relatively common problem
is there’s a thing that shouldn’t be allocating that is, in the hot
path.
18:55
Andrew
Yeah, that’s right. And then a good optimized C++ system, it should be
spending 5-10% of its time memory allocating and sometimes you just
have to do this, it’s necessary, but maybe you’re allocating twice as
much as you really need to be and you can look at a memory profile and
take a look at it. It’s important to remember that profiles aren’t
just about time. They’re about measuring the overall consumption of or
use of resources.
19:15
Ron
So that’s how perf essentially works to get you the information that
you need. But there’s a fundamental trade-off in a tool like perf
where it’s sampling information and that itself is essentially a
performance optimization. Part of the reason that you sample rather
than capturing all of the information is you want to make it fast. You
want to not distort the performance behavior of the program by
grabbing information out of it. But in some sense there’s a real
tension there because you’re saying I don’t want some broad
statistical sampling of the overall behavior of my program. I want to
know in detail how it’s behaving at the times that matter most.
19:47
Andrew
I think a really instructive example of that was an optimization that
I hit on last year where we had a system, it was trying to send some
orders and it was doing it in a totally reasonable way and it was
doing it in a really cheap way. The thinking about whether or not I
want to send an order was really cheap. It happened really fast. The
physical sending of the order really cheap, happened really fast.
20:08
Ron
The thinking, what do you mean by the thinking?
20:10
Andrew
The business logic to decide the looking at the markets and saying, oh
yeah, I want to buy or and then do the physical act of sending the
message, right? Both these were really cheap. If you pointed a
profiler at them, even a profiler that was magically restricted to the
times of interest, it would tell you, yep, 5% of your time was doing
this. It’s all good. That’s not a hotspot. Here’s the problem:the
order sending was happening 200 microseconds after the thinking was,
and the reason was it was being put on a low priority queue of work to
do later. It was a misconfiguration of the system. It was using this
kind of older API that it needed for complicated reasons that did
reasonable things under most circumstances. But it assumed that
network traffic you wanted to send couldn’t be that latency
sensitive. So it just waited to do it in an idle moment and this was
not a good thing to wait about. But the profiler tells you nothing
about this because I didn’t care about the overall cost, I didn’t care
about the overall time. I cared that it happened promptly. And so
fixing this again was really easy. I just switched to an eager API
that didn’t wait. But a profiler tells you nothing about this.
21:07
Ron
So what kind of tools do you use to uncover things like that?
21:10
Andrew
Magic Trace.
21:11
Ron
So what’s Magic Trace?
21:12
Andrew
Magic Trace is a tool we wrote that gives you a view into what a
system was doing over a short window of the past. It’s retrospective
in some sense. And what I mean is that any point in time you can just
yell, stop, tell me what you were doing for the last two milliseconds,
three milliseconds maybe. And you write it down. And exactly like you
said earlier, this is not a profile, this is not some statistical
average of what you’re doing at various times. This is the exact
sequence of where your code went. It went from this function to this
function to this function, and you get a different visualization of it
that just shows you what things was happening over time. And in fact,
exactly like you say, there’s more overhead for using this, but it
gives you this really direct view into what happened at an interesting
time that a profiler fundamentally can’t give. Traces are in some
sense really better. In fact, traces aren’t restricted to Magic
Trace. I said there’s memory profiles that are really useful. Memory
allocation traces are another thing that we care about a lot. We have
a really good one, in fact, that gives you the literal trace of–
22:06
Ron
Although a memory profiler is actually statistical also, right? That’s
a sample profile.
22:09
Andrew
Memory tracing is in fact, it’s not a profiler, it’s a tracer, right?
22:12
Ron
Maybe it’s misnamed actually there’s a lot of annoying sort of
terminological fuzz around this. I think people
22:16
Andrew
We are not unique in this, I’ll say
22:18
Ron
Right, at least the terms I’ve come to the most are people use
“profilers” for what you might call statistical profilers. And then
people use “tracing” when they’re talking about capturing all of the
data. So a common form of tracing that shows up all over the place is
there’s all these nice systems for doing RPC tracing where you write
down the details of every message that goes by. And this is sort of a
common thing to do if you want to debug, why was that web query slow?
And some query came in and it kicked off a bunch of RPCs, it kicked
off another bunch of RPCs and you can pull up an API that lets you see
the whole cascade of messages that were sent. So that’s like a nice
example of tracing. And then we also, as you mentioned, we have a
thing called memtrace, which sadly I think is actually a profiler in
that it is a statistical sample and does not actually capture
everything,
23:01
Andrew
But it does give you a time series of events, which is a key thing
that a profiler can’t.
23:07
Ron
That’s interesting. I guess in some sense, all of these systems start
by giving you a time series of events and then it’s how you process
them. Perf is sampling at some rate and grabbing data, and then you
turn that into an overall statistical summary. But you could look at
that information temporarily. You just don’t. And in any case, the
information is sampled. I think what I just said is also true about
memtrace. You get this information just like perf, sampled randomly
from the memory allocations. Then you can interpret it statistically
as being about the overall state of the heap.
23:35
Andrew
A key difference here is that memtrace gives you all of the
information about a statistical sample of some of the allocations. It
tells you when this guy was allocated and then freed.
23:48
Ron
That’s true
23:48
Andrew
Whereas you might get a set of stacks out of perf that here’s some
allocations and here’s some frees, but you have no guarantee it’s the
same thing. This lifecycle example, it’s exactly like the RPCs. A
thing people frequently do is just capture traces for 1% of their
RPCs. And giving the whole lifecycle of individual one is way more
interesting than 1% of the individual moments.
24:09
Ron
I mean, maybe this just highlights this is a total terminological car
crash. It’s a little hard to separate it out.
24:13
Andrew
All of these tools are way too hard to use and very inexact and all of
them use the same terminology in different ways.
24:18
Ron
Okay, so we’ve talked about the trace part of the name Magic Trace,
and the key thing there is it’s not just sampling, it’s giving you the
complete summary of the behavior. It’s maybe worth talking a little
bit about the magic, which is how do you do this thing? You just said,
oh, something interesting happened and then you retrospectively look
at what happened before and grab it. How can that be done in an
efficient way? What’s mechanically happening here?
24:38
Andrew
Magic. No, there’s two parts to this, right? First is how do you get
the data? And the second part is how do you decide when to take the
sample? Let’s take those in order. So how do you get the data? Well,
there’s this, I’m really just going to call it magic. I don’t know how
they managed to do this efficiently, but Intel has this technology,
it’s called Processor Trace, that just keeps a ring buffer of
everything the CPU does in a really compressed format, like it uses
one bit per branch or something along those lines. And it just
continually writes this down and the ring buffer is the size that it
is and it contains some amount of history—in practice, it’s a couple
milliseconds—and at any point in time you just snap your fingers and
say, give me that ring buffer.
25:14
Ron
And the critical thing is this is a feature integrated into the
hardware.
25:18
Andrew
Yeah, we couldn’t possibly implement this. The kernel couldn’t
possibly implement this. This is in the silicon and it’s a huge
advantage for Intel processors.
25:26
Ron
Yeah, although I really do not understand Intel’s business strategy
around this, which is to say they built this amazing facility in their
CPUs, but they never released an open source toolkit that makes it
really easy to use. They did a lot of great work at the perf level. So
perf has integrated support for Processor Trace. In fact, we relied
heavily on that. But I think Magic Trace is the first thing that’s
actually a nice usable toolkit built around this. There’s various
companies that have built internal versions of this, but it seems like
such a great competitive advantage. I’m surprised that Intel hasn’t
invested more in building nice, easy to use tools for it, because it’s
a real differentiator compared to say AMD chips.
26:01
Andrew
There are a lot of performance analysis tools in the world, and
there’s very limited hours of the day. I really don’t feel like I know
what everything the world does, but I generally agree with you that a
really underinvested thing is good, easy to use, obvious, idiot proof
APIs, right? And tools that just work in the obvious way you want them
to.
26:18
Ron
I was mentioning before how part of being a good performance engineer
is building up a lot of experience and knowledge and
intuition. Another part of it is just building encyclopedic knowledge
of all the bizarre ins and outs of the somewhat awkward tooling for
doing performance analysis. Perf is a great tool, in many ways it’s
beautifully engineered, but the user experience is designed for
experts and the command lines kind of work the way they do. And
sometimes their meanings have evolved over time and the flag still
says the old thing but you have to know that it has the new
meaning. And there’s really, I think a lot of space for building tools
that you just turn on and hit the button and it just does the obvious
thing for you and gives you the result.
26:55
Andrew
So it’s a user interface built by experts for experts, and I think
it’s easy for them to forget that most of the people have not used it,
just like you say, a lot of random esoteric knowledge about what CPUs
do. And I also just have a lot of memorized little command lines that
I happen to know will point at certain problems. And it’s just an
issue of having done this a lot. And I don’t know a good way of
teaching this other than getting people to do the reps and a better
way would be to give them tools that just give them obvious
defaults. But I haven’t figured out how to do this universally.
27:23
Ron
But Magic Trace is one good example of a tool like that, where the
defaults have been pretty carefully worked out. The UX is really nice
and you can just use it. It’s not perfect. It doesn’t work in all the
contexts, but it usually gives you something usable without a lot of
thinking.
27:34
Andrew
It is exactly like all the best tools I know in that I’m frequently
furious at it for not doing quite the right thing, and that’s a sign
of how much I want to use it. I would just, oh, can it also do this?
Can it also do this? Can it also do this? But yeah, I mostly just use
it in the obvious way with the one obvious flag, which gives me what I
want to know.
27:50
Ron
One of the interesting things about a tool like Magic Trace is you’ve
told a whole narrative of when doing easy mode optimization, you care
about these broad-based things. Sampling profilers are mostly the
right tools. When you care about focused latency oriented performance,
then you want this kind of narrowed in time analysis tools and Magic
Trace is the right thing for that. But a thing that’s actually struck
me by seeing people using Magic Trace inside of the organization is
it’s often a better tool for doing easy mode style optimization than
perf is because just the fact that you get all of the data, every
single event in order and when it happened, with precision of just
like a handful of nanoseconds makes the results just a lot easier to
interpret. I feel like when you get results from perf, there’s a
certain amount of thinking and interpretation and how do I infer from
this kind of statistical sample of what’s going on what was my program
actually probably doing and where really is the hotspot? But with
Magic Trace, you can often just see in bright colors exactly what’s
happening. I’ve seen people just pick a magic trace at a totally
random point in the day and use that and be able to learn more from
that than they’re able to learn from looking at perf data.
28:59
Andrew
Yeah, this is not universally true, but it’s very frequent that you
can get a bunch of extra information. I think one of the really good
examples is that a profiler tells you’re spending 40% of your time in
the hotspot of the send order function. But here’s an interesting
question. Is that one call to send order that’s taking forever or is
that a thousand calls each of which was cheap, but why are you doing
this in a tight loop? And it turns out it’s really easy to make the
mistake we were calling some function in a tight loop you didn’t
intend to. And in a profiler these two things look pretty
identical. There are really esoteric tricks you can use to tease this
out, but in Magic Trace, you just see, oh God, that’s really
obvious. There’s a tight loop where it has a thousand function calls
right in front of one another. That’s embarrassing. We should fix it,
right? You actually develop this weird intuition for looking at the
shape of the trace that you get the physical shape on your screen and
the visualization and like, oh, that weird tower. I’m clearly calling
a recursive function a thousand deep. That doesn’t seem right. You get
to see a lot of these things.
29:54
Ron
Yeah, it makes me wonder whether or not there’s more space in the set
of tools that we use for turning the dial back and forth between stuff
that’s more about getting broad statistical samples and stuff that
gives you this more detailed analysis and just more in the way of
visualizations, more ways of taking the data and throwing it up in a
graph or a picture that gives you more intuition about what’s going
on.
30:15
Andrew
A huge fraction of what I do in my day-to-day is visualization work,
rather looking at them or trying to build better ones. It’s really
important and we have barely scratched the surface of how to visualize
even the simplest profiles.
30:26
Ron
Yeah. One thing I’ve heard you rant a lot over time is that probably
the most common visualization that people use for analyzing
performance is flame graphs. And flame graphs are great. They’re easy
to understand, they’re pretty easy to use, but they also drop some
important information. And you’re a big advocate of pprof, which is a
tool that has a totally different way of visualizing performance
data. Can you give a quick testimonial for why you think people should
use pprof more?
30:49
Andrew
Yeah. Flamegraphs were way better than anything that came before
it. By the way. They were a revelation when they were invented, I
think. And so this is one of those things we have to be really careful
not to say something is bad. I just think something better has been
invented. So it’s called a flame graph because it’s a linear line that
has a bunch of things going up out of it that look like flames. And
what this is is that—
31:05
Ron
And people often use orange and red as the color, so it really looks
like flames.
31:09
Andrew
Exactly right wouldn’t look as cool if it was green. And so the first
level of this is broken down 40%, 60% and that 40% of your stack
traces start this way and 60% start this way. And then the next level
is just each of those gets refined up. And so every stack trace
corresponds to an individual peak on this mountain range. And then the
width of that peak is how many stack traces looked like that. So this
is good. It tells you where is your time going?
31:32
Ron
And one nice property is that at a glance, it makes it really easy to
intuitively see the percentages, right? Because the width of the lines
as compared to the width of the overall plot gives you the percentage
that part of the flame graph is responsible for.
31:43
Andrew
If there’s one really fat mountain that’s 60% wide, you know what
you’re doing. It’s that. Here’s the problem with it. It misses a thing
I like to call join points, which are points where stack traces start
differently and then reach the same important thing. Because what
happens is suppose you’ve got 15 or 16 little peaks, none of which is
that big, and then right at the tippy top of each of them in tiny,
tiny narrow things. It’s all calling the same function. It’d be really
easy to dismiss that you don’t even notice the thing at the top, but
it turns out if you add them all together, that’s 40% of your
time. And different visualizations can really show you everything
comes together here.
32:17
Ron
So how does pprof try and deal with this?
32:19
Andrew
So we’re now going to try to proceed to describe a representation of a
directed acyclict graph over a podcast. Which of all the dumb ways
people have visualized directed acyclic graphs might be the worst. But
what it does is it draws a little DAG on your screen where each node
is a function you end up calling and each arrow is a path. You can
imagine that you have one node for every function you’re ever in. And
then for each stack trace, you just draw a line through each node in
order and they go towards the things that were called. And then you
highlight each node with the total percentage of time you spent there
and you put some colors on it like you say, and you make the arrows
thicker, thinner for bigger or smaller weights. And that’s the basic
idea. If you close your eyes, and imagine with me for a second, I
claim that what will happen in the scenario I described is that you’ll
see a bunch of bushy random small paths at the top of your screen and
then a bunch of arrows all converge on one function. And that function
now is really obviously the problem.
33:20
Ron
And then underneath it’ll also branch out to lots of different things.
33:23
Andrew
Yeah, that’s actually a really good point. It tells you in fact that
maybe it’s not that function that is the time, it’s the things it
calls, but now you at least know where all this is coming from. And
there’s a single example of this that is the most common thing, at
least if you’re working in say C++, that function’s always malloc.
33:37
Ron
Oh, interesting.
33:38
Andrew
Like I said, with the business, logic may be very diverse, but
everything allocates memory because it’s really easy to realize, oh,
it’s doing a little bit of malloc here. It’s doing a little bit of
malloc here. It’s doing a little bit of malloc here.
33:49
Ron
And I guess this ties a little bit into what you were talking about of
that style of trying to look at the foundations and get the common
things that everyone sees. It becomes really important to see those
common things. Although I would’ve thought one of the ways you could
deal with this just with flame graphs is there’s just two different
orientations. You can take your flame graph and turn it upside down,
and so you can decide which do you want to prioritize? Thinking about
the top of the stack or the bottom of the stack.
34:11
Andrew
It’s the bush below malloc that screws you over there. It’s when the
thing’s in the middle that you get problematic. It’s when the really
key function is in the middle where everything converges through one
particular, it’s not a bottleneck, but you can think about it as a
bottleneck in the graph, that the flame graphs really fail. And to
their credit, people have built flame graph libraries where you can
say, give me a flame graph that goes up and down for malloc, but you
need to know to focus on that function.
34:33
Ron
I see. And so pprof somehow has some way of essentially figuring out
what are the natural join points and presenting them.
34:39
Andrew
I think that it outsources that problem to the graph drawing library
that tries to do various heuristics for how people view graphs tends
to work.
34:46
Ron
I’ve looked at flame graphs and I’ve looked at pprof visualizations. I
do think pprof visualizations are a little bit harder to intuitively
grok what’s going on. So I feel like there’s probably some space there
yet to improve the visualization to make it a little more intuitively
clear.
34:59
Andrew
I definitely agree. I think that just like we were saying earlier with
this is one of those things that experts know, new people don’t, you
just kind of have to get used to staring at these for a couple days
and then you get used to it. But it’d be nice if we didn’t have
to. It’d be nice if it was just as obvious as the other things are.
35:12
Ron
So we’ve talked a bunch here about performance engineering tools and
measurement tools more than anything else, which I think makes
sense. I think the core of performance engineering is really
measurement. And we’ve talked about ways in which these focused
tracing tools like Magic Trace in some ways can kind of outperform
sampling tools for a lot of the problems that we run into. What are
the kinds of cases where you think sampling tools are better?
35:33
Andrew
To me, the case I wish I could use sampling the most is once I’ve
identified a range of interest. The thing we care about a lot when we
think about latency and optimization of trading systems is tails. If
your system 99% of the time responds in five microseconds and then 1%
of the time responds in a millisecond, that’s not great. You don’t
always get a choice of which of those is the one you trade on, right?
35:57
Ron
And also there’s the usual correlation of the busiest times are often
the times that are best to be fast at.
36:02
Andrew
That’s right.
36:02
Ron
And so exactly where it’s bad is the case where you care the most.
36:06
Andrew
And Magic Trace is pretty good at finding tails because you can just
do some interesting tricks and hacks to get magic trace to sample at a
time where you’re in a 99-percentile tail. Now sometimes you look at
those trails and magic trace and you see, oh, I stopped the world to
do a major GC. I should maybe avoid that. I need to allocate less or
some other bizarre weird event. Sometimes it’s a strange thing that is
happening. But a remarkably common pattern we see is that your tails
aren’t weird, aren’t interesting, they’re just your medians repeated
over and over. And like you said, you’re having a tail because the
market is very busy because it’s very interesting. You saw a rapid
burst of messages and each of them takes you a microsecond to process,
but that’s not the budget you have. You have 800 nanoseconds and
you’re just falling behind.
36:50
Ron
It’s just a classic queuing theory result.
36:51
Andrew
It’s nothing but queuing theory.
36:52
Ron
Get lots of data in. You’re going to have large tails when it all
piles up in time.
36:57
Andrew
So you pull up this magic trace and you say, oh, I processed 10,000
packets. What do you do now? And sometimes you can think of obvious
solutions. Sometimes you realize, oh, I can know I’m in this case and
wait and process all of these in a batch. That’s a great optimization
that we do all the time. But sometimes you’re just, oh wow, I just
really need to make each of these packet processes 20% faster. And
what I really wish you could do is take that magic trace and select a
range and say, Hey, show that to me in pprof because it’s just like
the flame graph. You have all these little tiny chunks in the magic
trace and I really want to see them aggregated and you have some
trouble doing it.
37:31
Ron
So I asked one question and I think you answered a different one.
37:33
Andrew
I do that a lot.
37:35
Ron
I asked the question of when do you want sampling? You answered the
question of when do you want a profile view? And I totally get that,
and that seems like a super natural thing and maybe a nice feature
request for the Magic Trace folk have a nice way of flipping it into
the profile view. But I really wanted to poke at the other question
which is when is sampling itself a better technique versus this kind
of exhaustive tracing that Magic trace is mostly built around?
37:57
Andrew
One easy answer is the easy mode problems we have. Things like
historical research, things like training machine learning models,
things that are really are throughput problems and we do have
throughput problems. And there it’s just easier to look at the
sampling profile or you can really target it at cache misses or
whatever you want. So that’s a case where we definitely want it.
38:15
Ron
And maybe another thing to say about it is it is definitively
cheaper. The whole point of sampling is you’re not grabbing the data
all the time. I guess we didn’t talk about this explicitly, but Intel
processor trace, you turn it on and you end up eating 5-15% or
something of the performance of the program. There is a material—
38:31
Andrew
Don’t say that out loud or they’ll stop letting me use it on our
trading systems.
38:35
Ron
I mean, it is the case that we don’t just have it turned on everywhere
all the time. We turn it on when we want it.
38:40
Andrew
And that’s a thing that the hyperscalers do. They just leave a
sampling profile on across the fleet just getting 1% of things. And
that gives you kind of a great sense of what the overall world is
doing. And that is actually a thing I kind of wish we had. It would be
less valuable for us than it would be for them. But I would love if I
could just look at it like a global view of hotspots. I think the best
thing in the world would be like, can I get a sampled profile of all
the things all of our trading systems did when they weren’t spinning
idly? If I could know that, oh, overall if I could get a little bit
more return from optimizing the order sending code versus the market
data parsing code, I think that would be a really valuable thing to
me.
39:15
Ron
So another interesting thing about the way in which we approach and
think about performance is our choice of programming language. We are
not using any of the languages that people typically use for doing
this kind of stuff. We’re not programming in C or C++ or Rust. We’re
writing all of our systems in OCaml and that changes the structure of
the work in some ways. And I’m kind of curious how that feels from
your perspective as someone who very much comes from C++ background
and is dropped in weird functional programming land. what have you
learned about how we approach these problems? What do you think about
the trade-offs here?
39:44
Andrew
Well, the best thing about it’s employment guarantee, anyone can write
fast C++, but it takes a real expert to write fast OCaml, right? You
can’t fire me.
39:54
Ron
Although I think that’s actually totally not true. I didn’t mean the
part about firing you, but the point about writing fast C++, I
actually think there’s a kind of naive idea of, oh, anyone can write
fast C++. It’s like, oh man, there’s a lot of ways of writing really
slow C++. And actually a lot of the things that you need to get right
when designing high performance systems are picking the right
architecture, the right way of distributing the job over multiple
processes, figuring out how to structure the data, structure the
process of pulling things in and out. There are lots of stories of
people saying, oh, we’ll make it faster in C or C++. And sometimes you
implement it in some other language and it can be made faster still
because often the design and the details of how you build it can
dominate the language choice.
40:30
Andrew
I think it’s really easy for people who are performance obsessed like
myself to just get a little too focused on, oh, I’m going to make this
function faster. And maybe the better answer is, can we avoid that
function being called? Can we not listen to that data source? Can we
outsource this to a different process that feeds us interesting
information?
40:47
Ron
The single most important thing in performance engineering I think is
figuring out what not to do. How do you make the thing that you’re
actually doing as minimal as possible? That is job one.
40:56
Andrew
Honestly, I think one of the reasons that I really like performance
optimization as a topic to focus on. I don’t like writing code very
much. I’m not very productive. It takes me a long time to do good
work. So I want to do the stuff that requires me to write the fewest
lines of code and have the biggest impact. This is like one of those
hypotheticals, you make a two line change and everything gets 10%
faster. The hard part was the three weeks of investigation proving
that it was going to work. Right? And I think this is actually a good
example of you really have to think about the whole board. You have to
think about how you’re structuring the code and how you’re structuring
the system, how many hops this is going to go through, how many
systems is it going to go through, how can you get assistance from
hardware, any of these things? And micro optimizing the fact that
clang has a better loop optimizer than GCC than the OCaml
compiler. It’s really annoying to look at the bad loop. Is that really
what’s killing you? No. What’s killing you is that you’re looping over
something that you shouldn’t be looking at at all.
41:46
Ron
Right. So there’s a bunch of stuff you just talked about that would
love to talk about more about, in fact the hardware stuff in
particular. But I don’t want to lose track of the language issue. So I
believe what I said that people often overfocus on the details of the
language, but the language does matter and I think it matters in
particular for performance. And I’m kind of curious, what’s your
feeling about how that affects how we approach the work and how your
own kind of interaction and engagement with it?
42:08
Andrew
I break it down into three categories. The first category in which
OCaml provides us a challenge is, I call it the most annoying, but the
least important. And that’s what I was saying earlier about, oh, our
code generation isn’t as good. We’re branching too much. We have too
many silly register spills. It’s annoying to look at. It’s really not
what’s killing you. I would wish it was better. And really the limit
isn’t OCaml, the limit is scale. The C compiler people have been
working for 30 more years than the OCaml compiler people have, and
there’s more people working on optimizing clang right now across the
world than we have probably employees at Jane Street. We’re never
going to catch up. That’s okay. It’s not really what’s killing
you. The second category is things that are just maybe more of an
actual problem, but hard to deal with, but not really the key issue.
Our memory model requires us to do slightly more expensive things in
some cases. A good example is we’re a garbage collected language, our
garbage collector inspects values at runtime. Therefore, uninitialized
data can be really problematic. And so we have to do stupid things in
my brain like, oh, it’s really important to null out the pointers in
this array and not just leave them behind or they’ll leak, or you
can’t just have an uninitialized array that I promise I’ll get too
soon. Because what happens if you GC in that range? And I do actually
think this is meaningfully costly in some scenarios, but I’m willing
to put up with it. In most cases, there are things you can do about
it. The thing that I think is most problematic for our use of a
language like OCaml gets back to mechanical sympathy. And I said that
the world is not made out of parentheses that Lisp uses, and it’s also
not made out of algebraic data types. OCaml’s fundamental
representations of the world are very boxy. There’s a lot of
pointers. There’s a lot of this object contains a pointer to something
else where in C++ it would just be splatted right there in the
middle. And the reasons we do this, there are reasons that make it
easy to write good, clean, safe code, but it is fundamentally costly
and it lacks—the language, if anything, some mechanical sympathy.
44:02
Ron
Right? Or at least it makes it hard to express your mechanical
sympathy because getting control over the low level details is
challenging. And I don’t want to go too much into the, we’re actually
doing a lot of work to try and make OCaml better exactly at this. But
a question I’m more interested in talking about with you is how do you
see us working around these limitations in the language, in the code
base that we have?
44:20
Andrew
There’s a couple options here. The first is you can kind of write it
the hard way because OCaml’s a real language. You can write whatever
you want. It’s just a question of difficulty. If nothing else, I can
just, I could in theory, allocate a 64 gigabyte int array at the
beginning of startup and then just write C in OCaml that just
manipulates that as memory, right? It would work, it never GC, it
would do all the things you wanted it to. It’d just be miserable. And
clearly I’m not going to do that. But given that we’re a company that
has a lot of people who care about programming languages, one thing
we’re pretty good at is DSLs. And so we have some DSLs, for example,
that let you describe a layout and we’re going to embed this layout
into some low level string that doesn’t know a lot, but it’s still, if
you glance at it the right way, typesafe, in that the DSL doesn’t let
you write out of bounds accesses or anything like this.
45:09
Ron
Right? And the DSL, you sit down and write down what’s the format of a
packet that you might get from the NASDAQ exchange. And then it
generates some actually relatively reasonable, easy to understand
interfaces that are backed by the low level, horrible manipulation of
raw memory. And so you write a DSL, you generate some code, and what
you surface to the user is a relatively usable thing, but you get the
physical behavior that you want with all the flattening and inlining
and tight representation of data.
45:35
Andrew
You’re hitting on a really good point that a lot of these originated
from our need to parse formats that were given to us. But it turns out
you can also just use them for representing your data in memory. I can
build a book representation of the state of the market that’s just
laid out flatly and packedor me. It’s much less pleasant to use than
real OCaml. It’s difficult, and we only do this in the places that it
matters, but you can do it. This is what I like to call a dialect of
OCaml. We speak in sometimes and sometimes we gently say it’s zero
alloc OCaml. And the most notable thing about it, it tries to avoid
touching the garbage collector, but implied in that zero alloc dialect
is also a lot of representational things. We have little weird corners
of the language that are slightly less pleasant to use, but will give
you more control over layout and more control over not touching the GC
and using malloc instead. And it works pretty well. It’s harder, but
you can do it. In the same way, another thing we think about a lot is
interoperability. Again, sort of out of necessity that our libraries,
we have to interact with that only work in C. So we have these little
C stubs that we can call into and it’s really cheap. It’s not like
Java. It’s not like one of those languages where there’s this huge
costly process for going cross language, just make a function call and
it just works, right?
46:40
Ron
Yeah. The overhead, I think for a function call to C at the least is,
I dunno, three or four nanos. And I think in Java it’s like three or
400 nanos because the JNI is a beast for reasons I’ve never
understood.
46:50
Andrew
Option two is cry a little bit and deal with it. Yeah, we fight a
fundamental disadvantage. We’re working on reducing it. I’m super
excited about getting more control over the layout of OCaml
types. This is like the biggest change to me that maybe will ever
happen in the compiler. Being able to write down a representation of
memory that is what I want it to be in a real OCaml type that is fun
to play with, but fundamentally we’re kind of at a disadvantage and we
just have to work hard and we have to think more about, okay, we’re
going to have a higher cache footprint. What does this mean about our
architecture? How can we get cache from other places? How can we
spread out the job across more steps, more processes, pre-process this
one place. It gets back to, you don’t want to focus on over optimizing
this one function. You want to make your overall architecture do the
right things and just informs infrastructural changes.
47:33
Ron
And I think you make an important point that it’s not that any of the
optimizations you want to do are impossible, it’s that they’re a
little bit more awkward than you would like them to be and you have to
do a little extra work to get them to happen. And that means
fundamentally that we don’t always do them. And so we really do pay a
cost in performance in that getting people to do the right thing. The
harder you make that, the less it happens.
47:53
Andrew
One of the hardest things to learn when you’re doing this sort of work
is discipline. I have to go through the code base every day and say,
no, I’m not fixing that. Yes, it’s like offensive to me on a personal
level that it’s slow and it allocates and do these things, but it just
doesn’t matter. It’s legitimately hard for me not to stop whatever I’m
doing and just fix this optimization that I know is sitting there. If
this doesn’t bother you on a fundamental physical level, I just don’t
understand.
48:17
Ron
But you have to prioritize.
48:18
Andrew
You have to prioritize. There’s so much more important things to be
doing.
48:21
Ron
So another thing I’m wondering about is how you think about the role
of hardware in all of this. In some sense, if you’re thinking about
making things as low latency as possible, why do we even bother with a
CPU? Right? You look at the basic latency of consuming and emitting a
packet, and on any ordinary computer you’re going to cross twice over
the PCI express bus, it’s going to cost you about 400 nanos each
way. Know that and a little bit of slop between the pieces. It’s kind
of hard to get under a mic really for anything where you’re like, I’m
going to consume some data off the network, do something and respond
to it. And on an FPGA attached to a NIC, you can write a hardware
design that can turn around a packet in under a hundred
nanoseconds. So there’s an order of magnitude improvement that’s just
physically impossible to get to with a computer architecture. And so
in some sense, if all you cared about is, well, I just want the
absolute lowest latency thing possible, it’s like, why are we even
using CPUs at all? So how do you think about the role of hardware as
integrating and how you think about performance in the context of
building these kinds of systems?
49:18
Andrew
It informs the architecture you choose. Yeah, nothing’s ever going to
be as fast as hardware, but it’s really hard to write hardware. It
can’t do complicated things. And even the things it can do are just
exponentially harder to write. I have never in my life written
Verilog, which feels like a personal sin. I am reliably informed that
it is miserable and unpleasant and your compiler takes 24 hours to
run. So we have a lot of strategies with really complicated logic, and
that logic is important and it’s valuable. And implementing that in
hardware is, I’m just going to say flatly impossible. You couldn’t do
it. And so the question becomes what can you outsource to hardware
that is easy? How do you architect your system so that you can do the
really, really hyper-focused speed things in a simple, simple hardware
system that only does one thing and you feed that hardware the right
way, but the rest of the software system still has need to be fast. It
has to be fast on a different scale, but it turns out there’s
optimizations that matter on roughly every single timescale you can
imagine. We have trades at this firm that, like you say, complete in
less than a hundred nanoseconds or you might not even bother. We also
have trades that we send someone an email and the next day you get
back a fill, right? And every level in between there turns out you can
do useful optimization work.
50:32
Ron
And even with stuff that has no humans in the loop, we really do think
about nanoseconds, microseconds, and milliseconds. Depending on what
you’re doing and how complicated it is, you really do care about many
different orders of magnitude.
50:43
Andrew
Yeah. There’s a system that I’ve worked on where our real goal is to
get it down from having 50 millisecond tails to one millisecond tails,
and we’ve celebrated when we get there, and it still does a lot of
great trading. We have other systems that are doing simpler, more
speed competitive things where your software needs to be 20
microseconds or 10 microseconds or five microseconds. And that’s
achievable. It’s harder and you have to do simpler things just like
with the hardware, but it’s achievable and you care about both of
these latencies. And I think another good thing to point out is that
you said systems that don’t interact with humans, but it turns out
some of the most important latencies of systems we interact with is
the systems that do interact with humans. I don’t know about you, Ron,
but when my editor freezes up for five seconds while I’m typing, I
just want to put a keyboard through the window.
It just drives me nuts and putting aside the aggravation. Human
responsive systems are just really important too, both on you’re
actively trading and you want by hand and you want to have a good
latency on the thing that’s displaying the prices in front of
you. That matters a lot, but also I think it matters a lot for just
your ability to adapt and improve your systems over time. I said
earlier, think about historical research. That’s a throughput problem,
but it’s also a latency problem on a human scale. This thing that will
give you feedback on whether your trading idea was good in a minute is
worth so much more than one that’ll gives you an idea if it’s worth
anything in a day.
52:00
Ron
Yeah, that’s absolutely right. I think for lots of different creative
endeavors, and I think trading and software engineering are both count
from my perspective and also all sorts of different kinds of
research. The kind of speed of that interactive loop of I have an
idea, I try an idea and I get feedback on how well that idea works
out. The faster you can make that loop, the more that people can
experiment with new things and the more the creative juices can get
flowing and more ideas you can create and try out and evaluate.
52:25
Andrew
The thing I’m obsessed with telling people about is this Air Force
colonel from the fifties or the sixties, his name was John Boyd, he
invented this idea called the UDA loop, OODA, I believe. It’s observe,
orient, decide, act, just like the four stages you go through and
figuring out like, oh, I see something. I think about it. I decide how
I’m going to adjust to that. I implement the adjustment. And the
faster this loop happens, the more control you have over the system
and the better you can iterate. I think a great example of this
outside software, oddly enough, is whiskey. I like bourbon a lot. And
it turns out that to make good bourbon takes 5, 7, 10 years. And so
you don’t get a lot of opportunity to iterate. And there are some
people who are doing a really controversial thing, which is they’re
using technology to rapidly age spirits.
And some people call this sacrilege and it’s never going to be quite
as good as doing it the hard way. But on the other hand, it lets you
taste it in a month and be like, I think I’m going to change this and
they’re going to get 12 iterations in the time someone else might get
one. And I just think this sort of process turns out to matter a lot
in software too, of being able to rapidly iterate on what you’re doing
and get feedback either on the quality of the trading or for that
matter on the quality of the performance. One of the things I really
care about a lot is building systems that let me really quickly
evaluate the performance of a system. There’s a huge difference
between, I think this is going to be faster. I did a profile, I know
this is a hotspot. I’ve made it better. Okay, I’ll run it in prod for
a couple days and okay, I’ve made this change. I think it’s going to
be better. I’m going to run it on this really realistic testbed. I
know in 10 minutes if it’s better and I can change it and I can try it
again.
53:51
Ron
Yeah, iteration speed matters kind of almost everywhere you are. And I
really like your point about this kind of performance analysis
mattering for trading systems and also mattering for systems with
human interaction. And actually I feel like the performance mindset
isn’t really so different. You look at someone who’s really good at
thinking hard about and optimizing the performance of stuff in the
browser. There’s a very different instruction set and oh my God, is
that a weird and complicated virtual machine. But a lot of the same
intuitions and habits of mind and that focus on being really
interested in details that are really boring. All of that really
matters a lot. You really have to care about all the kind of gory
details of the guts of these things to do a really good job of digging
in.
54:30
Andrew
And this is why I kind of wonder if there’s just a mindset that can’t
be trained. You have to just look at this and go, what the hell are
you talking about, Ron, this isn’t boring. I get why you say that, but
I just look at this stuff and go like, you don’t have to pay me to
look at this. Sorry, I take that back. You do have to pay me to look
at this. I would not do this for free. I promise.
54:49
Ron
Boring in quotes. I love this stuff too, and totally understand why it
is, but it is from the outside. It’s a little hard to explain to your
friends and family why you like this stuff. They’re like, you can’t
even explain in words out loud, what are the details that are going
on. People will fall asleep.
55:00
Andrew
Do you know my dad once sat me down in college and said, are you sure
you want to do this CSS thing and not go into something where you can
find a job? Like being a lawyer? I’m the only person who disappointed
his parents by not becoming an English major.
55:12
Ron
Well, maybe that’s a good point to end it on. Thanks so much for
joining me. This has been great.
55:16
Andrew
Thanks for having me on. This was a really good talk.
See
. This means that their FLT proof only works for
For well over a decade, copyright holders have used U.S. federal courts to unmask anonymous BitTorrent users.
See all settings
.
.
