The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: OK, so by this time, most of you should have
had your meetings with your masters.
How many of you had meetings with your masters?
Who didn't yet?
AUDIENCE: Today.
PROFESSOR: Today?
And Friday.
Who hasn't scheduled a meeting?
So are you talking to them, and do you know--
AUDIENCE: [INAUDIBLE]
PROFESSOR: OK, make sure you get masters meetings scheduled
because this is a very important part.
We are getting feedback from the masters.
I think they are giving you guys a lot of good advice, and
so make sure you schedule, you go, you meet, get that result,
use that result.
It's very hard to arrange that kind of a senior person
spending a lot of time with only two of
you at a given time.
That's a really good ratio to have, so take
advantage of that.
OK, so also, I think at this point, most of the beta issues
for project one, we have been probably worked with, and they
have some issues some people looked at that they are
performance code.
They were not happy because the thing is this.
You can always run everything instantaneously if it doesn't
have to produce a correct answer.
You can just return, and so part of performance is also it
has to have some correctness.
So certain amount of correctness has to be passed
before we can give performer grade.
So some people have this question.
Why are you giving me a correctness grade because it
seemed to run?
But if you haven't done the right checks and stuff like
that, then it's unfair for other people because there's
no way people who do the correct implementation can
match that performance because we're using
that as upper bound.
So make sure you do that.
And another issue people had was when you go to masters,
you're going to work with one or two other students.
And in your group, you're going to share all the
material with your group member and nobody else.
But between the beta and final, you get opportunity to
basically get input from your masters, and also see what the
other people who are coming to a project in your
discussion have done.
So it's OK for you to see those code.
You can't take the code home.
You can't just say give me a print-out, I am going to take
home and re-implement.
That doesn't count, but if you see some clever things they
have done, it's all here to learn.
Class is all the mostly about learning.
However we have to grade you, so it's not all about grading,
it's about learning.
So we are trying to get a balance where we opportunity
for you to learn from your peers.
So if you go look at other people's code, if you find
something cool they have done, learn.
And also if you [UNINTELLIGIBLE]
for you [UNINTELLIGIBLE]
say, look, I did something cool.
It's a good learning experience in there, so that's
one opportunity you have to talk to other people on other
groups, to basically learn something or find some
interesting tidbits of how to get good performance.
And hopefully, you learn some interesting things and able to
implement that in your final.
OK?
So today we are going to talk about memory systems and
performance engineering.
So if you look at basic idea of what memory systems, you
want to build a computer that seemed to have
a really fast memory.
You don't want things to be slow.
For example, a long time ago, people build these computers
where memory's always slow, everything [UNINTELLIGIBLE]
further away.
It doesn't help anybody.
The way we know how to do that is to build small amount of
memory very close to the processor.
You can't build a huge amount of things all close to you.
That doesn't work, doesn't scale like that.
And we made a cache just like that, and you build larger and
larger memory [UNINTELLIGIBLE] going down, and the illusion
is you want to give the illusion of you have huge
amount of memory.
Everybody's very close to you.
OK, so it looks like you have millions of friends, that you
talk to everybody, and that doesn't happen.
But how do you give that illusion?
And the way that you give that illusion is when you use
normal programming practice, when you normally using data,
people have found there are two properties.
The first thing is called temporal locality.
That means if I use some data item, there's a good chance I
use that data item again very soon.
There is something I haven't used for a long time, probably
has a good chance I will not use it for a
very, very long time.
So can I take advantage of temporal locality?
So that mean some data will be get for use a lot of time, in
a very quick [UNINTELLIGIBLE], so those things should be very
near to you because you might.
Other one is spatial locality.
That means if I use some data item, there's a good chance
you'll be using data items next to that, closer to that.
So can we take advantage of that?
So those two properties help you help the compiler to fool
the system, fool everybody, thinking that it has this huge
amount of memory very close to you, but internally doesn't.
Unfortunately, since it's trying to fool you, it doesn't
work all the time, and when it doesn't work, it's good to
recognize and able to fix those things.
So I showed this picture sometime ago, too.
Memories in a big hierarchy.
L1, L2, in fact, we have L3 cache and a memory, disk,
tapes, it can go up and down.
And the key thing is when you go here, you're starting from
registers, you have very small amount of
things, very fast access.
Here, we have almost infinite amount of
things, very slow access.
And the two reasons why this has to be smart because first
of all, those things are very expensive.
And second of course, you can't [UNINTELLIGIBLE] too
much, and when you go down, things get somewhat cheaper.
So that was one of the kind of economic incentive here.
So I talked about cache issues, and I'm going to go
through this again because we are really going to dig deep
into these things.
So when you have a cache, that means you have a small memory
close to you that's basically shadowing something sitting
behind you.
So when you ask for the data in the cache, there are many
reasons why it's not there.
First thing is cold miss.
What that means is you're asking for something that you
have never asked before.
It's the first time you're asking, so the data is
probably sitting way behind in your memory where,
[UNINTELLIGIBLE]
in a disk.
We haven't even loaded it in there.
And the first time you have to get it, so that get pulled in,
and so this seems to be a place where there's nothing
you can do.
But another part of locality is a thing called prefetching.
These modern processors have a crystal ball.
They carry most of the things that branch prediction and
stuff is trying to predict what you want to do.
You can do the same thing in memory.
You can look at what you have done.
And Intel has this huge amount of circuitry that they don't
tell anybody what they do, but they're trying to predict what
you might fetch next, what data you might look for next.
And if it is working well, something you had never, ever
looked before, it might deduce that you might need it and go
get it for you.
So if that works, great.
So then there's thing called capacity miss.
What that means is as I said, I am trying to keep my best
friends, or the people I will be working with a lot--
Oh, nice.
What's going on here?
Please start later.
OK.
--the people I'm going to work a lot with, close to me.
The problem is I can have only certain number of friends, and
if you have more than that, I can't keep everybody.
So hopefully, your program is going to use a lot what we
call the working set that can fit in the cache.
And at that point, you get all of those people, like I said,
get everybody into one room, and the problem is if your
party spills over from the room, then
there's a lot of issues.
But if everybody fits in the room, things are very nice.
You can have a good interaction, continuous
interaction, with them.
And what happens is, if not, the worst case is the caches
have eviction policy, normally is called least recently used.
That means if I don't have room in the cache, I figure
out the cache line that I haven't touched for longest,
and I get rid of that, bring the next one.
So that's a good policy.
It fits in your locality thing, but when
will did not work?
[UNINTELLIGIBLE]
going to figure out when least recently used will create a
really bad problem in capacity.
AUDIENCE: [INAUDIBLE]
just one.
Say you're [INAUDIBLE] set, and that set is, by one,
bigger than the [INAUDIBLE]
missing.
PROFESSOR: Very good.
So what you're going through, you're going round some data
again and again, and that amount of data is one bigger
than your cache line.
You are going to have no locality because
[UNINTELLIGIBLE], and when you go to, just before you use
that data, they said, oops, I need to bring one more data.
I need to evict something, and who am I going to evict?
The guy I'm going to use next because it was the least
recently used thing.
So it goes, and then you go there, and you bring that.
Who's it going to evict?
You're going to evict the one I just brought.
You're going to evict the next one that I am just about to
use and bring that one.
And you go use that, and then to use the next one, you're
going evict the one you're just about to use.
So by going and doing that, you basically get no locality.
So that's a really bad problem in here.
So another interesting thing is called a conflict misses.
Conflict misses says normally, when you have a cache--
so what we have is you have this large storage, which
[UNINTELLIGIBLE]
the memory, you're going to map into smaller storage in
here, which is cache.
So one way to say is any of this value line can be
anywhere here.
That's called fully associative cache.
Implementing that is somewhat complicated, so the other
extreme is called direct map cache.
What that means is this segment, the same size here,
can map into here, and then the next segment also
can map into here.
That means if you get this cache line here, it can be
only mapped into one place in this here.
This cache line can only be here, and also this cache line
can also only be here if it is the same offset.
So that means for every cache line, there's only one
location in the cache.
So here, what would be really sad scenario?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Yeah, I mean I assume I have a
lot of these slots.
I read something here, and the next time I read something
here, next time I read something here, and I go round
and round this way.
I might be touching only very few number of data items, but
they also mapping into one cache line even though my
working set is very small than the cache.
I still don't have any cache usage
because I am having conflicts.
And then there are two other misses when you're going to
multi-process.
We'll go through them in later lectures.
One is called true sharing.
That means there are caches private to each core, and what
happens is if one core uses some data, you've have to get
the cache line there.
If the other core wants to use that data, you have to bring
the cache line back there, and the cache lines can ping-pong
[UNINTELLIGIBLE] that.
False sharing is even a worst way.
So what happens is if the first processor here touches
this value of the cache line, second processor touches this
value of the cache line.
We're not sharing anything, but unfortunately, the two
things I'm using sits next to each other, so when I ride
this thing, I get the cache line, he doesn't.
When he need to ride this thing, he has to
get the cache line.
So I am seemingly using independent data, but I am
bouncing cache line in between, and if that happened,
that going to have a huge big performance impact.
So these other things.
The last two, we'll get to a little bit later.
So today I am going to start with modeling how caches work
with a very simplistic cache.
So here's, assume, my cache.
I have 32 kilobytes in here.
This is a direct map cache.
That means that in my memory, 32 kilobyte chucks get mapped
to direct [UNINTELLIGIBLE].
This 32 kilobyte get mapped here.
That 32 kilobyte get mapped here.
And the cache line inside is 64 bytes, 64, and that means I
have basically [UNINTELLIGIBLE PHRASE].
32 kilobyte in here.
OK?
So just remember this as we go on doing this, we will use
this as a formula.
OK?
And we assume if you getting the cache, [UNINTELLIGIBLE]
single cycle, if you miss 100 cycles.
OK, so it's nice numbers to do that.
And so the first thing is we have a reference like this.
OK?
You go from [i] equals 0 to [i] less than very large
number, A[i].
I just go to accessing memory like one after another after
another after another.
So you see how this is going on in here?
So also I am accessing integers, so that means four
bytes at a time in cache line.
So what happens here?
So I assume size of int is four, so you're getting four
bytes in here.
So yeah, you're doing s reads to A. I'm accessing
s elements in here.
And you have 16 elements of a per cache line.
OK, you see why 16 if there?
Because I have 64 in here, 64 bytes in here.
Each time axises four bytes, so I have 16 of integers in
cache line.
AUDIENCE: [INAUDIBLE]
bits or bytes?
PROFESSOR: Bytes.
Byte.
Did I say bits?
No, bytes.
So what happens is if it axises the same element, 15 of
every 16 is in the cache because when we are going one
after the first guy, cache miss, I had to go bring it,
and the next 15, I have brought to it.
It's in the cache.
So what should be my cost?
[UNINTELLIGIBLE]
cost of memory access.
I am accessing s data, and 1/16th of that is a cache
miss, 15/16th is a cache hit.
So basically total [UNINTELLIGIBLE]
is a 15/16th of s is a cache hit, and other 1/16th I have
100 times cycles I need because it's a cache miss.
Everybody good so far?
OK, so what type of locality do we have here?
First of all, do we have locality?
Who thinks we have locality?
OK, lot of people thinks we have locality.
What type?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Spatial locality because what we are doing is
we are accessing the nearby elements even though we are
not getting back any of the data yet, so we get some
spacial locality axis here.
That's good, so this is our very -- most simple
thing we can do.
So what kind of misses are we going to have in the cache?
Cold misses.
I'm missing because I have never seen that data before,
so it's a cold miss in here.
OK, so that's that.
Let's look at this one.
I'm accessing A[0]
s times.
OK?
What's the total access time?
How many cache misses am I going to get?
One, so I basically have 100 time for the first cache miss,
and s minus 1 for the rest basically it's a hit.
OK?
That's good, so what kind of locality's this?
Spatial or temporal.
There's not too many choices.
Or none.
How many people think spatial?
How many people think temporal?
How many people said there's none locality?
OK, that's a lot of temporal.
OK, you want to get [UNINTELLIGIBLE].
That's temporal locality here because you're accessing to
the same thing again and again and again, same data.
So this is--
Oh, OK.
Want to restart, OK.
Wait a little.
So this is only time we notice in a hurry.
Every time, I go to do something, I get hourglass.
So what kind of misses are we getting?
Trick question.
I got one cold miss, and that's about it.
And the rest I don't have any misses.
So if I have a miss, it's a cold miss.
OK, so here I am doing something interesting.
So what the heck is in this list?
So I'm accessing A[i]
[UNINTELLIGIBLE]
this 2 to the power number, one shifted to the entire
miss, 2 to the power N. I shifted between 4 and 13.
What this 13?
Why is 13?
Less than 13.
What's 2 the power of 13?
I think I got this right.
AUDIENCE: 8,192.
PROFESSOR: What's 2 to the power of 13?
AUDIENCE: 8,192.
PROFESSOR: Yeah, 8K.
And I have 32K.
So 8K of 32 [UNINTELLIGIBLE].
8K of integers, which is each as 4 bytes is how much?
32K.
So what this says is everything I access should fit
into the cache.
I am accessing data like that.
I'm going back and accessing data like that.
I'm going back and accessing data like that, and it should
all fit into the cache.
Do you see what I do?
I access up to 2 to the power of 18, and I go back and
access that again.
I'm just going back and back because
of the model operation.
Everybody see what's going on?
So how many cache misses should I have?
AUDIENCE: There are four cache [INAUDIBLE].
PROFESSOR: OK, how many cache lines I would be accessing if
I am doing i1 2 to the power N?
[i]
mod 2 to the power N.
AUDIENCE: [INAUDIBLE]
PROFESSOR: So one miss for each axis line
the first time around.
Afterward, it's only in the cache.
So how many axis lines?
2 to the power N.
Is that right?
I think this is wrong.
Oh, yeah, this is right because every axis--
So in the cache line, how many--
AUDIENCE: [INAUDIBLE]
PROFESSOR: This is four.
So there are 16 entries here.
16 entries in here.
OK, only one of them basically misses that, so that means if
2 to the power N axises--
AUDIENCE: [INAUDIBLE]
PROFESSOR: You are doing 2 to the power N axises before you
go back in here.
So what you're doing is you are--
you make this many axises before you go back again.
OK?
And how many axises assigned to be in the same cache line?
You have 64 bytes in the cache.
OK each axis is four.
16.
AUDIENCE: [INAUDIBLE]
PROFESSOR: 16.
So that's true [UNINTELLIGIBLE].
AUDIENCE: [INAUDIBLE]
PROFESSOR: Yeah, but what I'm saying is I might not access
the [UNINTELLIGIBLE]
because if any small.
I only access a certain amount of cache line, so I'm
accessing a part of the cache.
I might not go through the-- can everybody see this, how
this is working?
Because if I'm only accessing, we'll say, accessing one, if
this is, we'll say, 2 to the power of 5.
OK, I am not going to access the entire cache line.
[UNINTELLIGIBLE]
cache is I'm going to [UNINTELLIGIBLE]
go through the cache.
So this many cache lines I'm accessing.
OK, and the first time I access that, I get a cache
miss, and after that, it say everything is in the cache.
Everybody's following me?
Or are we like lost in here?
How many people are lost?
OK, let's do this.
So what happens is if you have a modular 2 to the power N.
What that means is I'm going to keep accessing one to
somewhere 2 to the power N, and the next one, I am going
back here again.
That's my axises, basically, because it goes
[UNINTELLIGIBLE]
accessing to the power of N.
I nicely said n is between 4 and 13.
So n is 13 means the maximum I can do is 8K.
8K increase.
8K increase equals 32K kilobytes of memory.
So that means you don't have any option of
overwriting the cache.
So you go to cache, [UNINTELLIGIBLE].
You don't wrap around in the cache.
Do you see that?
OK, you know wrap arounding is so bad, that means all these
things is going to fit in the cache, and then when you get
[UNINTELLIGIBLE], this is going to be in the cache
because the data you access is smaller than the entire cache.
OK, so what that means is only the first line has to have
some cache misses.
So how many cache misses you are going to have
in the first line?
Or any of the lines going to have cache misses because the
first line still fit in the cache.
[UNINTELLIGIBLE]
anything.
We don't need everything.
How many cache misses have in here?
The interesting thing is because these are four bytes,
we have a 64-byte cache line, and each is four bytes.
So that means I can do 16 axises every time I have a
cache miss.
OK, so up to here to here is 2 to the N. My cache misses I do
[UNINTELLIGIBLE]
basically.
My cache misses is basically this much.
Everything else is in the cache, and after the first
line, everything else is in the cache again,
[UNINTELLIGIBLE] nicely, got my working set
to fit in the cache.
I'm really happy.
Everybody see this?
How many people don't see it now?
OK, good.
So let's move to something a little bit more complicated.
What kind of locality do we have?
AUDIENCE: Temporal and spatial.
PROFESSOR: Good.
We have temporal and spatial.
You have spatial because every time you go, you only get the
first line of the cache.
Rest is in the cache.
From that, I got a 16x improvement because I have
temporal locality.
And since I only access when I go back to the data, after
accessing this, since it's already in the cache, I get
spatial locality because of the 16.
16 things are in the cache, I am going through that.
I get spatial locality.
I get temporal locality because next time I access,
it's still in the cache.
I haven't taken anything out of the cache.
OK, what kind of misses?
Should be cold misses again.
Cold misses because the first time I [UNINTELLIGIBLE]
things.
I haven't seen that data.
After I get it, everything is in the cache.
So now, here's interesting case.
Now, I am doing 2 to the power N, where N is great than 14.
Now, what happens?
Now, what happens in our picture here is I am going--
So as you might cache is somewhere here.
At this point, I fill out the cache.
I still keep going, and minute I access something
[UNINTELLIGIBLE]
cache, what's going to happen?
So I went through accessing 34 kilobyte.
If I access the next one, what happens?
OK, no, no, no shutting down, please.
OK, what happened?
So I am accessing more than I can fit into the cache.
Next time access, what's going to happen?
Anyone want to take a wild guess?
AUDIENCE: [INAUDIBLE]
PROFESSOR: [UNINTELLIGIBLE PHRASE]
before that.
[UNINTELLIGIBLE]
in there.
So what happens is then am I ever going to have any
temporal locality?
No.
OK, so now, first access to each line basically misses
forever because by the time I get back to the
[UNINTELLIGIBLE], it's not there.
So basically, it's gone out of the cache.
So what that means is my total access time is basically every
16th element, I am getting cache miss forever.
So what that means, every 15th of 16, I have a hit.
So what I have is 15 out of 16, I have a cache hit.
1/16th of the time, I have a cache miss forever.
So what kind of locality do I have now?
I have spatial locality.
I don't have any temporal locality.
I don't ever go back to something in the cache again.
So what I have is spatial locality, so what type of
misses now do I have?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Cold, right.
AUDIENCE: Is it like shared misses?
[INAUDIBLE]
PROFESSOR: It's not shared.
When you fill out the cache, you go back to
[UNINTELLIGIBLE].
You fill the cache.
So what type of miss is that?
AUDIENCE: Capacity?
PROFESSOR: Capacity miss.
OK, so you have basically cold and capacity
misses happening now.
OK?
AUDIENCE: One question.
[INAUDIBLE]
that you multiplied whenever you're trying to load from
memory, is that like an arbitrary number, or is that
the actual cost of going--
PROFESSOR: Every machine, you can get this beautiful table.
I will go off what your machines have.
We still have a number saying this is your miss
[UNINTELLIGIBLE], this is the thing--
I mean, so some of these things you realize because of
all of the complexity.
It's not that nice and simple, but this, I just pull out of
hat, and it's kind of normal.
So what do I do, now?
I am doing the mod, but I am multiplying [i] by 16.
So now, I am accessing this value, this value, this value,
this value in here.
I'm accessing this value, this value, this value, this value.
One value in each cache line.
OK?
How much cache misses do you think I'm going to get?
AUDIENCE: [INAUDIBLE]
PROFESSOR: 100 for every access is basically beginning
a cache line, you're taking one value and if I go back to
that value, I have already filled up the cache.
So basically, I have first access in the cache.
And what's your total access time?
Anyone will take a guess?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Yep, 100 times x because I have no--
OK, so this is a [UNINTELLIGIBLE]
locality.
That was clear.
What kind of misses am I getting now?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Now, I'm getting conflict misses.
I get a cold miss at the beginning, first time around,
and then every time I do something, I'm getting
conflict miss because the thing is, now, this N, I am
only accessing small amount of data.
It doesn't fit in the cache, but I'm not still getting any
kind of sharing, so I'm having conflict misses.
[UNINTELLIGIBLE] second time, I guess I will
probably jump over this.
I just did OK, if I go random access, what happens?
So let me jump over that.
Actually can do a calculation to figure out how many,
probabilistically, how much you might have
and stuff like that.
OK.
So now, if you look at what's going on, when you have no
locality, you'd have no locality.
That's a pretty obvious statement.
And then if you have spatial locality and no temporal
locality, we are streaming data.
That means you are going through the data, but you're
not getting back to it fast enough, so I
have temporal locality.
So I stream through data, so that means I go through data
in a streaming fashion.
OK?
So what we have is if working set fits in cache, you have
InCache mode.
If working set is too big for cache, you can get some
streaming mode and still get some locality because we are
bringing the lines in here.
And you can do other things like optimizers like
prefetching we'll get to.
And other issues [UNINTELLIGIBLE]
this last one, but to deal with cache axises.
So if you have more than one axis-- so here what I have is
too nice arrays.
[UNINTELLIGIBLE] is 2 to the power size array 2 to the 19
power arrays.
OK, so now what happens when I put this in the cache?
Look what happens in here.
So what happens is this array get mapped in this one.
This array get maps this nicely in here.
So what happens is this line can only be in this cache
line, and this line also only can be in
this line of the cache.
Do you see what's going on, now?
So if you do this one, basically every time you
access something--
See, assume I'm accessing this data item, so I'm doing A, B.
I access A, I get this line.
I access B, what happens?
AUDIENCE: [INAUDIBLE]
PROFESSOR: So this [UNINTELLIGIBLE] is gone.
It is gone.
Now, I am accessing the next one in that
cache line we made.
AUDIENCE: How do you know what cache line-- or how
[INAUDIBLE]
cache?
[INAUDIBLE]
PROFESSOR: Normally, in language like C, if people two
arrays next to each other, they should be mapped next to
each other in memory.
And if you want to be more adventurous, you can go to the
assembly and then see what their locations is.
So if you put A, A next to each other, you normally kind
of get it next to each other.
So what this one is this is pretty bad.
I should have gotten at least some spatial locality.
I'm getting nothing.
Do you see what's going on here?
Do you know what's going on here?
Know why I'm not getting any spatial locality?
Because I am bouncing.
When I access this one, this one is gone even though I have
more data in the same cache line, so I'm kind of bouncing
two cache lines because of that.
What's a good solution for this?
Back there.
AUDIENCE: [INAUDIBLE]
A and B could be mapped to the same cache line.
PROFESSOR: The thing A is a nice 2 to the power.
A size is a multiple of the cache size.
AUDIENCE: [INAUDIBLE]
PROFESSOR: The size of A is a multiple of the cache size.
So what happens is if you have memory in here, you map
something like A, C here.
If this is a multiple of cache line, so assume this is 32K
times some number, and you map B here, and then this is
normally what happens in C. You put all adjacent map
allocations next to each other in memory.
Do you got A here, and you start having B in
here, next to it.
OK, so what happens is now if A starts to assume address,
we'll say 100, this is 100 plus 32K times N, basically.
And you do this mod 32K, this mod 32K, this same number.
It maps into the same line.
OK, so I kind of gave you the problem.
What can be the solution?
AUDIENCE: [INAUDIBLE]
allocate one line.
PROFESSOR: Yes, it's called padding.
OK, so I have no locality in here, let me go to the next
slide, what kind of misses?
I have cold and conflict.
What I can do is just basically add a
little bit of padding.
I did 16 here.
Normally, you do a prime number so it will not clash
with anything.
Add a little bit of padding at the end of array, and that
means the next array will not be
conflicted most of the time.
So a lot of times when you declare things next to each
other, it's always good to add a little bit of
a padding in between.
Normally, a smaller prime number would be a good thing
to add a padding.
Now, I start getting back my nice locality because the two
things are mapping the two cache lines.
What type of locality do I get here?
Anybody want to take any guess what type of
locality I have here?
AUDIENCE: Question, sorry.
[INAUDIBLE]
you actually added just an extra 16.
PROFESSOR: Yes.
AUDIENCE: Why would that just make the A and B [INAUDIBLE]?
PROFESSOR: Because what happens is normally it doesn't
make A and B into leaving the cache, but it makes A[0]
not exactly matching to B[0].
A[0]
will interleave with something like B[16]
or something.
This thing might map into the same place.
AUDIENCE: I mean, but I [INAUDIBLE].
PROFESSOR: Yes, normally in a computation,
people do that, A[i]
equals B[i]
plus something.
AUDIENCE: Oh, OK, so [INAUDIBLE].
Like I understand what you're saying, but I don't understand
how that [INAUDIBLE]
because they're both accessing [i] at the same time.
Why would they [INAUDIBLE]?
PROFESSOR: So here's the problem.
Assume I have [UNINTELLIGIBLE]
32K.
I'm assuming [UNINTELLIGIBLE].
OK, so first one is, we'll say, you start with 100, 100
plus [i], and the other one is 100 plus 32K plus [i].
OK, then you take a mod 32K, and this end up
being 100 plus [i]
with another 100 plus [i].
So this is basically [UNINTELLIGIBLE].
You still have one or one element in the cache it's all
going to map into.
You see that?
But now, if I add 16 more to this one, then this map into
116 plus [i].
AUDIENCE: I don't see that reflected in the code.
You just [INAUDIBLE], so how is B different
from A in your code?
PROFESSOR: Because [UNINTELLIGIBLE] my S. I added
16 to S, so my A is a little bit bigger, now.
AUDIENCE: But isn't B also [INAUDIBLE]?
PROFESSOR: Yeah, B's also bigger here.
I don't care.
B goes down here.
B, I add a little bit more to the B. It doesn't matter.
I padded B, too.
So what that means is the first A[i]
and B[i] are not in the same cache line.
AUDIENCE: How do we allocate A and B right
next to each other?
PROFESSOR: So normally in C, if you declare something like
int A [100], int B [100], more or less, they will be
allocated next to each other.
If you look at the assembly listing, it'll say code
allocated next to each other in memory.
Even in the stack, if you do that, if you allocate things,
compiler has no incentive to go and move things around.
AUDIENCE: This is only for global and [INAUDIBLE]?
PROFESSOR: Global variable, local variables.
AUDIENCE: Does any of it still apply if you use malloc?
PROFESSOR: The thing about malloc is--
AUDIENCE: [INAUDIBLE]
PROFESSOR: It might do something [UNINTELLIGIBLE],
but on the other hand, if it is fitting on the same page,
it might also do something like that, too.
So it depends on how we do that.
At the beginning, if you keep mallocing things, it might be
[UNINTELLIGIBLE].
They might be a little bit off because malloc put some
metadata with each status.
If you ask for [UNINTELLIGIBLE], you're not
getting exactly 100.
You're getting a little bit of a bigger size.
But you might have some configuring to do, but after
some time, when you have done a lot of [UNINTELLIGIBLE], we
have no idea where it's going to go.
It's random.
Anymore questions?
OK, good, that is a good question.
So what kind of locality do I have here?
[UNINTELLIGIBLE]
two lines, I am accessing each, and I got 16 times 15
out of 16 hits.
What's that?
AUDIENCE: Spatial?
PROFESSOR: Spatial locality.
And of course, if you have spatial locality, you get cold
misses, basically.
And I think [UNINTELLIGIBLE], you get what
other type of misses?
AUDIENCE: [INAUDIBLE]
PROFESSOR: [UNINTELLIGIBLE] conflict here.
Basically, capacity.
Capacity miss, exactly.
So in order to avoid this, what we have done is we make
cache-- instead of mapping into one place before, and
what we said was we had this memory, and
assume we have 32 sets.
And if you have a cache like this, we make this entire map
into this one, and this map into this one.
What people have done is instead of doing that, what
you can make is make the cache, instead of one big
block, two small blocks.
So what that means is this can happen to this one.
My drawings are not that great, but I
hope you get the picture.
This map into this one, so what that means is each
reference here, so maps here, and these
two basically conflicts.
But now with that [UNINTELLIGIBLE]
too, I can put this one here or here.
I have two places to put the value, basically, and that
[UNINTELLIGIBLE].
OK, so that means if I access something conflicting,
[UNINTELLIGIBLE], you cannot just two things.
Still, there are places in the cache for them to go.
And one way to do people is fully associative cache.
It can go anywhere, but that's very hard to do in hardware,
but you can do some small number of ways.
So in here, if you do two basic associative cache, in
the first axis in here, even though these are conflicted,
on can go to one [UNINTELLIGIBLE] set, another
one can go to the other set.
OK, so two associative cache [UNINTELLIGIBLE].
So total access time, again, I have nice locality in here,
spatial locality, and I have both cold and, I guess,
capacity misses.
But if you have two associative cache, what's the
next problem?
How about if you have three different things?
So we have two different things like that.
OK, now that I only have two associated [UNINTELLIGIBLE],
it doesn't work because, in a [UNINTELLIGIBLE], I will
replace the last thing, and you might still end up with
nothing in here because you are replacing nothing, and the
next thing replaced the other thing.
And basically after you gave B, C, then you go
to A. A Is not there.
When you go to B, B is not there, C is not there.
You kind of go back to the same problem in here, and then
basically have to access everything again.
There is some spatial locality but cache can't use it.
Basically, you have conflict misses.
OK, so you see that?
So associative is good because normally people assume you
don't need that much
associativity, but it's limited.
So if you have too many things, you might run into
this problem again.
AUDIENCE: [INAUDIBLE]
PROFESSOR: You mean capacity?
AUDIENCE: Yeah, [INAUDIBLE] capacity.
PROFESSOR: Less.
AUDIENCE: [INAUDIBLE]
elements in a cache before you pick [INAUDIBLE] because it
looks as though the matrix is is using half
the cache by itself.
PROFESSOR: Because it's associated, you mean?
You have multiple there?
I mean, the thing is since it's associated, no, because
you have enough room.
I mean, if you're going [UNINTELLIGIBLE]
everything, you have two places to put the
name because it maps.
That's a good question.
So what happens is now, assume this many things fit into the
cache before.
Now, cache has two banks, and in each bank, only this many
things fit in, but the next thing can go
to the other bank.
So if you go through [UNINTELLIGIBLE], you have the
same behavior even if you have associativity.
OK, you can go and--
because if the cache is the same size, the size is what
matters for if you are going through [UNINTELLIGIBLE] axis.
AUDIENCE: So far, [INAUDIBLE] machines that we're working
on, do they have a fixed kind of cache, or
can we decide like--
PROFESSOR: The fixed kind of cache.
It's built into the hardware.
I will show the table to say what type of
associativity is in there.
AUDIENCE: Is there not a way to say [INAUDIBLE]
three-way associative?
Can you have something on top of it so that it looks like
it's two-way associative or something?
PROFESSOR: I mean, you can do things like that in software,
but they're very expensive.
I mean, think about cache is trying to fool the cache in
that direction cannot be that easy because they are like
really hard-baked into hardware. that's a good
question, but there are things you can do, not exactly
modeling something, but kind of putting offsets and stuff
like that so it doesn't get into these kind of conflicts.
So yeah, I will answer this thing.
Why don't you have more?
Because if you go back to your [UNINTELLIGIBLE]
lectures.
Perhaps it might have explanation in it's hardware
why it's much harder to build.
And if you have the linked lists now--
so assume I have linked lists we are going through in here.
And what's my [UNINTELLIGIBLE]?
Basically, depending on allocation and using some
things, so best case is everything is in cache, so I
have a small list.
And I go through again and again, round and around my
list, everything's in cache and [UNINTELLIGIBLE]
S. That's really great.
Next best is everything is adjacent.
OK, everything might not be in the cache, but everything at
least in adjacent memory.
So basically, first time I allocate my malloc did a nice
thing for me, and at that time, I basically get a nice
basic spatial locality since my size is 16 in here.
So I get this much.
And the worst case is random.
Malloc puts it all over the place.
Question?
AUDIENCE: [INAUDIBLE]
cache, does the thing behind it [UNINTELLIGIBLE] in front
of it [UNINTELLIGIBLE] also going to?
Do those both have a spatial locality?
PROFESSOR: What happens is normally, we assume 64 element
cache, so it's at that boundary, so basically 2 to
the 8 boundaries is what happens.
So what happens is cache line begins at 0 address, 64
address here.
So if you get 63, that means this value is not in the
cache, so you get this cache line.
So the cache line doesn't shift.
It's fixed chunks in here, so that if you get address 0, you
get this one, and all these things.
If you get something in the middle, you'll get the left
and right [UNINTELLIGIBLE], but if you get here, you only
get the right.
So this basically 0 and 64, they are boundaries
[INAUDIBLE].
If you like 65, that's it.
But that said, if you're accessing [UNINTELLIGIBLE],
that's this thing called prefetcher that say you are
going through linear pattern, and you might need that next,
and it'll bring this up for you.
And what you find, I'll show later, that in fact, the
machine you're using has a really powerful prefetcher, so
it can hide a lot of these things.
So we'll get to that.
So this is my total access time in here.
Random case.
Random case is basically, every time I access something,
it's not in the cache.
And then if you have a bigger struct in here--
but assume I have this big struct in here, but I am only
accessing one element of this struct in here.
I do a lot.
I created this large struct, and I am [UNINTELLIGIBLE]
things in the struct, and I just basically
looking at my data.
Only one element.
What's the problem here?
AUDIENCE: [INAUDIBLE]
PROFESSOR: Exactly, so I bring the entire cache line, but I'm
only using a little bit of it.
Because I bring all these things into the cache, it
comes with axis and that, but I'm only using data.
So basically, what happens is, if everything in the cache, I
have access time S. That's great.
That's not a problem, [UNINTELLIGIBLE]
streaming.
Even though I'm getting spatial locality, I only get
half of it because even though I'm bringing the cache line,
only two have datas because this is 32 bytes long.
It's in my cache.
Even though I am bringing all this data, I only two of them
in the cache because rest of this other data is kind of
hanging in there with me, and so I don't get that.
And random, of course, you have no locality.
So how can I basically make this better?
How can I make this better?
OK, I just showed it very fast.
OK.
AUDIENCE: [INAUDIBLE]
PROFESSOR: So instead, if I ask you, I have this large
data structure, but most of time, I'm only accessing a
small part of it, this is kind of a little bit of a hack.
I mean, this is kind of going against, I guess 6005 kind of
building, but this actually works.
What you can do is, you can go back and instead of doing
structs, what you can do is you can do arrays
of each data item.
And now what happens is that I am basically bringing these,
and then the data I am accessing is basically right
now next to each other.
So you might have this very large data structure
representing something you have a lot of, then.
But most of the time, you're only using few of them, so you
don't have to be structs of arrays.
You might have couple of different data structures.
One is the one that you access regularly hopefully you get
them next to each other.
And the nice thing about arrays is it guarantees things
are next to each other.
Of course, was managing it is harder.
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: You can do both.
So that means instead of doing the structs, you use arrays of
each field.
AUDIENCE: [INAUDIBLE]
PROFESSOR: Instead of.
So instead of doing this one, you just do this one.
AUDIENCE: [INAUDIBLE]
instead of being able to pass stuff over.
Now you have to kind of be able to know [UNINTELLIGIBLE],
so what that is where because you cannot just pass the
struct [UNINTELLIGIBLE].
PROFESSOR: No, you can't pass the struct.
You basically have to pad the index in here for this
structure, basically.
So you have to know something in there.
So this is a lot messier, but it can actually give you a
performance if you do this type of access, so it's not
something you want everywhere.
That's why you still want to use structures and nice
classes and stuff.
But this, if you're doing very large data, but most of the
you're accessing only small amount, this kind of
representation can help.
So I want to get a little bit into matrix multiplying.
So what we're looking at is this four by four matrix, and
you can represent it using this representation.
But to make it clear, what I'm going to do is I'm just going
to allocate this entire thing as a single array, and I'm
going to do that [UNINTELLIGIBLE] calculation
myself because if you go a little bit down, you realize
this is allocated in a little bit of a weird way.
And so this is what we call row-major because all rows are
next to each other, and you can access allocated
column-major.
Basically that means now A001, instead of that, 1, 0 is next
to 0, 0, 1, 0, 2, 0.
And then basically saying same thing, [UNINTELLIGIBLE]
instead of saying 4i plus j, it's just 4j plus i.
You can do both.
So this is my nice matrix multiply function, because the
reason I did it this way, it's just very clear how many axis
I'm going to use.
So if you look at C, let's look at only the inner loop.
Most of the mapping matters in the inner loop.
I go for k in 0 to size.
Only first axis on C is going to miss because it's going up
[UNINTELLIGIBLE]
same K, da-da-da-da-da, so C is very nice in here.
Only get one miss in here.
You have a question?
OK.
A has a streaming pattern because every time I update K,
I go to the next element, next element, next
element, next element.
[INAUDIBLE].
What type of a pattern do I have in B?
AUDIENCE: Looks like [INAUDIBLE].
PROFESSOR: Yeah, exactly.
I have jumping on memory.
I'm jumping one to another.
So if you look at that, I am going through A like that, and
B, and going through C element at a time, basically.
So if you look at doing this in order to calculate one
element of A, [INAUDIBLE] calculating these, what you
have to do is I am getting B and C this way.
I'm going through row here and column, and this row
[UNINTELLIGIBLE]
fit into the cache line through that because this is
the memory.
But C is just jumping all over the memory, and I have no
locality in here.
So what can we do here?
What's the simplest thing in here?
How do we get some locality into the C?
AUDIENCE: [INAUDIBLE]
PROFESSOR: So yeah, you're coming, you're getting it.
That's the next thing that's most important.
That's the more advanced thing you can do.
We'll get there.
What was a simple thing you can do for C?
AUDIENCE: [INAUDIBLE]
PROFESSOR: [UNINTELLIGIBLE]
So here's the [UNINTELLIGIBLE]
execution time, original, and voila, when you transpose
that, you can actually get a huge benefit in here because,
now, I get spatial locality in both sides in here.
And so now when you're transposed, I am going
through like this.
So now what I did was I am giving this CPI, and what
happens is when you transpose, I have a 5x
improvement in sync CPI.
That means instructions actually not waiting, not
doing nothing.
I have a pretty good reduce of cache misread.
because, now, I'm actually going
through cache very nicely.
So the next thing we can do is, what she just
pointed out, blocking.
So instead of matrix multiply, just multiplying this matrix
at a time, you can multiply submatrices and calculate
small submatrices.
And if you do that, the submatrix can nicely fit in
the memory, so let me show you that.
This is the blocking code.
Let me show you what that means.
So what we are doing is you are multiplying this matrix by
this matrix in there and getting this matrix.
So what that means is, normally, what you do is, to
get one element, you have to multiply a row and a column.
Normally, we are calculating a row of A at a time.
In order to get a row of A, I had to get a row of B
[UNINTELLIGIBLE]
entire C has to participate into calculate A. So by the
time I have calculated 1024 elements, I have gone through
this many axises to new elements basically because I
am getting B again and again, but I had to go through the
entire C in here.
So if this much data is bigger than the cache, I have no
locality in here.
But if, instead of calculating a row, if I calculate a 32 by
32 block, which is again 1024 elements I am calculating.
If I do that, I need a block here and a block here, and I
only type this much data.
OK, I'm getting the same amount of output, but because
of reuse, I touch a lot less inputting here.
And just by doing this transformation, I can really
reduce the amount of data.
Basically, I can create a smaller working set, go
through that much more, and reduce the amount of cache
misses you get.
So by doing that, I basically go out another 3x improvement
in my CPI, really brought down my L1 cache misses, and
basically eliminated L2 cache misses.
OK, so I got a really good speed up in here.
So you can block multiple levels, L1, L2, L3, because
every time you figure out the workings and size, you block
so the data in that will be safe in the cache in here.
And this is kind of nasty to do by hand.
We'll see.
So the other interesting thing is call stack locality.
So if you do something like fib in here, traditional fib,
if you do fib(4), you calculate a fib(4), fib(3),
fib(2), you kind of build a stack, go up
back again in here.
So if you keep doing that, the nice thing is you're
going up, but its--
Hopefully, one of these days, I am going to hit the wrong
button, and--
What kind of locality do we have if we are accessing data
like this in my call stack?
What kind of locality do we have?
I mean, I access here, fib(4), and I get back to fib(4) here.
So within each frame, I am accessing data next to each
other, so I can have a little bit of a spatial locality
because, each frame, all the data is next to each other.
But across frames, normally, I will have some temporal
because I will get back to that data some time.
OK?
OK, before I summarize, this one.
If you are reading a lot of data, processing through stage
one, [UNINTELLIGIBLE]
and producing [UNINTELLIGIBLE], that means
you're reading a huge amount of data through stage one,
save, save data again, do stage two, you have really bad
cache locality.
One thing you can do is read a small chunk, do all the stages
on that small chunk, save it, and go to another chunk.
So instead of doing the entire data, doing something
[UNINTELLIGIBLE]
producing, you can do this chunk-ifying, and by doing
that, you can get a lot of locality.
So once you bring in data, you like to do as many things as
possible to that data before it gets out of the cache.
That's the entire thing about locality, and you can
restructure code through that.
So if you look at that, what we have done today, we have
looked at things like associativity, cache lines,
cache sets in here, and how to address and map to a cache, we
kind of did this thing, and we look at [UNINTELLIGIBLE]
streamings versus InCache versus no
locality type patterns.
And then when you look at data structure transformation, we
look at how to basically replace structured arrays and
split data structures to make sure that you get a lot of
nice locality if you access a certain amount of data, and
pack data to get smaller cache footprints kind of things.
And we also look at computer transformations, things like
transpose copying, compute copy, outtype things you can
do, so you can actually get nice locality, so you can do
both data and compute transformations.
And [UNINTELLIGIBLE]
axises and [? re-cast ?]
calculation stages, so this is kind of summary.
Next thing I want go is look at what machines
you guys are using.
So this is the machines we used last year.
These are quad cores, and it had L1 cache, so these
[UNINTELLIGIBLE].
So this has 32 kilobyte L1 cache, data cache
[UNINTELLIGIBLE]
two caches in here for data and instructions, each core,
and then the cache line size is normally 64 bytes
[UNINTELLIGIBLE].
And here, latency [UNINTELLIGIBLE], you get
three cycles.
If not, you get 14 cycles to go to to L2 in here if data
doesn't meet there.
And associated with this [UNINTELLIGIBLE PHRASE].
So that was what we did last year.
This year, we have a bigger machine that has multiple
levels of caches.
So each core has L1 instruction and data cache.
It's kind of the same, 32 kilobyte cache, four
nanosecond latency in here.
Eight, three and four [UNINTELLIGIBLE], they kind of
reduced that, but they added a 256-kilobyte L2 cache in here,
and then a pretty large humongo
12-megabyte L3 cache in here.
The interesting thing is here, this pretty fast access to
these ones.
If you miss in here, this is about 2 and 1/2 times cost
from going from here to here.
From here to [UNINTELLIGIBLE], about a five times cost, so
here to here is a five times cost.
The interesting thing that actually surprises a lot us
was here to here, it's cost is not that different.
[UNINTELLIGIBLE PHRASE]
So actually memory, even though it's sitting out in
there, they have managed to get it pretty cheap access.
Normally, my intuition say this should be much higher
because you're going off-chip, so this is the structure here.
It's a lot more complicated.
So here's something I produced last year.
So what I did was I ran this program.
And in these machines, there's a way to turn off the prefetch
of the previous [UNINTELLIGIBLE].
So when you've turned on the prefetch and then you run,
what you are doing is I am accessing data with a working
set, small working set, little bit bigger working set.
So this is basically when you consider working set size.
So you get a graph like this.
I am hitting my L1 cache, and if it is more than that, I
[UNINTELLIGIBLE]
L1 cache, I go to my L2 cache.
And if L2 misses, I have to go to main memory.
Very traditional kind of a graph in here.
Another a little bit of an interesting thing, which is
instead of accessing next item, I kind
of skipped in here.
So this trying to force how to get conflict misses.
And so if you line up things like this exactly, what
happens is you get these conflict misses, and you get
this number.
So I have interesting story to tell.
When I was a graduate student, we were running this bunch of
benchmarks, and normally when you run a benchmark, you try
it for an equal
[UNINTELLIGIBLE], stuff like that.
And we got this graph like this.
I mean, we expect a graph going like this.
OK, that how a benchmark [UNINTELLIGIBLE].
Then at one point, by accident, instead of typing an
equal 64-something, I typed an equal 63, and suddenly, I got
a performance like this.
Wait a minute, what's going on?
So in here, what I got was not a line.
I got 16, 32, 64, 128, nice graph like that.
And then I just did 63, like down to here.
I'm like, geeze, that can't be right.
And then I start looking at 62, which is here.
61 is here.
So I realize if I enumerate all those things, I got a
graph like this, basically.
Kind of something like this.
And the thing is, if I had never tried the 63, I would
have basically thought my performance was like
this, so far worse.
Never realized because it was all having conflicts in memory
because I'm [UNINTELLIGIBLE]
2 to the power numbers.
So this is what happens if you just look at 2 to the power
numbers, you'll think, yeah, this is the performance you're
getting, without realizing that, in fact, you
can get this far.
So this is good.
So I ran it on the new machine.
The problem with our new machine is there's no way to
turn off prefetching.
OK, so this is what you're getting.
Everything, because I am accessing this nice linear,
looks same.
This is exactly what you want to happen.
What it's doing is, minute you go to access, it realizes you
accessing linearly.
It start fetching, fetching, fetching data in front you,
it's going to catch up.
It's not feeding you data, so you're not going to have any
cache missed because you're going through this nice,
linear pattern, and you just basic end up with this
beautiful graph, which gives the feeling your working set
could be, now, gigabytes long.
At this point, I am fitting into the entire L3 cache, and
still look like this works beautiful because this is what
you want for it to figure out.
So now I'm like, geeze, this is great.
This means my cache works, it's very nice, but I want to
show you guys what's going on, so that doesn't help.
So I came up with this program.
So this program, what I did was I created a working set,
but I'm accessing randomly within that working set.
I calculated this address.
This is my poor man's random number generator, so basically
I'm adding number in here and multiplying this, and then the
masking these to get the right [UNINTELLIGIBLE]
so I get within that set in here.
And then I just access data, so I'm getting
[UNINTELLIGIBLE].
So I get a number like this, and this jump up here, and
then I had this kind of behavior.
This is a lot more complicated.
I need to figure out what the hell is going on in here, so I
said, OK, let's now look at performance counters.
I look at L1 cache miss.
OK, up to 32K, I have no cache misses.
Boom, cache misses jump up.
Perfect.
And I have this little bit of a kink in here, this beautiful
describes because I have 32K L1 cache, and until then, I
have no things, and then when the cache start missing, I'm
jumping up.
Perfect.
I got that description here.
Then I look at my L2 cache misses.
This is, I think, the L2 cache in this machine might have a
little bit more complex [UNINTELLIGIBLE]
telling us because this is not going like
here had jumping up.
That's a 256K [UNINTELLIGIBLE]
get.
Is this 256?
Yeah.
[INAUDIBLE]
250 is [UNINTELLIGIBLE]
these things.
OK, so we have 32 256 2L megabytes in here.
Even if you have 256 in here--
AUDIENCE: [INAUDIBLE]
PROFESSOR: Hm?
AUDIENCE: Should it be 256 or 32--
PROFESSOR: [UNINTELLIGIBLE]
because everything's backed up, it's hierarchy
[UNINTELLIGIBLE], so it's not added to, it's not next to
each other, because at this point, L1 is now useless.
It'll run everything is just going to L2, so L2 is the one
who was serving it.
Come on.
So what happens in here, this normally should start jumping
here, 256, but it start jumping earlier, so there
might be some other interesting
thing going on here.
It might be interesting to investigate.
So this is OK, so still I am pretty close.
I realized this jump is basically because of L2
happen, this jump in here.
L2 start missing.
Then I look at L3.
It's here.
What's happening here?
So this was a big mystery the two of us were trying to debug
what one hour ago.
Not one hour.
Class is going.
Like two hours ago.
We was trying to figure out, OK, wait a minute, this
doesn't make sense.
My performance is here.
L3 should be just basically a nice flattening out here, and
just jumping.
It's not doing that, and we are like what's going on here.
So then we were scratching our heads, and we start looking.
I say, ha, there were other things going on.
We looked at this thing called TLB.
Let me tell you what TLB is.
What we realize is at this point, we are [? seeing ?]
TLB misses.
How many people know what a TLB is?
OK, good, so let me spend the last couple of minutes
explaining what a TLB this, and that kind of explain
what's going on in this graph.
So TLB means--
Normally, data is stored in memory using real addresses.
That means there's a memory size in here, there's an
address format, there's certain [UNINTELLIGIBLE].
And then the program for it use virtual memory.
That means programs want to feel like they have a lot more
memory than actual, physical memory available.
So what happens is you have virtual memory address, but
somebody has to map virtual memory to physical memory.
So the way it work is there's a thing called translation
lookaside buffer, TLB, that says if you give this virtual
address, here's the right physical address.
So of course you can't do it for each address by address.
What you have is you have 4K pages, so the address space is
broken down into 4K pages.
Each page will have an entry in this TLB, saying if you get
this virtual address, here's the right physical address.
So normally what happens is when the operating system they
created this large table, for every place in memory, you use
this table, and it loads into memory.
Then the hardware will fetch and cache some of those
entries in this TLB cache in there.
So normally what happens is in here, it caches 52L entries.
It can keep 52L entries to 52L pages,
saying here's the mapping.
The problem with 52L pages is in a 4K byte page size, 52L
pages can only hold two megabytes of data.
If you're accessing more than two megabytes of data, you
have to go more than 52L pages, and at that point, that
mapping is not in the TLB, and that means that you have to go
do a mapping translation.
The problem for this is these 4K pages is a
very ancient artifact.
There's the new machines, you can I ask for what we call
large-pages or super, but you can ask for 2-megabyte or
4-megabyte pages.
If you ask for super-pages, then of course, 52L entry's
good enough.
It can hold enough things not to have TLB miss, at least
while you're in the cache.
But since Linux, without doing anything, only asks for 4K
pages, before you run out of your L3 cache, you run out of
the TLB table.
And it has to start fetching and [UNINTELLIGIBLE]
data table, that means you had to actually fetch the table
entries, and so you have to do more fetchings.
So that is why this is even running higher, because it's
trying to fetch this TLB entry and stuff like that.
Question?
AUDIENCE: Is there a way to then actually fiddle with the
TLB so that you have to fit your pages rather than--
PROFESSOR: So in allocation, you can ask for super-pages or
large-pages.
But in Linux, normally just ask for 4K pages, so that the
operating systems haven't kind of caught up
to the current hardware.
And this is was built [UNINTELLIGIBLE]
doesn't make sense TLB to run out before
you run out the cache.
I mean, that's kind of mismatch in here.
But it's built assuming you'll get large-pages, but if you
don't, you're going to run out of the TLB in there.
So this is why sometimes performance engineering is
interesting, exciting, and frustrating, because you
assume you know where the hell this is going.
You assume this is going to go here and that, and suddenly,
this miss shows up, and so we have to scratch our heads.
And luckily, we found what's going on, but we could have
spent hours or days trying to figure out what's going on
here because this is one [UNINTELLIGIBLE]
say OK, let's try TLB, and it worked.
That was really good, but I have, many cases, spent days
trying to figure out why something is going on because
performance is kind of a [UNINTELLIGIBLE]
of everything.
There's no nice abstractions and stuff like that.
If you think abstraction wise, somewhere outside the
abstraction, that's going to kill you.
So that is why it can be interesting, because you have
to know everything.
And the fun thing is, once you understand performance, you
basically understand end-to-end, soup to nuts,
what's going on.
OK, with that encouraging thought, let's see how you
guys do in the next part of project two.