Why is mmap()ed reserved memory so slow?

[David Chau]

Hi all,

I'm working on a driver for the Broadcom 1250, and I am using reserved
memory for some data buffers. The board comes with 256 MB of RAM, so I
boot Linux with "mem=253M" to reserve some RAM at the top of memory, and
then mmap() /dev/mem starting at 253 MB.

The problem is that accessing this memory is ridiculously slow. A simple
benchmark revealed that it takes about 200 cycles to read a 64-bit
number. If I mmap() /dev/zero instead, a read takes under 3 cycles.

For those of you who knows how the Linux VM works, could you tell me why
the memory access is so slow? It look like it might be invoking the
page-fault handler on every read. How can I make memory access faster?

Thanks,
David

[Dan Malek]

Post by David Chau
For those of you who knows how the Linux VM works, could you tell me
why the memory access is so slow? It look like it might be invoking
the page-fault handler on every read. How can I make memory access
faster?

How about a little more info, like what kernel are you using and what
are
the parameters you are sending to mmap()?

One of the things that happens here is /dev/mem is thinking this memory
is
not real memory (because you said the system has only 253M of real
memory), so it treats it like IO space. This causes changes to the
attributes
of the pages, most notably the CCA type for cache or pipeline behavior,
which isn't what you want in this case.

The better way to approach this is to place an mmap() function in the
associated driver that works in conjunction with the application to gain
shared access as you expect. This also closes a hole where an errant
application could write into unexpected places through /dev/mem.

Thanks.

-- Dan

[David Chau]

How about a little more info, like what kernel are you using and what are
the parameters you are sending to mmap()?

Linux (none) 2.4.31 #412 SMP Fri Jul 15 16:26:05 EDT 2005 mips unknown
(unmodified kernel from linux-mips.org).
It's running on the SB1 on a Broadcom 1250 board.

I mmap() with:
int mem_fd = open("/dev/mem", O_RDWR);
void* mem_base =
mmap(NULL, DRIVER_MEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED,
mem_fd, DRIVER_MEM_PHYS_BASE);
Where driver_mem_phys_base = 253M, and driver_mem_size=1M.

The better way to approach this is to place an mmap() function in the
associated driver that works in conjunction with the application to gain
shared access as you expect. This also closes a hole where an errant
application could write into unexpected places through /dev/mem.

Could you point me to an example of this so I can figure out how to do it?

Thanks,
David

[Ralf Baechle]

Post by David Chau
I'm working on a driver for the Broadcom 1250, and I am using reserved
memory for some data buffers. The board comes with 256 MB of RAM, so I
boot Linux with "mem=253M" to reserve some RAM at the top of memory, and
then mmap() /dev/mem starting at 253 MB.
The problem is that accessing this memory is ridiculously slow. A simple
benchmark revealed that it takes about 200 cycles to read a 64-bit
number.

mmap will create uncached mappings for anything above the highest RAM
address.

Post by David Chau
If I mmap() /dev/zero instead, a read takes under 3 cycles.

Because you have a cache hits. No RAM is that fast.

Above 200 cycles really is how horribly slow RAM is compared to a moderatly
clocked system.

Post by David Chau
For those of you who knows how the Linux VM works, could you tell me why
the memory access is so slow? It look like it might be invoking the
page-fault handler on every read. How can I make memory access faster?

Ralf