I am forwarding this message for a co-worker; his email to "fedora-list" keeps geting bounced. Having said that, I have worked with him on this issue and will be able to answer questions/describe the issues well enough for anyone who is kind enough to reply. What he tried to send follows below: ------------------------------------------------------------------------------------------------------------------------- I have run into an issue with memory bandwidth using the Fedora Core 2 kernels and I need help. I don't know what is wrong, but something killed performance of my custom driver when I ported it from RedHat 7.3 to Fedora Core 2. I believe I have narrowed it down to the kernel. My driver requires a large amount of contiguous physical memory for DMA from a PCI device. I use the 'mem=YYY' command line parameter to reserve the top of physical RAM for my driver. Then I allow mapping via mmap() calls to user space. The user space app then uses this pointer to save the data to disk. Normally the user space app writes to disk using the mmap()'d pointer as the source. With the new kernels these writes are taking way too long (around 20 MB/s). Even when the write goes to /dev/shm, the speed is limited to around 20 MB/s. A memcpy from the mmap()'d memory seems to have no such slowdown. This driver has been in use for some time on a RedHat 7.3 (2.4) kernel with no issues. To narrow the problem down, I removed all code that talks to the HW and created a driver that only maps host memory. The pattern I use is shown below. It is almost identical to the code in the kernel mem driver (...drivers/char/mem.c). dev_mmap(...) { ... u32 remap_addr = num_physpages*PAGE_SIZE; // Top of memory ... vma->vm_flags |= VM_IO; vma->vm_flags |= VM_RESERVED; status = remap_page_range( vma, vma->vm_start; remap_addr, vma->vm_end - vma->vm_start, vma->vm_page_prot ); if( status ) return -EAGAIN; ... } I created a test program that opens the device, calls mmap() to get a pointer, then saves 32 MB to /dev/shm and times it with the wall clock, as follows: dev_fd = open("/dev/mydevice",O_RDWR,0); shm_fd = open("/dev/shm/foo.dat",O_O_TRUNC|O_CREAT,0666); void *devptr = mmap(0,0x2000000,PROT_READ,MAP_SHARED,dev_fd,0); msync(devptr,num_bytes,MS_SYNC|MS_INVALIDATE); double t1 = /* time in seconds using gettimeofday() */ int n = write(shm_fd,devptr,0x2000000); double t2 = /* time in seconds using gettimeofday() */ /* check for errors */ I have tried this on several platforms and kernels and the results vary, but the common denominator seems to be: Fedora kernel + 32-bit Intel = poor performance (see below) Processor Kernel Chipset Arch Results Opteron 2.6.5-1.358smp AMD 64-bit Pass Opteron 2.6.7-1.492smp AMD 32-bit Fail Xeon 2.6.7-1.492smp Intel E7505 32-bit Fail Xeon 2.6.6-1.435.2.3 Intel E7505 32-bit Fail Xeon 2.6.6-1.435.2.3smp Intel E7505 32-bit Fail Xeon 2.4.18-24smp Intel E7505 32-bit Pass Xeon 2.4.18-24smp Intel E7501 32-bit Pass P4 2.6.7 (kernel.org) Via ??? 32-bit Pass Notes: * The Fails are always around 20 MB/s * When it passes, the performance depends on the chipset (e.g. 700+ MB/s) * The E7505 is hosted in an HP xw8000. * The E7501 is hosted on an Intel SE7501WV2 motherboard. * The P4 is my home PC, which is a VIA chipset - don't ask me which. Any help is appreciated. Thanks, John Fusco ------------------------------------------------------------------------------------------------------------------------- Does anyone have any ideas where to begin with this one ? And is there some other list that this question should be passed to ? Thanks, Jim Foris