Hi, To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda), which given that device support 1/3/6 Gb/s I/O speeds and the device is plugged into a 6Mb/s Sata port on the motherboard, is very poor performance. Doing some research on disk performance I found a reference to bad disk performance be attributable to BTRFS and that the disk should be formatted as EXT4, is this sentiment reasonable and if so why is Fedora pushing the use of BTRFS.
regards, Steve
Once upon a time, Stephen Morris steve.morris.au@gmail.com said:
To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda), which given that device support 1/3/6 Gb/s I/O speeds and the device is plugged into a 6Mb/s Sata port on the motherboard, is very poor performance.
That's a misunderstanding of how things work. The SATA port speed is just an upper-bound on transfer, but has nothing to do with how fast a device can actually read data (similar to having a 1G network card and even Internet service doesn't mean sites will serve data to you at 1G). Traditional spinning hard drives typically do top out in the neighborhood of 150 MB/s... and in fact, the official spec from Seagate for that drive is an average read rate of 156 MB/s.
On 17/11/24 11:54, Chris Adams wrote:
Once upon a time, Stephen Morrissteve.morris.au@gmail.com said:
To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda), which given that device support 1/3/6 Gb/s I/O speeds and the device is plugged into a 6Mb/s Sata port on the motherboard, is very poor performance.
That's a misunderstanding of how things work. The SATA port speed is just an upper-bound on transfer, but has nothing to do with how fast a device can actually read data (similar to having a 1G network card and even Internet service doesn't mean sites will serve data to you at 1G). Traditional spinning hard drives typically do top out in the neighborhood of 150 MB/s... and in fact, the official spec from Seagate for that drive is an average read rate of 156 MB/s.
If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
regards, Steve
On 17 Nov 2024 at 15:44, Stephen Morris wrote:
Date sent: Sun, 17 Nov 2024 15:44:51 +1100 Subject: Re: Fedora F41 Hard Disk Performance To: users@lists.fedoraproject.org From: Stephen Morris steve.morris.au@gmail.com Copies to: steve.morris.au@gmail.com Send reply to: Community support for Fedora users users@lists.fedoraproject.org
On 17/11/24 11:54, Chris Adams wrote: Once upon a time, Stephen Morris steve.morris.au@gmail.com said:
To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda), which given that device support 1/3/6 Gb/s I/O speeds and the device is plugged into a 6Mb/s Sata port on the motherboard, is very poor performance.
That's a misunderstanding of how things work. The SATA port speed is just an upper-bound on transfer, but has nothing to do with how fast a device can actually read data (similar to having a 1G network card and even Internet service doesn't mean sites will serve data to you at 1G). Traditional spinning hard drives typically do top out in the neighborhood of 150 MB/s... and in fact, the official spec from Seagate for that drive is an average read rate of 156 MB/s. If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
I've always hdparm -Tt to test drives. hdparm -Tt /dev/sda
/dev/sda: Timing cached reads: 10716 MB in 1.99 seconds = 5377.96 MB/sec Timing buffered disk reads: 700 MB in 3.00 seconds = 232.99 MB/sec
In this case the cached speed is reading data from the drives ram buffer. The second one is reading directly from the disk without using the cache.
My other notebook is newer and has a SSD drive, and nvme drive, and a memory card. hdparm -Tt /dev/sda
/dev/sda: Timing cached reads: 32260 MB in 1.98 seconds = 16287.27 MB/sec Timing buffered disk reads: 1484 MB in 3.00 seconds = 494.39 MB/sec root@setzcodell:~# hdparm -Tt /dev/sdb
/dev/sdb: Timing cached reads: 34946 MB in 1.99 seconds = 17574.52 MB/sec Timing buffered disk reads: 1342 MB in 3.00 seconds = 447.31 MB/sec root@setzcodell:~# hdparm -Tt /dev/mmcblk0p1
/dev/mmcblk0p1: Timing cached reads: 35092 MB in 1.98 seconds = 17717.93 MB/sec Timing buffered disk reads: 62 MB in 1.34 seconds = 46.19 MB/sec
regards, Steve
+------------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor (Retired) mailto:mikes@guam.net mailto:msetzerii@gmail.com mailto:msetzerii@gmx.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ +------------------------------------------------------------+
On 17/11/24 16:02, Michael D. Setzer II wrote:
I've always hdparm -Tt to test drives. hdparm -Tt /dev/sda
/dev/sda: Timing cached reads: 10716 MB in 1.99 seconds = 5377.96 MB/sec Timing buffered disk reads: 700 MB in 3.00 seconds = 232.99 MB/sec
For me on my hard disk its giving:
/dev/sdd: Timing cached reads: 52114 MB in 1.99 seconds = 26165.57 MB/sec Timing buffered disk reads: 492 MB in 3.01 seconds = 163.69 MB/sec
and on my SSD it is giving:
/dev/sdc: Timing cached reads: 52352 MB in 1.99 seconds = 26285.62 MB/sec Timing buffered disk reads: 1498 MB in 3.00 seconds = 498.96 MB/sec
regards, Steve
Stephen Morris composed on 2024-11-18 09:07 (UTC+1100):
For me on my hard disk its giving:
/dev/sdd: Timing cached reads: 52114 MB in 1.99 seconds = 26165.57 MB/sec Timing buffered disk reads: 492 MB in 3.01 seconds = 163.69 MB/sec
and on my SSD it is giving:
/dev/sdc: Timing cached reads: 52352 MB in 1.99 seconds = 26285.62 MB/sec Timing buffered disk reads: 1498 MB in 3.00 seconds = 498.96 MB/sec
Try it on individual partitions and you may see more than trivial variation.
# hdparm -t /dev/nvme0n1
/dev/nvme0n1: Timing buffered disk reads: 2872 MB in 3.00 seconds = 957.06 MB/sec # hdparm -t /dev/sdb
/dev/sdb: Timing buffered disk reads: 436 MB in 3.00 seconds = 145.19 MB/sec # hdparm -t /dev/nvme0n1p8
/dev/nvme0n1p8: Timing buffered disk reads: 3652 MB in 3.00 seconds = 1216.79 MB/sec # hdparm -t /dev/sdb7
/dev/sdb7: Timing buffered disk reads: 440 MB in 3.01 seconds = 146.41 MB/sec # hdparm -t /dev/nvme0n1p7
/dev/nvme0n1p7: Timing buffered disk reads: 3056 MB in 3.00 seconds = 1018.60 MB/sec # hdparm -t /dev/sdb8
/dev/sdb8: Timing buffered disk reads: 434 MB in 3.00 seconds = 144.51 MB/sec #
On 2024-11-17 14:32, Felix Miata wrote:
Stephen Morris composed on 2024-11-18 09:07 (UTC+1100):
For me on my hard disk its giving:
/dev/sdd: Timing cached reads: 52114 MB in 1.99 seconds = 26165.57 MB/sec Timing buffered disk reads: 492 MB in 3.01 seconds = 163.69 MB/sec
and on my SSD it is giving:
/dev/sdc: Timing cached reads: 52352 MB in 1.99 seconds = 26285.62 MB/sec Timing buffered disk reads: 1498 MB in 3.00 seconds = 498.96 MB/sec
Try it on individual partitions and you may see more than trivial variation.
What trivial variation are you seeing. The non-cached reads are going to be maxing out the SATA connection with the SSD.
# hdparm -t /dev/nvme0n1
/dev/nvme0n1: Timing buffered disk reads: 2872 MB in 3.00 seconds = 957.06 MB/sec # hdparm -t /dev/sdb
/dev/sdb: Timing buffered disk reads: 436 MB in 3.00 seconds = 145.19 MB/sec # hdparm -t /dev/nvme0n1p8
/dev/nvme0n1p8: Timing buffered disk reads: 3652 MB in 3.00 seconds = 1216.79 MB/sec # hdparm -t /dev/sdb7
/dev/sdb7: Timing buffered disk reads: 440 MB in 3.01 seconds = 146.41 MB/sec # hdparm -t /dev/nvme0n1p7
/dev/nvme0n1p7: Timing buffered disk reads: 3056 MB in 3.00 seconds = 1018.60 MB/sec # hdparm -t /dev/sdb8
/dev/sdb8: Timing buffered disk reads: 434 MB in 3.00 seconds = 144.51 MB/sec #
You have an nvme drive, so it's not going over SATA and will be much faster than a regular SSD. The variations in your measurements are just noise.
Chris Adams:
That's a misunderstanding of how things work. The SATA port speed is just an upper-bound on transfer, but has nothing to do with how fast a device can actually read data (similar to having a 1G network card and even Internet service doesn't mean sites will serve data to you at 1G). Traditional spinning hard drives typically do top out in the neighborhood of 150 MB/s... and in fact, the official spec from Seagate for that drive is an average read rate of 156 MB/s.
Is that really megabytes or megabits per second?
And the converse applies to Stephen, remember when you're measuring one thing against another, and they use the two different things. Convert gigabits per second into megabytes per second, and it seems far less impressive. Even more so when they mix up the usage of 1024 or 1000 bits and bytes multipliers.
Stephen Morris:
If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
Because big numbers are a marketing ploy... Sure, there's *something* that the SATA port can do at that speed, but it's not continuously churn your data through in the way that you'd like.
If they put high speed cache between SATA port and internal storage, they can increase data through-put up to a point (to the point where it's filled the cache). So it's quick for storing one or two very large files, because they're measuring the SATA data speed. But internally, the cache is transferring over to the storage medium at a much slower rate.
And maybe you could get a RAID device which has SATA ports to the PC, so it can spread the load internally across several drives and keep up with a very high data speed. I've never looked to see if anyone has actually done that.
I remember when IDE went over to UDMA (same two inch wide [approx] fat ribbons, with twice the wires in them). It could achieve much higher data speeds across the cable, but the drive medium was a bottle neck. Then they put cache RAM in the drives, and that allowed more data to be quickly dumped across the cable to the drive, but the same problem existed: Once the cache was full, you're down to the slow speed of the storage medium inside the drive.
On 2024-11-17 02:02, Tim via users wrote:
And maybe you could get a RAID device which has SATA ports to the PC, so it can spread the load internally across several drives and keep up with a very high data speed. I've never looked to see if anyone has actually done that.
Back in the early 00's I build a server with 16 SATA disks. I believe they were WD 500GB. There were two HW RAID controllers for 8 disks each configured as RAID-6. Both controllers were then striped into HW RAID-0 doubling the B/W. With a quad Opteron MoBo I was getting around 550MBps measured throughput if memory serves me well.
The purpose was the throughput, money was no concern.
Cheers Frank
On 2024-11-17 14:52, Frank Bures wrote:
On 2024-11-17 02:02, Tim via users wrote:
And maybe you could get a RAID device which has SATA ports to the PC, so it can spread the load internally across several drives and keep up with a very high data speed. I've never looked to see if anyone has actually done that.
Back in the early 00's I build a server with 16 SATA disks. I believe they were WD 500GB. There were two HW RAID controllers for 8 disks each configured as RAID-6. Both controllers were then striped into HW RAID-0 doubling the B/W. With a quad Opteron MoBo I was getting around 550MBps measured throughput if memory serves me well.
The purpose was the throughput, money was no concern.
Sorry, Errata:
HW RAID-0 should read SW RAID-0. OS was RH, I don't remember the version.
Frank
On 17/11/24 18:02, Tim wrote:
Chris Adams:
That's a misunderstanding of how things work. The SATA port speed is just an upper-bound on transfer, but has nothing to do with how fast a device can actually read data (similar to having a 1G network card and even Internet service doesn't mean sites will serve data to you at 1G). Traditional spinning hard drives typically do top out in the neighborhood of 150 MB/s... and in fact, the official spec from Seagate for that drive is an average read rate of 156 MB/s.
Is that really megabytes or megabits per second?
The bench mark speeds are megabytes per second and the quoted disk performance speeds and the port speed is megabits per second, and looking at the performance I might try hdparm and see what it says for raw reads and cached reads, assuming my hard disk has cache (I haven't checked that spec yet). I have an issue at the moment that seems to have started with an update in F40, where SDDM takes a long time to load when the gui boot screen goes black and KDE takes a long time to load after the SDDM login screen goes black, plus KDE sits there thrashing my disk drives for several minutes after the desktop is displayed, and reducing the performance of starting up new applications. I'm trying to determine whether it really is KDE or whether my disks are not performing up to scratch. It also doesn't help when the I/O summary display in HTOP says the disk reads are at 100% but its detail display says nothing is performing any I/O.
And the converse applies to Stephen, remember when you're measuring one thing against another, and they use the two different things. Convert gigabits per second into megabytes per second, and it seems far less impressive. Even more so when they mix up the usage of 1024 or 1000 bits and bytes multipliers.
Stephen Morris:
If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
Because big numbers are a marketing ploy... Sure, there's *something* that the SATA port can do at that speed, but it's not continuously churn your data through in the way that you'd like.
Yes, I understand that but when the device specs specify that the device can operate at 1Gb/s, 3Gb/s and 6Gb/s and the device is plugged into a 6Gb/s port, I expect it to operate at the faster end of the range, but a benchmark speed to 156GB/s says that it is not.
If they put high speed cache between SATA port and internal storage, they can increase data through-put up to a point (to the point where it's filled the cache). So it's quick for storing one or two very large files, because they're measuring the SATA data speed. But internally, the cache is transferring over to the storage medium at a much slower rate.
Yes, a full cache can cause performance issues, but I would expect to be able to play around with the caching algorithms to control what gets cached and what doesn't, particularly when looking at sequential vs random access, combined with disk fragmentation. I know EXT4 is a journaling file system which is supposed to make fragmentation a non-issue, but I don't know whether BTRFS is the same.
And maybe you could get a RAID device which has SATA ports to the PC, so it can spread the load internally across several drives and keep up with a very high data speed. I've never looked to see if anyone has actually done that.
I was doing that for a while where I was installing Windows 10 in a Raid 10 environment provided by my motherboard, but for the OS to be installed in that environment it required drivers to be installed at OS install time and the motherboard only provided Windows drivers, so I was running my Linux distributions in a VM. At the time the research I did specified that Fedora Workstation did not support being installed in RAID, only Fedora Server had that support.
I remember when IDE went over to UDMA (same two inch wide [approx] fat ribbons, with twice the wires in them). It could achieve much higher data speeds across the cable, but the drive medium was a bottle neck. Then they put cache RAM in the drives, and that allowed more data to be quickly dumped across the cable to the drive, but the same problem existed: Once the cache was full, you're down to the slow speed of the storage medium inside the drive.
I know that can be an issue and the issue can be compounded by deferred writes causing data to remain in cache for longer.
regards, Steve
On 2024-11-17 14:00, Stephen Morris wrote:
On 17/11/24 18:02, Tim wrote:
Chris Adams:
That's a misunderstanding of how things work. The SATA port speed is just an upper-bound on transfer, but has nothing to do with how fast a device can actually read data (similar to having a 1G network card and even Internet service doesn't mean sites will serve data to you at 1G). Traditional spinning hard drives typically do top out in the neighborhood of 150 MB/s... and in fact, the official spec from Seagate for that drive is an average read rate of 156 MB/s.
Is that really megabytes or megabits per second?
The bench mark speeds are megabytes per second and the quoted disk performance speeds and the port speed is megabits per second, and looking at the performance I might try hdparm and see what it says for raw reads and cached reads, assuming my hard disk has cache (I haven't checked that spec yet).
The cached reads are referring to the system cache, not the hard drive. It's not accessing the hard drive at all for that test.
And the converse applies to Stephen, remember when you're measuring one thing against another, and they use the two different things. Convert gigabits per second into megabytes per second, and it seems far less impressive. Even more so when they mix up the usage of 1024 or 1000 bits and bytes multipliers.
Stephen Morris:
If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
Because big numbers are a marketing ploy... Sure, there's *something* that the SATA port can do at that speed, but it's not continuously churn your data through in the way that you'd like.
Yes, I understand that but when the device specs specify that the device can operate at 1Gb/s, 3Gb/s and 6Gb/s and the device is plugged into a 6Gb/s port, I expect it to operate at the faster end of the range, but a benchmark speed to 156GB/s says that it is not.
This was explained in another email. It doesn't matter how fast the bus is, the hard disk can only read so fast. The specs say that the disk will work with any of those bus options. And the data that is read will be transferred at those rates. But the overall rate is still limited by the disk.
If they put high speed cache between SATA port and internal storage, they can increase data through-put up to a point (to the point where it's filled the cache). So it's quick for storing one or two very large files, because they're measuring the SATA data speed. But internally, the cache is transferring over to the storage medium at a much slower rate.
Yes, a full cache can cause performance issues, but I would expect to be able to play around with the caching algorithms to control what gets cached and what doesn't, particularly when looking at sequential vs random access, combined with disk fragmentation. I know EXT4 is a journaling file system which is supposed to make fragmentation a non- issue, but I don't know whether BTRFS is the same.
That's not the cache that's meant. He was referring to the hard disk internal cache. You have no control over that. The filesystem could be a factor for seek times, but it shouldn't be that much.
I remember when IDE went over to UDMA (same two inch wide [approx] fat ribbons, with twice the wires in them). It could achieve much higher data speeds across the cable, but the drive medium was a bottle neck. Then they put cache RAM in the drives, and that allowed more data to be quickly dumped across the cable to the drive, but the same problem existed: Once the cache was full, you're down to the slow speed of the storage medium inside the drive.
I know that can be an issue and the issue can be compounded by deferred writes causing data to remain in cache for longer.
Again, this was referring to the cache on the hard disk, not the system cache.
On 18/11/24 10:36, Samuel Sieb wrote:
Yes, a full cache can cause performance issues, but I would expect to be able to play around with the caching algorithms to control what gets cached and what doesn't, particularly when looking at sequential vs random access, combined with disk fragmentation. I know EXT4 is a journaling file system which is supposed to make fragmentation a non- issue, but I don't know whether BTRFS is the same.
That's not the cache that's meant. He was referring to the hard disk internal cache. You have no control over that. The filesystem could be a factor for seek times, but it shouldn't be that much.
I know there is a huge difference in price points, but 40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality. I was so much concerned about seek overheads, but rpsmiss overheads and the impact fragmentation has on that particularly if a load of a file has to traverse all over the disk to do so. But having said this I'll put this to bed, I may have found what is causing the KDE start up hard disk thrashing.
regards, Steve
On Mon, Nov 18, 2024 at 4:46 PM Stephen Morris steve.morris.au@gmail.com wrote:
On 18/11/24 10:36, Samuel Sieb wrote:
Yes, a full cache can cause performance issues, but I would expect to be able to play around with the caching algorithms to control what gets cached and what doesn't, particularly when looking at sequential vs random access, combined with disk fragmentation. I know EXT4 is a journaling file system which is supposed to make fragmentation a non- issue, but I don't know whether BTRFS is the same.
That's not the cache that's meant. He was referring to the hard disk internal cache. You have no control over that. The filesystem could be a factor for seek times, but it shouldn't be that much.
I know there is a huge difference in price points, but 40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality.
Some do. For example, Intel Optane memory is a storage accelerator. It was a stock component of my Dell XPS 8930. And I understand Apple is shipping storage devices that optimize file access using a combination of RAM+Disk in one solid state drive package. (I can't find the name Apple uses for its drives).
I was so much concerned about seek overheads, but rpsmiss overheads and the impact fragmentation has on that particularly if a load of a file has to traverse all over the disk to do so. But having said this I'll put this to bed, I may have found what is causing the KDE start up hard disk thrashing.
Jeff
On Tue, 2024-11-19 at 08:39 +1100, Stephen Morris wrote:
40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality.
I'm wondering how a storage device is going to know which files to treat differently from other files? That's going to be OS dependent.
On Tue, 2024-11-19 at 12:52 +1030, Tim via users wrote:
On Tue, 2024-11-19 at 08:39 +1100, Stephen Morris wrote:
40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality.
I'm wondering how a storage device is going to know which files to treat differently from other files? That's going to be OS dependent.
I had the same thought. Possibly MVS had a way to do this, I wouldn't know, but it doesn't sound like anything Linux-like.
poc
On 19/11/24 13:22, Tim wrote:
On Tue, 2024-11-19 at 08:39 +1100, Stephen Morris wrote:
40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality.
I'm wondering how a storage device is going to know which files to treat differently from other files? That's going to be OS dependent.
The storage device doesn't, you tell it what to do.
regards, Steve
Stephen Morris wrote:
40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality.
Tim:
I'm wondering how a storage device is going to know which files to treat differently from other files? That's going to be OS dependent.
Stephen Morris:
The storage device doesn't, you tell it what to do.
Then why did you think that disc drives had advanced to the point where they could control what's cached or not? (Your message on 19 Nov.)
Storage control is an OS-controlled thing through device controller on your motherboard or daughterboard. Or, much more likely, completely in software with the hardware just being a simple interface.
The storage device - the disc drive itself - is a pretty dumb device.
To optimise use of a storage device (to that degree) you'd need to make OS- and file-system-specific decisions. That's really outside the scope of a simple storage device that's used on many (very) different kinds of systems. Otherwise, you'd need custom disk drives for each system. Generic hardware is just one of the reasons PCs were cheaper than mainframe equipment.
And yes, your (outside the disc drive) controller could possibly send commands to the drive about caching priority, and maybe the drive could do different things, but the decisions would be external. And I suspect new bottlenecks would arise.
Disks in mainframe days had no onboard cache, and the O/S had to do all that. Today that has been built into much cheaper disks. While there is no mechanism to determine what is cached by the onboard disk memory and what is not, there are situations where you need disk cacheing to be disabled for the full disk, and that can be done in Linux. Its not something commonly done anymore with the use of COW filesystems and actually-working redundant UPS systems, but there are complex mission-critical db apps and complex multi-machine apps that need to make guarantees that something is really on disk after a 2-phase commit completes. The default disk performance suffers a huge hit when doing this though, as the rotational latency becomes the critical speed limit instead of the bus write speed. In general, these apps would use their own dedicated disks and the rest of the system is not then limited by the same handicapping of the disk.
In the Disks app, select the disk, go to the 3 dots on the top right, select write cache and select disable. I'm unsure, but you might need to reboot as well. You can also do this in the CLI, but its been years since I have had to do that.
On 2024-11-20 22:45, Tim via users wrote:
Stephen Morris wrote:
40 years ago mainframe storage controllers provided functionality to control what files were loaded into the storage cache and how much of the file was loaded, I just thought hard disks had advanced enough to now provide similar functionality.
Tim:
I'm wondering how a storage device is going to know which files to treat differently from other files? That's going to be OS dependent.
Stephen Morris:
The storage device doesn't, you tell it what to do.
Then why did you think that disc drives had advanced to the point where they could control what's cached or not? (Your message on 19 Nov.)
Storage control is an OS-controlled thing through device controller on your motherboard or daughterboard. Or, much more likely, completely in software with the hardware just being a simple interface.
The storage device - the disc drive itself - is a pretty dumb device.
To optimise use of a storage device (to that degree) you'd need to make OS- and file-system-specific decisions. That's really outside the scope of a simple storage device that's used on many (very) different kinds of systems. Otherwise, you'd need custom disk drives for each system. Generic hardware is just one of the reasons PCs were cheaper than mainframe equipment.
And yes, your (outside the disc drive) controller could possibly send commands to the drive about caching priority, and maybe the drive could do different things, but the decisions would be external. And I suspect new bottlenecks would arise.
On 21 Nov 2024, at 13:57, John Mellor john.mellor@gmail.com wrote:
In the Disks app, select the disk, go to the 3 dots on the top right, select write cache and select disable. I'm unsure, but you might need to reboot as well. You can also do this in the CLI, but its been years since I have had to do that.
This should not be required. The OS will send a command to the disk to write through its cache when the OS needs this for data integrity reasons.
In the bad old days disk manufacturers used write cache tricks to win bebchmarks. But Microsoft added tests to Windows that detected the tricks and refused to use the drives. They did this as the disk failures were being blamed falsely on Windows bugs. Because of this, and other changes, disks became well behaved around caching.
Barry
Tim:
Because big numbers are a marketing ploy... Sure, there's *something* that the SATA port can do at that speed, but it's not continuously churn your data through in the way that you'd like.
Stephen Morris:
Yes, I understand that but when the device specs specify that the device can operate at 1Gb/s, 3Gb/s and 6Gb/s and the device is plugged into a 6Gb/s port, I expect it to operate at the faster end of the range, but a benchmark speed to 156GB/s says that it is not.
If you could speed test data through the drive's internal cache only (something it whizzed through, and didn't need to read or write to the internal storage medium), you might find you get close to idealistic claimed SATA speeds.
The other thing to consider with drive thrashing, that I don't recall seeing being mentioned in this thread is a drive with errors. If it's having trouble reading/writing its media, performance will be awful.
On Mon, Nov 18, 2024 at 2:27 AM Tim via users users@lists.fedoraproject.org wrote:
The other thing to consider with drive thrashing, that I don't recall seeing being mentioned in this thread is a drive with errors. If it's having trouble reading/writing its media, performance will be awful.
At one time the standard issue laptops at my work would become very
slow. Some tools showed the disks re-reading a sector multiple times. IT had boxes of new replacement drives and was filling the previous box with failing drives.
-- George N. White III
On 17 Nov 2024, at 04:45, Stephen Morris steve.morris.au@gmail.com wrote:
If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
Because that is the spec of the connectors on the motherboard. It provides an upper bound on transfer speed, but for an HDD its the physics of rotation that limit speed of transfer and access latency.
Barry
Once upon a time, Stephen Morris steve.morris.au@gmail.com said:
If that is the case why does the specs for that device under performance say it will support speeds of 1Gb/s, 3Gb/s and 6Gb/s.
Because that is the speed of the link between the drive and the controller/motherboard. It's possible for an individual sector to be sent to the controller at that data rate (short burst of data, especially if the requested sector is in the drive's cache), but that's not what can be sustained, due to the mechanical process of spinning platters.
Also, don't forget that SATA speeds are rated in bits, while drive performance is typically rated in bytes... so 156 MB/s is 1.25 Gb/s (really more than that because of link overhead). SATA connections were upgraded to 3 Gb/s as hard drives approached the 1.5 Gb/s speed (especially for short bursts), and then SATA was upgraded to 6 Gb/s to handle SSDs (and then NVMe replaced SATA for much faster SSDs).
Basically... nobody gets spinning hard drives for high performance. SSDs, especially NVMe, are the high performance target. Spinning drives are for bulk storage, backups, and certain write patterns like video recordings (monitoring systems and DVRs).
Stephen Morris composed on 2024-11-17 11:22 (UTC+1100):
To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda),
That's typical of what I've seen on any of my Barracudas. I usually use hdparm -t for checking.
On Sat, Nov 16, 2024 at 7:22 PM Stephen Morris steve.morris.au@gmail.com wrote:
To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda), which given that device support 1/3/6 Gb/s I/O speeds and the device is plugged into a 6Mb/s Sata port on the motherboard, is very poor performance. Doing some research on disk performance I found a reference to bad disk performance be attributable to BTRFS and that the disk should be formatted as EXT4, is this sentiment reasonable and if so why is Fedora pushing the use of BTRFS.
You might also have a look at Debian's page at https://wiki.debian.org/SSDOptimization.
(I am not aware of a similar page for Fedora).
Jeff
On 22/1/25 09:37, Jeffrey Walton wrote:
On Sat, Nov 16, 2024 at 7:22 PM Stephen Morris steve.morris.au@gmail.com wrote:
To test my hard disk performance I have run Kdiskmark and used the Real World Performance Profile. For its sequential read performance it is showing around 156 MB/s on my ST3000DM007-1WY1 (3TB Seagate Barracuda), which given that device support 1/3/6 Gb/s I/O speeds and the device is plugged into a 6Mb/s Sata port on the motherboard, is very poor performance. Doing some research on disk performance I found a reference to bad disk performance be attributable to BTRFS and that the disk should be formatted as EXT4, is this sentiment reasonable and if so why is Fedora pushing the use of BTRFS.
You might also have a look at Debian's page at https://wiki.debian.org/SSDOptimization.
(I am not aware of a similar page for Fedora).
Thanks Jeff, I've had a look at that page and even though it seems to be aimed at ssd's it is still useful. My main performance issue is fragmentation impacting rpsmiss and seeks. I've read that fragmentation on journaling file systems like EXT4 is a non-issue, but I don't know why, and I haven't seen anything to indicate whether or not BTRFS is similar.
regards, Steve
Jeff