Thanks Richard. Yes, I talked with Titan; they suggested trying the pcie-m.2 adapter. I will try them again. I have not checked for bios updates. Not sure how to go about that (last time I did that it required an msdos floppy disc). Haven't tried the SSDs in another device because I don't have one. But the fact that replacing the SSD causes it to work, where it wasn't working before, tells me they were damaged. I have at least once power off/on the workstation, and the bios did not find any ssd to boot from. So power cycle didn't fix it, but replace ssd did fix it. I will try Titan again later today, but just looking for ideas. Thanks, Neal On Tue, Feb 22, 2022 at 8:44 AM Richard Shaw <hobbes1069(a)gmail.com&gt; wrote: -- *Those who don't understand recursion are doomed to repeat it*

On Tue, 22 Feb 2022 at 10:04, Neal Becker <ndbecker2(a)gmail.com&gt; wrote: With this history, I'd probably replace the workstation power supply. I would also scan the the system board for capacitors on bulging tops or overheated components. Are there any externally powered devices connected to the workstation (other than the monitor)? Are you in an area with frequent lightning storms? How stable is your power? Is the system connected to a UPS? I had a similar experience with spinning disks in a system that contained a drive-bay radio receiver and was connected to a satellite dish and GPS receiver on the roof, and an antenna controller. Everything was powered by a high quality UPS. I added a heavy wire connecting the antenna controller case to the workstation case and the failures stopped. I gather you now have space for two m.2 SSD's. If you haven't discarded the non-working devices, it would be interesting to see if any are detected and what smartmontools says about them, but you also have the option to put /var on a separate drive. Smartmon tools can monitor a drive and report any problems it detects, but you may also want to run self-tests periodically. -- George N. White III

On Tue, Feb 22, 2022 at 11:19 AM Neal Becker <ndbecker2(a)gmail.com&gt; wrote: If it was just erasing the partition table the drive would still be visible using lsblk, and you could re-partition it with fdisk, etc. You mentioned a surge suppressor strip - any chance it has already suppressed a surge in the past? If so, it might not be functioning as a surge suppressor anymore.

On Wed, 23 Feb 2022 at 06:49, Tim via users <users(a)lists.fedoraproject.org&gt; wrote: Various magical incantations do sometimes seem to work: Solid State Drive SSD not seen in computer bios data recovery repair - YouTube <https://www.youtube.com/watch?v=BgddtpSEhQQ> Fix your dead SSD with the power cycle method - The Silicon Underground (dfarq.homeip.net) <https://dfarq.homeip.net/fix-dead-ssd/> fmadio | Recover Bricked SSD with JTAG <https://fmad.io/blog-ssd-bricked-restore.html> I used to add surge protection to power bars. We had a tree fall on the cable coax that they had installed without a proper anchor, just a zip tie to the mast for the AC power. The mast was pulled off the house, which meant the neutral line got disconnected first, and lightbulbs went off like flash bulbs. My Victor 9000 PC with the home-made surge protection survived, but we lost the doorbell transformer and a radio (and the thyristers in the surge protector). My father in law also used home-made surge protectors. A truck hit a high-voltage distribution line, causing the high voltage wires to fall onto the lower voltage feed to the house. His computer survived, but some appliances died. My power bar had a circuit breaker which did trip. I grew up in an area with frequent power outages. My parents were careful to unplug computers when not in use. We often put an overhand knot in power cords. One day I had just turned on an electric kettle when lightning hit near the house. The kettle had a metal base with a hole where the cord passed thru and a 2-conductor cord. The cord burned off at the hole -- I assume an induced current took a shortcut from one conductor to the other. Nothing else was damaged. At work we had a small machine room with a window in the door. One day during a storm I was walking past the room when lightning hit the building. I saw a bright trail down the corner of the room and across one of the SGI Octane workstations. Those systems had a heavy metal chassis under a plastic cover, but there was a light bar outside the chassis. One of the incandescent bulbs in the light bar burned out, but that was the only damage. Better designed equipment costs more. I generally try to buy gear that has been on the market a couple years, which gives time for drivers to make it into the kernel and for design flaws to be noticed. Vendors often reduce prices just before introducing new models. I once scored a PowerEdge server with full complement of ECC memory for the price of the memory days before a newer model was announced. -- George N. White III

On Wed, 23 Feb 2022 at 12:20, Neal Becker <ndbecker2(a)gmail.com&gt; wrote: Some articles mention drives talking very long times to respond while running internal "grooming" -- I presume moving data off areas with high "wear" to areas of lower wear. There are also reports that frantically moving the drive around to different external adapters eventually allows drive to mount. All could be explained by drive taking too long to respond while grooming is in progress. Once grooming is finished, drive works normally. -- George N. White III

Well the fact that the failed ssd is working again tells me it wasn't zapped by static or power transients. The comment about taking a long time for the ssd to reorganize itself is interesting, but here it failed 1 day, and I went to fix it the next day, where it still was not detected by F35 live usb stick. Came back to life after I installed the sata drive. The only thing I can think of is a motherboard component that fails when warm, came back to life after it cooled while I installed the sata? But since the same failure was seen with SSD plugged into m.2 socket on motherboard, and different SSD plugged into m.2 socket on pcie->m.2 adapter; I don't think there is any common component? Don't know enough about these motherboard architecture to say for sure. Must be something in common though to explain these symptoms. On Wed, Feb 23, 2022 at 8:20 PM Tim via users <users(a)lists.fedoraproject.org&gt; wrote: -- *Those who don't understand recursion are doomed to repeat it*

On Thu, 24 Feb 2022 at 09:07, Tim via users <users(a)lists.fedoraproject.org&gt; wrote: This is similar to the pattern reported on other forums. Drive is appears to be dead, after trying various external adapters it starts working, so we don't know if there is something special about a particular adapter or a change in the internal status of the drive. One recovery recipe is to apply power without the data connection. This could allow the drive to get its act together without the diversion of commands on the data connection. It makes we wonder if the drive doesn't have whoops mode where it reinstalls firmware and rescans the drive to get its own internal mapping into a consistent state. I assume vendors try to design this (if it even is a real things) in a way that most users won't notice, but there may well be edge case usage patterns that the vendor didn't anticipate.

Did you fully pull power and/or turn off power at the power supply prior to installing the sata drive? Ie was the sata drive install the first time the machine was actually off and not just quickly rebooted/reset? If that was the first full power off it would be likely that the drive was in some sort of internal firmware loop and until power was fully removed it never actually was rebooted and/or reset.. I had a spinning sata disk that I was forcing badblocks in an attempt to clean it up to be usable, and 1 out of 50 times or so when it was handling the bad blocks it would stop responding, and a simple reboot never fixed it, but a turn off power for a few seconds fixed it every time. On Thu, Feb 24, 2022 at 6:42 AM Neal Becker <ndbecker2(a)gmail.com&gt; wrote: > > Well the fact that the failed ssd is working again tells me it wasn't zapped by static or power transients. The comment about taking a long time for the ssd to reorganize itself is interesting, but here it failed 1 day, and I went to fix it the next day, where it still was not detected by F35 live usb stick. Came back to life after I installed the sata drive. > > The only thing I can think of is a motherboard component that fails when warm, came back to life after it cooled while I installed the sata? But since the same failure was seen with SSD plugged into m.2 socket on motherboard, and different SSD plugged into m.2 socket on pcie->m.2 adapter; I don't think there is any common component? Don't know enough about these motherboard architecture to say for sure. Must be something in common though to explain these symptoms. > > On Wed, Feb 23, 2022 at 8:20 PM Tim via users <users(a)lists.fedoraproject.org&gt; wrote: >> >> On Wed, 2022-02-23 at 10:41 -0400, George N. White III wrote: >> > I used to add surge protection to power bars. We had a tree fall on >> > the cable coax that they had installed without a proper anchor, just >> > a zip tie to the mast for the AC power. The mast was pulled off the >> > house, which meant the neutral line got disconnected first, and >> > lightbulbs went off like flash bulbs. My Victor 9000 PC with the >> > home-made surge protection survived, but we lost the doorbell >> > transformer and a radio (and the thyristers in the surge protector). >> >> I've always been a bit wary of ones in power boards (the multi-socket >> adaptors on short leads. Ideally surge protection should be at the >> distribution box (to protect the whole house from surges from the >> street). You really want the house to disconnect from power under >> dangerous conditions, not just a board with potentially (now) hazardous >> wiring still energised between it and the wall. >> >> If you pull apart power boards, you often notice how thin the wiring >> is, and the metal strips used to form the sockets. You've got a good >> chance at weakening or blowing the board instead of the main fuse. Even >> under good wiring circumstances the fuses and breakers may not be quick >> enough to break the circuit. >> >> Some of the boards use inappropriate over-voltage clamping that's >> always being driven warm by triggering too close to the normal mains >> voltage (possibly this is manufacturers of 110 volt equipment selling >> their products into 240 volt markets). I've come across some that are >> always too warm for my liking, and wouldn't like them being buried and >> out of sight. I've seen them discoloured and deformed plastic. >> >> People have a habit of daisy chaining power boards. EMI filters and >> surge protection at the end of the line puts heavier workload on ones >> nearer to the socket, and all the joins between. Advice was that if >> you have to have more outlets than available on one board, the first >> one plugged into the wall should be the one with filters and >> protection, then plug other plain unprotected boards directly into it, >> rather than string a series of boards through each other. The first >> board protects the rest, rather than the rest stressing out everything. >> >> -- >> >> uname -rsvp >> Linux 5.11.22-100.fc32.x86_64 #1 SMP Wed May 19 18:58:25 UTC 2021 x86_64 >> >> Boilerplate: All unexpected mail to my mailbox is automatically deleted. >> I will only get to see the messages that are posted to the mailing list. >> >> _______________________________________________ >> users mailing list -- users(a)lists.fedoraproject.org >> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org >> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > > > > -- > Those who don't understand recursion are doomed to repeat it > _______________________________________________ > users mailing list -- users(a)lists.fedoraproject.org > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

On Thu, 24 Feb 2022 10:35:11 -0500 John Mellor wrote: Yep. I had a cursed computer that something different broke on every few days. I eventually dumped it in the recycle bin at the local electronics recycling place and built a whole new system. It was impossible to track down what was really wrong with the cursed one (for all I know it really was a curse :-).

And not related... I've have a keyboard go wonky in my Dell laptop ever since I stressed it out running Folding @ Home for a year. It was running ... hot... and the thermal stress has caused a permanent, intermittent flakiness.

Endgame: So now the system is running from the sata ssd drive, but the nvme ssd is plugged into the M.2->pcie adapter with the failed ssd now working again. So I ask, what's the difference in speed between the nvme ssd plugged into the pcie and the sata? I run bonnie++ on the nvme and then: Feb 24 16:59:11 nbecker8 kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 Oh, look at that! Just as I suspected, the nvme controller fails when heavily used. The thing that surprised me is that I would have thought the nvme controller which just went down would have been on the pcie interface card. But now it seems that the nvme controller is on the motherboard and even when the nvme is plugged into the pcie the same motherboard nvme controller is still being used. That's the only explanation I can think of. Sorry for the top posting and excessive quoting: I've given up on running my own mail server and am using gmail, which makes it too easy to follow bad habits.

On Fri, Feb 25, 2022 at 6:20 AM Patrick O'Callaghan <pocallaghan(a)gmail.com&gt; wrote: I've used Gmail exclusively for all the mailing lists I'm on and it doesn't bother me anymore but I have some automatic keystrokes :) Click reply Cntl-a (unhides everything) Cntl-Home, delete, delete (goes to the top, deletes extra lines on top) Then I just navigate through and reply to what I want and delete what's unneeded (shift-up/down, delete). Works well enough for me. Thanks, Richard

On Fri, 2022-02-25 at 12:19 +0000, Patrick O'Callaghan wrote: Better that, than the mass quoting. Another option is just word your reply so it makes sense on its own. -- uname -rsvp Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64 Boilerplate: All unexpected mail to my mailbox is automatically deleted. I will only get to see the messages that are posted to the mailing list.

On Fri, 25 Feb 2022 at 08:00, Neal Becker <ndbecker2(a)gmail.com&gt; wrote: The NVME controller is the big chip on the SSD that gets hot. It has a PCIe controller and a CPU. Google the above kernel message, you should get to: < https://bugzilla.kernel.org/show_bug.cgi?id=195039> On my Fedora box with a Samsung NVMe: % sudo lspci | grep NVMe 03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 The bugzilla thread is long and has lots of detail. One post mentions connector issues. Auto parts stores sell "contact enhancer" that is used on connectors in recent model highly computerized cars. -- George N. White III

The 2 crucial bx drives I was losing, I replaced with an older smaller mx drive and that one has been working just fine for a couple of months, thinking about my issue and Neal's issue here is what springs to mind. So in my case, if mine was a power supply issue, it would have to be that something about the new ssds is excessively sensitive to power or ground loops. The thought of my issue being a power supply issue/sata issue burning the device did occur to me. And that issue I have is heavily reported in the 1-star reviews for the crucial device, several people having more than 1 failure and returning the device for refund. The people that have the failure seem to be able to repeat, and I assume others work just fine. So it would seem that there must be some component used in recent ssd's may be super sensitive to something either power supply wise or sata port wise, or the design has a internal grounding issue and is sensitive to ground loop wise that does not cause an issue with the older devices (I have 2 older SSD's and 8 hard drives that have been running in said machine for months to years just fine). I would think on an NVME device that it would be well grounded to the motherboard/case. In my case my ssds were in a plastic drive holder so the only ground would have been via the sata connection and the power supply, and so if the drive design had components expecting a screw hole ground that won't exist in some cases, and could have floating voltages then that might damage something. How was your nvme drive mounted in your case? On mine the normal screw holes were not connected to ground (plastic drive case) so the "chassis" of the drive would not have been externally grounded, and had said drive unit chassis not had a direct connect to to power or SATA ground that could end up with floating voltages on the drive chassis and any components tied to it internally. And ground loops are tricky. I have a wind meter on my roof hooked to a device that counts it's rotations, and that serial port device would randomly stop working requiring a reset of the usb-to-serial communication to get it to function again (I had a cron job to reload/reset the usb nightly because it was happening often enough). I guessed ground loop ran a ground wire to house ground and grounded the hw device doing the counting years ago, and that solved the issue. On Tue, Feb 22, 2022 at 9:47 AM George N. White III <gnwiii(a)gmail.com&gt; wrote: > > On Tue, 22 Feb 2022 at 10:04, Neal Becker <ndbecker2(a)gmail.com&gt; wrote: >> >> Thanks Richard. Yes, I talked with Titan; they suggested trying the pcie-m.2 adapter. I will try them again. >> I have not checked for bios updates. Not sure how to go about that (last time I did that it required an msdos floppy disc). >> >> Haven't tried the SSDs in another device because I don't have one. But the fact that replacing the SSD causes it to work, where it wasn't working before, tells me they were damaged. I have at least once power off/on the workstation, and the bios did not find any ssd to boot from. So power cycle didn't fix it, but replace ssd did fix it. >> >> I will try Titan again later today, but just looking for ideas. > > > With this history, I'd probably replace the workstation power supply. I would also scan the > the system board for capacitors on bulging tops or overheated components. > > Are there any externally powered devices connected to the workstation (other than the monitor)? > > Are you in an area with frequent lightning storms? How stable is your power? Is the system > connected to a UPS? > > I had a similar experience with spinning disks in a system that contained a drive-bay radio receiver > and was connected to a satellite dish and GPS receiver on the roof, and an antenna controller. Everything > was powered by a high quality UPS. I added a heavy wire connecting the antenna controller case to the > workstation case and the failures stopped. > > I gather you now have space for two m.2 SSD's. If you haven't discarded the non-working devices, > it would be interesting to see if any are detected and what smartmontools says about them, but > you also have the option to put /var on a separate drive. Smartmon tools can monitor a drive and > report any problems it detects, but you may also want to run self-tests periodically. > > >> >> >> Thanks, >> Neal >> >> On Tue, Feb 22, 2022 at 8:44 AM Richard Shaw <hobbes1069(a)gmail.com&gt; wrote: >>> >>> On Tue, Feb 22, 2022 at 7:34 AM Neal Becker <ndbecker2(a)gmail.com&gt; wrote: >>>> >>>> I know this is a bit OT, but you guys are great at answering all questions. >>>> >>>> I bought a workstation from Titan computers around 1/2020 (dual EPYC cpu). After about 1 year it stopped working. I could ssh to it, and almost any command would return Input/Output error. Unfortunately journalctl gave input/output error so I can't see logs. cat /proc/partitions did not show any nvme device (the root device) on which the OS was installed. >>>> >>>> I replaced the SSD with a samsung 980 pro. Reinstalled fedora. It then worked a few weeks, then the exact same symptoms. >>>> >>>> I replaced the SSD with another samsung 980 pro, this time with heatsink. Reinstalled fedora. It worked a few weeks. Then same symptoms. >>>> >>>> Then I replaced with a 4th samsung 980 pro, but this time instead of using the M.2 socket I used a pcie-m.2 adapter (in case something was wrong with the m.2 socket). Also added a surge protector outlet for good measure. Reinstalled. Watched the smartctl. No errors. Temperature was always low. >>>> >>>> Now it's failed again, exactly same symptoms. >>>> >>>> Any ideas? >>> >>> >>> I remember your other email about a month or so ago and thought it was really strange. Have you tried the drives in another system to confirm they're truly dead? >>> >>> I would check for BIOS updates just for good measure. Other than that, have you had any communication with Titan about it? >>> >>> Thanks, >>> Richard >>> _______________________________________________ >>> users mailing list -- users(a)lists.fedoraproject.org >>> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org >>> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure >> >> >> >> -- >> Those who don't understand recursion are doomed to repeat it >> _______________________________________________ >> users mailing list -- users(a)lists.fedoraproject.org >> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org >> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure > > > > -- > George N. White III > > _______________________________________________ > users mailing list -- users(a)lists.fedoraproject.org > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

On Tue, 2022-02-22 at 08:34 -0500, Neal Becker wrote: Rinse, lather, repeat... A thought just occurred to me, did you take proper (*) anti-static precautions when handling the drives? Remembering back to my days in college studying electronics, one thing drummed into us was proper handling of components. Static shock (charge and/or discharge) doesn't always immediately kill a product, but it usually weakens it and shortens its lifespan. Ages after some device fails you're never going to attribute failure to how you built it, but laboratory testing confirms this kind of failure. We were told that might mean something only lasts one year instead of a decade, or a decade instead of your lifespan. While you might think 10 years of product life is adequate, there's plenty of situations where that is not, and I might say you've got accustomed to the low standards of modern manufacturing. And it's not impossible that mishandling could result in something only lasting a couple of weeks. It doesn't have to be you, either. For instance, if the supplier is in the habit of walking across the carpet with drives outside of anti- static protection (or even inside anti-static bags, if they happen to generate enough static). Or moving things generating static past where they store the drives. Or destroying them in transit. /They/ could be the cause of your problems, and I'd be inclined to try another supplier when you've had several strange failures in a row. * While "proper" precautions traditionally meant anti-static floor matting, high resistance desk matting, wrist straps, etc, it's not essential to do things like you're in a research laboratory. Doing things like: Putting all your components on the desk, still bagged, with the computer, so they all reach the same potential slowly. Sitting down and staying seated while you open the bags. Keeping in contact with the desk or the PC as you pick up things and fit them to the PC. If you have to get up in the middle of the job, you keep your hand on the desk when you sit down again before touching anything else. If you work standing up, you lean against the desk the entire time. Staying in contact with your work area is easier than wearing wrist straps, and safer if you have to step away from the desk and forget to unplug (you can yank things off the desk), and safer for those of us who work with personally dangerous voltages (not being attached means you can back off and not take it with you). Goes a long way to ensuring that no large static differences build up between the things then suddenly discharge when they come into contact with each other. -- uname -rsvp Linux 5.11.22-100.fc32.x86_64 #1 SMP Wed May 19 18:58:25 UTC 2021 x86_64 Boilerplate: All unexpected mail to my mailbox is automatically deleted. I will only get to see the messages that are posted to the mailing list.