I'm no stranger to diagnosing hardware problems on Linux systems. Even though most of my professional work over the past few years has involved virtualization, I still enjoy crouching under desks and fumbling around with devices and memory modules. Well, except for the "crouching under desks" part. But none of that means that persistent and mysterious bugs aren't frustrating. I recently faced off against one of those bugs on my Ubuntu 18.04 workstation, which remained unsolved for months.
Here, I'll share my problem and my many attempts to resolve it. Even though you'll probably never encounter my specific issue, the troubleshooting process might be helpful. And besides, you'll get to enjoy feeling smug at how much time and effort I wasted following useless leads.
It all began when I noticed network latency and download speeds on my workstation were awful. Well, let me back up. It really began a few years ago when I moved and couldn't be bothered to run cable through my new home to all the rooms that would need connectivity. Instead, I bought a bunch of USB WiFi adapters. The performance wasn't as good as Ethernet, but it was good enough—usually between 2Mbps and 4Mbps between machines.
But then one gloomy winter day, my workstation connection deteriorated. As far as I could tell, it seemed to have hit around the time I briefly put my interface into monitor mode to experiment with network sniffing. Latency was now high, I was often lucky to get 300kbps download speeds, and ifconfig was reporting high numbers of transfer errors. It was so bad, I would regularly download larger internet-based packages to a laptop and transfer them to my workstation using a USB. What century is this?
Eliminating the impossible
Could my USB WiFi interface have degraded over time? No problem: there are probably 10 of them—all different models—scattered through the house. So I tried two or three others, but with similarly depressing results.
While a long shot, it couldn't hurt to check my ISP service connection. Multiple DSL speed test websites all confirmed that my provider was giving me everything they said they would. Perhaps the WiFi router provided by my ISP was to blame. Nope. The laptop was getting good connectivity to the same router even while sitting right next to the workstation.
Was I sure it was a WiFi problem? Perhaps it was a general networking issue. To find out, I combined two old CAT5 cables and ran them from the router, under the dining room table, through the living room, across a hall, and into my office. Instant connection happiness: 25Mbps downloads. But I sensed that, over the long term, the wife might not respond well to the cable in the dining room. Not a solution, but it did confirm that the issue was WiFi-specific.
Did my brief switch to monitor mode somehow mess with some obscure WPA_supplicant setting? (WPA_supplicant is a package for managing client connections using the WPA, WiFi Protected Access, protocol.) The quickest way to find out for sure was to boot my workstation to a live USB Ubuntu session to get a pristine, out-of-the-box operating system experience. The result? Even using the live session, I was still getting poor connectivity. Which pretty much ruled out software configurations, didn't it?
So what's left? In desperation, I emailed an old friend who gave me some logical suggestions, including checking dmesg for errors, lsmod for conflicting drivers, and the firmware versions of my wireless chipsets. Given how much progress has been made over the past few years incorporating such a wide range of WiFi chipsets into the Linux kernel—and how complicated it could be to dive down the firmware version rabbit hole—I skipped that last item and hoped for the best.
Finally, my friend advised me to purge wpa_supplicant and reinstall it. That "reinstall" bit could get tricky, considering that I'd have no WiFi connection at that stage. But it's doable. So I did it and, indeed, my connection improved… for a week or so. Then it was back to the dark ages of 250kbps.
Desperate, I tried uninstalling NetworkManager and replacing it with the Wicd package. Two interesting observations about that:
- Did you know that uninstalling NetworkManager in GNOME 3 automatically causes gnome-control-center to uninstall as well? That's considered a feature, by the way, not a bug.
- And did you know why Wicd isn't the default WiFi tool used with any Linux distros with which I'm familiar? I'm now pretty sure that—while it can be a life-saver in some edge scenarios—it's because the module is more likely to introduce lots of annoying buggy behavior.
New hardware, same old problem
So there things stood for a few uncomfortable weeks until, for mostly unrelated reasons, I decided to build a replacement workstation.
Happily, my connection to the new machine was robust. Until it wasn't.
Within a few days, I was back to watching the internet through a thick fog. Speed tests showed me I was getting latency in the range of 100ms, and transfer speeds were dead slow. What was going on? The only piece of hardware that both machines shared was the power supply, and there's no way that was a factor here.
In fact, the workstation upgrade made things simpler for me, because it ruled out just about everything. And, as Sherlock Holmes famously said, when you've ruled out the impossible, what's left is the probable.
So what was left? There was really only one thing, but it took all those dead ends and failures to get me to it. Because my workstation lives under my desk, and because WiFi reception can suffer behind too many physical barriers, I wanted to lift my USB WiFi adapter as high as I could. To do that, I connected it to my workstation through a three-foot USB extender and sat it on the surface of my desk. On a hunch, I pulled the extender and plugged the adapter directly into one of the integrated USB ports.
Instant success. Perhaps using the extender was a bad idea from the start or, more likely, it degraded over time, but everything was humming along happily now. I'll bet you didn't guess that ending. I certainly didn't.
7 Comments