Elusive Crashing Problem Solved

Four times in the last month Kimmy or I have woken up to a dead Linux computer. At first I thought it was a kernel panic (Linux BSOD), that took out my screen. It was always happening during the job runs in the middle of the night, backups, virus scans, etc. So my first step was to turn on all debugging and wait for the problem to happen again.

Well it did but there was still nothing written to any of the log areas. Then I decided to reload Fedora on a new machine, and moved all the hard drives and eSATA cages over to the new machine. Well the problem happened there as well.

So I got this crazy idea. If the drives had a problem and went in read-only mode it would explain the absense of logs. So I put the /var /tmp and main root drive on different disks.

Then the problem happened again, this time at 3:00. I had started a F13 respin build just before I left to pick up my daughter from school. Although this time the problem was a little different. Instead of a black screen with nothing on it, I had a screen saver that wouldn't wake up. Turns out the /tmp drive had gone read only. The reason nothing would wake up is programs were all waiting for the /tmp to go back into read/write. It was like a train where the engine suddenly stopped dead, and all the other cars came piling into it.

Apr 29 14:38:00 liandra kernel: ata8.00: exception Emask 0x10 SAct 0xd SErr 0x100 action 0x6
Apr 29 14:38:00 liandra kernel: ata8.00: irq_stat 0x02060002, device error via SDB FIS
Apr 29 14:38:00 liandra kernel: ata8.00: SError: { UnrecovData }
Apr 29 14:38:00 liandra kernel: ata8.00: failed command: READ FPDMA QUEUED
Apr 29 14:38:00 liandra kernel: ata8.00: cmd 60/80:00:bf:a8:b9/00:00:23:00:00/40 tag 0 ncq 65536 in
Apr 29 14:38:00 liandra kernel:         res 60/80:00:bf:a8:b9/00:00:23:00:00/40 Emask 0x10 (ATA bus error)
Apr 29 14:38:00 liandra kernel: ata8.00: status: { DRDY DF }
Apr 29 14:38:00 liandra kernel: ata8.00: error: { ICRC }
Apr 29 14:38:00 liandra kernel: ata8.00: failed command: READ FPDMA QUEUED
Apr 29 14:38:00 liandra kernel: ata8.00: cmd 60/80:10:3f:a8:b9/00:00:23:00:00/40 tag 2 ncq 65536 in
Apr 29 14:38:00 liandra kernel:         res 41/84:2f:90:a8:b9/04:00:23:00:00/40 Emask 0x410 (ATA bus error) 
Apr 29 14:38:00 liandra kernel: ata8.00: status: { DRDY ERR }
Apr 29 14:38:00 liandra kernel: ata8.00: error: { ICRC ABRT }
Apr 29 14:38:00 liandra kernel: ata8.00: failed command: READ FPDMA QUEUED
Apr 29 14:38:00 liandra kernel: ata8.00: cmd 60/80:18:3f:a9:b9/00:00:23:00:00/40 tag 3 ncq 65536 in
Apr 29 14:38:00 liandra kernel:         res 60/80:18:3f:a9:b9/00:00:23:00:00/40 Emask 0x10 (ATA bus error)
Apr 29 14:38:00 liandra kernel: ata8.00: status: { DRDY DF }
Apr 29 14:38:00 liandra kernel: ata8.00: error: { ICRC }
Apr 29 14:38:00 liandra kernel: ata8.02: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Apr 29 14:38:00 liandra kernel: ata8.02: failed command: FLUSH CACHE EXT
Apr 29 14:38:00 liandra kernel: ata8.02: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 1
Apr 29 14:38:00 liandra kernel:         res 0e/12:02:02:00:00/00:00:00:10:0e/00 Emask 0x2 (HSM violation)
Apr 29 14:38:00 liandra kernel: ata8.02: status: { DRQ }
Apr 29 14:38:00 liandra kernel: ata8.02: error: { IDNF }
Apr 29 14:38:00 liandra kernel: ata8.00: hard resetting link
Apr 29 14:38:01 liandra kernel: ata8.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Apr 29 14:38:01 liandra kernel: ata8.02: hard resetting link
Apr 29 14:38:01 liandra kernel: ata8.02: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Apr 29 14:38:01 liandra kernel: ata8.00: configured for UDMA/100
Apr 29 14:38:01 liandra kernel: ata8.02: configured for UDMA/100
Apr 29 14:38:01 liandra kernel: end_request: I/O error, dev sdg, sector 0

My best guess, under heavy load the controller card on my eSATA drive cage heats up and pause for a minute or two. That's what causes the glitch. It also explains why the problem followed me when I re-installed Fedora and put it on a new machine. It also explains why I could never re-produce the problem. By the time I got back to it the unit had cooled down.

Guess I'm looking for a new external drive cage. I'm starting to think I should just build my own.

More doesn't mean all, and less doesn't mean none.

I've said quite a few times that I like Linux because it's less restrictive. I'm not saying there are no restrictions.

Examples:

• Windows 7 is restricted to 2 processors (EULA Limit)
• Linux (Desktop) is limited to 32 cores for 32 bit system and 255 cores for 64 bit system.

• Windows 7 limits one user on each machine concurrently (EULA Limit)
• Windows Home Server is limited to 3 users. 1 - console and 2 rdp (EULA Limit) However;

  "Terminal Services – You may only use Terminal Services functionality to the extent required to manage the
  server software in Remote Administration Mode. You may not use Terminal Services for any other purpose."

• Linux (Desktop) I couldn't find a limit. But as a test I tried making 10 connections to my X11vnc server and it went no problem.

• Windows Home Server: it's not allowed to be connected to a free Primary Domain Controller.
• Linux (Desktop) you can connect it to whatever you want.

• Windows 7 I can install iTunes
• Windows Home Server I can't install iTunes (Repeated problems with iPod/iPhone service when tried to force it)
  
• Windows 2008 Server I can't install iTunes (BSOD when I tried to force it)
  
• Linux I can't install iTunes (will mostly, wine can emulate part of it)
  

There are clearly restrictions in Linux. However, the restrictions in Linux don't seem to cause me as many issues. Most of the time I never hit them. Where with windows I'm hitting the restrictions all the time. Not to mention cost involved. To legally get the Terminal Server option I use all the time on Linux, it would cost $1029 for Win2008 and $749 for a 5 pack RDS. Ref (again I'm sure you can get it cheaper somewhere, but it's still going to be more than my family's monthly budget.)

Support for Linux?

Whenever I mention, that Linux is a viable alternative to Windows in the desktop market, the comment of "but there's no support for Linux" always comes up. There is actually lots of support for Linux. In fact, for my problems, better support because the guys answering the questions/tickets in Linux are usually the experts. Where big companies, and their product support have a tier system and the first level support usually has no clue what I'm talking about.

Calling Microsoft Support:

First, I'll pick on Microsoft, only because it was the most recently mention. -- I have similar stories with Rogers, Bell, Futureshop, Staples, etc.. -- Recently I learned with the $200-350 purchase of the Windows 7 Retail you get free support from Microsoft. (well not really free your paying for it from the purchase price.) This explains the big price difference between the OEM and Retail versions.

I have two stories:

First is from several years ago. We (Lenbrook Industries) had a problem with excel, it couldn't handle a spreadsheet one of the accountants designed. When we did the calculation run, it would GPF (General Protection Fault). When I called Microsoft Support, it took me a few weeks just to convince the guy we had a problem. After that he said the only solution was to add more memory to the system. The system was a 386/486 (can't remember which) with 16M of memory, which was the max for any system at the time. The average for a windows system was 1-2M, not to mention the 640k barrier at the time. The support ticket was closed as that was the only solution Microsoft was going to offer. Needless to say we started looking at other companies products.

Note: even thought the story is old, the trend of my support experience as a customer has not changed. I only bring this up to illustrate my point below.

The second story is more recent. About a week ago I logged a ticket on Windows 7. The question is rather complex, but after 1 phone call and 6 emails and +1 week later, they still don't get the question, they keep sending me irrelevant solutions. I even sent them the blog post Ross sent me on how to accomplish the task. The question to support/customer service is how to make it legal without breaking the EULA. Again, how helpful is support?

This is a common experience for me. Any time I call support it takes weeks to three months (rogers support) just to explain the problem to them, or even convince them I have a problem. After that it's still 50/50 if they can even help me. Granted my problems are usually much more complex that the average user. However, my mom and dad call me first for any support because the feel they get no help from these companies. My dad mostly thinks the support guys are idiots. (and since I'm a support person for a large company I'm not sure if I should take offence to that)

Update: make that 2 calls. Microsoft just called while I was typing this. Their going to arrange an escalation technician call on Tuesday.

So my POINT just because their is a support network it doesn't automatically guarantee it's going to be any help to you. Most big company support networks (example: rogers and bell) are setup to handle only common questions. -- My computer won't turn on, is it plugged in, oh thanks for the help. -- Anyone who's called support and had their problem solved on the first try, good for you! Not all of us have been so lucky.

Side Note: Microsoft support desk is one of the better ones. Even thought it's taken a week just to have the techs understand, at least their chasing/calling me. I don't have to chase them. (which is really nice for a change.)

Linux Support:

With that out of the way let's talk about where you can get Linux support. First off, there is many more support people for Windows, and many more companies supporting windows. That's a given because of the +80% desktop market share. But as stated above, quantity doesn't guarantee quality. (Increases your chances.. but it's still hit and miss)

For the rest, I'm going to talk Fedora Linux here because it's what I have the entire family running. For Ubuntu support, maybe Henry can start a blog.

The first place I go to for support is the forums. http://www.linuxquestions.org is the main site I go to. I can usually get an answer to my question with in 2-24 hours. There is a really strong community there for helping out Linux users. -- now if I can only get the family to start posting questions rather than bugging me all the time --

The second place is the bugzilla. http://bugzilla.redhat.com/ Bugzilla is your direct line to the developers, contributors and administrators in charge of the Fedora project. Level of support here varies depending on the problem and area. There's also a automatic bug logging tool in Fedora 12 so if your program crashes, it packages up all the debug logs and can fire it off to your bugzilla account. These are the two places I goto for help.

For the new user there is still other places you can go for support. There's the online help pages (which I find just a useless as Microsoft's help pages, but that's just my experience) Then there is all the options listed on http://fedoraproject.org/en/get-help The online classrooms is an interesting thought.

Then for the person who wants some accountability there's the "paid" support. Companies like Redhat, Oracle, and others, you can pay a yearly subscription fee and get the same level of support that you would with any other provider.

Finally there's the "ask a friend". Most of the computer geek friends you have can support any computer. It might take them a bit longer to support you on Linux, if they've never seen it, but a computer is a computer. Both have a file system, linux is / for path names like /home/me where windows is c:\Users\me. Both have some type of menu system with programs. Both have GUI tools for managing the system. Most computer geeks can easily work through these differences to help the general users. -- Especially if the question is, how do I get on the web --

So there is plenty of support options for Linux. Makes you wonder where they get this "no support" logic? Recently I've been told that "the vast majority of "conspiracy theories" from the open source community are complete bunk" but by the same token the "no support" or "Linux isn't a viable alternative" are created from the Microsoft community and are also complete bunk.

Why do I rant?

Simple, I get tired of people telling me windows is the best operating system because they say it is. Or Windows is the best operating system because somehow Microsoft managed to make it a mandatory course requirement in all schools in Ontario. Or that Windows is the best operating system because you have to buy it when you purchase a new computer. I'm tired of people spreading rumours like Linux is not a viable alternative to Windows. Or that Linux can't do simple things like browse the internet. Then there's the entire TOS type studies where Microsoft had to ask a company to "adjust" the numbers to make Microsoft products look better.

So I rant. I use Linux as my primary desktop platform, and so does many of my extended family, so anyone who says it can't be done we're living proof. Also for those who say that the "average user" can't use Linux because it's too complicated, has never used a LiveCD/DVD before. Pop it in, click on the red fox, and surf the net. Besides my Mom uses Linux and she barely knows where the on switch is.

As for "but you support those people", ya but I had to support those people using windows too, and I'm supporting the same issues, so I'd prefer to support them on Linux. Not to mention what about the "average user" who buy's Windows. Don't they get support from Futureshop, plus Microsoft? Fedora users can easily get support from the multiple forms, live chat feeds, online classrooms, help pages, and on and on and on. The only different between Windows and Linux support networks is ... well none. You can even get paid support for Linux just like Microsoft.

So why do I rant... to correct some of the people out there who want to show Windows as a better product by spreading, what I consider to be lies about Linux.

Linux can run your desktop...

Why I prefer Linux - Reasons #5 - Features

Both Linux and Windows are rich with features. But when you get used to one over the other it's amazing how the little things start to annoy you.

Yesterday the motherboard on one of my servers decided to go belly up. As a quick fix I dismantled both and swapped them. As a result I had no computer to work on as my work virtual machine was sitting in the server. But, I still needed to get some work done. I ended up using Window XP on a laptop as it was the only computer setup for work.

The first thing I missed was the Virtual Desktop. In Linux we can have multiple virtual desktops. I used them to group tasks. One is for mail, one for web, one for virtual machines, one for terminal windows.... etc. Any program not in that virtual window doesn't show on the window list bar. It can really reduce clutter, especially with the group like windows feature. Also if your missing a program you can just middle-click on the desktop to bring up a window of what's running.

Now I'm sure there is a way to hack or add software to get this effect in Windows. But, I'm almost never using my own windows computer. It's always a borrowed, or temporary machine. So customizing it with all these hacks is not always the best option. I prefer Linux as this is all out of the box.

The next feature I miss is the gnome-terminal (bash). Yes Windows has powershell, but powershell doesn't have all the features I used every day in Linux. I'm an old hat when it comes to Linux/Unix systems. I prefer the text windows over a gui screen for most tasks. So I prefer Linux for is powerfull out of the box command line. Not to mention everything in Linux can be done from the command line, and with out the GUI.

Now there's cygwin which basically installs Linux bash on top of Windows. (but I always ask myself, if I have to install Linux/Unix on top of Windows for my regular tasks... why not just use Linux) Same can be said the other way except I find myself always gravitating to using Linux for all those tasks. Not to mention cygwin has a lot of quarks on windows that slow me down. No tabbed windows for starters.

The third annoyance is the text editor. Out of the box we have gedit. It's a powerful text editor with spell checking, colour syntax highlighting, regexpr find and replace, and many more. With Windows we have notepad. Since I write a lot of scripts, SQL for starters, I jump into the text editor regularly. When on windows for a day I was disparately looking for my old copy of UltraEdit for Windows. The notepad was slowing me down considerably. Again I'm talking out of the box here. I'm sure there's new, maybe even better software out there. But if I'm on a foreign computer installing all this software for temporary use is a pain, if they will even let me.

These are just 3 of the frustrations I find when going back to windows. Most is all preference and a lot of windows users would probably not even see my problem. But this is my ranting... and why I prefer Linux. If you prefer Windows, write your own blog.

Why I prefer Linux - Reasons #4 - Easy to Use

Ease of use is a raging debate, but it really boils down to one thing. Software! So which do you find easier to use of these two screen shots? Linux or Windows? For those who have found my point too obscure (yes I know I'm in my own little world most of the time) both Windows and Linux can run Firefox. Both Linux and Windows can run flash and youtube. Both Linux and Windows can run many pieces of software, you can even get Microsoft Office and Photoshop running under a product called wine. Companies like CodeWeavers and Cedega even create nice gui screens making the installing of Windows programs on Linux simple.

Personally I think the easy of use for distributions of Fedora and Ubuntu released in 2010/09 are just as easy to use as Windows 7. I have my Parents using Linux and I've had far less support calls from them with Linux than I ever had on Windows. (Granted I moved them off Windows from 2000/XP)

Most of the arguments presented to me against Linux is a comparison of Windows 7 pre-installed on a computer by 7 computer techs, to a download-able CD/DVD off the web. Sure for the most part new Linux users are on there own. But Linux has come along way from needing to download those 17 floppies and trying to figure out how to create images. These days we have LiveDVD/USB. You just burn it to a disk, put it in a computer and turn it on. Now your in Linux using firefox in all it's glory. Most people don't even use the OS these days. Which probably explains where Gnome Shell came from. Video: #1, #2, #3, #4. So for all those people saying Windows is easier to use than Linux I have to ask, did you install it yourself? Have you used a new version of Linux like Fedora or Ubuntu? What have you tried to do with your computer?

Now this is just the using part... we haven't even got to the installing of software or installing a new PC...

Why I prefer Linux - Reasons #3 - No Restrictions

I'm an uncommon user. I have a Windows Machine connected to my TV running iTunes. (only for running iTunes I might add) If I need to use the machine I access it remotely via RDP from my Linux Desktop. However, when I do, my wife and daughter are locked out of the Windows Desktop until I log out. Windows Desktop Licenses only allows one user to be logged into a given machine at one time. (and it's software enforced) If you want more Microsoft wants you to purchase Windows Server with CALs and Terminal Server CALs. Which would be no big deal, as Windows Home Server can do the trick at $400 for a OEM licensed copy, except many providers (like iTunes) don't allow their products to be run on Windows Servers.

With Linux, my main machine, has both Kimberly and I connected at the same time using the same programs. I can have an unlimited number of people connected via the X11 protocol. Well, unlimited as far as licenses. The reality is I couldn't handle much more that 10 active users. System just doesn't have enough RAM for that.

Not saying Linux is better here, just less restrictive. With Windows you do have options, purchase Windows Server, illegally hack Windows Desktop... or you could just use Linux as your primary platform and consolidate your computers. Dumb Terminals are not hard to setup, and Fedora has everything you need included in the box.


To setup for X11 vnc: (there are gui ways but cutting and pasting is faster)

Install:

yum install tigervnc-server

add to /etc/gdm/custom.conf: (gives access to connect to the X11 server)

[xdmcp]
Enable=true
HonorIndirect=true
MaxSessions=32
DisplaysPerHost=2

Create /etc/xinetd.d/VirtualMachines: (sets vnc connections to X11)

service vnc-mini
{
disable = no
socket_type = stream
port = 5901
protocol = tcp
group = tty
wait = no
user = nobody
server = /usr/bin/Xvnc
server_args = -inetd -query localhost -geometry 1024x600 -depth 16 -once -fp /usr/share/X11/fonts/misc -securitytypes=none -desktop=MyMachine
}

Add /etc/services: (I'm using named services in xinetd)

vnc-mini 5901/tcp # VNC to GDM


To Connect:

to use just open vncviewer and log into machinename:1 and you will be presented with your login screen. If you want to get fancy you can use persistent connections and allow your computer to stay on even when you disconnect.

Why I prefer Linux - Reasons #2 - Toy?

Every time I look at XP or Windows 7 I think ahhh shiny. But I just can't help but think of Windows as anything more than a kids toy. It does have power, and can do many tasks but I still can't get past the toy part.

To give an example. When doing the intro video for Derek's Wedding and later the into video for a Michelle's DVD I needed to setup a rendering farm. Rendering a 20 minute video can take days depending on the complexity. Setting it up with a gui in windows just seemed like a waist. How much memory does windows take up? 100M? 512M? Isn't that memory I could use to dedicate to the rendering farm? With Linux I can shut down pretty much everything, the graphics, the printing interface, everything. I managed to get a foot print down to under 50M for the rendering farm, and even better could load the thing off a CD as not to interfer with what was on the system already. That way I could slave several computers together to dedicate the entire system to the rendering process.

So I'm not saying Linux is better here. In fact, I'm a true believer that they both have their strengths and weaknesses that make them equal. But, the more shinny added to Windows the more I'm still going to see it as a toy. But that's me, I'm all about function rather than form.

Why I prefer Linux - Reasons #1 Cost

Recently I've been back and forth on trying to setup a iTunes computer. I'm quickly reminded why I prefer Linux over Windows. What can you do with $340? I could buy two RC Helicopters. I could buy a two new monitors for my desk. I could buy a new bare-bones PC kit with one of the latest CPUs in it. I could take the family on a weekend vacation. So what could you buy?

Windows Professional is listed in our local computer store for $340. Where Fedora 12 is free, and only costs the blank DVD (if your not using PXE booting like I am) Sure I can get that discounted, or steal it, or get a copy from e-bay... but the point is Linux is free. No hassle working about legal copies.

All arguing about which is better aside you can't beat the initial price.