Patrick’s Rants

I haven’t read Bob Cringely as much as I used to, but it’s nice to see that he somewhat agrees with me on social notworking He suggests downsizing your friends list and I can’t disagree with him. At least on here, there aren’t many commenters regardless of the number of readers so everything here is important. And everything on Bob’s blog is important to Bob and sometimes, such as here, important to our readers as well.

Another hard drive bites the dust. It looks like it’s been hanging around a while… the last drive that it looks like I replaced in my trusty dusty firewall was in 2005… can that be? Hmm. It’s possible I guess. It’s the main drive for my firewall. No Operating System, no internet access. No nothing. I pull out my trusty box of old hard drives and look for a promising candidate. A 2g drive!

Of course I skipped a step or two. First thing I did was pull the firewall off the shelf that it sits on above my head. When I pulled the cover off I noticed a note written in permanent marker warning me that this CD drive has intermittent read errors. Aha… that’s the problem. I hadn’t been able to get the firewall completely up to date with the latest version of NetBSD because the CD drive was crapped out. Now it’s time to pull the old drive out and put it in the “Quality Connections pile”.

The same shelf as the firewall holds a stack of CD drives. I try them one by one to find one that doesn’t keep ejecting (BIOS issue?) all on its own. Then to the aforementioned box of hard drives. While I’m browsing the aisles, I look for faster NICs – that whole cable internet thing chewing at the back of my mind. And yes, like any true geek, I have aisles of old computer crap. Or at least shelves. I pulled a couple of D-Link 10/100 NICs from the shelf. They are marked DFE-530TX and DFE-530TX+. I vaguely recall recovering them from (Windows) computers they failed in. Like the pack rat that I am, I kept them. I pulled the 10Base-T cards, 3 Com Etherlink III NICs from the firewall and replaced them with the 10/100 D-Links.

A quick install of NetBSD 5.0.2 and I was ready to start figuring out all of my now gone configurations. It has taken a day of futzing around just to get the home network back online, but I’m there. I now have to remember what all of my customizations were and how to recreate them. Maybe this time I’ll make a backup copy of the configuration files, or not. I only seem to have to perform the Lazarus trick every five years or so.

My interim boss just sent me 3 emails with attached documents. There was no message body. Just a single attachment per email. Then she sent me an instant message to tell me that she sent the emails. I won’t go into the content of the attached documents as I didn’t spend much time looking at them to compare them to existing documents. It is the rest of the comment that really has me scratching my head. She asked me to create a “web presence” that will indicate when a bus is running over 15 minutes late. I’m guessing, since she did not respond to my “WTF?” reply, that she told our incoming boss that this was easy to do. “Web presence”? That doesn’t sound like the interim boss. The new guy is supposed to be tech savvy. That sounds like a PHB or someone who really knows what it takes.

Even if I could find a widget that would do what I she wants, the underlying technology isn’t there. The buses would have to be equipped with some type of GPS unit. The unit would have to report and compare times to the routing software and determine when the routes are overdue. Unless of course, we just log into the web site every time a bus is running late and update the page. Oh wait, we run bussing from 4:00am to 7:00pm and I’m the only one with a login to that web page. I’m guessing that installing and configuring GPS units and then installing the tracking software – complete with connector software to talk to the routing software – will run $300.00 – $500.00 per bus. That would be roughly $25,000.00 to $50,000.00. I’m thinking that whoever thought up this idea did not take into consideration the costs to do this automatically nor did they take into consideration that I’m currently the only person who can update the Transportation website nor the fact that I work 11:00am to 7:00pm and I’ll be mufuggin damned if I’m getting up at 4:00am so little Johnny’s mom can see if a bus is running 15 minutes late to Johnny’s stop. I don’t usually say this, but they don’t pay me enough for that. They don’t even pay me enough to update it during my shift as, due to budget cuts, my office is the de facto receptionist. I don’t have, nor will I have time to update the damned thing when the phone is ringing off the hook.

The switch is done. We have cable internet, phone and tv. I would have been happy with just the phone and internet, but my wife insisted on tv too since it’s not that much more than just the phone and internet.
Observations so far:
I still see reruns even though the channels I’m seeing shows on reruns I’ve only had for about three weeks.
I had to fight with the cable modem to get onto the internet. I guess having ancient 10baseT NICs in the firewall don’t talk to the modem so well. Adding a four port Linksys 10/100 hub between the modem and the firewall fixed the connection. I might have to think about upgrading the network to gigabit and adding at least 100baseT cards to the firewall. The internet is much faster than it used to be.
I don’t like using the cable provider’s DNS servers. I’ll keep using my own, thanks.

I broke down and bought a Windows 7 laptop. I have some software that just won’t run on the Windows 2000 computer so I felt like it was necessary. We purchased a Toshiba satellite, a more or less middle of the line laptop with 4gigs of ram. Twice now the laptop would not let us log in because it was “Low on Resources” and would not allow a login for a “new” user. The first time I logged into my account which was still running. Just now, no one was logged in and only background programs were running. What the heck? I had to restart the laptop. I’m just curious what Windows 7 does, with the laptop cover closed that uses up all of the resources.

I’m working on getting some reports done for the bus barn. My boss asked me for a pretty complex set of numbers. Numbers that you just don’t get from a spreadsheet. Don’t get me wrong, spreadsheets are and will be involved, they just aren’t complex enough to get what I want without learning some serious Visual Basic (yech!). I guess I’m just more familiar with running a quick and dirty query against the data. But that’s just me. It took a couple of days (and working in Dispatch, that’s just the nature of the beast) to get multiple spreadsheets dragged into Access (because it’s there) and run the query that I needed. But I did learn that I can use OpenOffice.org Base just as easily.

What I ended up doing was taking the spreadsheet data, exporting that to csv files and then importing into Access. Access is pretty easy to do, inside and Access file (with Access open) click on new table, import and browse to the csv file you want. You can modify the field types and then import the file. Do that with each table you want. Access works fairly quickly importing the file.

In OpenOffice.org Base, first you have to create a new database. Once your database is created you have to open the csv file. The csv file will open in a spreadsheet (OpenOffice.org Calc). Select the data you want, (Ctrl-A for all, Ctrl-C to copy to clipboard) and then on the table section of Base, click Ctrl-V or Edit -> Paste on the menu. Depending upon the size of the table this might take a little while. Repeat for each csv file you want as a table. And now my joins work! (They don’t work when you use the csv files as your database. You must use the built in hsqldb engine to get real database behavior)

Looking back over the previous entry I realize that I made the install seem just slightly easier than it was. When I wrote,

format the scsi drive, copy over the diskboot.img from my Debian workstation to the scsi drive, then dd the image onto the ide drive.

I forgot to mention that I had already (sorta) done this once. I copied the diskboot.img to the scsi drive and ran the install. The problem with installing to the same drive that you are using the disk image on is – it doesn’t really work. Oh, it pretends to work, mounting the image as a loop back file system, but it doesn’t completely and correctly install. At least not with CentOS 5, which is my baseline. This is why I ended up with the IDE drive in the machine too. It’s also why I installed the OS twice in one night. But enough about that time.

A few days after this server was installed as a crutch, I got another phone call. No network booting. Which is where I started. I went through a couple of things that might be wrong. The ethernet cable was in a location that it could get bumped so I had them wiggle the cable. It worked. Until I got the next phone call. I resigned myself to going into the office to work on this machine again.

I had to grab a chair, connect up a keyboard, mouse and monitor and I sat down in front of the beast. The screen did not come alive. Several boot cycles later I decided the old 18g scsi drive must have given up the ghost. It didn’t work in any of the hard drive slots and Knoppix would not see it when I booted up that way. So following my 2 hard drive install I came home, picked up 2 IDE drives from the shelf (no spare Ultra 320 drives here) and drove back to the office. I cracked open the box and stuck in the 2 drives. It was the same dance as before, boot Knoppix, copy the netboot image, boot from the netboot image and run the install from the crippled office server. I was able to keep the failing server running long enough to get my install done. I rebooted and everything looked beautiful.

Walking to the workstations I realized I wasn’t done. The screens showed a gray hash-marked background with an X cursor. No logon prompt. I spent until midnight or later that night trying to edit this config file and that config file. Nothing worked. And the thing that was bugging me is that I was using the same (copied from the old server) config files, that until the scsi drive died, worked. I finally stumbled upon the fix, you have to go into the gui login on the server:

Now goto System -> Administration -> Login Window
Now click on “Remote”
On the drop down menu of styles select “Same as Local.

This is the first time I have had to set this since I started using K12LTSP in 2002. I’m not sure why this install – done mere days after the last one – required this change, but it did. Even worse, the fix was not at the top of my searches or I might have tried it first.

I also started running into trouble with backups running from the Windows 2008 Server to the Linux server. It turns out that using Cygwin rsync over ssh has some potential problems. The first is that rsync hangs. And my little bash script wasn’t set to only run one copy at a time (by using a lock file) so rsync was running multiple times and hogging all the CPU and RAM. The final solution was to run Rsync outside of SSH and use lock files.

In the end this crutch held me over until the arrival of the new T300 Poweredge (next in the saga)

On January 5, I was having breakfast with J at Mike and Rhonda’s when my cell phone interrupted my casual coffee enjoyment. It was the office. The workstations were all off line. To me it sounded like a network issue. I stepped outside to finish my call, people talking on their cell phones in restaurants is one of my pet peeves (closely followed by people on their cell phone in any public place). To me it sounded like a network issue. Step by step each suggestion failed in turn. I resigned myself to cutting breakfast short and heading into the office before the paying job.

Once at the office I went through the steps that I was assured had already been taken. Power cycle the network switches, the server and then workstations. Nothing. Each phase that should have – could have – worked didn’t. I tailed the logs, watched as the workstations/clients booted up and nothing made sense. Then the screen went dark and the BIOS screen appeared. The server had just spontaneously rebooted; never a good sign. Nothing seemed to work and the office was shut down for the day. I went to my full time job for the remainder of the day.

After getting out of work at 7:00pm, I headed back to the tax office and again to read through the logs to see if there was anything that I missed. Workstations still would not PXE boot. The server spontaneously rebooted on me a couple more times and I resigned myself to the fact that the server at somewhere around 7 years old had reached the end of its life. The hard drives were reporting (using SMART) that they were aging, occasionally showing sectors not available. I knew at some point the server would need to be migrated, but I wasn’t ready. I really did hope to get another tax season out of that machine – it was not to be.

On January 6, I unstacked the stash in the corner. Imagine a tower of tower computers placed next to the wall width-wise and two lengths left to right. Imagine that tower at two to three high. Yep the bane of every geek’s non-geek wife (or non-geek husband as the case ma be). The overt hoarding of old computers just waiting for the day when they can be salvaged and recombined into a working machine. These machines are only awaiting the day when their geek overlord, master of their existence, has the chance to evaluate and resurrect them. I unpiled that stack looking for a gem that I knew was there – the beige beast.

The beige beast is pretty impressive. It houses the Intel® Server Board SE7501BR2, has dual hot swap power supplies, 5 hot swap fans with internal wind tunnels (firing this puppy up gave me wind chill) 5 scsi hard drives (well, 5 possible. Only one actually was installed a comparatively small 18g Ultra 320 drive), intrusion detection, dual Xeon 2.4ghz chips. Now this was a hand me down (thanks Steve) so there are no complaints. Some of the hardware is absolutely impressive – dual Xeons in a box that was decommissioned sometime around 2007 and ran Windows 2000 Server. That box cost a pretty penny when it was originally deployed. Today you could grab the board (used) on ebay for less than $20.00. Of course the case is not included at that price.

Wednesday night I began my installation journey. I burned the K12LTSP v5EL dvd. I let the drive select the speed and it warned there might be an underburn . Naturally I stuck it in the drive and booted – what could go wrong? It did not see the disk. It did not boot. That’s what could go wrong. I grabbed another blank dvd and set the drive speed to 5x. No warnings. Excellent. I swapped the dvds in the drive and… same thing. Then it dawned on me. The drive in the machine was CDROM. I pulled a dvd drive that I have on the shelf, powered the server off, and temporarily installed the dvd drive. I powered up the server and it still choked. (more…)

Anyone who has any number of servers that they manage will eventually see failures. It’s just natural, hardware gets old and dies. Or you run completely over the hardware’s ability to keep up with demand.
Both of those things happened this year.

First – and I have to say this, forgive me – Windows Server 2003 has served me well. But the hardware just would not keep up with three to five users who kept three to five programs open each. This is a machine that has dual PIII 1ghz chips and a whopping 1.5gb ram. Three full time users on Windows 2003 Server, even I’m a little impressed. But it was straining under the heavy load. And the load was heavy.

My partner ordered the new Dell to replace the VisionMan gray box server that served us so well for these last several years. The server itself – hardware – is still up and strong. I have had to replace CPU fans on one of our machines, but that happens sometimes.

The migration from Windows 2003 Server (using Terminal Services for Windows based software access) to Windows Server 2008 64bit (also with Terminal Services) has been a little bumpy.¹ I nearly fell out of my chair when the current year tax software installed without any major hitches. Our Windows Server is now online – all major software has been migrated and the Windows 2003 server has been powered off awaiting the day it is wiped clean.

1. Mostly chronicled in Geek News and Stuff

Anyone who runs anything online has had to deal with bots. WordPress has plugins to fight spam of all kinds, bulletin boards and mailing lists have to remain vigilant and firewalls have to be erected on personal computing and company networks. This last week the district closed up shop for two days to get the place cleaned up after a heavy snowfall on Monday. I took the opportunity to do a little reading and to try to fortify my web server. I started out trying to get my Apache logs cleaned up and found perishablepress.com to have a nice beginning. I don’t have everything working the way Jeff writes about over there, but let me tell you what I have been able to do. First, I’m using a simple add on to httpd.conf that looks like this:
RewriteEngine On
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|/|\\\.\.\\).* [NC,OR]
RewriteCond %{REQUEST_URI} ^.*(\=|\@|\[|\]|\^|\`|\{|\}|\~).* [NC,OR]
RewriteCond %{REQUEST_URI} ^.*(\'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} http\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]
RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
RewriteRule ^(.*)$ - [F,L]

RedirectMatch 403 \/\/(.*)


To be honest I think the biggest work horse is the double slash redirecting to 403 “Forbidden”. I’m not even sure that the rewrite stuff is even working (I’m going to spend some time on it in the future to crank up the logging on it to see if it truly is working). Now all the danged bots looking for vulnerabilities on my web server hit those 403s. I set up a simple script to look for 403 and 404 failures. It takes those failures and adds a rule to iptables to drop the host if it happens to hit too many times. I screen out the requests for the favicon.ico file, robots.txt and even other image types and I’m left with small list of hosts that try to pound away at my server. I currently have it configured to look at the current Apache server logs for the 403 and 404 errors. It then (hourly) inserts a simple drop rule for that host into iptables, which then logs additional attempts to connect. And by additional attempts I mean on any port. So if somebody’s Windows computer is compromised and they have a bot that tries to compromise my server, I block any future attempts to connect to my server via email, IRC, etc. And those attempts are logged while my server looks like it dropped off the face of the earth to the infected machine. So I won’t be getting spam from those infected hosts either.

Now if infected (or mis-configured – msn search is hitting a lot of 404s – stupid bot) machines try connecting after getting the firewall treatment they will stay blocked for a month. Otherwise hosts that are cleaned up will only be in the block list for a couple of weeks the way it’s all configured. Then they can be back reading my rants. I also added a twist. I have a script that dumps the addresses of the stupid bots where my home firewall can grab the list nightly and add those IPs to a squidGuard blacklist. So if those hosts happen to be running rogue web servers, at least no one here will try to connect to them.

I’m sure there are much better solutions – and I might add hosts that run ssh connection sweeps to a block list – but I’m having fun thinking about all the ways I can make something like this work and cut down on the break-in attempts on the web server.