Patrick’s Rants

I’m working on getting some reports done for the bus barn. My boss asked me for a pretty complex set of numbers. Numbers that you just don’t get from a spreadsheet. Don’t get me wrong, spreadsheets are and will be involved, they just aren’t complex enough to get what I want without learning some serious Visual Basic (yech!). I guess I’m just more familiar with running a quick and dirty query against the data. But that’s just me. It took a couple of days (and working in Dispatch, that’s just the nature of the beast) to get multiple spreadsheets dragged into Access (because it’s there) and run the query that I needed. But I did learn that I can use OpenOffice.org Base just as easily.

What I ended up doing was taking the spreadsheet data, exporting that to csv files and then importing into Access. Access is pretty easy to do, inside and Access file (with Access open) click on new table, import and browse to the csv file you want. You can modify the field types and then import the file. Do that with each table you want. Access works fairly quickly importing the file.

In OpenOffice.org Base, first you have to create a new database. Once your database is created you have to open the csv file. The csv file will open in a spreadsheet (OpenOffice.org Calc). Select the data you want, (Ctrl-A for all, Ctrl-C to copy to clipboard) and then on the table section of Base, click Ctrl-V or Edit -> Paste on the menu. Depending upon the size of the table this might take a little while. Repeat for each csv file you want as a table. And now my joins work! (They don’t work when you use the csv files as your database. You must use the built in hsqldb engine to get real database behavior)

Looking back over the previous entry I realize that I made the install seem just slightly easier than it was. When I wrote,

format the scsi drive, copy over the diskboot.img from my Debian workstation to the scsi drive, then dd the image onto the ide drive.

I forgot to mention that I had already (sorta) done this once. I copied the diskboot.img to the scsi drive and ran the install. The problem with installing to the same drive that you are using the disk image on is – it doesn’t really work. Oh, it pretends to work, mounting the image as a loop back file system, but it doesn’t completely and correctly install. At least not with CentOS 5, which is my baseline. This is why I ended up with the IDE drive in the machine too. It’s also why I installed the OS twice in one night. But enough about that time.

A few days after this server was installed as a crutch, I got another phone call. No network booting. Which is where I started. I went through a couple of things that might be wrong. The ethernet cable was in a location that it could get bumped so I had them wiggle the cable. It worked. Until I got the next phone call. I resigned myself to going into the office to work on this machine again.

I had to grab a chair, connect up a keyboard, mouse and monitor and I sat down in front of the beast. The screen did not come alive. Several boot cycles later I decided the old 18g scsi drive must have given up the ghost. It didn’t work in any of the hard drive slots and Knoppix would not see it when I booted up that way. So following my 2 hard drive install I came home, picked up 2 IDE drives from the shelf (no spare Ultra 320 drives here) and drove back to the office. I cracked open the box and stuck in the 2 drives. It was the same dance as before, boot Knoppix, copy the netboot image, boot from the netboot image and run the install from the crippled office server. I was able to keep the failing server running long enough to get my install done. I rebooted and everything looked beautiful.

Walking to the workstations I realized I wasn’t done. The screens showed a gray hash-marked background with an X cursor. No logon prompt. I spent until midnight or later that night trying to edit this config file and that config file. Nothing worked. And the thing that was bugging me is that I was using the same (copied from the old server) config files, that until the scsi drive died, worked. I finally stumbled upon the fix, you have to go into the gui login on the server:

Now goto System -> Administration -> Login Window
Now click on “Remote”
On the drop down menu of styles select “Same as Local.

This is the first time I have had to set this since I started using K12LTSP in 2002. I’m not sure why this install – done mere days after the last one – required this change, but it did. Even worse, the fix was not at the top of my searches or I might have tried it first.

I also started running into trouble with backups running from the Windows 2008 Server to the Linux server. It turns out that using Cygwin rsync over ssh has some potential problems. The first is that rsync hangs. And my little bash script wasn’t set to only run one copy at a time (by using a lock file) so rsync was running multiple times and hogging all the CPU and RAM. The final solution was to run Rsync outside of SSH and use lock files.

In the end this crutch held me over until the arrival of the new T300 Poweredge (next in the saga)

On January 5, I was having breakfast with J at Mike and Rhonda’s when my cell phone interrupted my casual coffee enjoyment. It was the office. The workstations were all off line. To me it sounded like a network issue. I stepped outside to finish my call, people talking on their cell phones in restaurants is one of my pet peeves (closely followed by people on their cell phone in any public place). To me it sounded like a network issue. Step by step each suggestion failed in turn. I resigned myself to cutting breakfast short and heading into the office before the paying job.

Once at the office I went through the steps that I was assured had already been taken. Power cycle the network switches, the server and then workstations. Nothing. Each phase that should have – could have – worked didn’t. I tailed the logs, watched as the workstations/clients booted up and nothing made sense. Then the screen went dark and the BIOS screen appeared. The server had just spontaneously rebooted; never a good sign. Nothing seemed to work and the office was shut down for the day. I went to my full time job for the remainder of the day.

After getting out of work at 7:00pm, I headed back to the tax office and again to read through the logs to see if there was anything that I missed. Workstations still would not PXE boot. The server spontaneously rebooted on me a couple more times and I resigned myself to the fact that the server at somewhere around 7 years old had reached the end of its life. The hard drives were reporting (using SMART) that they were aging, occasionally showing sectors not available. I knew at some point the server would need to be migrated, but I wasn’t ready. I really did hope to get another tax season out of that machine – it was not to be.

On January 6, I unstacked the stash in the corner. Imagine a tower of tower computers placed next to the wall width-wise and two lengths left to right. Imagine that tower at two to three high. Yep the bane of every geek’s non-geek wife (or non-geek husband as the case ma be). The overt hoarding of old computers just waiting for the day when they can be salvaged and recombined into a working machine. These machines are only awaiting the day when their geek overlord, master of their existence, has the chance to evaluate and resurrect them. I unpiled that stack looking for a gem that I knew was there – the beige beast.

The beige beast is pretty impressive. It houses the Intel® Server Board SE7501BR2, has dual hot swap power supplies, 5 hot swap fans with internal wind tunnels (firing this puppy up gave me wind chill) 5 scsi hard drives (well, 5 possible. Only one actually was installed a comparatively small 18g Ultra 320 drive), intrusion detection, dual Xeon 2.4ghz chips. Now this was a hand me down (thanks Steve) so there are no complaints. Some of the hardware is absolutely impressive – dual Xeons in a box that was decommissioned sometime around 2007 and ran Windows 2000 Server. That box cost a pretty penny when it was originally deployed. Today you could grab the board (used) on ebay for less than $20.00. Of course the case is not included at that price.

Wednesday night I began my installation journey. I burned the K12LTSP v5EL dvd. I let the drive select the speed and it warned there might be an underburn . Naturally I stuck it in the drive and booted – what could go wrong? It did not see the disk. It did not boot. That’s what could go wrong. I grabbed another blank dvd and set the drive speed to 5x. No warnings. Excellent. I swapped the dvds in the drive and… same thing. Then it dawned on me. The drive in the machine was CDROM. I pulled a dvd drive that I have on the shelf, powered the server off, and temporarily installed the dvd drive. I powered up the server and it still choked. (more…)

Anyone who has any number of servers that they manage will eventually see failures. It’s just natural, hardware gets old and dies. Or you run completely over the hardware’s ability to keep up with demand.
Both of those things happened this year.

First – and I have to say this, forgive me – Windows Server 2003 has served me well. But the hardware just would not keep up with three to five users who kept three to five programs open each. This is a machine that has dual PIII 1ghz chips and a whopping 1.5gb ram. Three full time users on Windows 2003 Server, even I’m a little impressed. But it was straining under the heavy load. And the load was heavy.

My partner ordered the new Dell to replace the VisionMan gray box server that served us so well for these last several years. The server itself – hardware – is still up and strong. I have had to replace CPU fans on one of our machines, but that happens sometimes.

The migration from Windows 2003 Server (using Terminal Services for Windows based software access) to Windows Server 2008 64bit (also with Terminal Services) has been a little bumpy.¹ I nearly fell out of my chair when the current year tax software installed without any major hitches. Our Windows Server is now online – all major software has been migrated and the Windows 2003 server has been powered off awaiting the day it is wiped clean.

1. Mostly chronicled in Geek News and Stuff

Anyone who runs anything online has had to deal with bots. Wordpress has plugins to fight spam of all kinds, bulletin boards and mailing lists have to remain vigilant and firewalls have to be erected on personal computing and company networks. This last week the district closed up shop for two days to get the place cleaned up after a heavy snowfall on Monday. I took the opportunity to do a little reading and to try to fortify my web server. I started out trying to get my Apache logs cleaned up and found perishablepress.com to have a nice beginning. I don’t have everything working the way Jeff writes about over there, but let me tell you what I have been able to do. First, I’m using a simple add on to httpd.conf that looks like this:
RewriteEngine On
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|/|\\\.\.\\).* [NC,OR]
RewriteCond %{REQUEST_URI} ^.*(\=|\@|\[|\]|\^|\`|\{|\}|\~).* [NC,OR]
RewriteCond %{REQUEST_URI} ^.*(\'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} http\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]
RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
RewriteRule ^(.*)$ - [F,L]

RedirectMatch 403 \/\/(.*)


To be honest I think the biggest work horse is the double slash redirecting to 403 “Forbidden”. I’m not even sure that the rewrite stuff is even working (I’m going to spend some time on it in the future to crank up the logging on it to see if it truly is working). Now all the danged bots looking for vulnerabilities on my web server hit those 403s. I set up a simple script to look for 403 and 404 failures. It takes those failures and adds a rule to iptables to drop the host if it happens to hit too many times. I screen out the requests for the favicon.ico file, robots.txt and even other image types and I’m left with small list of hosts that try to pound away at my server. I currently have it configured to look at the current Apache server logs for the 403 and 404 errors. It then (hourly) inserts a simple drop rule for that host into iptables, which then logs additional attempts to connect. And by additional attempts I mean on any port. So if somebody’s Windows computer is compromised and they have a bot that tries to compromise my server, I block any future attempts to connect to my server via email, IRC, etc. And those attempts are logged while my server looks like it dropped off the face of the earth to the infected machine. So I won’t be getting spam from those infected hosts either.

Now if infected (or mis-configured – msn search is hitting a lot of 404s – stupid bot) machines try connecting after getting the firewall treatment they will stay blocked for a month. Otherwise hosts that are cleaned up will only be in the block list for a couple of weeks the way it’s all configured. Then they can be back reading my rants. I also added a twist. I have a script that dumps the addresses of the stupid bots where my home firewall can grab the list nightly and add those IPs to a squidGuard blacklist. So if those hosts happen to be running rogue web servers, at least no one here will try to connect to them.

I’m sure there are much better solutions – and I might add hosts that run ssh connection sweeps to a block list – but I’m having fun thinking about all the ways I can make something like this work and cut down on the break-in attempts on the web server.

In an article titled Windows 7 Vodka and the Microsoft Hangover, John C Dvorak takes Microsoft’s marketing to task. He laments the “personal touch” of years gone by. He blasts the Microsoft newsletter but touches little on the Operating System that could. I’m waiting for the reviewer whose panties are neither in a bunch nor soaked with excitement to give a true review of Windows 7.

Anyone? Bueller?

The Windows 2008 Server suffered an unexpected shutdown a couple of days ago and I blame UAC (Ok, so UAC and the tax software vendor). As I noted in UAC What is it good for? with UAC enabled I had to set all users as Administrators or hand out an administrator password. Prompting for a password every time a program is opened was not going to work so I set the users as Administrators, trying to keep the UAC security intact. And so, when the remote user chose “Shut Down” instead of “Log Off” the office screens went dark.

So. I changed the user to a regular user account and disabled UAC. Because there isn’t an option to save the administrator account for “this program only”. And despite what Lasse Petterrson says over on technet

UAC is a security feature that should be turned on.

Turning on UAC breaks too much other stuff.
You moved your mouse. <Allow> <Deny>?

It’s old news to most geeks, but the London Stock Exchange is dumping their Windows-based trading system for a Linux-based trading system.
Apparently the Windows solution just couldn’t keep up.

LSE to dump Windows
Open source makes big gains at the LSE
LSE buys MilleniumIT so all of the exchange software is now created “in house” in Sri Lanka.

I’m in the process of migrating the tax office server from Windows Server 2003 to Windows Server 2008 which of course caused me to start thinking about everything else that I eventually want to add or migrate. There’s the firewall – which will probably get a newer machine that most recently was a thin client. The firewall machine will probably be re-purposed as storage. The Windows 2003 server box will be decommissioned as a brick (meaning it will site quietly unplugged in the back room for at least a year to make sure we don’t lose anything in the transfer). Our K12LTSP server will continue to chug along as a K12LTSP server. Although it is the absolute base of our setup – and our oldest “new purchase” computer we lean much, much heavier on the Windows servers and the Windows 2003 server was actually creaking under the load we had on it at any given point in time. At some point in time I also want to add an Asterisk server, but I’ve wanted that for some time now.

Given all of these upcoming changes I was reading up on FreeNAS, something that I have read about before, but not had the chance to deploy. I don’t rush into deployments and try to hash things out and around with people that I know. One of the drivers had just finished up for the day and I casually asked him if he knew anything about computers (you never know what you’re gonna get). He told me that he could check his email and browse the internet – which is what a huge number of people can do – but that was about it. Another driver who had been standing outside my view jumped in asking what I needed; he might be able to help. He had set up his network at home and knew a little bit. I replied that I was thinking about deploying FreeNAS, based on BSD and was just nosing around for experiences. I could visibly see his eyes glaze over as what I just said was more than he had even heard of. At that point we both knew the conversation was over. Not that he’s not a nice guy who really thought he might be able to help with something that a mere radio dispatcher might not be familiar with, but he just stepped into the deep end of the pool. And he forgot his floaties.

Did you hear that? It’s the whoosh of the conversation passing overhead.

I just got my 1400+ page copy of Windows Server 2008 Inside Out. Even this book can’t help me. Windows Vista, Server 2008 and the upcoming Tupperware version, Windows 7 all have UAC. Now, UAC is supposed to prevent programs from just running and installing malware or tweak the registry and is supposed make your computer safer.

The truth is that it normally just annoys the users.

At the tax office I am in the process of migrating from a Windows 2003 server to Windows 2008 (with 64bit) server. I have had a few stumbling blocks, but yesterday I managed to finalize installation of seven years worth of tax software. Out of all of the years the 2006 and 2007 versions have the stupid “protection shield” on the desktop icon meaning the user has to enter and administrator password every single time they run the program. What exactly is the point of having regular user accounts if they have to know an administrator password to use the software installed on their computer?

There are three options for UAC.

1. I can turn it off completely – defeating the purpose of UAC.
2. I can give the regular user the Administrator password – defeating the purpose of Administrator accounts vs. user accounts.
3. I can set all users as Administrators – defeating the purpose of Administrator accounts vs. user accounts.

There is no option to save the Administrator password for a specific program so that next time the user uses the program it won’t prompt for the password. There is an option to Run as Administrator and to set that for all users… but it’s not checked so why is it prompting like it is checked? There is also a program that I found that will allow me to set UAC for individual programs… but from what I’ve read it’s bloated and for one feature it’s not something that I want to install.

So all of the users are now in the Administrators group because MS didn’t provide a way to override UAC on a per program basis.