Detect broken links on a Web site using wget

The other day, I started thinking about writing a simple Web site validator that detects broken links on a Web site, similar to W3C Link Checker. After looking at various code samples in Java, Ruby, etc, I figured out that GNU wget 1.12 on my Linux machine could do the job just fine, with no programming required. It even detected broken resource links in CSS, not just broken <a> links.

Here is how to write a simple script to check the site. First, pretend to be a Mozilla-based browser and spider the site to the depth of one level:

wget --spider -r -l 1 --header='User-Agent: Mozilla/5.0' \
-o wget_errors.txt http://the_site_i_want_to_validate

Then, simply look at the return code to determine if there is any error. If the code is larger than zero, there is an error.

EXIT_CODE=$?
if [ $EXIT_CODE -gt 0 ]; then
    echo "ERROR: Found broken link(s)"
    exit 1
fi

To find out the actual links in question, just grep for 404 in the wget error log.

BROKEN_LINKS=`grep -B 2 '404' wget_errors.txt`

The -B 2 outputs the 2 lines above any matching line, which in this case contains the broken link in question.

Mandriva 2009 on Toshiba Portege M800

(Screenshot of KDE 4 desktop with Dolphin file manager, photo and desktop widgets)

I've just installed the newly released Mandriva 2009.0 Powerpack on a company supplied Toshiba Portege M800. Almost everything worked out of the box... almost.

The following required some tweaking:

Video Playback
I did not like DragonPlayer that was configured by the installer. so the first thing I did was install MPlayer (my favourite video player that plays everything) from PLF, together with all the popular codecs. Initially, my video did not play. After reading the Mandriva Errata page, it turned out that because I had enabled desktop effects using Compiz, I needed to also enable the Compiz Video Playback plugin, using ccsm. Once that was done, MPlayer worked fine. However, other video players like Kaffeine and Totem did not seem to work, but I didn't care much for those.

Wifi
After trawling the Internet, I realized that I needed to reinstall the iwlwifi-4965-ucode package, because the installer had removed some firmware files required by the built-in Intel 4965AGN wireless adaptor. After that, wifi seemed to connect fine to my home wifi network that used WPA authentication.

MandrivaUpdate over proxy
At work, MandrivaUpdate must be configured to go through an authenticated proxy before it could go out to grab updates from the Internet. Unfortunately, there seemed to be an inconsistency between how the proxy login and password were specified in the urpmi configuration and Network Settings. As a result, it was not possible to perform an update via MandrivaUpdate. The workaround was just to do updates from the command line, using:

urpmi --proxy server:port --proxy-user login:password ...

MandrivaUpdate worked fine at home where it did not need to go through a proxy.



Things I did not quite like include:

Auto-partitioning
The Portege M800 came with a 130GB harddisk. I chose to let the installer manage the partitioning this time and told it to copy the contents of the installation DVD onto disk. I had expected it to give several tens of GB for the "/" partition, a separate partition just to hold the DVD contents, and everything else for "/home". I was disappointed to see that the installer allocated a mere 8GB for "/" that also contained the DVD contents (~4GB). That left only 4GB for the entire OS plus any other programs I wanted to install. No where near enough because I needed some large applications like an Oracle database. To fix this, I moved the DVD contents to a new directory under "/home", and made a symlink to it from the original location. That freed up 4GB that should hopefully be enough until my next upgrade to Mandriva 2009.1. Next time, I'll choose custom partitioning.

Missing applications

Missing altogether were KDE 3.x applications that had not been ported over to KDE 4.x, notably K3B. Hopefully, this situation is remedied in the near future.


Now, the good bits...

Overall, I was impressed with this release. KDE 4 looked a lot better than KDE 3 in 2008.1. The desktop widgets were a pleasure to use, desktop effects were nicely integrated, performance was acceptable, and there was an abundance of applications from the PLF and contribs repositories. The"Start Application" menu had a new tabbed-style that I found more user-friendly on a laptop. Even the SD card reader worked properly.

---