Detect broken links on a Web site using wget

The other day, I started thinking about writing a simple Web site validator that detects broken links on a Web site, similar to W3C Link Checker. After looking at various code samples in Java, Ruby, etc, I figured out that GNU wget 1.12 on my Linux machine could do the job just fine, with no programming required. It even detected broken resource links in CSS, not just broken <a> links.

Here is how to write a simple script to check the site. First, pretend to be a Mozilla-based browser and spider the site to the depth of one level:

wget --spider -r -l 1 --header='User-Agent: Mozilla/5.0' \
-o wget_errors.txt http://the_site_i_want_to_validate

Then, simply look at the return code to determine if there is any error. If the code is larger than zero, there is an error.

EXIT_CODE=$?
if [ $EXIT_CODE -gt 0 ]; then
    echo "ERROR: Found broken link(s)"
    exit 1
fi

To find out the actual links in question, just grep for 404 in the wget error log.

BROKEN_LINKS=`grep -B 2 '404' wget_errors.txt`

The -B 2 outputs the 2 lines above any matching line, which in this case contains the broken link in question.