I have two Ubuntu servers running at home which both have large RAID volumes on them, set up via mdadm. This summer I had a total disk failure in one of my RAID5s which luckily didn’t result in data loss. Thanks you RAID5!

In any case, it caused me to write a script that logs SMART-data to a MySQL database. I also wrote an admin webpage that displays this data for me in an easy to follow way. The monitoring script itself is written in php5 and so are the admin pages. I used php5 because it is easy to communicate with MySQL from it, and it has the needed string manipulation commands. It could probably be done as easily in Python though. The script is called as a cron job every 2 hours on the servers and every hour on the desktops when they’re running. Examples of the code I’m using is attached below and includes the cron-ed script and the code generating the log output and plot.

What to look for?

Well, that is the big question. How do you know if a drive is about to fail? Google Labs has looked into this topic back in 2007 in this paper: «Failure Trends in a Large Disk Drive Population». An interesting read if you are at all concerned about harddrive failure in servers. The results for how temperature affects the lifetime and failure rate in harddrives are especially interesting. It turns out, at least in their data, that low temperature isn’t such a good thing for the drives contrary to what many people seem to assume. I have up until now been concerned that my drives get too hot, but in fact they seem to be almost overcooled the way I have things set up now.

When I wrote these scripts this summer I decided to log temperature and reallocated sector count primarily, which is what is emphasized in the log display scripts. Seems now I also should be including scan errors as well after reading that paper. The colour coding I use in the temperature plot below is loosely based on Figure 4 in the paper and reflects what seems to be the optimal operating temperature for harddrives.

Screenshots

Screenshot 1: The overview page.
Screenshot 2: Details of one of the RAID-drives.
Screenshot 3: Details of one of the drives with reallocated sectors.

Code

The php source code I wrote is available in this file: hd-mon.tar.gz

It is specifically designed to work with my setup and hardware and probably isn’t universal, but it gives an idea of how I set it up. There are probably better ways of doing this though. I just call shell commands from php and parse the returned text-string and do simple search on it and input the data into a MySQL database.

I also included php-snippets showing how the admin page is generated. These are not standalone php files, they need to be wrapped in a template. However they reproduce what is seen in the screenshots above.

Packages needed for these scripts to run:

• php5-cli for the php5 command line.
• mysql-client, php5-mysql for the database connection.
• smartctl to access the SMART-data.
• mdadm to access RAID-data (assuming you use mdadm for RAIDs in the first place)

All are available in the Ubuntu repository.

Just a quick MySQL tip …

I have a RSS reader that I coded myself a while back with php/MySQL. The news table was set to latin1 character encoding and I wanted to change that into utf8, but couldn’t quite figure out how. Eventually I stumbled upon this solution:

9.1.11.2. Converting 4.0 Character Columns to 4.1 Format

Basically what I needed to do to convert the field “Title” in the table “news” was running the following SQLs:

ALTER TABLE news MODIFY Title BLOB;
ALTER TABLE news MODIFY Title VARCHAR(250) CHARACTER SET utf8;

Easy peasy …

As I posted a while back, I have bought myself an Amazon Kindle. I am very happy with that little piece of technology. I have already made a template for generating pdf-files that fits well on my Kindle.

Last weekend I decided to try to compile my own mobi files. It is actually fairly simple (although preparing the text itself can be very time-consuming). First you need the text you’d want to convert into a book. I had read this post on pandasthumb.org and downloaded Matt Young’s book «No Sense of Obligation: Science and Religion in an Impersonal Universe». Since the pdf was not very well suited for my Kindle, I decided to try and convert it to mobi-format.

Firstly you need a decent tool. Amazon has provided a tool that prepares books for the Kindle, it can be found on their Kindle Publishing Programs page here. However the conversion tool isn’t all that practical. It’s command-line which is no problem for me, but it also have very few options. A much better program is Calibre, which is multi-platform and is a full eBook management tool that connects and sync with the Kindle and other eBook readers. It can import many formats and also export/convert to many formats.

I found that the easiest way to format an eBook is by making a html-document of it first and the convert the html to mobi. As an example I have made a short eBook out of the essay about Homeopathy I posted about earlier. I will provide all the source files in a bit.

The html file needs to start with a header. Something like:

<?xml Version='1.0' Encoding='utf-8'?>
<html Xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Homeopathy and Its Kindred Delusions</title>
    <meta Http-Equiv="Content-Type" Content="text/html; charset=utf8">
    <link Rel="Stylesheet" Type="text/css" Href="Style.css">
</head>
<body>
</body>
</html>

The text of the book goes inside the body-tag.

Calibre generates a pagebreak for each <h1> and <h2> heading by default, so I use the <h2> tag for chapter headings. You’d also want to put in an anchor tag before each chapter and make a table of contents at the beginning of the book. It is the usual syntax:

<a Name="Chapter1"></a>

for the chapter tag, and

<a Href="#Chapter1">Chapter 1</a>

for the TOC link.

Otherwise the text formatting is straight forward with <i> and <b> tags for italics and bold as usual. One point worth mentioning is paragraph formatting. I myself prefer the indented paragraph as opposed to the extra linebreak style for books. To achieve this I used <div> tags instead of <p> tags in the text, and use the CSS style “Text-Indent: 1.5em” to indent the text at the start of each paragraph. The example files should demonstrate this well enough.

When the html version looks fine (you can preview it in your web-browser), Then you import it to calibre via “Add Book”. Then you do a conversion to “MOBI” (in the right-click dropdown menu). You may play around with the settings, but there were no need to change anything when I converted this book, except I disabled the option to let calibre generate your TOC. Sync with the Kindle, and read away!

Example files:
Oliver Wendell Holmes – Homeopathy and Its Kindred Delusions (1842)

• HTML source files
• Final MOBI file

(The files are gzipped.)

Wrote a script today that parses a torrent file and retrieve the name of the torrent. The reason I made this is that I have a small web page running at home that parses an RSS feed and upload selected torrents to another computer. Since the name of the torrent on disk (the data folder) doesn’t always match the name of the actual .torrent file, I need to get the name from the file itself. It is quite simple really, and it falls back to the .torrent file name if it fails.

// Find torrent name
$sTorrent     = "torrent_file_name.torrent";
$sTName       = trim(substr($sTorrent, 0, strrpos($sTorrent, ".")));
$sTorrentData = file_get_contents("path/to/torrent/file/".$sTorrent);
preg_match("(:name[0-9]+:)",$sTorrentData,$aTemp,PREG_OFFSET_CAPTURE);
if(count($aTemp) == 1) {
  $sTemp = str_replace(":","",$aTemp[0][0]);
  $sTemp = str_replace("name","",$sTemp);
  if(is_numeric($sTemp)) {
    $sTName = substr($sTorrentData,$aTemp[0][1]+strlen($aTemp[0][0]),(int)$sTemp);
    $sMessage = "<div>Torrent successfully parsed.<br>";
    $sMessage = "Name: ".$sTName."</div>";
  } else {
    $sError = "<div>Parseing failed.<br>Using name: ".$sTName."</div>";
  }
} else {
  $sError = "<div>Parseing failed.<br>Using name: ".$sTName."</div>";
}

Ever needed a script that automatically extracts all rar archives in a directory and subdirectories, unrar them into their respective directories and then delete the archive files, but only if the extraction was successful? Well in that case, I wrote this little bash script today:

#!/bin/bash
if [ -n "$1" ]
then
    echo "Unraring from $1"
    find $1 -type f -name '*.part01.rar' -execdir rar x -y -o- {} \;
    find $1 -type f -name '*.part001.rar' -execdir rar x -y -o- {} \;
    find $1 -type f -name '*.rar' -execdir rar x -y -o- {} \;
fi

It takes the path of the folder you want to scan as argument: unrarscript.sh /path/to/folder

Edit Jan 31 2011: It seems the if statements checking if the extraction failed or not only checks if last extraction was successful. For it to work correctly the unlink should be nested inside the finds. I have not yet made any corrections to the script, but be aware of this weakness.

Edit Nov 11 2011: Just removed the unlink part as it wasn’t working properly anyway.

Much of the code I write these days is for numerical methods used in physics. What I often need is some output that tells me how long a program has come in for instance a Monte Carlo integration. I had to look a bit around on the net as a regular cout command doesn’t seem to work within a loop, and I want the progress counter to overwrite the same line as it progresses. Here is the solution:

j=0;
cout << "Progress: 0%";
for(i=0; i<iMax; i++) {
  //Algorithm here..

  if(i%(iMax/100)==0) {
    j++;
    cout << "\rProgress: " << j << "%";
    fflush(stdout);
  }
}
cout << endl;

Key elements here are the “\r” to return the cursor to the beginning of the line (for the overwrite) and “fflush(stdout)” to flush the buffer, something which isn’t done automatically if you do not have an endl.

I got fed up with the comment spam I get on my blog. I googled a bit and found a few solutions, but ended up with a plugin that adds CAPTCHA to the comment form.This little gem can be downloaded from here.

It is fun to observe the spammer-scripts bang their virtual heads against this simple little tool. I can observe all the spam attempts in my website log.

Was writing a formula sheet for my quantum mechanics exam today, and needed to maximize usage of my available space on a piece of A4 paper. So I googled some solutions to get LaTeX to do just that. In addition to using the /tiny command to make small text of course.

Firstly I needed to minimize the margins, so I added this in the header:

\setlength\topmargin{-3cm}
\setlength\textheight{27cm}
\setlength\textwidth{19cm}
\setlength\oddsidemargin{-1.5cm}
\setlength\evensidemargin{-1.5cm}

Then I needed to split the page into 3 columns so I wouldn’t waste too much space at each lineshift:

\usepackage{multicol}

\begin{document}
\tiny
\begin{multicols}{3}
...text...
\end{multicols}

And last but not least, get rid of that annoying huge indent that the align environment has.

\documentclass[fleqn]{article}

\usepackage{amsmath}

\makeatletter
\setlength\@mathmargin{0.2cm}
\makeatother

After spending most the afternoon coding php scripts for a new website, I decided to just stick with WordPress afterall. There are many things I don’t like about it, but I managed to find plugins for the stuff I was missing. Not to mention the animated Tag Cloud I saw the other day. Looks fancy, just need more, uhm, tags…

The current theme is fine too, spent a lot of time yesterday cleaning it up. It had several bugs. I also modified a lot of the code. I may just replace the banner (have a few made) and change the colors too, at which point there won’t be much left of the original theme. I have kept the link to it on the footer tho, but removed the ads.

Tweaking is half the fun of a website!