Category Archives: Coding

Harddrive failure and monitoring

Posted by on October 11, 2011 1 comment | 3,335 views

Broken harddiskI have two Ubuntu servers running at home which both have large RAID volumes on them, set up via mdadm. This summer I had a total disk failure in one of my RAID5s which luckily didn’t result in data loss. Thanks you RAID5!

In any case, it caused me to write a script that logs SMART-data to a MySQL database. I also wrote an admin webpage that displays this data for me in an easy to follow way. The monitoring script itself is written in php5 and so are the admin pages. I used php5 because it is easy to communicate with MySQL from it, and it has the needed string manipulation commands. It could probably be done as easily in Python though. The script is called as a cron job every 2 hours on the servers and every hour on the desktops when they’re running. Examples of the code I’m using is attached below and includes the cron-ed script and the code generating the log output and plot.

What to look for?

Well, that is the big question. How do you know if a drive is about to fail? Google Labs has looked into this topic back in 2007 in this paper: «Failure Trends in a Large Disk Drive Population». An interesting read if you are at all concerned about harddrive failure in servers. The results for how temperature affects the lifetime and failure rate in harddrives are especially interesting. It turns out, at least in their data, that low temperature isn’t such a good thing for the drives contrary to what many people seem to assume. I have up until now been concerned that my drives get too hot, but in fact they seem to be almost overcooled the way I have things set up now.

When I wrote these scripts this summer I decided to log temperature and reallocated sector count primarily, which is what is emphasized in the log display scripts. Seems now I also should be including scan errors as well after reading that paper. The colour coding I use in the temperature plot below is loosely based on Figure 4 in the paper and reflects what seems to be the optimal operating temperature for harddrives.

Screenshots
HD-Mon Screenshot 1HD-Mon Screenshot 2HD-Mon Screenshot 3
Screenshot 1: The overview page.
Screenshot 2: Details of one of the RAID-drives.
Screenshot 3: Details of one of the drives with reallocated sectors.

Code

The php source code I wrote is available in this file: hd-mon.tar.gz

It is specifically designed to work with my setup and hardware and probably isn’t universal, but it gives an idea of how I set it up. There are probably better ways of doing this though. I just call shell commands from php and parse the returned text-string and do simple search on it and input the data into a MySQL database.

I also included php-snippets showing how the admin page is generated. These are not standalone php files, they need to be wrapped in a template. However they reproduce what is seen in the screenshots above.

Packages needed for these scripts to run:

  • php5-cli for the php5 command line.
  • mysql-client, php5-mysql for the database connection.
  • smartctl to access the SMART-data.
  • mdadm to access RAID-data (assuming you use mdadm for RAIDs in the first place)

All are available in the Ubuntu repository.

Converting Charset in MySQL

Posted by on August 6, 2011 No comments | 1,101 views

Logo MySQLJust a quick MySQL tip …

I have a RSS reader that I coded myself a while back with php/MySQL. The news table was set to latin1 character encoding and I wanted to change that into utf8, but couldn’t quite figure out how. Eventually I stumbled upon this solution:

9.1.11.2. Converting 4.0 Character Columns to 4.1 Format

Basically what I needed to do to convert the field “Title” in the table “news” was running the following SQLs:

ALTER TABLE news MODIFY Title BLOB;
ALTER TABLE news MODIFY Title VARCHAR(250) CHARACTER SET utf8;

Easy peasy …

MOBI-Books

Posted by on June 9, 2011 No comments | 2,259 views

As I posted a while back, I have bought myself an Amazon Kindle. I am very happy with that little piece of technology. I have already made a template for generating pdf-files that fits well on my Kindle.

Last weekend I decided to try to compile my own mobi files. It is actually fairly simple (although preparing the text itself can be very time-consuming). First you need the text you’d want to convert into a book. I had read this post on pandasthumb.org and downloaded Matt Young’s book «No Sense of Obligation: Science and Religion in an Impersonal Universe». Since the pdf was not very well suited for my Kindle, I decided to try and convert it to mobi-format.

Firstly you need a decent tool. Amazon has provided a tool that prepares books for the Kindle, it can be found on their Kindle Publishing Programs page here. However the conversion tool isn’t all that practical. It’s command-line which is no problem for me, but it also have very few options. A much better program is Calibre, which is multi-platform and is a full eBook management tool that connects and sync with the Kindle and other eBook readers. It can import many formats and also export/convert to many formats.

I found that the easiest way to format an eBook is by making a html-document of it first and the convert the html to mobi. As an example I have made a short eBook out of the essay about Homeopathy I posted about earlier. I will provide all the source files in a bit.

The html file needs to start with a header. Something like:

<?xml Version='1.0' Encoding='utf-8'?>
<html Xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Homeopathy and Its Kindred Delusions</title>
    <meta Http-Equiv="Content-Type" Content="text/html; charset=utf8">
    <link Rel="Stylesheet" Type="text/css" Href="Style.css">
</head>
<body>
</body>
</html>

The text of the book goes inside the body-tag.

Calibre generates a pagebreak for each <h1> and <h2> heading by default, so I use the <h2> tag for chapter headings. You’d also want to put in an anchor tag before each chapter and make a table of contents at the beginning of the book. It is the usual syntax:

<a Name="Chapter1"></a>

for the chapter tag, and

<a Href="#Chapter1">Chapter 1</a>

for the TOC link.

Otherwise the text formatting is straight forward with <i> and <b> tags for italics and bold as usual. One point worth mentioning is paragraph formatting. I myself prefer the indented paragraph as opposed to the extra linebreak style for books. To achieve this I used <div> tags instead of <p> tags in the text, and use the CSS style “Text-Indent: 1.5em” to indent the text at the start of each paragraph. The example files should demonstrate this well enough.

When the html version looks fine (you can preview it in your web-browser), Then you import it to calibre via “Add Book”. Then you do a conversion to “MOBI” (in the right-click dropdown menu). You may play around with the settings, but there were no need to change anything when I converted this book, except I disabled the option to let calibre generate your TOC. Sync with the Kindle, and read away!

Example files:
Oliver Wendell Holmes – Homeopathy and Its Kindred Delusions (1842)

(The files are gzipped.)

Getting Torrent Name from File in php

Posted by on January 24, 2011 No comments | 1,980 views

Wrote a script today that parses a torrent file and retrieve the name of the torrent. The reason I made this is that I have a small web page running at home that parses an RSS feed and upload selected torrents to another computer. Since the name of the torrent on disk (the data folder) doesn’t always match the name of the actual .torrent file, I need to get the name from the file itself. It is quite simple really, and it falls back to the .torrent file name if it fails.

// Find torrent name
$sTorrent     = "torrent_file_name.torrent";
$sTName       = trim(substr($sTorrent, 0, strrpos($sTorrent, ".")));
$sTorrentData = file_get_contents("path/to/torrent/file/".$sTorrent);
preg_match("(:name[0-9]+:)",$sTorrentData,$aTemp,PREG_OFFSET_CAPTURE);
if(count($aTemp) == 1) {
  $sTemp = str_replace(":","",$aTemp[0][0]);
  $sTemp = str_replace("name","",$sTemp);
  if(is_numeric($sTemp)) {
    $sTName = substr($sTorrentData,$aTemp[0][1]+strlen($aTemp[0][0]),(int)$sTemp);
    $sMessage = "<div>Torrent successfully parsed.<br>";
    $sMessage = "Name: ".$sTName."</div>";
  } else {
    $sError = "<div>Parseing failed.<br>Using name: ".$sTName."</div>";
  }
} else {
  $sError = "<div>Parseing failed.<br>Using name: ".$sTName."</div>";
}

Auto UnRar Script

Posted by on January 16, 2011 4 comments | 3,557 views

Ever needed a script that automatically extracts all rar archives in a directory and subdirectories, unrar them into their respective directories and then delete the archive files, but only if the extraction was successful? Well in that case, I wrote this little bash script today:

#!/bin/bash
if [ -n "$1" ]
then
    echo "Unraring from $1"
    find $1 -type f -name '*.part01.rar' -execdir rar x -y -o- {} \;
    find $1 -type f -name '*.part001.rar' -execdir rar x -y -o- {} \;
    find $1 -type f -name '*.rar' -execdir rar x -y -o- {} \;
fi

It takes the path of the folder you want to scan as argument: unrarscript.sh /path/to/folder

Edit Jan 31 2011: It seems the if statements checking if the extraction failed or not only checks if last extraction was successful. For it to work correctly the unlink should be nested inside the finds. I have not yet made any corrections to the script, but be aware of this weakness.

Edit Nov 11 2011: Just removed the unlink part as it wasn’t working properly anyway.