Category Archives: Computer Stuff

Harddrive failure and monitoring

Posted by on October 11, 2011 1 comment | 3,340 views

Broken harddiskI have two Ubuntu servers running at home which both have large RAID volumes on them, set up via mdadm. This summer I had a total disk failure in one of my RAID5s which luckily didn’t result in data loss. Thanks you RAID5!

In any case, it caused me to write a script that logs SMART-data to a MySQL database. I also wrote an admin webpage that displays this data for me in an easy to follow way. The monitoring script itself is written in php5 and so are the admin pages. I used php5 because it is easy to communicate with MySQL from it, and it has the needed string manipulation commands. It could probably be done as easily in Python though. The script is called as a cron job every 2 hours on the servers and every hour on the desktops when they’re running. Examples of the code I’m using is attached below and includes the cron-ed script and the code generating the log output and plot.

What to look for?

Well, that is the big question. How do you know if a drive is about to fail? Google Labs has looked into this topic back in 2007 in this paper: «Failure Trends in a Large Disk Drive Population». An interesting read if you are at all concerned about harddrive failure in servers. The results for how temperature affects the lifetime and failure rate in harddrives are especially interesting. It turns out, at least in their data, that low temperature isn’t such a good thing for the drives contrary to what many people seem to assume. I have up until now been concerned that my drives get too hot, but in fact they seem to be almost overcooled the way I have things set up now.

When I wrote these scripts this summer I decided to log temperature and reallocated sector count primarily, which is what is emphasized in the log display scripts. Seems now I also should be including scan errors as well after reading that paper. The colour coding I use in the temperature plot below is loosely based on Figure 4 in the paper and reflects what seems to be the optimal operating temperature for harddrives.

Screenshots
HD-Mon Screenshot 1HD-Mon Screenshot 2HD-Mon Screenshot 3
Screenshot 1: The overview page.
Screenshot 2: Details of one of the RAID-drives.
Screenshot 3: Details of one of the drives with reallocated sectors.

Code

The php source code I wrote is available in this file: hd-mon.tar.gz

It is specifically designed to work with my setup and hardware and probably isn’t universal, but it gives an idea of how I set it up. There are probably better ways of doing this though. I just call shell commands from php and parse the returned text-string and do simple search on it and input the data into a MySQL database.

I also included php-snippets showing how the admin page is generated. These are not standalone php files, they need to be wrapped in a template. However they reproduce what is seen in the screenshots above.

Packages needed for these scripts to run:

  • php5-cli for the php5 command line.
  • mysql-client, php5-mysql for the database connection.
  • smartctl to access the SMART-data.
  • mdadm to access RAID-data (assuming you use mdadm for RAIDs in the first place)

All are available in the Ubuntu repository.

Converting Charset in MySQL

Posted by on August 6, 2011 No comments | 1,103 views

Logo MySQLJust a quick MySQL tip …

I have a RSS reader that I coded myself a while back with php/MySQL. The news table was set to latin1 character encoding and I wanted to change that into utf8, but couldn’t quite figure out how. Eventually I stumbled upon this solution:

9.1.11.2. Converting 4.0 Character Columns to 4.1 Format

Basically what I needed to do to convert the field “Title” in the table “news” was running the following SQLs:

ALTER TABLE news MODIFY Title BLOB;
ALTER TABLE news MODIFY Title VARCHAR(250) CHARACTER SET utf8;

Easy peasy …

Ubuntu 11.04

Posted by on July 8, 2011 No comments | 1,405 views

NattySo, a new version of Ubuntu is up. Named: Natty Narwhal, or just Natty for short. I skipped the last 2010 version, 10.10 (Maverick) as I had problems running Kile on it. I have upgraded my desktop and my laptop, and left my server and media centre running 10.04 (Lucid). I had no issues on my desktop, a custom built computer from last year with an ASUS M4A89TD Pro mainboard, an AMD Phenom II Hex core CPU and a XFX Radeon HD5770 GFX card. I had no issues at all installing 11.04, neither did I have any problems with 10-04, only Windows7 gave me problems. On my laptop (HP ProBook 4515s) I do have some funky issues with LCD backlights. I also needed to add the option “nomodeset” to the grub loader, otherwise no problems there either.

11.04 comes with Firefox 4, which wasn’t in the Lucid-repository when I upgraded, and LibreOffice has replaced OpenOffice.org, otherwise I found all my regular software packages in working order, including Kile. Although I do have audio sync issues on VLC player on my laptop that I haven’t resolved yet, it was fine on Lucid.

Natty comes with a new desktop as well, which I did not like at all, but you can get the old one back by choosing “ubuntu-classic” at login.

Harddrive Encryption

One thing I did on my laptop was installing Ubuntu on a completely encrypted harddrive—completely except for /boot that is. I have not tried this before, so I am not sure when this feature was implemented, but you need to download the alternate install CD to get these extended partitioning options unless you want to do this manually. There is a guided option, but what the installer does is to create a small boot partition (I made mine 1GB, but the wizard made it 256 GB) with ext2 file system and a logical volume spanning the rest of the space. Then this partition is encrypted and handed over to the LVM where I partitioned it into a swap and a root partition. With this method you will only need to type the decryption passphrase once during boot.

Now the whole harddrive is encrypted with 256 bit AES encryption—good enough for the US government. I do however notice the demand on the CPU, most notable when playing a HD videofile when the data both needs to be decrypted and decoded and played. I don’t use my laptop for anything demanding, but it is worth noting that 256 bit encryption on the entire harddrive—including the system—is a strain on the CPU. You of course do not have to set the encryption to 256 bit though.

MOBI-Books

Posted by on June 9, 2011 No comments | 2,261 views

As I posted a while back, I have bought myself an Amazon Kindle. I am very happy with that little piece of technology. I have already made a template for generating pdf-files that fits well on my Kindle.

Last weekend I decided to try to compile my own mobi files. It is actually fairly simple (although preparing the text itself can be very time-consuming). First you need the text you’d want to convert into a book. I had read this post on pandasthumb.org and downloaded Matt Young’s book «No Sense of Obligation: Science and Religion in an Impersonal Universe». Since the pdf was not very well suited for my Kindle, I decided to try and convert it to mobi-format.

Firstly you need a decent tool. Amazon has provided a tool that prepares books for the Kindle, it can be found on their Kindle Publishing Programs page here. However the conversion tool isn’t all that practical. It’s command-line which is no problem for me, but it also have very few options. A much better program is Calibre, which is multi-platform and is a full eBook management tool that connects and sync with the Kindle and other eBook readers. It can import many formats and also export/convert to many formats.

I found that the easiest way to format an eBook is by making a html-document of it first and the convert the html to mobi. As an example I have made a short eBook out of the essay about Homeopathy I posted about earlier. I will provide all the source files in a bit.

The html file needs to start with a header. Something like:

<?xml Version='1.0' Encoding='utf-8'?>
<html Xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Homeopathy and Its Kindred Delusions</title>
    <meta Http-Equiv="Content-Type" Content="text/html; charset=utf8">
    <link Rel="Stylesheet" Type="text/css" Href="Style.css">
</head>
<body>
</body>
</html>

The text of the book goes inside the body-tag.

Calibre generates a pagebreak for each <h1> and <h2> heading by default, so I use the <h2> tag for chapter headings. You’d also want to put in an anchor tag before each chapter and make a table of contents at the beginning of the book. It is the usual syntax:

<a Name="Chapter1"></a>

for the chapter tag, and

<a Href="#Chapter1">Chapter 1</a>

for the TOC link.

Otherwise the text formatting is straight forward with <i> and <b> tags for italics and bold as usual. One point worth mentioning is paragraph formatting. I myself prefer the indented paragraph as opposed to the extra linebreak style for books. To achieve this I used <div> tags instead of <p> tags in the text, and use the CSS style “Text-Indent: 1.5em” to indent the text at the start of each paragraph. The example files should demonstrate this well enough.

When the html version looks fine (you can preview it in your web-browser), Then you import it to calibre via “Add Book”. Then you do a conversion to “MOBI” (in the right-click dropdown menu). You may play around with the settings, but there were no need to change anything when I converted this book, except I disabled the option to let calibre generate your TOC. Sync with the Kindle, and read away!

Example files:
Oliver Wendell Holmes – Homeopathy and Its Kindred Delusions (1842)

(The files are gzipped.)

Kindle LaTeX Template

Posted by on January 31, 2011 11 comments | 6,933 views

So I finally got around to buy myself a Kindle from Amazon. Very nice device, but it is not very good with standard pdfs. I have a number of books laying around in text format, and the internet is full of good classics that are public domain (i.e. not under copyright). Check for instance Project Gutenberg which has a number of books available, also prepared for Kindle and other mobile devices. Anyway, as I said, I needed a LaTeX template that is formatted and suitable for generating pdfs for the Kindle. I googled a bit around and found one that, with some modification, worked well.

Here is the template.

…and a few books I’ve formatted using that template:

Jules Verne – A Journey to the Interior of the Earth [PDF] [LaTeX]
Jules Verne – Around the World in 80 Days [PDF] [LaTeX]
Jules Verne – Twenty Thousand Leagues Under the Sea [PDF] [LaTeX]


Edit on 10.05.2011:
Based on a comment below I tested the template with an equation. To do this i added the following packages:

\usepackage{amsmath}
\usepackage{amssymb}

The result is shown here (click for larger image):Kindle