Ruizs Place

This is the first script I created using the NexentaStor SA-API, the install of which is detailed in my previous post here. We use a combination of Nagios and Zabbix to monitor our infrastructure, and a very helpful component of that has been the check_netapp plugin available here, which reports any sort of failure on the netapp, including array health, disk space/quota’s, etc. Netapp provides most of this data with a custom SNMP MIB, but unfortunately NexentaStor doesn’t have one (yet at least). I wanted the same functionality with our NexentaStor boxes, without having to remember to add a volume to our monitoring system every time I create a volume on the storage.

This script examines the state of each volume and the space used/free for each volume with a quota, and reports back any errors. The layout is copied from the check_netapp plugin, and is fairly simple to follow along. Of note, be sure to set the appropriate @INC path for your installation, and the location of your public key if using PKI with NexentaStor. Unfortunately some of the API is not documented, and the functionality is incomplete – I had to play around with the $folder->execute() method to find the correct command to get the space used and quota allocated. It seems like this should be a built in, documented method in the API, so here’s to hoping it makes it in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

#!/usr/bin/perl
 
BEGIN { unshift (@INC, "/opt/build/NexentaStor-SDK-Linux/examples/perl");}
 
use NZA::Common;
use strict;
 
$ENV{'NEXENTA_ID_RSA'} = '/home/nagios/.ssh/id_rsa.pub';
 
#Configure space % threshold here for warnings
my $space_threshold = .9;
 
my $IPADDR = $ARGV[0];
if (!$IPADDR) {
        print "Usage: checkhealth.pl <ip -ADDR>\n";
        exit 1;
}
my $volume = nms_remote_obj($IPADDR, '/Root/Volume');
my $folder = nms_remote_obj($IPADDR, '/Root/Folder');
my @errors;
 
my $volnames = $volume->get_names('');
for my $vol (@$volnames) {
        #errors: No known data errors
        #state: ONLINE
        my $volstatus = $volume->get_status($vol);
        if (${$volstatus}{'state'}[0] ne 'ONLINE') {
                push (@errors, "Bad volume status for $vol: ", ${$volstatus}{'state'}[0] );
        }
        if ( ! ${$volstatus}{'errors'}[0] =~ /No known data errors/ ) {
                push (@errors, "Errors for $vol: ", ${$volstatus}{'errors'}[0] );
        }
 
        my $folder_names = $folder->get_names('');
        for my $fol (@$folder_names) {
                $folder->reconstruct($fol);
                my $quota;
                my $used;
                my $lines = $folder->execute($fol, "get -H used $fol");
                for my $line (@$lines) { $used .= $line };
                my $lines = $folder->execute($fol, "get -H quota $fol");
                for my $line (@$lines) { $quota .= $line };
                my @data = split (/\s+/, $used);
                $used = $data[2];
                my @data = split (/\s+/, $quota);
                $quota = $data[2];
                if ($quota =~ /none/i) {
                        next;
                }
 
                $used = convert_space_units($used);
                $quota = convert_space_units($quota);
                #print "$fol - used: $used, quota: $quota\n";
                if ($quota > 0) {
                        if ( ($used/$quota) > .9 ) {
                                push (@errors, "Volume almost full: $fol");
                        }
                }
        }
}
 
if (scalar(@errors) > 0 ) {
        print "Nexentastor healthcheck failed! ";
        for my $error (@errors) {
                print "  $error.";
        }
        print "\n";
        exit(1);
}
else {
        print "Nexentastor healthcheck OK\n";
        exit(0);
}
 
 
sub convert_space_units($) {
        my $string = shift;
        # K, M, G, T
        my %factor = (
                K => 1000,
                M => 1000000,
                G => 1000000000,
                T => 1000000000000,
        );
        if ($string =~ /([.\d]+)(\w+)/) {
                if ($factor{$2}) {
                        return $1 * $factor{$2};
                }
        }
        return "";
}
</ip>

Nexentastor, a great storage appliance based on solaris/ZFS, has a pretty powerful API built in for remote management, but building the API on anything other than Ubuntu is an exercise in frustration. Included in this post are my notes on getting it installed and working, after a good bit of work by myself and their support/development team. I’m not going to clean it up too much, but if you follow the steps below on a Centos 5 machine you should be able to get it working. The API tarball has a couple good readme’s that you will definitely want to read/reference, especially for setting up API authentication.

yum install gcc make patch iso-codes-devel libnotify-devel NetworkManager-glib-devel libxml2 libxml2-devel dbus-libs dbus-glib-devel dbus-glib dbus-devel expat-devel perl-XML-Parser perl-XML-Twig

cd /root/NexentaStor-SDK-Linux/examples/perl/
./test.pl 192.168.1.100


To setup authentication using ssh keys, do the following:
On the client machine, generate a rsa key if one doesn’t already exist:
ssh-keygen -N "" -q -f ~/.ssh/id_rsa
And on the nexentastor appliance, add that key. The Key ID should be a name you recognize to keep track of the key, and the key value should start with ssh-rsa AAAAzxvdsa….. and end with XXX== – make sure you leave off the name at the end of the key, this is confusing:

nmc@nexentastor:/$ setup appliance authentication keys add
Key ID : myclient
Key Value : ssh-rsa AAAAB3Nza
......
.....
.... 1I89Q==

If you found this through a search, you probably have seen something like this in your logs:
ERROR: missing chunk number 0 for toast value 87246446

Searching for this issue on the net, I found that this is usually caused by a corrupt row in the database. The solution is to find the corrupt row, and delete it (and/or update it with proper, non-corrupt values).

Unfortunately, there is no easy way to find the row that is giving you grief (or even the table if you don’t know what is causing this error). You basically have to narrow down your search by looking at parts of the database. If you have some sort of date or timestamp column in your table, you can narrow it down a lot more easily, if you know approximately when the error started:


mydb=# select id from mytable where date_created > timestamp '2009-02-27 00:00:00' and date_created < timestamp '2009-03-01 23:59:00';
id
--------------------------------------
3ab226cf-a972-463d-b5a1-148fe39672b5
10daca73-b2b3-470c-a258-5c92d21cfbb6
(2 rows)


Luckily it's only two rows, and the first one is the culprit:

mydb=# select * from mytable where id='3ab226cf-a972-463d-b5a1-148fe39672b5';
ERROR: missing chunk number 0 for toast value 87246446


Now, on to delete it:
mydb=# delete from mytable where id='3ab226cf-a972-463d-b5a1-148fe39672b5';

If you don’t have a date/timestamp column in your table, you could use LIMIT and OFFSET to narrow it down. I.e. a series of statements like (increasing the offset each time):

select * from mytable order by id limit 1000 offset 5000;


I’ve been meaning to get around to it for the longest time, but I finally switched over my Mac’s to use bash as the default shell (previously I had it set to tcsh). I’ve become more accustomed to bash the past few years as its the default shell on pretty much any unix/linux nowadays, so it was time to migrate.

The migration itself was pretty easy, but I wanted to gain some of bash’s features, specifically the emac-like keystrokes for moving forward/back a word in the command line, and also to the beginning and end of the line. The keystrokes, among others, are:

ctrl-a Move cursor to beginning of line
ctrl-e Move cursor to end of line
meta-b Move cursor back one word
meta-f Move cursor forward one word
ctrl-w Cut the last word
ctrl-u Cut everything before the cursor
ctrl-k Cut everything after the cursor
ctrl-y Paste the last thing to be cut
ctrl-_ Undo


I’m a big fan of iTerm – I use it as my primary terminal and have been for the last few years, as it matured in terms of performance and has a lot of great features. The most compelling feature it has over the built in OSX Terminal.app, for me, is that you can specify which characters to include when selecting words. A common operation in a terminal is to select a path, i.e. /home/myuser/testfile. In Terminal.app, if you double click on home, it will just select home – so you have to go back, double click on the first “/” character, and mouse right to get the whole word. With iTerm, you can set a preference with characters to include when connecting, i.e. “/”, so when I double click on home in iTerm, it selects the full path. Specifically, in iTerm -> Preferences -> Mouse, i have “Characters considered part of word:” set to /-_.

One thing I couldn’t figure out was how to get the “meta” key in iterm set to the alt/option key on the mac keyboard. Turns out it’s buried in a non-intuitive place, but it can be done. In iTerm, go to Bookmarks->Manage Profiles, and under Keyboard Profiles select Global. On the right side, you need to set Option Key as “+Esc” for this to work – yes, its strange that it has to be Esc instead of Meta, but hey, at least it all works.