Tuesday, 10 October 2017

Plotting Lustre MDS stats

At $dayjob we have several large filesystems - for example our /scratch system has 3.1 PB of space using over 1000 HDDs. Although each vendor offers their own dashboard for monitoring they're all a little bit crap and don't integrate with anything else.

Cue an afternoon setting up influxdb (trivial) and grafana (also trivial) on a spare VM and a simple python script run on the metadata servers:

[admin@snx11038n003 ~]$ cat push_mdt_stats.py
import urllib
import time

def grabbit(mds):
 post = ""
 with open(('/proc/fs/lustre/mdt/%s/md_stats' % mds), 'r') as f:
    for line in f:
        k,v,null = line.split(None,2)
        if k == "snapshot_time":
            post += 'metadata,fs={3} {0}={1} {2}\n'.format(k,v,ts,mds)
 with open(('/proc/fs/lustre/mdd/%s/changelog_users' % mds), 'r') as f:
    tmp = f.read().split()
    # we can cheat here as they have the same format - 3rd item in list is current changelog count, and then
    # from the 6th item on we get changelog id / position to pull into a dict
    head = int(tmp[2])
    clog = dict(zip(tmp[5:][0::2], tmp[5:][1::2]))
    post += 'changelog,fs={2} head={0} {1}\n'.format(head,ts,mds)
    for cl,count in clog.items():
        post += 'changelog,fs={3} {0}={1} {2}\n'.format(cl,count,ts,mds)

 p = urllib.urlopen('http://influxbox:8086/write?db=lustre&precision=u',post)

while True:
    sys.exit("Whoa, that went a bit Pete Tong!")

And a couple of clicks in Grafana can soon knock up a dashboard:

Monday, 28 August 2017

PSU tinkering, Part 1

As previously blogged, I've got a couple of 12v 88.7A PSUs that I'm trying to control under arduino. Stage 1 complete - It powers up with a trivial bit of code

/* Arduino control for (ex) server PSU 
 * Andrew Elwell <andrew.elwell@gmail.com> August 2016
 * Released under BSD licence

 /* Controls / Pins based on data sheet available at 
  *  https://belfuse.com/resources/PowerSolutions/SFP1050/bcd20031_ab_sfp1050-12bg.pdf
  *  A6/B4/C4/D4         +3.3 standby (power to arduino)
  *  A3/B1/B3/C1/C3/D3   Return 
  *  B5(SDA) / C5(SCL)   I2C
  *  B6                  Bring low for PS ON
  *  C6                  AC OK (if high)
  *  D6                  PWR OK (if high)

#include <wire.h>

int ACOK  = 2;
int PSON  = 3;
int PWROK = 4;
int LED   = 13;

void setup() {
  Wire.begin();                // join i2c bus (address optional for master)
  pinMode(ACOK, INPUT);
  pinMode(PSON, OUTPUT);
  pinMode(PWROK, INPUT);
  pinMode(LED,  INPUT);
  digitalWrite(PSON,HIGH) ;   // Stay off until ready

void loop() {
  if (digitalRead(ACOK) == HIGH) {
    digitalWrite(PSON,LOW) ;
  if (digitalRead(PWROK) == HIGH) {
    digitalWrite(LED,HIGH) ;

The one gotcha that I needed to get it working was to also bring PS A0 low (I2C address) and suddenly green led and 12v out!

Tuesday, 4 July 2017

I've got the power

(It's getting, it's getting, it's getting kinda hectic)

So, another "I should really get round to that" project that's worked its way to the top of the desk is repurposing a skip-dived server PSU (or 4) to be more usable.

Exhibit A - One ex-sun 'SPASUNM-03G' PSU, which spits out a fairly chunky 12v at 86.7A
Since these were pulled from a bunch of servers, the output is a less than friendly set of three paired contacts for +12 and another set of three pairs or the ground. It won't start spitting out 12v when you plug a mains lead in as it needs the PS_ON connector bringing low. Power-one seem to gave been bought out by bel, and the datasheet is available here.

Rather than the (sometimes) crude way people have modified these and similar server PSUs over at RC Groups, I thought I'd hook up an arduino and be "smart"

So - Grand Plan (TM)
* Nice big illuminated push button for on/standby
* LCD display to show status (output / alarms / temp)
* No screaming 'fan-at-maximum' setting all the time

This shouldn't be that hard, right? Arduinos can do i2c and I have a bunch of 3.3v ones to hand, so I can drive this off the stby 3.3v (even that's at 3A on this thing)

TO THE SOLDERING IRON! ... to be continued

Sunday, 25 June 2017

Handheld animal RFID reader teardown

As some of you may know, we have a small herd of llamas and although each is easily recognisable legislation and the Llama Association of Australasia Inc. require that the animals be microchipped. As a geek, I'd also like to integrate routine monitoring of weight, so my idea is to use the embedded microchips (same as your pet FDX-B 'grain of rice' thing) to save the scale output into the relevant animal record (yup, another use of a Raspberry Pi in the steading)

Aliexpress provided the reader, and 5 small screws later (and another 2 for the PCB) has the innards exposed at https://www.flickr.com/photos/elwell/albums/72157683149501201

Sunday, 24 July 2016

The writing's on the wall

For the last couple of jobs I've had, having some sort of status display has proved itself really useful. Things like an overloaded nagios dashboard help to drill down to see what system issues you may have, but on large systems there'll always be some component that's not green (however your service should work around these transparently to the end user).  In a smaller team without 24/7 operations staff and shift handovers, how do you know things aren't on fire when you walk into the office? - I'll ignore the fact that you probably read your email over breakfast.


At $job[-1] we put a spare monitor on the office wall and ran concerto on a PC feeding it. The backend at that time was php, and made assumptions such as assuming that short tags were OK - I hacked on a branch to make it more standard with the scientific linux systems we were using at the time. Given this was (is?) a student project out of the Rensselaer Institute ir's hardly suprising as young developers want LATEST SHINY. They then went through a second system effect, rewriting from scratch and completely missing the launch of the raspberry pi, which could have made a killer combination.
The problem is that many browser based clients need the overhead of X11 and all the various hacks to remove mouse cursors and make them more 'kiosky'


Fast forwards a few years and I have mini-magnus on my desk showing status and a set of 3 unused monitors on a wall looking to display some info. A quick bit of research flagged up info-beamer which has been used in production at the CCC events for several years. I've been playing with it for a couple of days now in the standalone pi and hosted variant.

hosted info-beamer

The install of this is very smoothly done. Small zip file download to populate an SD card with the raspberry pi bootloader files and a squashfs  and it self-installs the rest of the distribution from S3 and prompts you to register the node to your info-beamer account (in a similar way to a chromecast).
Florian has made some really nice touches to the setup - little things like setting cec_osd_name if your TV screen uses it, a custom kernel logo, and suppression of all but the player-related boot messages. In a public area, this makes it look a lot more professional than most of the other solutions which show the operating system before launching a player should they reboot. I've only played with a couple of the sample packages, but the install process is slick, if initially confusing terminology between packages, setups and playlists.

I'd love to see this integrated with indico - the meeting software used at CERN. Hint :-)

standalone info-beamer

The personal use player is distributed as a binary executable - I can understand the reasons for that, but it doesn't feel right. It doesn't (yet?) come as a debian package - It should be trivial for me to do myself according to the docs. When that's done, I'll install and run using systemd rather than the daemontools method (which is used by the various syncer scripts used in the hosted version). My concern is that the logging is presently noisy (good for debugging, bad for SD lifetime) 
I'm also planning on managing these R-Pi nodes using Ansible (Puppet is overkill for this as I want something that can bootstrap up a fresh SD card without needing extra daemons running) so I suspect there'll be future blogging on that. 

I've not yet investigated sending values directly to the info-beamer listening port (hoping I can do something with an MQTT subscriber) but pushing json files from various subsystems seems to work well. Obligatory screenshot:
Prototype display testing
Overall, I'm very impressed. Given I'd started learning lua for some nodemcu work I should be able to develop something functional. I've asked the work graphic designer to assist, so hopefully it won't end up "engineer style"

Wednesday, 15 June 2016

The Physical Web. Yeah, thats a good idea.

In the last week I've discovered the Physical Web from google, and I'm sold on the idea. Apart from the "what's around here" geeky stuff, it's a great idea for sensible 'distant' digital signage. For example, $dayjob is at the Pawsey Supercomputing Centre, but we don't plaster our URL over the visitor area - what if guests could be gently prompted to the right URL by beacon?

Again tonight (while watching WASO play the Indiana Jones score) I noticed a set of three A3 posters explaining to users of another part of the conference centre how to connect to wifi and download <exhibit> app. This isn't even Scott Jensen's complaint of a 'dos prompt on the browser' - it's more a dig out the index card from the library, then go to the dos prompt...

Saturday, 21 May 2016

All around the water tank, waiting for the rain...

Having the luxury of mains water means that I don't really care in fanatical detail about the state of the dam water levels for Perth (except that "it's lower than it should really be"). However with our new place being entirely dependant on rainwater collection off the roof into storage tanks, I'd like to know the levels of the various tanks (and therefore the volume remaining).
So, what's available - simplest is knocking on the side of the tank and guessing from the sound how full. Not terribly reliable or hi-tech, but is cheap. Dipstick also cheap, but requires removing filter cover. Next up are external gauges - Our tank supplier stocks the Yaktek Levetator, but that's not really any good when I'm not on site. Hence, it's time to investigate the electronic options:

This thread on whirlpool mentions the Electrosense Aquagauge, but at $265 each plus telemetry thats not cheap. The Jaycar ultrasonic one may be OK, but as it doesn't have a serial / usb  output from the display, I'd need to hack up something to parse the RF (as others have done). So datasheet hunting time:

The MaxBotix series looks like it's pretty much ideal - 1mm resolution, 5m max range, weather resistant. Annoyingly I can't see it cheaper than the ~100 USD on the manufacturer site (sparkfun et al. only have the MB7360 not the MB7369), but at least there's a local supplier.  A slightly different model (MB7060) is being used by the flood network.

Next up is trying to work out how to connect them - My options are:

  • Wired - Etherten or possibly something as simple as serial-ethernet adaptor
  • Wireless - wemos / ESP8266 for wifi (power hungry?)
  • Wireless - rfm type TX/RX setup to basestation
  • lora - nice but possibly expensive as I'd need a gateway.
I suspect this particular blog entry will get updated as things progress..

Plotting Lustre MDS stats

At $dayjob we have several large filesystems - for example our /scratch system has 3.1 PB of space using over 1000 HDDs. Although each vendo...