Monday, 15 January 2018

Don't count your chickens...

We have a cheapo Chinese incubator for hatching eggs. According to popular Internet postings, the calibration of the 'temperature setting' on the front vs reality inside isn't terribly accurate. Since I have a stack of 'Ruuvitags' (https://ruuvi.com) from their kickstarter, I decided to put combine them with Grafana so I could start logging the data and plotting trends.

First up, by default they broadcast an Eddystone beacon, so that you can simply see the data on a phone / tablet (via the Physical Web), however with Google dropping this feature, I decided to switch them to Raw mode which has a higher accuracy. This is done by simply opening the tag and pressing the 'B' button to toggle between Raw (LED blinks red) and URL (LED blinks green) mode.

I'm using a Raspberry Pi 3 as a Bluetooth receiver. Running Rasbian 9 (stretch) means that I get a recent (v 5.43) version of bluez which understands BLE. Although there's a Java app to push to influxdb, I'd rather use Python, so pip-installed the ruuvitag_sensor package.

Initially this worked 'OK', but the logs were full of noise on the listener, so I hacked up a quick script based loosely on the examples. When done, it was much cleaner than the original and was picking up more of the broadcasts.

Once again, trivial to add to influx with some templating

Tuesday, 10 October 2017

Plotting Lustre MDS stats

At $dayjob we have several large filesystems - for example our /scratch system has 3.1 PB of space using over 1000 HDDs. Although each vendor offers their own dashboard for monitoring they're all a little bit crap and don't integrate with anything else.

Cue an afternoon setting up influxdb (trivial) and grafana (also trivial) on a spare VM and a simple python script run on the metadata servers:

[admin@snx11038n003 ~]$ cat push_mdt_stats.py
#!/usr/local/bin/python2.7
import urllib
import time

def grabbit(mds):
 post = ""
 with open(('/proc/fs/lustre/mdt/%s/md_stats' % mds), 'r') as f:
    for line in f:
        k,v,null = line.split(None,2)
        if k == "snapshot_time":
            ts=int(float(v)*1000000)
        else:
            post += 'metadata,fs={3} {0}={1} {2}\n'.format(k,v,ts,mds)
 with open(('/proc/fs/lustre/mdd/%s/changelog_users' % mds), 'r') as f:
    tmp = f.read().split()
    # we can cheat here as they have the same format - 3rd item in list is current changelog count, and then
    # from the 6th item on we get changelog id / position to pull into a dict
    head = int(tmp[2])
    clog = dict(zip(tmp[5:][0::2], tmp[5:][1::2]))
    post += 'changelog,fs={2} head={0} {1}\n'.format(head,ts,mds)
    for cl,count in clog.items():
        post += 'changelog,fs={3} {0}={1} {2}\n'.format(cl,count,ts,mds)

 post=post.encode('ascii')
 p = urllib.urlopen('http://influxbox:8086/write?db=lustre&precision=u',post)
 #print(p.getcode())

while True:
  try:
    grabbit('snx11038-MDT0000')
  except:
    sys.exit("Whoa, that went a bit Pete Tong!")
  time.sleep(10)

And a couple of clicks in Grafana can soon knock up a dashboard:


Monday, 28 August 2017

PSU tinkering, Part 1

As previously blogged, I've got a couple of 12v 88.7A PSUs that I'm trying to control under arduino. Stage 1 complete - It powers up with a trivial bit of code

/* Arduino control for (ex) server PSU 
 * Andrew Elwell <andrew.elwell@gmail.com> August 2016
 * Released under BSD licence
 */

 /* Controls / Pins based on data sheet available at 
  *  https://belfuse.com/resources/PowerSolutions/SFP1050/bcd20031_ab_sfp1050-12bg.pdf
  *  
  *  A6/B4/C4/D4         +3.3 standby (power to arduino)
  *  A3/B1/B3/C1/C3/D3   Return 
  *  B5(SDA) / C5(SCL)   I2C
  *  B6                  Bring low for PS ON
  *  C6                  AC OK (if high)
  *  D6                  PWR OK (if high)
  *  
  */


#include <wire.h>

int ACOK  = 2;
int PSON  = 3;
int PWROK = 4;
int LED   = 13;

void setup() {
  Wire.begin();                // join i2c bus (address optional for master)
  pinMode(ACOK, INPUT);
  pinMode(PSON, OUTPUT);
  pinMode(PWROK, INPUT);
  pinMode(LED,  INPUT);
  
  digitalWrite(PSON,HIGH) ;   // Stay off until ready
}

void loop() {
  if (digitalRead(ACOK) == HIGH) {
    digitalWrite(PSON,LOW) ;
  } 
  if (digitalRead(PWROK) == HIGH) {
    digitalWrite(LED,HIGH) ;
  } 
}

The one gotcha that I needed to get it working was to also bring PS A0 low (I2C address) and suddenly green led and 12v out!



Tuesday, 4 July 2017

I've got the power

(It's getting, it's getting, it's getting kinda hectic)

So, another "I should really get round to that" project that's worked its way to the top of the desk is repurposing a skip-dived server PSU (or 4) to be more usable.

Exhibit A - One ex-sun 'SPASUNM-03G' PSU, which spits out a fairly chunky 12v at 86.7A
Since these were pulled from a bunch of servers, the output is a less than friendly set of three paired contacts for +12 and another set of three pairs or the ground. It won't start spitting out 12v when you plug a mains lead in as it needs the PS_ON connector bringing low. Power-one seem to gave been bought out by bel, and the datasheet is available here.


Rather than the (sometimes) crude way people have modified these and similar server PSUs over at RC Groups, I thought I'd hook up an arduino and be "smart"

So - Grand Plan (TM)
* Nice big illuminated push button for on/standby
* LCD display to show status (output / alarms / temp)
* No screaming 'fan-at-maximum' setting all the time

This shouldn't be that hard, right? Arduinos can do i2c and I have a bunch of 3.3v ones to hand, so I can drive this off the stby 3.3v (even that's at 3A on this thing)

TO THE SOLDERING IRON! ... to be continued


Sunday, 25 June 2017

Handheld animal RFID reader teardown

As some of you may know, we have a small herd of llamas and although each is easily recognisable legislation and the Llama Association of Australasia Inc. require that the animals be microchipped. As a geek, I'd also like to integrate routine monitoring of weight, so my idea is to use the embedded microchips (same as your pet FDX-B 'grain of rice' thing) to save the scale output into the relevant animal record (yup, another use of a Raspberry Pi in the steading)

Aliexpress provided the reader, and 5 small screws later (and another 2 for the PCB) has the innards exposed at https://www.flickr.com/photos/elwell/albums/72157683149501201

Sunday, 24 July 2016

The writing's on the wall

For the last couple of jobs I've had, having some sort of status display has proved itself really useful. Things like an overloaded nagios dashboard help to drill down to see what system issues you may have, but on large systems there'll always be some component that's not green (however your service should work around these transparently to the end user).  In a smaller team without 24/7 operations staff and shift handovers, how do you know things aren't on fire when you walk into the office? - I'll ignore the fact that you probably read your email over breakfast.

Concerto

At $job[-1] we put a spare monitor on the office wall and ran concerto on a PC feeding it. The backend at that time was php, and made assumptions such as assuming that short tags were OK - I hacked on a branch to make it more standard with the scientific linux systems we were using at the time. Given this was (is?) a student project out of the Rensselaer Institute ir's hardly suprising as young developers want LATEST SHINY. They then went through a second system effect, rewriting from scratch and completely missing the launch of the raspberry pi, which could have made a killer combination.
The problem is that many browser based clients need the overhead of X11 and all the various hacks to remove mouse cursors and make them more 'kiosky'

info-beamer

Fast forwards a few years and I have mini-magnus on my desk showing status and a set of 3 unused monitors on a wall looking to display some info. A quick bit of research flagged up info-beamer which has been used in production at the CCC events for several years. I've been playing with it for a couple of days now in the standalone pi and hosted variant.

hosted info-beamer

The install of this is very smoothly done. Small zip file download to populate an SD card with the raspberry pi bootloader files and a squashfs  and it self-installs the rest of the distribution from S3 and prompts you to register the node to your info-beamer account (in a similar way to a chromecast).
Florian has made some really nice touches to the setup - little things like setting cec_osd_name if your TV screen uses it, a custom kernel logo, and suppression of all but the player-related boot messages. In a public area, this makes it look a lot more professional than most of the other solutions which show the operating system before launching a player should they reboot. I've only played with a couple of the sample packages, but the install process is slick, if initially confusing terminology between packages, setups and playlists.


I'd love to see this integrated with indico - the meeting software used at CERN. Hint :-)

standalone info-beamer

The personal use player is distributed as a binary executable - I can understand the reasons for that, but it doesn't feel right. It doesn't (yet?) come as a debian package - It should be trivial for me to do myself according to the docs. When that's done, I'll install and run using systemd rather than the daemontools method (which is used by the various syncer scripts used in the hosted version). My concern is that the logging is presently noisy (good for debugging, bad for SD lifetime) 
I'm also planning on managing these R-Pi nodes using Ansible (Puppet is overkill for this as I want something that can bootstrap up a fresh SD card without needing extra daemons running) so I suspect there'll be future blogging on that. 

I've not yet investigated sending values directly to the info-beamer listening port (hoping I can do something with an MQTT subscriber) but pushing json files from various subsystems seems to work well. Obligatory screenshot:
Prototype display testing
Overall, I'm very impressed. Given I'd started learning lua for some nodemcu work I should be able to develop something functional. I've asked the work graphic designer to assist, so hopefully it won't end up "engineer style"

Wednesday, 15 June 2016

The Physical Web. Yeah, thats a good idea.

In the last week I've discovered the Physical Web from google, and I'm sold on the idea. Apart from the "what's around here" geeky stuff, it's a great idea for sensible 'distant' digital signage. For example, $dayjob is at the Pawsey Supercomputing Centre, but we don't plaster our URL over the visitor area - what if guests could be gently prompted to the right URL by beacon?

Again tonight (while watching WASO play the Indiana Jones score) I noticed a set of three A3 posters explaining to users of another part of the conference centre how to connect to wifi and download <exhibit> app. This isn't even Scott Jensen's complaint of a 'dos prompt on the browser' - it's more a dig out the index card from the library, then go to the dos prompt...


Don't count your chickens...

We have a cheapo Chinese incubator for hatching eggs. According to popular Internet postings, the calibration of the 'temperature settin...