Thursday, 14 June 2018

Overlaying SLURM job timings on Grafana plots

As you may have noticed, I'm quite fond of Grafana and use it at home and work. One of the dashboards I have at work is the general state of our lustre filesystems, showing IO and metadata traffic, collected by a custom python script (I'm working on converting this to a real collectd python plugin) which stores the data in an influxDB.

I've since written a small python script that talks to our SLURM accounting DB, so that given a jobID, we can get the start/end times and overlay those using the annotations API. One minor niggle in that the API expects epoch milliseconds, and seems to be tied to the TZ of the browser that generated the API key.

however...
~$ annotate_job 2924399
Found the following job:
  User: bskjerven (pawsey0001)
  Cluster: magnus, Partition: workq, QOS: normal
  Nodes: 768, CPUs: 36864
  Start: 2018-06-11 17:23:22, End: 2018-06-11 19:54:44
Got something back - Annotate? (y/n) y
200 - Annotation added

and lo - 


Thursday, 22 March 2018

weewx to home-assistant

At home I have a Fine Offset (this one branded by Jaycar) weather station that publishes to weather.two-fifteen.com via weewx (this is much simpler now I don't have to have the thing solar mounted in a field using a 3G dongle - at least the NBN is useful for some things) but I'd like to be able to use some of the measurements in home-assistant.

There isn't yet a direct plugin (spare time project anyone?) that I can see, but because I'm pushing the metrics locally to influxdb (have I mentioned I like drawing graphs of things?) for grafana, it's possible to use this in home-assistant via weewx-influx

Weewx config:
[StdRESTful]
      [[Influx]]
        host = localhost
        database = weather
        unit_system = METRIC

and on the home-assistant server:
  - platform: influxdb
    host:    203.0.113.88
    queries:
    - name: Outside Temp
      database: weather
      measurement: 'record'
      field: 'outTemp_C'
      group_function: last
      where: 'time > now() - 5m'
      unit_of_measurement: °C
      value_template: '{{ value | round(1) }}'

and lo and behold, you should end up with something like this:
which when compared to our nearest BOM observation site up the road in Grove correlates nicely (Grove is in a valley, we're at the top of a hill)


Monday, 15 January 2018

Don't count your chickens...

We have a cheapo Chinese incubator for hatching eggs. According to popular Internet postings, the calibration of the 'temperature setting' on the front vs reality inside isn't terribly accurate. Since I have a stack of 'Ruuvitags' (https://ruuvi.com) from their kickstarter, I decided to put combine them with Grafana so I could start logging the data and plotting trends.

First up, by default they broadcast an Eddystone beacon, so that you can simply see the data on a phone / tablet (via the Physical Web), however with Google dropping this feature, I decided to switch them to Raw mode which has a higher accuracy. This is done by simply opening the tag and pressing the 'B' button to toggle between Raw (LED blinks red) and URL (LED blinks green) mode.

I'm using a Raspberry Pi 3 as a Bluetooth receiver. Running Rasbian 9 (stretch) means that I get a recent (v 5.43) version of bluez which understands BLE. Although there's a Java app to push to influxdb, I'd rather use Python, so pip-installed the ruuvitag_sensor package.

Initially this worked 'OK', but the logs were full of noise on the listener, so I hacked up a quick script based loosely on the examples. When done, it was much cleaner than the original and was picking up more of the broadcasts.

Once again, trivial to add to influx with some templating

Overlaying SLURM job timings on Grafana plots

As you may have noticed, I'm quite fond of Grafana and use it at home and work. One of the dashboards I have at work is the general sta...