Wednesday, 2 September 2020

Thoughts on Statuspage.io integrations

 $dayjob uses statuspage.io for our public status page. Although I've got some things automated, there's still a bunch of components that need manual updating when there's an issue (enough to keep me out of mischief working out how to automate them without false alerts anyway). However this post isn't for getting info _in_ to statuspage, but for how to get it _out_, and what I want.

We're predominantly a command line shop (no, we don't yet have a JupyterHub frontend), so users are presented with a MOTD on login. Now, wouldn't it be good if that could be updated automatically with details of upcoming planned outages as well as any recent (and current) incidents that affected the service you're currently logged into?

So - Armed with the API, it should be possible to get upcoming maintenance[✓] and the impacted components[✓], but where do I map the cluster name to the statuspage group_id? or, for that matter any of the autogenerated id strings. Hard coding them into scripts is out, a lookup makes sense but how many CMDBs out there come with that sorta functionality built in. and we're back to another 'where is my source of truth?' problem. Sure I can string match components->name and check that "group": true and then pull the ID, but... yeah faffy. 

Anyway, after much parsing (all hail requests) it's possible to get the various incidents/component states/planned maintenance out and the resulting text snippets touched to the correct updated_at timestamp whereupon they can be pulled into the final output whenever the files are regenerated by a Makefile build under Jenkins. The goal of DRY is starting to be achieved by updating one place (currently statuspage) and having that trigger a build via webhook which then distributes the info out to the clusters.

I think a former colleague summed it up fairly well tho.


Thursday, 2 April 2020

Solar off-grid cabin

We have a small cabin on the site that used to have a 200w panel, 65Ah deep cycle lead acid battery, el-cheapo PWM charger to power some LED strip when we visited.

While this was OK for occasional use, there's no way it'd be any good for regular usage, let alone constant 12v fan for the composting loo, water pump, fridge, wifi etc. So, time for an upgrade.

First off, new panels. The roof faces pretty much north with not much shading. Jaycar sells a fairly decent range of rails and clamps, and I purchased 3*300w (seemed to be the sweet spot for size/cost/output) panels from ebay that were 800mm wide to fit on a 2560mm rail. 

Wired in series, they still fall under the wiring rules definition of 'Extra Low Voltage' (< 120v ripple free DC). Even on the sunniest of days I've only seen them putting out about 70v so far.

Batteries. The performance benefit of LiFePO4 is huge compared to lead-acid. I took a chance and purchased a set of 8*123 Ah cells from AliExpress ("BLS Official Store"). These came in at 778 USD delivered, nicely packaged and all 8 cells were within 0.02v of each other when I checked.
For protection / balancing I purchased a 'Daly' 8s BMS with a common port (again, AliExpress), and a 120A isolator from Jaycar. To keep cable runs as short as reasonably possible and not clutter up the limited space in the cabin, I decided to put the batteries in a lockable toolchest (bunnings) fixed to the supports for the bathroom shed. There's a small condensation drain in the corner, and the batteries themselves are raised from the base slightly with some strips of yellowtongue. My one gripe about this is the cables on the BMS are slightly too short to position a the BMS neatly.

Linking the two together is a fairly chunky charge controller - I went for a 50A EPEVER Tracer, mainly because they seemed to have a good reputation for the price, and published specs for the modbus-rs485 output. Once you go to the 50-100A level range of controllers, the load connections aren't on the charge controller, but need to be taken off the battery itself. There are a couple of switched outputs that can be used for LV disconnect and/or external charging relays. Oh, and it's a little larger than the old one we had.

Back to ebay for a switchboard and some DC rated MCBs (electric_super) to try and get a neat job. I've got plenty of space behind the melamine chipboard panel for the terminal posts (battery incomers), LV disconnect relay and a POE injector for the wifi access point

My plan is to have remote telemetry back to home-assistant / grafana long term, but for now I've put an 'MT50' meter into the cabin (with only moderate swearing for the cable run) 

So, what next?
I'd like to run an inverter occasionally, but am not convinced we have enough capacity in the batteries (it'd be used to run things like a vacuum cleaner). Today was heavily overcast and the bms dropped the batteries with LV protection, so perhaps an option to connect a charger (this would mean a lot of extension leads or a genset) - potentially one of the combined units? depends on wiring regs if its "fixed wiring"

Getting the telemetry off is becoming more important now we're approaching winter. I've discovered I can't use the wired tail on the UAP-AC-M when it's operating in 'wireless uplink' mode or it thinks its uplink is on the wired side and keeps disconnecting from its upstream partner. So, I'm looking for a low-power wifi SBC that'll easily let me script up modbus monitoring from the tracer, water usage from an impeller sensor, and the water tank volume from an ultrasonic sensor, and fling the whole lot at the on-site MQTT broker.

Inside? well I've added some upstairs lights, multiple DC-DC converters so there's places to charge a mobile phone, and some better kitchen lighting. As well as the fridge and waterpump (we now have both cold AND HOT running water)







Monday, 12 November 2018

Garden Irrigation, Pt 1

Some more background to my 'home assistant powered irrigation system' -- I'll write up a couple more articles as build progresses - I'm still waiting on some critical parts to arrive, like PVC pipe fittings and an Arduino Mega.

The Grand Plan.
The 2 main garden bed areas need to be automatically watered. The small 'hexagon' area contains 6 raised beds and a small central area. The main veggie patch (still being built. slow progress...) is around 20 beds. Later automation will include more smarts (hooking in my rainfall sensor, some buried gypsum sensors and soil temperature monitoring), but for now, the priority is to save on the 30-60 mins with a hosepipe in the evenings. Oh, and we're on tank (rainwater collection) water, so wastage is bad mmmkay.


The irrigation units themselves won't be 'smart' - All they need to do is send valve (relay) status, and respond to controls. I have an internal MQTT message broker, so a basic arduino + ethernet shield + 8 channel relay board does the small bed, and an arduino mega + ethernet (or an ethermega) + 3 8 channel relay shields the main veggie patch.

Sample arduino code is available on GitHub, (functions? who needs functions...) and yeah, it's not elegant. While it follows the syntax for the home assistant MQTT switch device, some of the defaults aren't the same, so I need to explictitly set them for now in the hass config. such as

switch:
  - platform: mqtt
    name: Hexagon bed 1
    icon: mdi:water
    availability_topic: status/hexIrrigation
    command_topic: control/hexIrrigation/valve1
    payload_on: "1"
    payload_off: "0"
    state_topic: status/hexIrrigation/valve1
    state_on: "On"
    state_off: "Off"

You can see here that I use status/hexIrrigation as LWT (so I know if the controllers on/offline) as well as the the relay state. I'm only using retain on the LWT, not the valves themselves. Power cycling the unit will (should!) reset back to all solenoids off, but I don't yet check and publish this, so hass may get confused if it loses comms. "Fixed in the next release" apparently :-)

I also want to integrate a flow sensor into each unit if possible (suspect I'm running out of digital IO on the small unit) to get an overall volume of water that's gone into the garden - fed, as normal into grafana.

I'll throw in another couple of postings (with pics) as the build progresses. but today I had an audience while working on the fence... (it will be fully netted to keep out the possums, pademelons and our hens)


Wednesday, 31 October 2018

Measuring empy space

If you've been following my recent posts on twitter, you'll be aware that I've finally got round to hooking up the MaxBotix rangefinders into our water tanks. So far I've done the main house tank and have sketches and buried duct ready to do the ones that water the garden.

Some more technical details here than I can fit in tweets:
* The sensors themselves sit through the roof of the tank. This involves drilling a 3/4" hole in the top of the tank and inserting the sensor from below, so you won't want to put it more than an arms reach from the access hole. The sensors I purchased have NPR thereads not BSP, so you may want to pick up matching nuts at the same time depending on local availability.
* I mounted some 6-way header pins on the sensor, and soldered the female end onto the cable so that it didn't twist as I tightened it up. All mounted in a 20mm conduit box on the tank.



Inside the shed, I'm just using a clone arduino with a wiznet ethernet shield (ebay/aliexpress) and a very basic sketch using the pulse width measurement from the sensor. As normal, it's available on GitHub for anyone interested. It pushes to an on-site MQTT broker (mosquitto) that's used to gather various sensor metrics and fed into home-assistant, where it also gets archived into InfluxDB for longer term plots

status/housetanksensor online
sensors/housetanks/depthbelowsensor 419
sensors/housetanks/litres 20762

I use the status for LWT messages, and the sensors has the raw air-gap measurement (mm below sensor - range is 300 -> 5000) as well as a calculated volume (see the arduino code for constants and calculation)

Importing into hass is as simple as
  - platform: mqtt
    name: House water tanks volume
    icon: mdi:water-pump
    availability_topic: status/housetanksensor
    unit_of_measurement: l
    state_topic: sensors/housetanks/litres
into the configuration.yaml



Next up, getting the irrigation system working...

Thursday, 14 June 2018

Overlaying SLURM job timings on Grafana plots

As you may have noticed, I'm quite fond of Grafana and use it at home and work. One of the dashboards I have at work is the general state of our lustre filesystems, showing IO and metadata traffic, collected by a custom python script (I'm working on converting this to a real collectd python plugin) which stores the data in an influxDB.

I've since written a small python script that talks to our SLURM accounting DB, so that given a jobID, we can get the start/end times and overlay those using the annotations API. One minor niggle in that the API expects epoch milliseconds, and seems to be tied to the TZ of the browser that generated the API key.

however...
~$ annotate_job 2924399
Found the following job:
  User: bskjerven (pawsey0001)
  Cluster: magnus, Partition: workq, QOS: normal
  Nodes: 768, CPUs: 36864
  Start: 2018-06-11 17:23:22, End: 2018-06-11 19:54:44
Got something back - Annotate? (y/n) y
200 - Annotation added

and lo - 


Thursday, 22 March 2018

weewx to home-assistant

At home I have a Fine Offset (this one branded by Jaycar) weather station that publishes to weather.two-fifteen.com via weewx (this is much simpler now I don't have to have the thing solar mounted in a field using a 3G dongle - at least the NBN is useful for some things) but I'd like to be able to use some of the measurements in home-assistant.

There isn't yet a direct plugin (spare time project anyone?) that I can see, but because I'm pushing the metrics locally to influxdb (have I mentioned I like drawing graphs of things?) for grafana, it's possible to use this in home-assistant via weewx-influx

Weewx config:
[StdRESTful]
      [[Influx]]
        host = localhost
        database = weather
        unit_system = METRIC

and on the home-assistant server:
  - platform: influxdb
    host:    203.0.113.88
    queries:
    - name: Outside Temp
      database: weather
      measurement: 'record'
      field: 'outTemp_C'
      group_function: last
      where: 'time > now() - 5m'
      unit_of_measurement: °C
      value_template: '{{ value | round(1) }}'

and lo and behold, you should end up with something like this:
which when compared to our nearest BOM observation site up the road in Grove correlates nicely (Grove is in a valley, we're at the top of a hill)


Monday, 15 January 2018

Don't count your chickens...

We have a cheapo Chinese incubator for hatching eggs. According to popular Internet postings, the calibration of the 'temperature setting' on the front vs reality inside isn't terribly accurate. Since I have a stack of 'Ruuvitags' (https://ruuvi.com) from their kickstarter, I decided to put combine them with Grafana so I could start logging the data and plotting trends.

First up, by default they broadcast an Eddystone beacon, so that you can simply see the data on a phone / tablet (via the Physical Web), however with Google dropping this feature, I decided to switch them to Raw mode which has a higher accuracy. This is done by simply opening the tag and pressing the 'B' button to toggle between Raw (LED blinks red) and URL (LED blinks green) mode.

I'm using a Raspberry Pi 3 as a Bluetooth receiver. Running Rasbian 9 (stretch) means that I get a recent (v 5.43) version of bluez which understands BLE. Although there's a Java app to push to influxdb, I'd rather use Python, so pip-installed the ruuvitag_sensor package.

Initially this worked 'OK', but the logs were full of noise on the listener, so I hacked up a quick script based loosely on the examples. When done, it was much cleaner than the original and was picking up more of the broadcasts.

Once again, trivial to add to influx with some templating

Thoughts on Statuspage.io integrations

  $dayjob uses statuspage.io for our public status page . Although I've got some things automated, there's still a bunch of compon...