diy

Project HiveSUMMary

1 Comment

How I built my IoT infrastructure

My father Peter is a passionate beekeeper [see honig.spreitzer.at] having some hives in his garden and is always interested about what’s going on in the houses of his many girls. Besides many things there is especially one indicator that is of high interest for a beekeeper: the current weight of the hive. Having some measurements over time gives you insights of how a bee colony develops.

So he had put some hives already on mechanical scales but – of course – he could only evaluate the weight sporadically when he was caring for his bees.

He soon heard about professional hive-scales that are continuously checking the weight and sending the data to some endpoint, may it be SMS, web or an SD-Card. But these professional beehive-scales are way too expensive for a hobby beekeeper. Doing some research on the web you quickly learn about a lot of projects where beekeepers from all over create their own beehive-scales. One project – beelogger.de – caught my interest and so I started experimenting with their solution. But quite soon I found out that it was not exactly the solution my father needed. E.g. he does not need to power the system of a battery and/or photovoltaic cells (power-line is nearby) or does not need to transmit the data via mobile (garden has wifi coverage).

Having played around a lot with Arduino and ESP8266 microcontrollers I started developing my own idea of how such a beehive-scale should work.

This blog post gives you an overview of how I designed the whole system – from automatically collecting the hive data to visualising it in an appropriate and generic manner and sending alert emails when special events in data occur. You wonder why the project title “HiveSUMMary” is written like that? Well, that is because SUMMen in german means “to hum”. :-)

Overview

My beehive-scale ecosystem basically consists of three independent parts:

  • Beehive-Scale hardware
    • Weighting Cell + ESP8266 microcontroller – one for each hive
  • IoT Device Manager
    • PHP script to provide a static endpoint for each ESP8266 and provides a basic gateway to send the received data to my IoTLog instance
  • IoTLog
    • WordPress Plugin to manage data storage, visualisation, monitoring and alerting

As you can see in the names of “IoT*” I did not design the system to be “just” a beehive-scale system but to be my generic idea of an IoT (“Internet of Things”) system and managing the hive-scales being just the first use-case for it. Over the time I also started using it for lots of other sensors – e.g.

  • HiGrow moisture sensor
  • MQ-X gas sensors
  • ESP32-Cam
  • personal weather station
  • many more to come

Lots of ideas are borrowed from microservice architecture and other well-known (software) development patterns. Most parts are designed to be stateless which keeps them easily adoptable and flexible. Data is mostly transferred as JSON for the same reason.

Basically the system should just provide a framing to solve the problem of collecting, storing, visualising, evaluating and reacting to data.

Beehive-Scale Hardware

ESP8266-12 + HX711 + Weighting Cell

  • First working prototype of the PCB + weighting cell

I based the hardware for the hive-scale on the dirt-cheap ESP8266 microcontroller. It has lots of GPIO ports and integrated WiFi capabilities. As it can be programmed via the Arduino IDE with its big ecosystem of libraries for various sensors it is a good choice for my requirements.

The basic hardware setup consists of the following parts:

  • ESP8266-12 microcontroller, ~2 Eur
  • HX711 Load Cell Amp, ~1 Eur
  • 200kg weighting sensor load cell, ~10 Eur
  • LD1117 3,3V Voltage Regulator, ~0.2 Eur
  • Some screw terminal blocks and capacitors

Additionally the firmware also supports the following sensors:

  • DS18B20 temperature sensor [1-wire], ~2 Eur
  • BME280 temperature, air pressure, humidity sensor [i2c], ~3 Eur
  • BMP280 temperature, air pressure sensor [i2c], ~1 Eur
  • BH1750 light sensor [i2c], ~2 Eur
  • BeeHive-Scales collecting data
  • First tries to calibrate the BeeHive-Scale
  • Iron frame holding the weighting cell

The principal functionality of taking the measurements is as follows: The ESP8266 microcontroller starts up, connects to one of the predefined WiFi networks and initialises any connected (and supported) sensors. Then it takes all the measurements, posts them to the IoT Device Manager and goes into deep sleep mode. After another 15 minutes the ESP8266 wakes up and performs all the actions again.

Firmware features

  • Capable of various sensors – whatever is connected is initialised and evaluated – no reprogramming needed if sensor-setup is changed
  • Scale calibration: ability to calculate scale factor and offset values remotely
  • Secure requests via https
  • If IoT Device Manager after a request responds with stayawake-flag the ESP8266 stays awake for 10 minutes.
    During this time it
    • allows OTA updates via Arduino IDE
    • runs a webserver and responds to the following endpoints
      • http://ip [Status]
      • http://ip/measure [measure and view values]
      • http://ip/calibrate [start weighting cell calibration]
      • http://ip/restart [reboots the ESP8266]

Arduino Build Script

I like the Arduino IDE for its simplicity – however as a professional software developer I was missing some important features to automate things – especially regarding building and compiling the source files. So after some research (see here) I learned that the Arduino IDE already has a CLI built in. The basic way to build the sources for the ESP8266 from the command line is as follows:

#!/bin/sh
rm *.bin
ARDUINO_BIN=/Applications/Arduino.app/Contents/MacOS/Arduino
$ARDUINO_BIN --verify zwt_beescale_esp8266.ino --board esp8266:esp8266:generic:eesz=4M2M --pref build.path=build

This script

  • removes any previous build (“rm *.bin”)
  • builds (“verify”) the given zwt_beescale_esp8266.ino.generic.ino file against ESP8266 with an EEPROM size of 4M2M and saves the compiled result into zwt_beescale_esp8266.ino.generic.bin

Note: This is basically the same as using the Arduino IDE feature Sketch -> Export compiled binary.

OTA Update Bash Script

To upload the compiled binary to any OTA-enabled ESP8266 from the command line I use the following script named flash_ota.sh

#!/bin/sh
ip=$1
path=$2
password="pwd-used-for-OTA"
python /Users/YOUR_USER_NAME/Library/Arduino15/packages/esp8266/hardware/esp8266/2.6.3/tools/espota.py -d -r -i $ip -p 8266 --auth="$password" -f "$path"

Sample usage to flash compiled binary to ESP8266 currently having IP 10.0.0.67:

> ./flash_ota 10.0.0.67 /PATH/TO/YOUR/PROJECT/zwt_beescale_esp8266.ino.generic.bin

Note: Again you can accomplish the same in the Arduino IDE: Select your currently online ESP8266 via Tools -> Port and then press the upload button.

The reason I still wanted bash scripts to do these jobs is that I can freely combine any tasks – e.g. I have scripts that

  • build and upload the compiled binary in one step
  • build and copy the compiled binary via VPN to the remote RaspberryPI at my fathers place. The RaspberryPI then starts the upload of the binary to the ESP8266 which – for the RaspberryPi – is in the local network.
    Note: The step of delegating the OTA-uploading to the RaspberryPI was necessary as I did not manage to get OTA updates working via my openVPN connection {Please drop me a line if you were able to do so over a VPN}.

Custom PCB

The first few devices I created manually on prototyping boards but that took me about two hours per board and is prone to failures while assembling.

So I started to experiment with Fritzing to design a PCB and had it produced. Now it just takes me some minutes to assemble a new hardware unit. Furthermore I designed the board with flexibility in mind so I can use it for whatever other purpose that comes to my mind.

  • My first fully functioning PCB – so proud :-)

Flashing firmware to ESP8266-12 via Serial

My current PCB does have a serial connection for debugging but is not prepared to put the ESP8266 into flash/upload mode. For that reason I built another board I just use for uploading the firmware. Such a board needs to pull

  • GPIO-0 to Ground
  • GPIO-2 to High and
  • GPIO-15 to Ground.

For more information on bootloader modes search for “ESP8266-12 modes” or check out this guide.

Software: IoT Device Manager

  • Buffered (meta) data of environmental sensor

Though I’m more into “real” object-oriented programming languages like Java I decided to implement the software-side in PHP as you can easily deploy it to most (shared) hosters and I just did not want to have the hassels of operating and an instance of a Java Application Server.

As IoT is highly distributed by nature I soon found out that I needed some sort of managing layer for all the devices. The basic idea of this IoT Device Manager is to have a known static entry point where each IoT Device (every ESP8266 in my case) can post data to. This layer then knows for each device what to do with the collected data. In my case I just forward it to my IoTLog but I could also easily just define any other action that should be taken.

Key features

  • Provides an endpoint for each IoT Device to post data to
  • Handles authentication of the devices
  • Each device has assigned a name, channel and device group
  • Decides what to do with the received data (forward, web hook, email)
  • Buffers latest (meta) data
    • last time online
    • (local) IP address – useful when updating ESP firmware OTA
    • RSSI (signal strength) of WiFi
    • SSID of WiFi
    • Data (very useful for debugging :-)
  • LoRa integration: The Idea is to have any LoRa device send data which is received by a dedicated ESP8266 and proxied to this IoT Device Manager. Also handles AES-CBC decryption (see my previous post)
  • REST-API: query for devices by device group, name, channel
    • In that way it can act as sort of name service to find out the current IP of a device that was assigned via DHCP.
  • Stay-Awake: As my ESP8266 microcontrollers are mostly always in deep sleep to save energy I normally can not update their firmware over-the-air (OTA). For this reason I can flag a device to stay awake the next time it sends me data. When this happens I receive an email that the device is now reachable and I can update its firmware. Because I also collect meta data I know its latest IP address. With some bash script magic (see above) I even managed to update the ESPs over a VPN-tunnel – cool ey?
  • Backed by just one database table

Software: IoTLog

  • Report cards -insights into your data

Again I decided to write this functionality in PHP but taking it even further I implemented it as a WordPress Plugin. Using WordPress as your development framework let’s you reuse a lot of features – Rendering Webpages, Data Storage via (my)SQL, deploying your plugin and so on. And just think about how easy it is to have an instance of WordPress Page – most hosters even provide one-click installers. – Just install the IoTLog plugin and the platform is up and running :-)

  • Line chart
  • Donut chart

Key features

  • REST Enpoints
    • to receive data (e.g. from IoT Device Manager)
    • for an overview of hive data requested by a dedicated E-Ink display (for details see this blog post)
  • Alerting
    • via Monitors, e.g.
      • “send an email any time a specific temperature sensor sends a temperature lower than 15 degrees
      • “send an email any time a humidity sensor reports a humidity of at least 85%”
      • prepared to implement web hooks
    • via Watchdogs, basically act like a dead man’s switch, e.g.
      • “send me an email if any beehive-scale did not send any data for two hours”
  • WordPress Shortcodes for visualisation
    • Generic Report Cards
      • freely configurable which sensor data to show
      • responsive – automatically adapt view to mobile and desktop devices
    • Charts
      • freely configurable line and donut charts
  • WordPress Admin area for simple configuration
  • Mobile friendly data visualisation
  • IoTLog Admin Area – Sensors
  • IoTLog Admin Area – Monitors
  • IoTLog Admin Area – Settings

Swarm Detection

  • Detect swarms as they happen and don’t lose your bee colony.

Specially for detecting swarming bee colonies I implemented a monitor that allows to send an email any time a beehive-scale reports a weight lower than 1.2kg compared to the value an hour ago. Of course the threshold and duration is customisable.

Conclusion

This is my interpretation of what a beehive-scale and/or IoT platform for personal use should look like. A big advantage for me is that I host the scripts on my own webhost and am therefore not depended by any other third-party system. Furthermore collecting data can fastly grow into privacy issues so it is always fine to know what exactly happens to your data.

Questions?

You have any more questions about my beehive-scale / IoT solution?
You need help with your own similar project?

Feel free to comment below or drop me a line via e-mail – see zeitwesentech.com for contact details.

Links

Please also check back at my fathers bees:

Honigmacherei Peter Spreitzer
Ihr Honig aus dem Mostviertel
Preview by Beautiful Link Preview

Building your own encrypted LoRa infrastructure

7 Comments

Lately I discovered these nice little LoRa boards that make it possible to transmit data over the air on an open frequency without the need to have an extra SIM-card and a cellular data network. They promise up to 10km range so I knew that this would be my new playground :-)

Basically with LoRa you just send out data in clear-text on a predefined channel and everyone else that listens on that channel can read it. Not very safe. Actually not safe at at.

I also learned about LoRaWan that already solves a lot of these problems – especially encryption, addressing and lots more. But at the place where I wanted to deploy LoRa is no LoRaWan coverage and I also did not want to buy+operate a LoRaWan gateway there.

So I decided to build my own LoRa infrastructure. It consists of some nodes sending data via LoRa and another node (my “LoRa Gateway”) listening to them and forwarding the data to my web-backend via WiFi.

Outline of my LoRa infrastructure

I’m using the ESP8266-12 modules programmed via Arduino on both sides as there is already a broad variety of libraries and a big and helping community.

After initial data transmission from the sensor-node to the web-backend finally was working I started to play around with encrypting the payload. There are already a lot of libraries to encrypt and decrypt data on the Arduino that work great. But I did not find a lot of examples to encrypt on the Arduino and decrypt on a webserver.

One that was quite promising was the one that I found here: https://github.com/kakopappa/arduino-esp8266-aes-encryption-with-nodejs. It uses AES-128-CBC and was the first example project that I got working for my requirement to encrypt on the ESP8266 and decrypt on another platform – a big thanks to https://github.com/kakopappa :-)

The only problem was that it is using nodejs to decrypt the data but my webhost only supports php. So I wrote a php-port of the decryption part:

<?php
function decryptAes($aesKey, $ivB64, $msgB64){
		$plain_iv = bin2hex(base64_decode($ivB64));
		$iv = hex2bin($plain_iv);
		$key = hex2bin($aesKey);
		$bytes = openssl_decrypt($msgB64, "AES-128-CBC", $key, $options=0, $iv);
		$plaintext = base64_encode($bytes);
		$decoded_b64msg = base64_decode($plaintext, true);
		return base64_decode($decoded_b64msg, true);
}
?>

Sample usage:

<?php
	$aesKey = "2B7E151628AED2A6ABF7158809CF4F3C";
	$ivB64 = "iv-in-base64"; 
	$msgB64 = "msg-in-base64";
	$decryptedMsg = decryptAes($aesKey, $ivB64, $msgB64);
	echo $decryptedMsg;
?>

Of course you need to change the $aesKey to the one you have used. The key you see here is the same as in the library of https://github.com/kakopappa. The key is defined as a HEX-String, the IV and message are given Base64-Encoded.

Literal youTUBE TV

6 Comments

To meet the nerdy me in me I created this “literal youTUBE TV”.
I found this super retro/vintage CRT of the Austrian brand Kapsch on the flea market (it even came with a receipt of 1977) and as it uses just RF as input it is basically worthless as TV-stations only broadcast digitally nowadays.
But… you can buy an HDMI-to-RF transmitter which does exactly that: Converting your HDMI-signal to RF that can be injected instead of an antenna signal. After that you just have to plug a Chromecast or whatever HDMI outputting device into it and you have upcycled your old CRT.
Sweet, isn’t it?

  • Kapsch VIENNASTAR CRT, built 1977
  • AIYIMA Transmitter HDMI To RF (find it in your fav. chinese retailer)
  • Google Chromecast
  • Some wiring and setting up

Big thanks to my father who – as a former TV and radio mechanic – took care of adjusting and maintaining the CRT.