Introduction

I have a Consumer Access Device (CAD) from Glow to enable access to near-real-time (10 second granularity) readings from the SMETS2 smart electricity meter. This is one of the few CAD options which can be easily configured to send MQTT messages to a local MQTT server, rather than always sending all the data out to the ‘Cloud’ (although it does that too, for access from the Bright app).

MQTT is convenient for me since that that’s where (almost) all the near-real-time data gets presented, whatever the source, for onward processing, storage and graphing. (Storage is in the InfluxDB time-series database and graphing is via Grafana.) All the other readings (temperature and humidity, water and heat meter readings etc.) are slower-moving and typically sampled every couple of minutes whereas the live power consumption readings from the electricity meter are much faster-moving and probably deserve some more intelligent treatment, such as down-sampling for longer-term storage, so they warrant being separated out from the others, into their own InfluxDB database.

When adding the readings from the Glowmarkt CAD into the existing MQTT / InfluxDB / Grafana data pipeline it seemed sensible to separate them out from the other MQTT-derived readings. It’s Telegraf that provides the ‘glue’ between MQTT and InfluxDB and since MQTT is the integration point for lots of different types of readings, from lots of different sources, the single InfluxDB database which currently gets all the readings from all the sensors is quite “busy” already.

Content of the Glow CAD MQTT Messages

The MQTT messages consist of JSON payloads and look like this:

{
  "electricitymeter":{
    "timestamp":"2023-08-26T12:54:29Z",
    "energy":{
      "export":{
        "cumulative":5746.315,
        "units":"kWh"
      },
      "import":{
        "cumulative":18161.713,
        "day":4.172,
        "week":72.270,
        "month":469.691,
        "units":"kWh",
        "mpan":"1170000287878",
        "supplier":"Octopus Energy",
        "price":{
          "unitrate":0.39350,
          "standingcharge":0.44480
        }
      }
    },
    "power":{
      "value":-0.160,
      "units":"kW"
    }
  }
}

and

{
  "software":"v1.8.13",
  "timestamp":"2023-08-26T12:54:35Z",
  "hardware":"GLOW-ESP-CAD-0v721-CAD-SMETS2",
  "wifistamac":"C44F336F5CA9",
  "ethmac":"C44F336F5CAC",
  "smetsversion":"SMETS2",
  "eui":"70:B3:D5:21:E0:01:01:3A",
  "zigbee":"1.2.5",
  "han":{
    "rssi":-52,
    "status":"joined",
    "lqi":192
  }
}

There’s a post on the Glowmarkt Forum relating to storing these in InfluxDB that basically says to declare the data_format as “json” and leave Telegraf / InfluxDB / Grafana to sort everything else out. While that does work, what’s happening under the covers is that the whole JSON text string is getting stored in InfluxDB (for every data point) – rather than just a simple numeric value – which places a lot of load on downstream processing tools (such as Grafana) to parse the JSON strings and extract the data. Sometimes Grafana shows ‘No Data’ when it’s still trying to process the JSON.

The fix is to process the JSON up-front then re-publish just the key values to discrete MQTT Topics, for easier processing by other data consumers. Sorting this out is on my TODO list…

References

https://www.bentasker.co.uk/posts/blog/house-stuff/connecting-my-smart-meter-to-influxdb-using-telegraf-and-a-glow-display.html

https://github.com/influxdata/telegraf/issues/1778

https://stackoverflow.com/questions/74564109/telegraf-multiple-input-multiple-output

Hildebrand / Glow Wired CAD MQTT Data Processing by Marsh Flatts Farm Self Build Diary is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.