Metadata

This document describes the metadata functionality of Edge Connect.

Metadata provides generic volatile and persistent storage for any data that may be shared or stored across pipelines.

Types

Two distinct types of metadata are supported in Edge Connect: device and global.

Metadata Rules:

Keyed with a string and can accept any value type (arrays, maps, etc. are permitted)
Persists for the lifetime of the Edge Connect process unless cleared by the metadataClear filter.
Optionally can be persisted to disk using the persistent configuration option.

Global Metadata

Global metadata is used to store data associated with the instance of Edge Connect.

Use cases:

Gateway location data
Gateway identification data
Authentication data (API keys, etc.)

Device Metadata

Device metadata is used to store data associated with a particular device.

A device is identified by the Bluetooth MAC (or similar) address, which must be present as the mac key in an event.

Use cases:

Identification data (map a physical device to an ID)
Authentication data (cloud or device)
Device data (battery level, sensor data, last refreshed, etc.)

Operations

The supported metadata operations are "Get", "Set" and "Clear". These operations are available for global metadata, all device metadata, or a single device's metadata.

The operation is set using the following fields in the available Metadata fields:

global: If set to true global metadata operations are performed.
keys: If non-empty will set the metadata keys to operate on. If empty, all available metadata keys are operated on. (default: empty).

Key dot-notation is allowed in keys.

Get (`metadataGet`)

The Get operation is used to retrieve metadata and add it into the event map.

Global Access (global set to true)

If keys is non-empty, only the requested keys in keys are retrieved (if present) from global metadata and added to the event map.
If keys is empty, all global metadata keys are added to the input map.

Device Access global set to false

The device metadata is looked up using the MAC.
If keys is non-empty, the requests keys in keys are retrieved (if present) from the device's metadata and added to the event map.
If keys is empty, all keys from the device's metadata are added to the input map.

Set (`metadataSet`)

The Set operation takes data from the event map and stores it in metadata. When a Set is performed on a metadata key that exists, the previous value is overwritten.

Global Access (global set to true)

If keys is non-empty, the keys in keys are retrieved (if present) from the event map and stored in global metadata.
If keys is empty, all keys from the event map are stored into global metadata.

Device Access (global set to false)

The device metadata is looked up using the MAC.
If keys is non-empty, the keys in keys are retrieved (if present) from the event map and stored in the device's metadata.
If keys is empty, all keys from the event map are stored into the device's metadata.

Clear (`metadataClear`)

The Clear operation is used to clear metadata when an event received.

In addition to the global setting, the allDevices setting allows performing device metadata operations on all device metadata.

Global Mode (global set to true)

If keys is non-empty, the keys in keys are deleted from global metadata.
If keys is empty, all keys present in the event map are deleted from global metadata.

All Devices Mode (global set to false and allDevices set to true)

If keys is non-empty, the keys in keys are deleted from every device's metadata.
If keys is empty, all devices are deleted from device metadata.

Single Device Mode (global false and allDevices)

The device metadata is looked up using the MAC.
If keys is non-empty, the keys in keys are deleted from device's metadata.
If keys is empty, the device is deleted from device metadata.

Usage

Metadata operations are permitted in pipelines and connection sequences.

In a Pipeline

To perform metadata operations in a data pipeline, add one of the metadata filters to your pipeline:

metadataGet
metadataSet
metadataClear

In a Connection

To perform metadata operations in a connection sequence, add one of the metadata actions to your connection sequence:

metadataGet
metadataSet
metadataClear

Initialization

On Edge Connect startup, global and device metadata are empty. If metadata needs to be initialized at startup, populate service.metadata in the top-level Edge Connect configuration.

Key dot-notation is allowed at the top-level of service.metadata.global and service.metadata.devices.

{
  "settings": {
    "rigado-edge-connect": {
      "service": {
        "metadata": {
          "devices": {
            "9454930000001": {
              "key": "secret88"
            },
            "9454930000002": {
              "key": "secret99"
            }
          },
          "global": {
            "apiKey": "secretStuff9000",
            "location.lat": "12",
            "location.lon": "34"
          }
        }
      }
    }
  },
  "type": "APP_SET"
}

Example Configuration

The following example configuration connects to two Thingy52 sensors, collects the battery level, and sends the data to MQTT every 2 minutes.

Gateway location metadata is initialized in service.metadata.global and attached to a device with the get-global-metadata filter.

Device alias metadata is initialized in service.metadata.devices per device.

Device battery metadata is set in the thingy connection sequence by the set-device-metadata action.

Device metadata is then attached to an advertisement with get-device-metadata filter.

The device metadata key, battery, is cleared for all devices when the aggregator fires an event so only recent battery levels are published.

{
  "settings": {
    "rigado-edge-connect": {
      "service": {
        "actions": {
          "read-battery": {
            "config": {
              "outputKey": "battery"
            },
            "type": "readBattery"
          },
          "set-device-metadata": {
            "config": {
              "keys": [
                "battery"
              ]
            },
            "type": "metadataSet"
          }
        },
        "cloud": {
          "connectors": {
            "mqtt": {
              "config": {
                "host": "192.168.1.1",
                "port": 1883,
                "scheme": "tcp"
              },
              "type": "mqtt"
            }
          }
        },
        "connections": {
          "thingy": {
            "actions": [
              "read-battery",
              "set-device-metadata"
            ]
          }
        },
        "filters": {
          "aggregate": {
            "config": {
              "interval": "2m",
              "key": "mac",
              "mode": "map"
            },
            "type": "aggregate"
          },
          "clear-device-metadata": {
            "config": {
              "allDevices": true,
              "keys": [
                "battery"
              ]
            },
            "type": "metadataClear"
          },
          "connect": {
            "config": {
              "id": "thingy",
              "interval": "1m",
              "timeout": 10
            },
            "type": "connect"
          },
          "get-device-metadata": {
            "config": {
              "keys": [
                "battery",
                "alias"
              ]
            },
            "type": "metadataGet"
          },
          "get-global-metadata": {
            "config": {
              "global": true,
              "keys": [
                "location"
              ]
            },
            "type": "metadataGet"
          },
          "publish": {
            "config": {
              "connector": "mqtt",
              "topic": "test/thingy"
            },
            "type": "publish"
          },
          "select": {
            "config": {
              "keys": [
                "mac",
                "rssi",
                "name"
              ]
            },
            "type": "select"
          }
        },
        "groups": {
          "thingys": {
            "config": {
              "include": [
                "c4efb9686279",
                "e91ae5a6831f"
              ]
            },
            "type": "static"
          }
        },
        "metadata": {
          "devices": {
            "c4efb9686279": {
              "alias": "chair"
            },
            "e91ae5a6831f": {
              "alias": "laptop"
            }
          },
          "global": {
            "location": "desk"
          }
        },
        "pipelines": {
          "connectThingy": {
            "filters": [
              "connect"
            ],
            "groups": [
              "thingys"
            ]
          },
          "publishThingy": {
            "filters": [
              "select",
              "get-device-metadata",
              "get-global-metadata",
              "aggregate",
              "clear-device-metadata",
              "publish"
            ],
            "groups": [
              "thingys"
            ]
          }
        }
      }
    }
  },
  "type": "APP_SET"
}