Elasticsearch and Arduino: better together!
Named Visionary by Gartner for the third year in a row, Elastic is the world’s leading platform for search-powered solutions – and a company we are proud to partner with.
Recently, we collaborated with this data powerhouse on a Portenta H7-based R&D project to provide a simple Elasticsearch client library (written in C++) that runs on Arduino modules. That’s right: you can now communicate with an Elasticsearch server directly from an Arduino board!
Among the many ways we immediately tested this new opportunity, we tried developing an IoT device that sends temperature data captured by sensors every five minutes to Elastic Cloud. This, combined with Elasticsearch’s geo features, could be the first step in building a solution that provides the current average temperature from all sensors 5 km away upon request.
Want to find out more? Here is a simple tutorial with all the details.
Portenta C33 
Portenta H7
Use case: temperature feedback from multiple IoT devices
We designed a use case for a company that needed to manage multiple IoT devices located in Italy. Each device sends data coming from sensors (e.g. temperature) to Elastic Cloud. Using Elastic Cloud the company can manage any scale of IoT devices, without the need of managing a dedicated infrastructure. Moreover, the company needs to adjust some internal parameters of each device from the average temperature of neighboring devices, in a range of 100 km. This is a typical scenario in control engineering applications.
Using Elasticsearch we can provide multiple feedback using search features, such as filtering, aggregation, multi-match, geospatial, vector search (kNN), semantic search, and machine learning.
In this use case, we used the average aggregation and geo-distance to retrieve all the devices between 100 km.
Using Kibana, the UI available in Elastic Cloud, we can easily create a dashboard to monitor the data coming from all the devices. Since we also have geo-data we can represent this information on a map.
This is a heat map created with different colors representing different temperatures (blue is cold and green, red are hot).
Setup of Elastic Cloud
The first step is to have an account for Elastic Cloud. If you don’t have one you can register for a trial here (no credit card required). Once you login you can create a new deployment, choosing the size of the Elasticsearch instances that you want to use.
Once you have created a deployment, you need to retrieve the endpoint URL and generate an API key of Elasticsearch. You can read this guideline for support on obtaining this information.
Preparing Elasticsearch index
We need to create an index to store the data coming from the Arduino boards. We want to store temperature values, position of the device using geo-location (latitude and longitude), a device identifier name, and a timestamp.
We can create an index “temperature” with the following HTTP request to Elasticsearch:
PUT /temperature
{
"mappings": {
"properties": {
"temperature": { "type": "float" },
"timestamp": { "type": "date" },
"location": { "type": "geo_point" },
"device-id": { "type": "keyword" }
}
}
}
To send this HTTP request you can use the Dev Tools of Kibana in Elastic Cloud.
We want to store the timestamp of the operation each time a device sends data. This can be done using the ingest pipeline feature of Elasticsearch. An ingest pipeline is an action that Elasticsearch executes before indexing (storing) a document. For instance, a pipeline can assign the value of a specific document field, based on some calculation.
In our case, we just need to store the timestamp and we can create a “set-timestamp” pipeline:
PUT _ingest/pipeline/set-timestamp
{
"description": "sets the timestamp",
"processors": [
{
"set": {
"field": "timestamp",
"value": "{{{_ingest.timestamp}}}"
}
}
]
}
Using this pipeline we can then send data to Elasticsearch as follows:
POST /temperature/_doc?pipeline=set-timestamp
{
"temperature": 21.45,
"device-id": "H7-001",
"location": {
"type": "Point",
"coordinates": [12.4923, 41.8903]
}
}
Here the device-id H7-001 is the name of the Arduino board and location is the geographic point expressed with 12.4923 (longitude) and 41.8903 (latitude), that is the position of the Colosseum in Rome (Italy).
Notice that we did not specify the timestamp value because this is automatically generated using the “set-timestamp” pipeline (specified in the URL as query string).
Geo-distance query
To retrieve the average temperature of the devices distance up to 100 km we can use the following Elasticsearch query:
GET /temperature/_search
{
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"geo_distance": {
"distance": "100km",
"location": [12.4923, 41.8903]
}
}
}
},
"aggs": {
"avg_temp": { "avg": { "field": "temperature" } }
}
}
This query will return an “avg_temp” aggregation field containing the average temperature of all devices within a radius of 100 km.
Usage of the Elasticsearch client for Arduino
It’s finally time to show some Arduino code! Below is a simple sketch that sends a temperature value to Elastic Cloud, gets the average temperature performing a geo-distance query and waits for 30 seconds.
The code reported here is available online in the examples folder of the elastic/elasticsearch-arduino github repository. The sketch uses an elasticsearch_config.h file as follows:
#define WIFI_SECRET_SSID ""
#define WIFI_SECRET_PASS ""
#define ELASTIC_ENDPOINT ""
#define ELASTIC_PORT 443
#define ELASTIC_CLOUD_API_KEY ""
#define DEVICE_GEO_LON 12.4923
#define DEVICE_GEO_LAT 41.8903
#define DEVICE_ID "x"
#define DEVICE_GEO_DISTANCE "50km"
In our example, we used Wi-Fi to connect the Arduino board to the internet.
The WIFI_SECRET_SSID and the WIFI_SECRET_PASS are the name of the SSID network to use and the Wi-Fi password.
The ELASTIC_ENDPOINT is the URL of the Elastic Cloud endpoint, the ELASTIC_PORT is 443 since Elastic Cloud uses TLS (https). The ELASTIC_CLOUD_API_KEY is the API key to be generated in the Elastic Cloud admin interface.
This file also contains other information related to the Arduino device. We have the longitude (DEVICE_GEO_LON) and latitude (DEVICE_GEO_LAT), the ID (DEVICE_ID) and the distance (DEVICE_GEO_DISTANCE) for the geo-query.
After filling all the previous information, we can have a look at the sketch, reported as follows:
#include <ArduinoJson.h>
#include <WiFi.h>
#include <WiFiSSLClient.h>
#include "ESClient.h"
#include "elasticsearch_config.h"
// WiFi settings
char ssid[] = WIFI_SECRET_SSID;
char pass[] = WIFI_SECRET_PASS;
// Elastic settings
char serverAddress[] = ELASTIC_ENDPOINT;
int serverPort = ELASTIC_PORT;
WiFiSSLClient wifi;
ESClient client = ESClient(wifi, serverAddress, serverPort);
int status = WL_IDLE_STATUS;
void setup() {
Serial.begin(9600);
Serial.println("Started");
while (status != WL_CONNECTED) {
Serial.print("Attempting to connect to Network named: ");
Serial.println(ssid);
// Connect to WPA/WPA2 network:
status = WiFi.begin(ssid, pass);
}
// print the SSID of the network you're attached to:
Serial.print("SSID: ");
Serial.println(WiFi.SSID());
// print your WiFi shield's IP address:
IPAddress ip = WiFi.localIP();
Serial.print("IP Address: ");
Serial.println(ip);
client.setElasticCloudApiKey(ELASTIC_CLOUD_API_KEY);
}
void loop() {
float temperature;
// Set the temperature from a sensor (removing the randomness)
temperature = random(10,30) + random(0,100)/100.00;
// Prepare the JSON with temperature and geopoint for Elasticsearch
StaticJsonDocument<200> doc;
doc["temperature"] = temperature;
doc["device-id"] = DEVICE_ID;
doc["location"]["type"] = "Point";
doc["location"]["coordinates"][0] = DEVICE_GEO_LON;
doc["location"]["coordinates"][1] = DEVICE_GEO_LAT;
String temp;
serializeJson(doc, temp);
Serial.println("Sending to Elasticsearch:");
Serial.println(temp);
ESResponse indexResult;
// Send the temperature to Elastic Cloud
indexResult = client.index("temperature", temp, "pipeline=set-timestamp");
DynamicJsonDocument result(1024);
deserializeJson(result, indexResult.body);
if (result["result"] == "created") {
Serial.println("Created with _id: " + result["_id"].as<String>());
} else {
Serial.println("Error sending data: " + indexResult.body);
}
StaticJsonDocument<512> query;
query["query"]["bool"]["filter"]["geo_distance"]["distance"] = DEVICE_GEO_DISTANCE;
query["query"]["bool"]["filter"]["geo_distance"]["location"][0] = DEVICE_GEO_LON;
query["query"]["bool"]["filter"]["geo_distance"]["location"][1] = DEVICE_GEO_LAT;
query["aggs"]["avg_temp"]["avg"]["field"] = "temperature";
query["size"] = 0;
String search;
serializeJson(query, search);
Serial.println("Geo-location query:");
Serial.println(search);
ESResponse searchResult;
// Send the temperature to Elastic Cloud
searchResult = client.search("temperature", search);
DynamicJsonDocument avg(512);
deserializeJson(avg, searchResult.body);
float avgTemp = avg["aggregations"]["avg_temp"]["value"];
int numDevices = avg["hits"]["total"]["value"];
Serial.println("Average temperature of " + String(numDevices) + " devices in " + DEVICE_GEO_DISTANCE + ": " + String(avgTemp));
Serial.println("Wait 30 seconds");
delay(30000);
}
This sketch requires Wi-Fi, WiFiSSLClient (for connecting using TLS) for the internet connection, the EsClient for connecting to Elasticsearch and the ArduinoJson library for serializing and deserializing Json data structure.
In the setup() function we start the Wi-Fi connection and we set the API key of Elastic Cloud using client.setElasticCloudApiKey(ELASTIC_CLOUD_API_KEY) function call. The client object is initialized in the main area passing the Wi-Fi object, the server address (endpoint) and the HTTP port.
In the loop() function we have the code that sends the temperature to Elastic Cloud. The temperature here is just a random float number between 10 and 30, typically coming from a sensor attached to the Arduino board. To prepare the document to send to Elasticsearch, we used the ArduinoJson library.
We used the following code to create a “doc” object:
StaticJsonDocument<200> doc;
doc["temperature"] = temperature;
doc["device-id"] = DEVICE_ID;
doc["location"]["type"] = "Point";
doc["location"]["coordinates"][0] = DEVICE_GEO_LON;
doc["location"]["coordinates"][1] = DEVICE_GEO_LAT;
This object is serialized in a JSON string as follows:
String temp;
serializeJson(doc, temp);
Finally, the document, stored in the “temp” variable, can be sent to Elasticsearch using the index API, as follows:
ESResponse indexResult;
indexResult = client.index("temperature", temp, "pipeline=set-timestamp");
This API adds the “temp” document in the index “temperature” using the “set-timestamp” pipeline. The result is stored in the “indexResult” variable that is a struct type as follows:
struct ESResponse {
int statusCode;
String body;
};
The “statusCode” is the HTTP status code of the response and “body” is the response body. The index operation is successful if the response contains a “result” field with value “created”.
To get the average temperature of the devices within a radius of 100 km, we used the following geo-distance query, expressed using ArduinoJson.
StaticJsonDocument<512> query;
query["query"]["bool"]["filter"]["geo_distance"]["distance"] = DEVICE_GEO_DISTANCE;
query["query"]["bool"]["filter"]["geo_distance"]["location"][0] = DEVICE_GEO_LON;
query["query"]["bool"]["filter"]["geo_distance"]["location"][1] = DEVICE_GEO_LAT;
query["aggs"]["avg_temp"]["avg"]["field"] = "temperature";
query["size"] = 0;
String search;
serializeJson(query, search);
ESResponse searchResult;
searchResult = client.search("temperature", search);
DynamicJsonDocument avg(512);
deserializeJson(avg, searchResult.body);
float avgTemp = avg["aggregations"]["avg_temp"]["value"];
int numDevices = avg["hits"]["total"]["value"];
The response of the search contains the average temperature, as an aggregation value. Moreover, we can retrieve the number of devices retrieved by the query using the [‘hits’][‘total’][‘value’] field in the JSON response from Elasticsearch.
Conclusion
Thanks to the collaboration with Elastic, we developed a very simple library that allows the usage of Elasticsearch directly from an Arduino board. In a few lines of code we can send data to Elasticsearch and execute complex elaborations, using geolocation and more.
We can’t wait to see what Arduino users will come up with, using Elasticsearch! For instance, if you are interested in generative AI you will certainly enjoy Elastic’s newest features. Give Elastic Cloud and the elasticsearch-arduino library a try!