Christoph's Blog about Digital Stuff

Posted on

Processing the Firmware Installation

2020-09-01 22:13:00 +0200

This is the sixth part of a multi-part blog post. You might want to start from the beginning with the first post.

In the previous part I showed you how to establish connection with the Bosch IoT Suite. In this part I will be showing you the necessary steps to receive firmware update commands from the IoT Suite and flash the firmware onto the device. Let’s start by adding a few headers that we are going to need for this project.

#include <Update.h>       // The OTA client of the ESP32
#include <HTTPClient.h>   // The HTTP Client for downloading the firmware
#include <Stream.h>       // We need this for streaming the file to the flash
#include <StreamString.h>

When a distribution set is assigned to a target in IoT Rollouts, IoT Manager will translate this into an install command on the device shadow in case that the device shadow has a SoftwareUpdatable function block. (Version 1 of the Vorto Functionblock will only be supported during the public Beta of the Device Management Package) So step 1 is registering the SoftwareUpdateable function block. Let’s extend the connect function to update the device shadow.

void connect() {
  //...

  // Update device shadow with SoftwareUpdateable function block
  char payload[1000];
  sprintf(payload, "{\r\n\t\"topic\": \"%s/%s/things/twin/commands/modify\",\r\n\t\"path\": \"/features/softwareupdatable\",\r\n\t\"value\": {\r\n\t \t\"definition\": [\r\n\t \t\t\"org.eclipse.hawkbit.swupdatable:SoftwareUpdatable:1.0.0\"\r\n\t \t],\r\n \t \t\"properties\": {\r\n \t\t \t\"status\": {\r\n \t\t\t \t\"agentName\" : \"m5stack\",\r\n \t\t\t \t\"agentVersion\" : \"1.0.0\",\r\n \t\t\t \t\"type\" : \"application\"\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n}", hubNamespace, deviceId );
  if(!client.publish("event", payload, false, 1)) {
    M5.Lcd.print("Failed publishing ");
    M5.Lcd.println(client.lastError());
  }
}

Something to point out is the type property that identifie what type of software module the device shadow accepts. In this case type has the value application, this needs to correlate with the type on the SoftwareModule we are creating in IoT Rollouts.

The message send from the Suite to the device will then look similar to this:

{
	"topic": "namespace/thing_id/things/live/messages/install",
	"headers": {
		"content-type": "application/json",
		"version": 2,
		"correlation-id": "p-pdid85-1fzfisu1ae6v5f-3vj06",
		"x-things-parameter-order": "[\"arg_0\"]"
	},
	"path": "/features/softwareupdatable/inbox/messages/install",
	"value": {
		"arg_0": {
			"softwareModules": [{
				"metaData": {},
				"name": "test",
				"version": "1.0.0",
				"artifacts": [{
					"filename": "generated_eclipseditto_1.0.0-SNAPSHOT_SoftwareUpdatable_thingJson.json",
					"size": 764,
					"hashes": {
						"sha1": "d12f37d9774e1cf2f9bfcc21a2f1091617775a58",
						"sha256": "e3399ace3f2cc70f337f3be894852d307e5a10abec163ac7dc53148984b70537",
						"md5": "d173d3d61191d5d8f4b630889905b054"
					},
					"links": {
						"download": {
							"http": "",
							"https": "https://link to artifact"
						},
						"md5sum": null
					}
				}]
			}],
			"dsMetaData": {},
			"weight": null,
			"correlationId": "32845",
			"actionProperties": {
				"actionType": "FORCED",
				"forceTime": null
			}
		}
	}
}

The key elements that we will need to process are the path property, the correlationId of the software installation request value.arg_0.correlationId and the url to the artifact value.arg_0.softwareModules.[].artifacts.[].links.download.https. Let’s add the handling code with some global variables for processing the message. (The arg_0 will be replaced after the public beta with the correct property name).

// New global variables
String INSTALLATION_PATH = String("/features/softwareupdatable/inbox/messages/install");

int installationState = 0;
StaticJsonDocument<2000> installationCommand;

void messageReceived(String &mqttTopic, String &payload) {
   // ...
   if(INSTALLATION_PATH.equals(path)) {
    installationState = 1;
    installationCommand = doc;
  }
  // ...
}

In case that we receive a installation command, we store the whole message and set the state to 1, to indicate that in the next run of the loop function we want to process the firmware update. We do this since we will need to communicate with the IoT Suite and we shouldn’t do this from the messageReceived callback.

Next I am going to extend the loop function for processing the software update state. It will extract the remaining information from the cached message, and trigger a download and installation function.

void loop() {
  // ...
  if(installationState != 0) {
    const char* correlationId = installationCommand["value"]["arg_0"]["correlationId"];
    JsonArray softwareModules = installationCommand["value"]["arg_0"]["softwareModules"].as<JsonArray>();
    for(JsonVariant v : softwareModules) {
      JsonObject moduleDoc = v.as<JsonObject>();
      String module = moduleDoc["name"];
      String version = moduleDoc["version"];
      
      client.publish("event", buildInstallationState(correlationId, module, version, "STARTED"), false, 1);
  
      JsonArray artifacts = moduleDoc["artifacts"].as<JsonArray>();
      for(JsonVariant artifactVariant : artifacts) {
        JsonObject artifact = artifactVariant.as<JsonObject>();
        String filename = artifact["filename"];
        String md5 = artifact["hashes"]["md5"];
        String url = artifact["links"]["download"]["https"];
        downloadAndInstallFirmware(correlationId, module, version, url);
      }
    }
    installationState = 0;
  }
}

The downloadAndInstallFirmware function will open a HTTPS connection to start downloading the artifact as a stream and pass it to the update handler of the ESP32. It will also create MQTT messages in order to update the device shadow. The IoT Suite transforms this shadow updates into state tranistions on the Distribution Set assignment in IoT Rollouts. You can find the code for the buildInstallationState helper functions down below.

The update itself is handled by four functions, that are part of the ESP32’s Update functionality. Update.begin(int update_size) needs to be called when you want to start an update, it responds true if the device is ready to be updated and the update size doesn’t exceed the parition size. Update.writeStream(stream) takes a binary stream that should be written to the flash storage. With Update.end() you can inform the ESP32 that the whole stream was transfered, with Update.isFinished() you can check if the ESP32 succesfully checked and accepted the update.

void downloadAndInstallFirmware(String correlationId, String module, 
    String version, String url) {
  WiFiClientSecure net2;
  Serial.printf("[HTTPS] Downloading Firmware %s\n", url.c_str());
  HTTPClient https;
  if (https.begin(net2, url.c_str())) {
    int httpCode = https.GET();
    // httpCode will be negative on error
    if (httpCode > 0) {
      // HTTP header has been send and Server response header has been handled
      Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

      // file found at server
      if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
        client.publish("event", buildInstallationState(correlationId, module, version, "INSTALLING"), false, 1);
        bool canBegin = Update.begin(https.getSize());
        if(canBegin) {
          Update.writeStream(https.getStream());

          if (!client.connected()) {
            connect();
          }
          if (Update.end()) {
            Serial.println("OTA done!");
            client.publish("event", buildInstallationState(correlationId, module, version, "INSTALLED"), false, 1);
            if (Update.isFinished()) {
              client.publish("event", buildInstallationState(correlationId, module, version, "FINISHED_SUCCESS"), false, 1);
              restart = true;
            } else {
              client.publish("event", buildInstallationState(correlationId, module, version, "FINISHED_ERROR"), false, 1);
              Serial.println("Update not finished? Something went wrong!");
            }
          } else {
              client.publish("event", buildInstallationState(correlationId, module, version, "FINISHED_ERROR", "Error Occurred. Error #: " + String(Update.getError())), false, 1);
            Serial.println("Error Occurred. Error #: " + String(Update.getError()));
          }
        }
      }
    } else {
      Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
    }

    https.end();
  }
  else {
    Serial.println("Failed downloading");
  }
}

In the following snippet you will find the buildInstallationState helper functions.

String buildInstallationState(String correlationId, String moduleName, 
    String version, String state) {
      char payload[1000];
    return "{\r\n\t\"topic\": \""+hubNamespace+"/"+deviceId+"/things/twin/commands/modify\",\r\n\t\"path\": \"/features/"+moduleName+"\",\r\n\t\"value\": {\r\n\t \t\"definition\": [\r\n\t \t\t\"org.eclipse.hawkbit.swmodule:SoftwareModule:1.0.0\"\r\n\t \t],\r\n \t \t\"properties\": {\r\n \t\t \t\"status\": {\r\n \t\t\t \t\"moduleName\" : \""+moduleName+"\",\r\n \t\t\t \t\"moduleVersion\" : \""+version+"\",\r\n \t\t\t \t\"status\" : {\r\n\t\t\t\t\t\"correlationId\": \""+correlationId+"\",\r\n\t\t\t\t\t\"operation\": \"install\",\r\n\t\t\t\t\t\"status\": \""+state+"\"\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n}";
}

String buildInstallationState(String correlationId, String moduleName, 
    String version, String state, String message) {
    return "{\r\n\t\"topic\": \""+hubNamespace+"/"+deviceId+"/things/twin/commands/modify\",\r\n\t\"path\": \"/features/"+moduleName+"\",\r\n\t\"value\": {\r\n\t \t\"definition\": [\r\n\t \t\t\"org.eclipse.hawkbit.swmodule:SoftwareModule:1.0.0\"\r\n\t \t],\r\n \t \t\"properties\": {\r\n \t\t \t\"status\": {\r\n \t\t\t \t\"moduleName\" : \""+moduleName+"\",\r\n \t\t\t \t\"moduleVersion\" : \""+version+"\",\r\n \t\t\t \t\"status\" : {\r\n\t\t\t\t\t\"correlationId\": \""+correlationId+"\",\r\n\t\t\t\t\t\"operation\": \"install\",\r\n\t\t\t\t\t\"status\": \""+state+"\",\r\n\t\t\t\t\t\"message\": \""+message+"\"\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n}";
}

As always, you can find the complete sources on GitHub.

Conclusion

Voilà, we are at the end. Pushed to the master branch in Github will now create a new firmware image, that is automatically pushed to the Bosch IoT Suite, which in turn automatically assigns it to the device, the device will then self update.

There is a lot of room for improvement for this simple example. Let’s make a short list of a few points I would consider. I bet there are many more that you might come up with.

device Configuration not externalized

In the above code, the Wifi configuration as well as the device credentials are hard coded into the code. This should be externalized to the flash or a SD storage. This way the firmware image is not device specific anymore.

Resilience for firmware flashing

At the moment the firmware image is downloaded and directly streamed to the ESP32’s unused firmware partition. Since the ESP32 uses a two partition concept for firmware update and checks the partition, we already have some resilience (OTA Process Documentation. It would still be good to download the file and verify the downloaded image against the checksum.

Also having a potential rollback after updating in case that the device is not able to for example connect would be nice. But this obviously also necessitates to externalize the update state to the flash or SD storage.

Code Cleanup

I tried to keep the whole code in a single file, to make it easier to understand for a person reading the blog. In a real production setting it would of course be better to separate the different concers into separate components.

Of course other things like the BAUD Rate would be more suitable in a #define statement.

State Handling

The way that the install state is handled is definitely not a good way to do it in production. There are multiple states that aren’t handled properly. In a real production scenario you should use a proper state machine to hande this, or maybe use RTOS task scheduling to handle the state transitions.

Thanks for reading this long blog post. I hope you were able to take away some information or ideas.