Persistence of outgoing packets in case of network loss (buffering)

[MathiasKoch]

Motivation

We need a way to allow users to persist outgoing MQTT packets to a non-volatile storage space, in case of network loss for longer periods. Ideally this should be a full opt-in feature.

Basic idea would be to make it two steps.

Persistent buffering to external flash

Work in progress i am thinking something along the lines of a key-value store, highly inspired by the persistence layer in Paho:

use embedded_storage::ReadWriteStorage;

pub enum PersistenceError {

}

pub struct Key();

pub trait MqttPerisistence: ReadWriteStorage {
    type Packet: MqttPayload;

    /// Initialize the persistent store.
    ///
    /// Either open the existing persistent store for this client ID or create a new
    /// one if one doesn't exist. If the persistent store is already open, return 
    /// without taking any action.
    ///
    /// An application can use the same client identifier to connect to many
    /// different servers. The clientid in conjunction with the 
    /// server_uri uniquely identifies the persistence store required.
    fn open(&mut self) -> Result<(), PersistenceError>;

    /// Close the persistent store referred to by the handle.
    fn close(&mut self) -> Result<(), PersistenceError>;

    /// Put the specified data into the persistent store.
    fn put(&mut self, key: Key, data: Self::Packet) -> Result<(), PersistenceError>;

    /// Retrieve the specified data from the persistent store.
    fn get(&mut self, key: Key) -> Result<Self::Packet, PersistenceError>;

    /// Remove the data for the specified key from the store
    fn remove(&mut self, key: Key) -> Result<(), PersistenceError>;

    /// Returns the keys in this persistent data store.
    fn keys(&mut self) -> impl Iter<Key>;

    /// Clears the persistence store, so that it no longer contains any persisted data.
    fn clear(&mut self) -> Result<(), PersistenceError>;

    /// Returns whether any data has been persisted using the specified key.
    fn contains(&mut self, key: Key) -> bool;


    /// A callback which is invoked just before a write to persistence.  This can be
    /// used to transform the data, for instance to encrypt it.
    fn before_write(&mut self, key) -> Result<(), PersistenceError> {};

    /// A callback which is invoked just after a read from persistence.  This can be
    /// used to transform the data, for instance to decrypt it.
    fn after_read(&mut self) -> Result<(), PersistenceError> {};
}

[MathiasKoch]

Relevant: rust-embedded/embedded-hal#241
Should be based on https://github.com/rust-embedded-community/embedded-storage/

[MathiasKoch]

Have a look at how Paho MQTT does it in C: https://github.com/eclipse/paho.mqtt.c/blob/master/src/MQTTClientPersistence.h