File-based MQTT emulation

Introduction

This communication package emulates a function of an MQTT broker by storing the sent messages in form of files in directories of a file system, and/or providing the messages stored in the file system to the receivers.

See Using communication packages for instructions on how to select the communication package, and configure it.

The file-based MQTT emulation is in the MQTT client role only.

The type name of the message channel object is: OpcLabs.BaseLib.Communication.Mqtt.FileMqttMessageChannel,OpcLabs.BaseLib (in QuickOPC publisher&subscriber, the namespace is automatically imported, and the type name can therefore be shortened to just FileMqttMessageChannel).

The communicate package stores messages in files. The files are are organized in directories under the specified "root". The names of directories under the "root" and their structure correspond to the names and structure of MQTT topics.

The file naming scheme is such that the message order corresponds to the order of the file names, across the whole directory structure. The file name includes a "tick" when the message channel started, and a sequential message number. This approach preserves the correct message order even if the application is restarted. The naming scheme is suitable for not more than one application publishing to any topic. If more applications publish to the same topic, the file name order will no longer correspond to message order.

Both communication directions, i.e. message publishing and subscribing to topics, are implemented.

Implicit Usage in MQTT Transport

When you use a file URI for the connection resource URI in OPC UA PubSub (either by specifying "file:" scheme, or implicitly with URI strings such as "\\MyPath"), and the specified path either has no file extension or has an extension of ".mqtt" (case-insensitive), QuickOPC assumes that the transport profile is for the MQTT protocol and that the file-based MQTT emulation should be used. The path specified in the file URI , and uses the specified path as the "root" directory of the MQTT storage.

Besides the ability to completely swap in the file-based MQTT emulation for the message channel (and therefore replacing the use of "real" MQTT broker by the file-based emulation), the MQTT transport in QuickOPC also supports "hooking" the file-based MQTT emulation to the existing send/receive pipeline used by the transport. This means that you can store file-based copies of messages sent to and/or received from the MQTT broker, without affecting the communication. The file-based copies can then be used for troubleshooting, replay, archival, or other purposes.

The MQTT transport will automatically store a copy of the generated messages into the file system, when its {http://opclabs.com/OpcUA/PubSub}MqttFileCopySentRoot property is set a non-empty string containing the root directory of where the messages should be stored. The property namespace can be shortened to a pre-defined replacement variable {OpcLabs} (but not in SDK-level tools like the OPC UA Demo Publisher).

The MQTT transport will automatically store a copy of the received messages into the file system, when its {http://opclabs.com/OpcUA/PubSub}MqttFileCopyReceivedRoot property is set a non-empty string containing the root directory of where the messages should be stored. The property namespace can be shortened to a pre-defined replacement variable {OpcLabs} (but not in SDK-level tools like the OPC UA Demo Publisher).

Opening the channel

URL string

When the UseInputUrlString is 'false' (the default), the URL string, if provided by the host software, is ignored.

When the UseInputUrlString is 'true', the URL string must have the "file:" scheme, and is interpreted as the physical root directory.

Interface name

The interface name, if provided by the host software, is ignored.

Configuration

The table below lists the available properties, their types, and descriptions.

Property	Type	Description
ClearRootDirectory	System.Boolean	Determines whether the root directory will be cleared (recursively) when the message channel starts. Default is 'false'. BE CAREFUL! Do not accidentally clear an unintended directory with this setting.
CreatePhysicalRootDirectory	System.Boolean	Determines whether the physical root directory will be created automatically (if it does not exist) when the message channel starts. Default is 'true'.
PhysicalFileProviderRoot	System.String	The directory path in the physical file system where the root topic will reside. Default is "/MqttRoot". The value form this property is only used when the UseInputUrlString is 'false' (the default).
UseInputUrlString	System.Boolean	Determines whether the URL string provided by the host, which must have the "file:" scheme, will be interpreted as the physical root directory. Default is 'false'.

Path Length Considerations

The file-based MQTT emulation transforms MQTT topic names (paths) to paths (directories) in the file system. Topic paths in MQTT can sometimes be very long, which means that the resulting paths in the file system can also be very long. Operating systems and file systems have limitations on maximum path lengths, and file-based MQTT emulation will not work correctly if the maximum path length is exceeded.

On Windows, normal file paths are limited to 260 characters. You can overcome this limitation by using the "\\?\" or "\\?\UNC\" prefix. For example, instead of specifying the physical root directory as "C:\MyMqtt", you can use "\\?\C:\MyMqtt". For more information, see Maximum Path Length Limitation.

Note that the "\\?\" and "\\?\UNC\" prefixes cannot be used in the "file:" URL. When you specify the root directory using the URL, however, the file-based MQTT emulation automatically extends the path with them.

Examples

OPC UA Demo Publisher

Publish to File Storage Only

You can instruct the UADemoPublisher utility to produce JSON messages for MQTT and store them into a directory/file structure under "C:\MqttRoot" using the following command:

publish --MqttJsonTcp --ConnectionProperty {http://opclabs.com/OpcUA/PubSub}MessageChannel!=[String]FileMqttMessageChannel

QuickOPC software automatically chooses the file-based MQTT emulation if the MQTT transport is specified, and the URL scheme is "file:" (in this case, the UseInputUrlString property is also automatically set to 'true'). It is therefore possible to achieve the same result with the following command:

publish --MqttJsonTcp --ConnectionResourceUri file:///MqttRoot

Publish to Broker, Copy to File Storage

It is also possible to publish simultaneously to the real MQTT broker, and store a copy of the generated messages into the file system. For example:

publish --MqttJsonTcp --ConnectionProperty {http://opclabs.com/OpcUA/PubSub}MqttFileCopySentRoot=[String]C:\MqttSent

When the {http://opclabs.com/OpcUA/PubSub}MqttFileCopySentRoot property (can be shortened to {{OpcLabs}}MqttFileCopySentRoot) is set in the MQTT transport, the transport automatically creates a "tee" in the send pipeline, and connects a file-based MQTT emulation on the specified root directory to it.

OpcCmd Utility

Subscribe to File Storage

You can subscribe to messages provided from the file storage - for example, those that you have saved earlier (see below), for troubleshooting. For example:

uaSubscriber subscribeDataSet file:///MqttReceived #

Subscribe to Broker, Copy to File Storage

You can subscribe to messages from an MQTT broker, and at the same time store a copy of the received message into the file system. This can be a big help for troubleshooting. For example:

uaSubscriber subscribeDataSet mqtt://opcua-pubsub.demo-this.com # --ConnectionProperty {{OpcLabs}}MqttFileCopyReceivedRoot=[String]\\\\?\C:\MqttReceived

When the {http://opclabs.com/OpcUA/PubSub}MqttFileCopyReceivedRoot property (can be shortened to {{OpcLabs}}MqttFileCopyReceivedRoot) is set in the MQTT transport, the transport automatically creates a "tee" in the receive pipeline, and connects a file-based MQTT emulation on the specified root directory to it.

Imperative Programming

Subscribe to File Storage

With the EasyUASubscriber class, you can subscribe to messages provided from the file storage - for example, those that you have saved earlier (see below), for troubleshooting. For example:

// Define the PubSub connection we will work with. By specifying a file (file URI) with the directory path, MQTT 
// messages will be provided from the file storage.
UAPubSubConnectionDescriptor pubSubConnectionDescriptor = @"C:\MqttReceived";

Subscribe to Broker, Copy to File Storage

You can also subscribe to messages from an MQTT broker, and at the same time store a copy of the received message into the file system, e.g. for troubleshooting. In order to achieve that, you need to configure the PubSub connection (using the UAPubSubConnectionDescriptor) accordingly, for example:

// Define the PubSub connection we will work with. Uses implicit conversion from a string.
// Default port with MQTT is 1883.
UAPubSubConnectionDescriptor pubSubConnectionDescriptor = "mqtt://opcua-pubsub.demo-this.com";
// Configure the PubSub connection so that a copy of the received messages is stored into the file system, under
// the specified "root" directory.
pubSubConnectionDescriptor.CustomPropertyValueDictionary[
    new UAQualifiedName("{OpcLabs}", "MqttFileCopyReceivedRoot")] = @"C:\MqttReceived";