OPC UA File Transfer internals

This article describes certain internal aspects of OPC UA File Transfer implementation in QuickOPC.

Object type checking

Before performing the actual file transfer operation, QuickOPC libraries check whether the target object exists and is of the intended type (file or directory). Without this, some operations would not behave predictably and could return errors that do not describe the problem well.

Connection locking

OPC UA file handles are only valid in scope of the OPC UA session. If the client disconnects from the server and establishes a new session, the previously obtained file handle is no longer (guaranteed to be) valid. Since QuickOPC normally automatically disconnects from the server when the session is not needed, this could cause a problem if the file handle was needed over a longer time period.

The file transfer implementation prevents this issue by locking the connection (forcing the session to stay open) until the file handle is disposed of.

Note that "forced" session terminations, such as those caused by long-lasting network disconnections, can still render the OPC UA file handle invalid, which will manifest itself by persistent errors returned from operations made with that handle.

Stream buffering

In OPC UA, it is important to prevent "chatty" communication: Operations should be grouped together and performed on larger bodies of data, if possible. This is because the communication between the client and the server can be (relatively) slow, and each service calls introduces its own time delay.

When OPC UA files are exposed as (.NET) streams, the developer is free to read any number of bytes from the stream, or write any number of bytes to the stream. It is not uncommon that the algorithm reads or writes small number of bytes many times in a sequence. This could lead to very poor performance with OPC UA, because every read or write requires a separate OPC UA service call. There is, however, nothing intrinsically wrong in accessing the stream in this way, and many algorithms with streams are abstract, written in such a way that they are not aware of the ramifications of the underlying technology.

In order to prevent performance degradation when stream data is accessed in small segments, QuickOPC (unless instructed otherwise) automatically buffers all operations on OPC UA streams in memory. The default buffer size is 4096 bytes (in version 2021.2). Specifying 0 for buffer size (e.g. in the QuickOPC methods that open the stream) turns off the stream buffering.

Stream expansion

The .NET stream abstraction allows seeking to a position that is beyond the current file length, and when new data is then written to the stream, the stream is required to expand accordingly. The methods defined in OPC UA file transfer behave (per OPC specifications) differently; with them, seeking to a position beyond the current file length is invalid.

QuickOPC provides correct stream expansion behavior even for streams based on OPC UA files. It detects the situation and automatically writes the necessary padding into the file.

Read/write chunking

QuickOPC can automatically split read and write operations that work with large amounts of data into smaller pieces. This is important in case there are limitations to the message size transferred between the client and the server, or when the server itself cannot process or provide data (byte arrays) over certain size. Without chunking, perfectly legitimate reads or writes on an OPC UA file could fail for implementation reasons that are outside of developer's control.

The maximum chunk size is defined separately for reads and writes. QuickOPC has default values for the maximum chunk sizes, based on experiments and experience. In addition, QuickOPC can adjust the maximum chunk sizes down, using the read/write size limits obtained from the server, or using the actually observed server (communication) behavior (adaptively). This is described further below.

This functionality can be turned off by a corresponding parameter.

Note: In a way, read/write chunking is the opposite of stream buffering. However, the two functionalities do not normally collide, because the chunk size is bigger than the buffer size.

Read/write size limits

The OPC UA specification allows servers to "declare" the limits of the server (the maximum size it supports in file reads/writes) using predefined variables in the address space. In addition, the client and server negotiate the maximum message size that can be used during communication, and the maximum message size also influences the true maximum data size that the client can use.

QuickOPC obtains the read/write size limits using all available methods from the OPC UA server, and the communication stack. It also uses some heuristics to calculate the size limits in case the necessary information is missing or incomplete.

Whether QuickOPC obtains the read/write size limits from the server can be controlled by a corresponding parameter (it is turned on by default).

The size limits determined are (by default) cached in memory, so that the server does not have to be interrogated repeatedly. This functionality can also be turned off by a corresponding parameter.

Adaptive read/write sizes

Some OPC UA servers have limits regarding the size of data that be used with files reads and/or writes, but do not provide the values of these limits to the client. The user is the left "guessing" what the limits are, which can be quite unreliable.

In order to overcome this problem, QuickOPC detects errors that are caused by exceeding the read/write limits, and instead of returning the errors to the user, it attempts to reduce the size of data transferred (using read/write chunking), until a success is encountered (the full algorithm is more complex and involves various other limits and logic). The net result is that things "just work" - the user only sees the the fact the operation has succeeded.

This functionality can be turned off by a corresponding parameter.

Metadata caching and model changes

When working with OPC UA file transfer objects, it is not uncommon that some objects are accessed repeatedly. This can also happen when higher API levels (such as the file provider abstraction) are used. While the data itself (file contents) can change frequently, the file system metadata (information model) typically changes much less often, based on some actions originating in the client, or in the server itself.

OPC UA file transfer can therefore be made quicker by caching the file system metadata in memory. The metadata being cached by QuickOPC include information about existing file and directory objects, how their names correspond to node Ids, and similar information.

QuickOPC subscribes to model change events that are provided by some OPC UA servers, and invalidates the cache accordingly when the server reports a change that can affect the validity of the cached information. The caching can be restricted or disabled when the server is unable to provide the model change events. Additional invalidation also happens when QuickOPC also detects when file system is being changed the client.

This functionality can be controlled and/or turned off by corresponding parameters.