WAL performance

[potrusil-osi]

Hi sakno,

I've been testing performance of a distributed application built on top dotNext/Raft and ran into couple of issues with the current implementation of WAL. I'm using DotNext.AspNetCore.Cluster v3.0.2 and custom binary serialization through the command interpreter framework.

1. The biggest issue has been the compaction into a snapshot. When I configure everything optimally it takes around 100ms which is a huge toll on overall performance. The snapshot file is around 10MB and given the application I have, that's the lowest size I can get. In order to keep the compaction around 100ms it needs to be called often, and so the partition size is only 100 entries. I have two questions:
   1. Why is compaction running during AppendEntries message processing? Could it be moved into background?
   2. The buffer used for snapshot file creation is the same buffer that is used for writing entries into partition files. That doesn't seem to be optimal. I have observed that the ideal buffer size for the snapshot file is 1MB, while the ideal buffer size for the partition files is 15kB. Do you understand why there is such a difference? Could two different buffer sizes be supported?
2. The second issue occurs when on the leader node multiple threads try to append uncommitted entries concurrently (without any break). Understandably, there is a lock in PersistentState.AppendAsync. Since I'm testing with 30 client threads the average time to only lock the audit trail is around 40ms. Do you have any suggestions how to deal with such a problem?

[sakno]

Hi @potrusil-osi ,

Let's start from things that can be done easily:

1. How is your binary serialization implemented? If your log records structured with the fields of primitive types then methods such as IAsyncBinaryWriter.WriteInt32Async may slow down the performance. If the size of single log records is less than 1 KB then you can serialize the object into memory. IAsyncBinaryWriter provides necessary methods for that:
   1. ValueTask WriteAsync<TArg>(Action<TArg, IBufferWriter<byte>> writer, TArg arg, CancellationToken token = default) allows to obtain buffer writer allocated by IAsyncBinaryWriter. Usually, it is acquired from the pool. Also, you can cache writer delegate instance to prevent allocations for each serialization. IBufferWriter has many synchronous extension methods for binary serialization located in DotNext.Buffers namespace so the migration will be simple.
   2. ValueTask WriteAsync(ReadOnlyMemory<byte> input, LengthFormat? lengthFormat = null, CancellationToken token = default) allows to write a raw block of memory. This method gives you full control of memory allocation mechanism.
2. 100 records per partition probably is too many. Log compaction performance depends on the number of log records in the partition. Try to set it to something between 5 and 10
3. Have you tried InitialPartitionSize configuration parameter? It allows to pre-allocate partition files on disk and reduces fragmentation of partition files.

Now let's take a look at existing implementation of PersistentState:

1. Background compaction is challenging for a few reasons. First of all, the rate of incoming log entries can be very high and compaction will not have enough time to erase unused partitions. In this case, disk space can grow infinitely. Another problem here is lock state: during compaction I need to have read lock on all partitions. Read lock prevent appending new entries. As a result, AppendEntries have to wait for compaction.
2. Splitting buffer size doesn't seem to be hard. I can do that. However, it's very strange that buffer size has different performance for partitions and snapshot file. Let's return back to this point when you are done with the first part.
3. Concurrent writes require to change a whole architecture of existing PersistentState class. And I'm still not sure that it's possible. Log records stored in partition file sequentially. Without knowledge of the size of previous log record I can't calculate the offset in the file. Moreover, two log record with the same log index cannot be stored simultaneously.

If you have clear ideas of how to do concurrent writes on disk without lock then you can implement IPersistentState from scratch or make PR.

[sakno]

You can check develop branch for this functionality. Unfortunately, I can't tell you ETA of release at this moment.

[potrusil-osi]

I think there is a bug in SkipAsync when StreamSegment is the stream used by the reader - updating its Position doesn't update the position of the underlying stream.

[potrusil-osi]

Or maybe I'm wrong where the problem is. But I'm getting EndOfStreamException when an entry is being read after the previous one has been skipped.

[potrusil-osi]

It is either EndOfStreamException or OutOfMemoryException. Both appear to be the result of a stream corruption when the entry is being read/deserialized.

[sakno]

I would suggest to move further discussions to #57. I'll check the problem with SkipAsync method.