Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modules/zstd: Add sequence decoding #1362

Draft
wants to merge 53 commits into
base: main
Choose a base branch
from
Draft

modules/zstd: Add sequence decoding #1362

wants to merge 53 commits into from

Commits on Jul 24, 2024

  1. modules/zstd: Add buffer library 

    This commit adds a DSLX Buffer library that provides the Buffer struct,
and helper functions that can be used to operate on it. The Buffer
is meant to be a storage for data coming from the channel. It acts like
a FIFO, allowing data of any length to be put in or popped out of it.
Provided DSLX tests verify the correct behaviour of the library.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    bccd3d3 View commit details
    Browse the repository at this point in the history

  2. modules/zstd: Add Buffer use-case example 

    This commit adds a simple test that shows, how one can use the Buffer
struct inside a Proc.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    04bf3e8 View commit details
    Browse the repository at this point in the history

  3. modules/zstd: Add library for parsing magic number 

    This commit adds the library with functions for parsing a magic number and
tests that verify its correctness.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    a43fa9c View commit details
    Browse the repository at this point in the history

  4. modules/zstd: Add library for parsing frame header 

    This commit adds the library with functions for parsing a frame header.
The provided tests verify the correcness of the library.

Internal-tag: [#49967]
Co-authored-by: Roman Dobrodii <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
    @rw1nkler @rdob-ant
    @lpawelcz @mtdudek
    3 people authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    d3048bf View commit details
    Browse the repository at this point in the history

  5. dependency_support/libzstd: Make zstd_errors.h public 

    Required for expected_status inference in C++ tests for ZSTD decoder
components

Internal-tag: [#53465]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    86de6a7 View commit details
    Browse the repository at this point in the history

  6. dependency_support: Add decodecorpus binary 

    Internal-tag: [#50967]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    3208de7 View commit details
    Browse the repository at this point in the history

  7. modules/zstd: Add data generator library 

    This commit adds a binary that calls decoding to generate data and loads
it into a vector of bytes.

Internal-tag: [#50967]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    9375be5 View commit details
    Browse the repository at this point in the history

  8. modules/zstd: Add zstd frame header tests 

    Internal-tag: [#50967]
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
    @rw1nkler @lpawelcz @mtdudek
    2 people authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    c65b32c View commit details
    Browse the repository at this point in the history

  9. modules/zstd: Add common zstd definitions 

    Internal-tag: [#51343]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    236d96e View commit details
    Browse the repository at this point in the history

  10. modules/zstd: Add raw block decoder 

    Internal-tag: [#51343]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4fecaca View commit details
    Browse the repository at this point in the history

  11. modules/zstd: Add rle block decoder 

    Adds RleBlockDecoder responsible for decoding Blocks
of RLE_Block Block_Type as specified in RFC 8878, paragraph 3.1.1.2.2.
https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2.2

RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

 * RleDataPacker
 * BatchPacker

Which are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

Internal-tag: [#51473]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    6f2d89b View commit details
    Browse the repository at this point in the history

  12. modules/zstd: Add block header parsing library 

    Internal-tag: [#51343]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4bd9c40 View commit details
    Browse the repository at this point in the history

  13. modules/zstd: Add SequenceExecutorPacket to common definitions 

    Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    3de695e View commit details
    Browse the repository at this point in the history

  14. modules/zstd: Add block data muxer library 

    This commit adds DecoderMux Proc, which collects data from specialized
Raw, RLE, and Compressed Block decoders and re-sends them in the correct order.

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    1c525a6 View commit details
    Browse the repository at this point in the history

  15. modules/zstd: Add block demuxer library 

    This DSLX proc responsibility is to dispatch encoded blocks to a correct
decoder: RAW, RLE, COMPRESSED.

It tracks and assigns block IDs.
The ID counter is reset on the frame's last block on the last data packet.

Internal-tag: [#51736]
Co-authored-by: Robert Winkler <[email protected]>
Signed-off-by: Maciej Dudek <[email protected]>
    @mtdudek @rw1nkler
    mtdudek and rw1nkler committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    27573e2 View commit details
    Browse the repository at this point in the history

  16. modules/zstd: Add block decoder module 

    This adds a decoder of block data. It decodes block header and
demuxes remaining input data into one of specific block decoders
depending on the type of the parsed block. Then it muxes outputs
from those decoders into single output channel.

Internal-tag: [#51873]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    323214e View commit details
    Browse the repository at this point in the history

  17. modules/zstd/common: Specify decoder output format 

    Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4f19952 View commit details
    Browse the repository at this point in the history

  18. examples/ram: Export internal RAM API to other modules 

    This commit marks SimultaneousReadWriteBehavior enum and
num_partitions function as public to allow for creating
simpler tests that interact with RAM models.

Internal-tag: [#53241]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    410d78c View commit details
    Browse the repository at this point in the history

  19. modules/zstd: Add Offset type to common zstd definitions 

    Internal-tag: [#54705]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    0b613d2 View commit details
    Browse the repository at this point in the history

  20. modules/zstd: Add RamPrinter Proc 

    This commit adds RAM printer block usefull for debugging
HistoryBuffer inside SequenceExecutor.

Internal-tag: [#54705]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    0ca18c4 View commit details
    Browse the repository at this point in the history

  21. modules/zstd: Add SequenceExecutor Proc 

    Add Proc responsible for handling ZSTD Sequence Execution step,
which is described in:
https://datatracker.ietf.org/doc/html/rfc8878#name-sequence-execution

Internal-tag: [#54705]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek
    rw1nkler authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    02fc78c View commit details
    Browse the repository at this point in the history

  22. modules/zstd: Add repacketizer 

    Internal-tag: [#52954]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    43a637d View commit details
    Browse the repository at this point in the history

  23. modules/zstd: Add ZSTD decoder 

    This commit adds a ZSTD decoder module that parses ZSTD frames.
The provided tests examine the model using C++ API, which is
a prerequisite for detailed tests using zstd library.

Internal-tag: [#50221]
Co-authored-by: Maciej Dudek <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Maciej Dudek <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @mtdudek @lpawelcz
    3 people committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    96b6efd View commit details
    Browse the repository at this point in the history

  24. modules/zstd: Add ZSTD Decoder documentation 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4c3b07c View commit details
    Browse the repository at this point in the history

  25. examples/ram: use assert!() instead of assert_eq() in RamModel 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    57afc0f View commit details
    Browse the repository at this point in the history

  26. CI: Add custom ZSTD module workflow 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    ab5fb05 View commit details
    Browse the repository at this point in the history

  27. [FIXUP] Update copyrights notice and add missing license header 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    495353d View commit details
    Browse the repository at this point in the history

  28. [FIXUP] Update README 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    593e40c View commit details
    Browse the repository at this point in the history

  29. [FIXUP] Add manual tags to test rules 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    dc6cb3c View commit details
    Browse the repository at this point in the history

  30. [FIXUP] restore original main CI workflow 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    d7463ba View commit details
    Browse the repository at this point in the history

  31. [FIXUP] Explicitly run test targets in ZSTD CI workflow 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    b46ee8c View commit details
    Browse the repository at this point in the history

  32. [FIXUP] document the usage of libzstd dependency 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    9ddd7ce View commit details
    Browse the repository at this point in the history

  33. [FIXUP] return reference to the data vector 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    9483420 View commit details
    Browse the repository at this point in the history

  34. [FIXUP] fix RLE block handling in decodecorpus 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    a16c27a View commit details
    Browse the repository at this point in the history

  35. [FIXUP] fix generating data when we don't force the block type 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    8c350b5 View commit details
    Browse the repository at this point in the history

  36. modules/zstd/frame_header_test: Reproduce bugs 

    Internal-tag: [#60906]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    5c5011c View commit details
    Browse the repository at this point in the history

  37. [FIXUP] return expect empty buffer on the unsupported windows size error 

    Internal-tag: [#60906]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    cc33b0b View commit details
    Browse the repository at this point in the history

  38. [FIXUP] Handle special case of difference between libzstd and ZSTD de… 

    …coder

Internal-tag: [#60906]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    2aef289 View commit details
    Browse the repository at this point in the history

  39. [FIXUP] Add DSLX test case for unsupported FCS 

    Internal-tag: [#60906]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    14fe3f0 View commit details
    Browse the repository at this point in the history

  40. [FIXUP] fix max_window_size calculation types 

    Internal-tag: [#52186]
Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e3804ff View commit details
    Browse the repository at this point in the history

  41. [FIXUP] dependencies cleanup 

    Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    cdeff69 View commit details
    Browse the repository at this point in the history

  42. [FIXUP] shorten test case names in frame_header tests 

    Signed-off-by: Pawel Czarnecki <[email protected]>
    @lpawelcz @mtdudek
    lpawelcz authored and mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    210e322 View commit details
    Browse the repository at this point in the history

  43. [FIXUP] Update to zstd-top 

    Signed-off-by: Maciej Dudek <[email protected]>
    @mtdudek
    mtdudek committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    ba7eda4 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. modules/zstd: Add buffer implementing desired API 

    Add a buffering proc that implements the desired API of the final buffering mechanism.
A more efficient implementation will be provided later.

Internal-tag: [#55149]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @m-torhan
    rw1nkler authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    f758928 View commit details
    Browse the repository at this point in the history

  2. modules/zstd: Add common FSE definitions 

    Internal-tag: [#57353]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @m-torhan
    rw1nkler authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    934ef92 View commit details
    Browse the repository at this point in the history

  3. modules/zstd: Add a proc handling RAM write completion 

    Internal-tag: [#57353]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @m-torhan
    rw1nkler authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    1fc12f1 View commit details
    Browse the repository at this point in the history

  4. modules/zstd: Add FSE probability frequency decoder 

    Internal-tag: [#57353]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @m-torhan
    rw1nkler authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    aab399d View commit details
    Browse the repository at this point in the history

  5. modules/zstd: Add proc for generating symbol order for FSE lookup 

    Internal-tag: [#57353]
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @m-torhan
    rw1nkler authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    807275b View commit details
    Browse the repository at this point in the history

  6. modules/zstd: Add CommandConstructor proc 

    Internal-tag: [#58422]
Signed-off-by: Ryszard Rozak <[email protected]>
    @RRozak @m-torhan
    RRozak authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    4935945 View commit details
    Browse the repository at this point in the history

  7. modules/zstd: Add Ram demux 

    Internal-tag: [#58557]
Co-authored-by: Ryszard Rozak <[email protected]>
Co-authored-by: Maciej Torhan <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
    @rw1nkler @RRozak @m-torhan
    3 people committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    4b23a24 View commit details
    Browse the repository at this point in the history

  8. modules/zstd: Add FSE table creator 

    Internal-tag: [#57353]
Signed-off-by: Robert Winkler <[email protected]>
    @RRozak @m-torhan
    RRozak authored and m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    c15b8d4 View commit details
    Browse the repository at this point in the history

  9. modules/zstd: Update FSE branch 

    Signed-off-by: Maciej Torhan <[email protected]>
    @m-torhan
    m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    7c0d935 View commit details
    Browse the repository at this point in the history

  10. modules/zstd: Add FseDecoder implementation 

    Signed-off-by: Maciej Torhan <[email protected]>
    @m-torhan
    m-torhan committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    9b273b2 View commit details
    Browse the repository at this point in the history