Auto flush on multiline codec not working

[ghost]

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: 7.6.2
• Operating System: Ubuntu 18.04.3 LTS
• Config File (if you have sensitive info, please remove it):

input {
    tcp {
        port => 10030
        codec => multiline {
                     pattern => "new_line"
                     negate => true
                     what => "previous"
                     auto_flush_interval => 1
                 }
    }
}
filter {

}
output {
    stdout {}
}

• Steps to Reproduce:
  • Write any line to tcp port. For example via nc localhost 10030 and line "some_test_line"
  • Nothing appear in stdout - auto flush is not working

[mj84]

I can confirm this with 7.10.0 using a tcp input as well.
4 years ago, someone mentioned that only file and beats inputs were supported with the multiline codec:
https://discuss.elastic.co/t/solved-auto-flush-interval-not-flushing-on-udp-input/60387

The current documentations of the tcp input or multiline codec don't mention such a limitation.

This is rather a limitation of the multiline codec and not the tcp input, and there is also an open issue for that:
logstash-plugins/logstash-codec-multiline#57

There also was a pull request work in progress regarding auto_flush, but it is closed and won't be merged for now:
logstash-plugins/logstash-codec-multiline#63

This issue provides a good roadmap in the last post which explains that an "input_type" option will be introduced for csv codec and later for the multiline codec:
elastic/logstash#11885

Sorry for pinging all the issues and PRs.
I though it was a good idea to summarize the state of auto_flush when using multiline with tcp since I ran into this behaviour today.
Google brought me to this issue, hopefully this explains the situation to others facing the same challenge.