Add a new perlrun option "-j" which allows JSON parsing in line

[catouc]

With this option we basically extend the already existing "-n" and "-p" flags to handle JSON using the inbuilt "JSON::PP" module. Basically we decode the STDIN from "$" with decode_json into $ again so you can run something like:

sudo journalctl -f -o json | ./perl -njle 'if ( $_->{"_GID"} == 0 ) {print $_->{"MESSAGE"}}'

To quickly filter out all root processes, this would make adhoc log analysis of most modern applications a more convenient.

I am by no means a Perl expert so this probably needs some adjustment to fit in, I also at this point have no idea where I would stash some tests for this, but it might be worth discussing merit before spending the time testing, the change works well enough for me now that the idea is reviewable.

---

• This set of changes requires a perldelta entry, and I need help writing it.

[Leont]

While useful, I'm not sure this belongs in perl itself. Can't this be a separate tool?

[guest20]

@Leont a whole separate tool that does all of perl plus one per-line calls to encode_json/decode_json?

Sounds like a pretty big tool to maintain, why you'd need a whole mailing list full of people for 20 to 30 years to build something like that ...

[guest20]

Not only does this does step into areas of recent discussion, wherein folks mention wanting -n/-p to not:

• trim white space in the file name list
  • perl -n mode eats trailing spaces in filenames #22112
• do a 2 arg open / accidentally be a piped open / warn about it being a piped open
  • <> gives strange warnings when '-|' is in @ARGV #21176
  • readline ARGV: don't try to open '|-' or '-|' and warn #21181

But it also coincides nicely with giving perl built-in HTTPS (#20739), so it might even make sense to go further and have -n/-p be able to do http requests!

% perldoc perlrun |grep '[::madeup_args::]'
  -j json_decode each line
  -h do http requests for -n/-p instead of reading local files
  -00 slurp 

# Get all my github notifications, but from dotcom, and the work github:
% perl -n -00 -j -E' say $_->{subject}{title} for @{$data}' -- \
   https://api.github.com/notifications \
   https://github.brand.com/api/notifications

... maybe -i here could also allow for POSTing back $data to the place it came from?

I think it's safe to assume this "The World Wide Web" thing is here to stay.

[Leont]

Sounds like a pretty big tool to maintain, why you'd need a whole mailing list full of people for 20 to 30 years to build something like that ...

That's not what I meant. What if you have a perl script that does exactly what you just described. It wouldn't even be a big script really.

[guest20]

@Leont In the past I've found that turning a one-liner into a script involves writing a decent chunk of software - those switches do a lot of stuff for you. A Deparse won't produce a script where you can turn -i on and off - it produces a script with -i hard-coded on, or hard-coded off ...

Also, it seems worth asking why perl-proper is the wrong place for a feature like this when autosplit (-a and -Fpattern and $,) does a very similar thing, just more ... CSVishly

[tonycoz]

If we add this do we also add similar support for YAML, XML, etc?

The line oriented processing has been there (probably) since perl 1, I know I used it in perl3.

[guest20]

I don't think you can do yaml with per-line, but -f--- might do it

[tonycoz]

Kind of ugly:

package JQ;
use v5.36;
use Filter::Util::Call;

sub import {
  my $done = 0;
  filter_add(
    sub {
      unless ($done) {
        # slurp in the whole source
        my $acc = "";
        while ((my $status = filter_read()) > 0) {
          $acc .= $_;
        }
        $acc = <<'PREFIX' . $acc . <<'SUFFIX';
use JSON::PP;
$_ = JSON::PP::decode_json($_);
#line 1
PREFIX
;
$_ = JSON::PP::encode_json($_);
SUFFIX
        $_ = $acc;
        ++$done;
        return 1;
      }
      else {
        return 0;
      }
    });
}

1;

and:

# journalctl -f -o json | perl -I... -MJQ -nle 'if ( $_->{"_GID"} == 0 ) {print $_->{"MESSAGE"}}'
Invalid user xa from x.x.x.x port 46424
pam_unix(sshd:auth): check pass; user unknown
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x
Failed password for invalid user x.x.x.x from x.x.x.x port 46424 ssh2
Received disconnect from x.x.x.x port 46424:11: Bye Bye [preauth]

[bulk88]

https://github.com/Perl/perl5/blame/blead/utils/json_pp.PL this exists, IDK what it does. Perl's bundled -x perm .pl standalone apps, egh, they "are there" but are woefully unused nowadays except for cpan. That can change with improvements, but perhaps a bundled .PL with a #/bin instead of using a a cmd lane larg should be used. p5 is running low on signle chr flags in its cli.

[Leont]

https://github.com/Perl/perl5/blame/blead/utils/json_pp.PL this exists, IDK what it does.

Most utils are really scripts from core modules, in this case JSON::PP

[guest20]

"We're saving up switches just in case"

If nobody's needed -j for something in the last 30 years, it might be safe to make it something useful in the year of our lord 2024

json_pp exists (and is installed via an executable script in a .PL)

... it doesn't do any of the things this PR does.

It also raises the question about porting the functionality to the scripts in assorted, non-core JSON::Any::CPanel::XS::Fast::Tiny too?

It's even mentioned earlier in this thread that round tripping other formats in this situation would be equally handy... That definitely doesn't fit the "Maybe jam it into json_pp" paradigm.

"You can do it with a source filter"

This one is the only usable suggestion, and it's a source filter. The religious teachings tell me I should hate and fear source filters, that it's correct and prudent to shun them. On the other hand I am enchanted by their heady and seductive power.

I have mixed feelings about this.

[catouc]

After a bit of thinking, I do think it is worth to use up -j for a data format that is not likely going to go away anytime soon.
I have driven this code for a bit at my daily work and it's been making adhoc log slicing and data manipulation a lot nicer.

I'm not sure what's necessary for a consensus here, but I will definitely compile this change into my own Perl for the time being.

[tonycoz]

For this type of change you're probably better off asking on the list and perhaps going through the PPC process.