PQL regexp character is unrecognized

[QueerCodingGirl]

Does anyone else have problems with PQL queries with regular expressions in them?

I am getting this error whenever i try to use the tilde ~ to indicate the regex (i removed fields like time or request id from this log):

{"component":"discovery","level":"error","msg":"Could not compile compound query 'pql:nodes[certname] { facts.apt_package_updates ~ '.*squashfs-tools.*' }': unrecognized character: U+007E '~' (1:50)\n | pql:nodes[certname] { facts.apt_package_updates ~ '.*squashfs-tools.*' }\n | .................................................^","subsystem":"discovery"}

which is thrown here:
https://github.com/choria-io/go-choria/blob/main/filter/compound/compound.go#L83

caused by the module github.com/antonmedv/expr on this line:
https://github.com/antonmedv/expr/blob/master/parser/lexer/state.go#L48

[ripienaar]

But it works for other pql queries? The expr code shouldn’t even be hit at this point as it’s not a expr query. So there is probsbly some logic error.

Can you show the command you run to cause this pls?

[QueerCodingGirl]

Disclaimer: This might not be the exact same query as in the initial post, but still the same problem/error.

This command works fine and gets me results:

mco find -I 'pql:nodes[certname] { certname in inventory[certname] { facts.apt_package_updates ~ ".*squashfs-tools.*" }}'

while this command with the same query fails with the error mentioned above:

mco tasks run exec -I "pql:nodes[certname] { certname in inventory[certname] { facts.apt_package_updates ~ '.*squashfs-tools.*' }}" --dm choria --command "id" -v --failonfail false

[ripienaar]

Really weird, I cannot reproduce this at all. I saw another issue that prevents this from working - but what you are seeing really no idea how that error from compound even happened.

[QueerCodingGirl]

There has to be a different query handling on mco tasks run and mco find, because mco findworks and the other one does not. That could be one angle to get to a fix for this.

The second best solution would be to get ~ recognized by github.com/antonmedv/expr. But that does not seem like a good solution and the maintainer there would have to agree to put the character into the lib...

[ripienaar]

If you can, patch this:

diff --git a/lib/mcollective/rpc/client.rb b/lib/mcollective/rpc/client.rb
index 7739c994..6abc80fc 100644
--- a/lib/mcollective/rpc/client.rb
+++ b/lib/mcollective/rpc/client.rb
@@ -847,6 +847,7 @@ module MCollective
             message.requestid = @stats.requestid

             message.discovered_hosts = hosts.clone.compact
+            message.type=:direct_request

             @client.req(message) do |resp|
               respcount += 1

Should be in /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/choria-mcorpc-support-2.25.1/lib/mcollective/rpc/client.rb

I bet that fixes it?

[ripienaar]

The PQL query should never get anywhere near exprvar, I have no idea how that code path is followed. Expr cannot parse pql queries and PQL queries arent compound, so there's a major bug somewhere but I cant trigger that code path on my side

[QueerCodingGirl]

That fixes it! Can you release this patch? (I am using choria-mcorpc-support-2.25.2 btw. so a little newer than you, could help with the reproducing.)

I have a really special talent for finding bugs in open source software i just want to use... Happens to me way too often. :D

[ripienaar]

haha yeah this bug has been there for like a decade :)

the combination of PQL filters, batch mode (that the tasks agent use internally) triggers it. I really dont think anyone uses PQL queries in this sense really, seemed like a good idea at the time but dont think the UX really work.

choria-io/mcorpc-ruby-support#176

[ripienaar]

I suspect I will do a release only next month though, I could do the gem only unannounced I guess if you are desperate

[QueerCodingGirl]

I can get along with PQL, but just out of curiosity: How would the above filter look like when you would write it?

I am always desperate the moment i open a github issue/discussion 😆
I try to solve my problems / read the docs, for a long time before i do this

[ripienaar]

I can get close:

choria find -S "include(fact('apt_package_updates'), 'exact package name')"

Unfortunately the include() function doesnt support regex, but I can certainly add such a function!

choria find -S "include(fact('processors.models'), 'Intel Core Processor (Broadwell)')"

works exactly for me and this is searching insie a array

[QueerCodingGirl]

I believe that would be a useful feature. I certainly would love to have it.

I took your second command and changed it to match my original purpose and it works like a charm. It selects the same hosts as my long PQL.

mco tasks run exec -S "include(fact('apt_package_updates'), 'squashfs-tools')" --dm choria --command "id" -v

As always, thank you for your amazing help and work! ❤️

[ripienaar]

nice, certainly a lot nicer than PQL :)

I will look to ship that gem tomorrow for you

[ripienaar]

-S "any(fact('processors.models'), {# matches('ntel')})"

this does a regex match on all items in the array

[QueerCodingGirl]

Works like a charm! Thank you very much!

[ripienaar]

https://rubygems.org/gems/choria-mcorpc-support/versions/2.25.3