easy partial support for regexp -expanded

[The-Markitecht]

<gush>Been using jimsh every day for many years. It's clearly the ultimate evolution of Tcl. It should just be mainstreamed on tcl.tk instead of tclsh.</gush>

But, I used regexp -expanded all the time in tclsh (such as #310 (comment)).
jimsh never had it, and I can see why.

But every time I use any regexp, I have to start like this:
set re [string map {{ } {}} { ^ \s* (\S+) \s+ (\w+) \s+ (\d+ \. \d+ \. \d+ \. \d+ / \d+)+ \s* }]
That adds a LOT of braces and loss of legibility in order to write an expression that's supposed to have GREATER legibility. One step forward, two steps back.

I propose an easy solution: support regexp -expanded which simply deletes all the spaces, as in my example. Don't bother with the comment feature offered by the same option in tclsh. I've never needed it.

That solution should run fast as well. Much faster than my example. And not add much executable size at all.

And it doesn't have to be called -expanded. There's an argument for avoiding that since the feature isn't identical to tclsh. So it would be fine to call it something else, like -compress. But not having it at all is a real nuisance.

If you can't make time for this, I understand. My life is crazy right now, but I might be able to make time for this small PR.

[The-Markitecht]

Deleting also tabs, newlines, and other whitespace-class characters in the same pass is also a good idea. Then we can have multi-line regex's, which is occasionally useful for legibility.

[msteveb]

Thanks for the feedback and the suggestion. I pushed something to master-next that you can try

[The-Markitecht]

Huzzah!!!

I tested that. It's great!!

Spaces, newlines, tabs, comments all worked the way you intended in regexp (i looked at the source). I didn't test regsub.

And it's even documented!

THANK YOU!!! This has pestered me for YEARS, ever since the first week i used jimsh.

Along the way I also noticed in action several of the other improvements of this year. Excellent!!

One question. In this part:

@@ -993,6 +1014,12 @@ cc_switch:
 					break;
 				}
 
+				/* For REG_EXPANDED, if we hit white space, stop */
+				if ((preg->cflags & REG_EXPANDED) && n == 1 && strchr(" \t\r\n\f\v", ch)) {
+					preg->regparse += n;
+					break;
+				}

What case is that for? It wasn't obvious what its effect is.

[msteveb]

You're welcome. If we don't stop when we hit white space, it will become part of the exact match.

[The-Markitecht]

Here are some -expanded test cases to immortalize in this thread. It's probably also a good idea to add these to the official build-test scripts and maybe the documentation. Most of the behaviors are intuitive enough to accept as-is, as long as everybody understands them. So this can be a little cheat-sheet.

Welcome to Jim version 0.84
. info patchlevel
0.82-134-gdcea42c
. regexp -inline -expanded { bcd } {abcde}  ;# spaces at ends of pattern are correctly ignored.
bcd
. regexp -inline -expanded { b c d } {abcde}  ;# spaces between atoms are correctly ignored.
bcd
. regexp -inline -expanded { ^ a b b c d } {abbcde}  ;# spaces around caret are correctly ignored.
abbcd
. regexp -inline -expanded { ^ c d } {abbcde}  ;# caret is still effective.
. regexp -inline -expanded { ^ a b{1,3} c d } {abbcde}   ;# quantifier works as usual.
abbcd
. regexp -inline -expanded { ^ a b{1,3} c d } {abbbbbbcde}   ;# quantifier works as usual.
. regexp -inline -expanded { ^ a b {1,3} c d } {abbcde}  ;# spaces before the quantifier make it ineffective, and no error is thrown.
. regexp -inline -expanded { ^ a b {1,3} c d } {ab  cde}  ;# quantifier doesn't apply to that space character preceding it either, so it applies to nothing!
. regexp -inline -expanded { b { 1 , 3 } c d } {abcde}  ;# spaces are recognized as part of the quantifier, an illegal part.
couldn't compile regular expression pattern: braces {} not balanced
. regexp -inline -expanded { b [cjk] d } {abcde}  ;# character set works as usual.
bcd
. regexp -inline -expanded { b [c jk] d } {abcde}  ;# character set still works even with a space in it.
bcd
. regexp -inline -expanded { b [c jk] d } {ab de}  ;# space is recognized as part of the character set, and it's effective.
{b d}
. regexp -inline -expanded " b \n d " {abcde}
. regexp -inline -expanded " b \n c \n d " {abcde}  ;# newlines are correctly ignored.
bcd
. regexp -inline -expanded " b \n # c \n d " {abcde}  ;# comment at start of line is correctly ignored.
. regexp -inline -expanded " b \n c # comment \n d " {abcde}    ;# comment at end of line is correctly ignored.
bcd
. regexp -inline -expanded { a (bb) c d } {abbcde}  ;# capturing group works as usual.
abbcd bb
. regexp -inline -expanded { a ( bb ) c d } {abbcde}  ;# spaces around parens are correctly ignored.
abbcd bb
. regexp -inline -expanded { a ( b b ) c d } {abbcde}  ;# spaces between atoms inside parens are correctly ignored.
abbcd bb
. regexp -inline -expanded { a b \s b c d } {abbcde}  ;# spaces can still be specified using a character class (or a bracketed character set with just a space in it) and they're effective.
. regexp -inline -expanded { a b \s b c d } {ab bcde}  ;# space class is effective.
{ab bcd}
. regexp -inline -expanded { a ( b b )+ c d } {abbbbbbcde}  ;# group quantifier is still effective.  note only the final submatch was captured.
abbbbbbcd bb
. regexp -inline -expanded { a (?: b b )+ c d } {abbbbbbcde}  ;# non-capturing group quantifier is still effective.
abbbbbbcd
. regexp -inline -expanded { a (?: b b )+ } {abbbbbbcde}  ;# default greedy quantifier works as expected.
abbbbbb
. regexp -inline -expanded { a (?: b b )+? } {abbbbbbcde}  ;# non-greedy quantifier works as expected.
abb
. regexp -inline -expanded { a ( b b )+ } {abbbbbbcde}  ;# capturing group with greedy quantifier works as expected.  note only the final submatch was captured.
abbbbbb bb
. regexp -inline -expanded { a ( b b ) + } {abbbbbbcde}  ;# space before quantifier SEGFAULTS!!  issue #313
Segmentation fault
$ jimsh
Welcome to Jim version 0.84
. regexp -inline -expanded { a ( b b )+? } {abbbbbbcde}  ;# capturing group with non-greedy quantifier works as expected.  note only the final submatch was captured.
abb bb
. regexp -inline -expanded { a ( b b )+ ? } {abbbbbbcde}  ;# space in the middle of non-greedy quantifier makes it ineffective, and throws no error.
abbbbbb bb
. regexp -inline -expanded { a ( b b )+ ? c d e } {abbbbbbcde}  ;# include those trailing characters in the match, as baseline for the next test.
abbbbbbcde bb
. regexp -inline -expanded { a ( b b )+ ? c d e } {abbbbbb cde}  ;# no, the spurious question mark does not cause the preceding space to be recognized.  good.
. regexp -inline -expanded { a ( b b ( c d )+ b b ) e f } {abbcdcdcdbbef}  ;# spaces in nested parens work as expected.  note only the final submatch was captured.
abbcdcdcdbbef bbcdcdcdbb cd
. regexp -inline -expanded { a ( b b ( c d )+ b b ) e f } {abbcdcdcdef}  ;# absence of trailing bb (part of outer group) correctly rejected.
. regexp -inline -expanded { a ( b b ( c d )+ b b ) e f } {abbbbef}   ;# absence of inner group correctly rejected.

[msteveb]

Looks good. Even better would be a pull request with these tests added :-)

[The-Markitecht]

sure, i'll look in the test suite and see if that change is practical for me...

[mingodad]

Maybe someone here can help with a problem I'm having trying to use https://github.com/devnull42/fickle with jimtcl it seems to be related with this topic because it does work with tclsh.

When trying to generate a scanner for this example https://github.com/devnull42/taccle/blob/master/examples/simple_scanner.fcl I'm getting this error:

jimsh ../../fickle/fickle.tcl simple_scanner.fcl
../../fickle/fickle.tcl:215: Error: couldn't compile regular expression pattern: count follows nothing
Traceback (most recent call last):
  File "../../fickle/fickle.tcl", line 906
    fickle_main
  File "../../fickle/fickle.tcl", line 883, in fickle_main
    build_scanner {...
  File "../../fickle/fickle.tcl", line 150, in build_scanner
    handle_rules_buf {...
  File "../../fickle/fickle.tcl", line 215, in handle_rules_buf
    regsub -all -- {{number}} {{number}} {([0-9]+)} pattern

The offending line is this one:

regsub -all -- "\{$sub_rule\}" $pattern "\($::sub_table($sub_rule)\)" pattern

I'm a beginer with tcl .

[msteveb]

I don't know anything about fickle, but the problem seems to be using "{number}" as a pattern. Jim Tcl treats the curly braces specially. It seems like you are doing a simple replacement, ala, string map which perhaps makes more sense here.
Otherwise perhaps use chars that aren't special in a regular expression like "@number@" instead.

[mingodad]

Thank you for reply !
Fickle is not my project, I was looking at it and taccle (LALR(1) parser generator) to evaluate if they could be used with jimtcl compiled to wasm to build something like I did here https://mingodad.github.io/parsertl-playground/playground/ that is in C++ but lacks scripting for code actions.
And searching this repo for related problems I found this discussion and thought that it could fit here.

[msteveb]

And this has nothing to do with regexp -expanded

[msteveb]

I created an issue for you: #323