Missing "/" operator in PHP tokenizer's result

[Dec1mo]

For this PHP code:

<?php
$x = 10;
$y = 4;

echo($x / $y);
echo "<br>";
echo "$x / $y";
echo($x % $y);
?>

Tokenizer's result is:

(<, TokenType.OPERATOR, (1, 1))
(?, TokenType.SPECIAL_SYMBOL, (1, 2))
(php, TokenType.IDENTIFIER, (1, 3))  
($x, TokenType.IDENTIFIER, (2, 1))   
(=, TokenType.OPERATOR, (2, 4))      
(10, TokenType.CONSTANT, (2, 6))     
(;, TokenType.IDENTIFIER, (2, 8))    
($y, TokenType.IDENTIFIER, (3, 1))   
(=, TokenType.OPERATOR, (3, 4))      
(4, TokenType.CONSTANT, (3, 6))      
(;, TokenType.IDENTIFIER, (3, 7))    
(echo, TokenType.KEYWORD, (5, 1))
((, TokenType.SPECIAL_SYMBOL, (5, 5))
($x, TokenType.IDENTIFIER, (5, 6))
($y, TokenType.IDENTIFIER, (5, 11))  // <-- I think "/" is missing here
(), TokenType.SPECIAL_SYMBOL, (5, 13))
(;, TokenType.SPECIAL_SYMBOL, (5, 14))
(echo, TokenType.KEYWORD, (6, 1))
(", TokenType.SPECIAL_SYMBOL, (6, 6))
(<br>, TokenType.STRING, (6, 7))
(", TokenType.SPECIAL_SYMBOL, (6, 11))
(;, TokenType.SPECIAL_SYMBOL, (6, 12))
(echo, TokenType.KEYWORD, (7, 1))
(", TokenType.SPECIAL_SYMBOL, (7, 6))
($x / $y, TokenType.STRING, (7, 7))
(", TokenType.SPECIAL_SYMBOL, (7, 14))
(;, TokenType.SPECIAL_SYMBOL, (7, 15))
(echo, TokenType.KEYWORD, (8, 1))
((, TokenType.SPECIAL_SYMBOL, (8, 5))
($x, TokenType.IDENTIFIER, (8, 6))
(%, TokenType.OPERATOR, (8, 9))
($y, TokenType.IDENTIFIER, (8, 11))
(), TokenType.SPECIAL_SYMBOL, (8, 13))
(;, TokenType.SPECIAL_SYMBOL, (8, 14))
(?, TokenType.SPECIAL_SYMBOL, (9, 1))
(>, TokenType.OPERATOR, (9, 2))

Please fix that @tranvien98

[xTrimy]

I know this is an old issue, but did you find any solution for it? I have the same issue.

[xTrimy]

Here's my problem

void mergeSort(T* a, T* b, int low, int high)
{
	int mid;
	if (low < high)
	{
		mid = (low + high) / 2;
		mergeSort(a, b, low, mid);
		mergeSort(a, b, mid + 1, high);
		merge(a, b, low, mid, high);
	}
}

(void, TokenType.KEYWORD, (46, 1))
(mergeSort, TokenType.IDENTIFIER, (46, 6))
((, TokenType.SPECIAL_SYMBOL, (46, 15))
(T, TokenType.IDENTIFIER, (46, 16))
(*, TokenType.OPERATOR, (46, 18))
(a, TokenType.IDENTIFIER, (46, 19))
(,, TokenType.OPERATOR, (46, 20))
(T, TokenType.IDENTIFIER, (46, 22))
(*, TokenType.OPERATOR, (46, 24))
(b, TokenType.IDENTIFIER, (46, 25))
(,, TokenType.OPERATOR, (46, 26))
(int, TokenType.KEYWORD, (46, 28))
(low, TokenType.IDENTIFIER, (46, 32))
(,, TokenType.OPERATOR, (46, 35))
(int, TokenType.KEYWORD, (46, 37))
(high, TokenType.IDENTIFIER, (46, 41))
(), TokenType.SPECIAL_SYMBOL, (46, 45))
({, TokenType.SPECIAL_SYMBOL, (47, 1))
(int, TokenType.KEYWORD, (48, 5))
(mid, TokenType.IDENTIFIER, (48, 9))
(;, TokenType.SPECIAL_SYMBOL, (48, 12))
(if, TokenType.KEYWORD, (49, 5))
((, TokenType.SPECIAL_SYMBOL, (49, 8))
(low, TokenType.IDENTIFIER, (49, 9))
(<, TokenType.OPERATOR, (49, 13))
(high, TokenType.IDENTIFIER, (49, 15))
(), TokenType.SPECIAL_SYMBOL, (49, 19))
({, TokenType.SPECIAL_SYMBOL, (50, 5))
(mid, TokenType.IDENTIFIER, (51, 9))
(=, TokenType.OPERATOR, (51, 13))
((, TokenType.SPECIAL_SYMBOL, (51, 15))
(low, TokenType.IDENTIFIER, (51, 16))
(+, TokenType.OPERATOR, (51, 20))
(high, TokenType.IDENTIFIER, (51, 22))
(), TokenType.SPECIAL_SYMBOL, (51, 26))
(2, TokenType.CONSTANT, (51, 30)) -> missing '/' before '2'
(;, TokenType.SPECIAL_SYMBOL, (51, 31))
(mergeSort, TokenType.IDENTIFIER, (52, 9))
((, TokenType.SPECIAL_SYMBOL, (52, 18))
(a, TokenType.IDENTIFIER, (52, 19))
(,, TokenType.OPERATOR, (52, 20))
(b, TokenType.IDENTIFIER, (52, 22))
(,, TokenType.OPERATOR, (52, 23))
(low, TokenType.IDENTIFIER, (52, 25))
(,, TokenType.OPERATOR, (52, 28))
(mid, TokenType.IDENTIFIER, (52, 30))
(), TokenType.SPECIAL_SYMBOL, (52, 33))
(;, TokenType.SPECIAL_SYMBOL, (52, 34))
(mergeSort, TokenType.IDENTIFIER, (53, 9))
((, TokenType.SPECIAL_SYMBOL, (53, 18))
(a, TokenType.IDENTIFIER, (53, 19))
(,, TokenType.OPERATOR, (53, 20))
(b, TokenType.IDENTIFIER, (53, 22))
(,, TokenType.OPERATOR, (53, 23))
(mid, TokenType.IDENTIFIER, (53, 25))
(+, TokenType.OPERATOR, (53, 29))
(1, TokenType.CONSTANT, (53, 31))
(,, TokenType.OPERATOR, (53, 32))
(high, TokenType.IDENTIFIER, (53, 34))
(), TokenType.SPECIAL_SYMBOL, (53, 38))
(;, TokenType.SPECIAL_SYMBOL, (53, 39))
(merge, TokenType.IDENTIFIER, (54, 9))
((, TokenType.SPECIAL_SYMBOL, (54, 14))
(a, TokenType.IDENTIFIER, (54, 15))
(,, TokenType.OPERATOR, (54, 16))
(b, TokenType.IDENTIFIER, (54, 18))
(,, TokenType.OPERATOR, (54, 19))
(low, TokenType.IDENTIFIER, (54, 21))
(,, TokenType.OPERATOR, (54, 24))
(mid, TokenType.IDENTIFIER, (54, 26))
(,, TokenType.OPERATOR, (54, 29))
(high, TokenType.IDENTIFIER, (54, 31))
(), TokenType.SPECIAL_SYMBOL, (54, 35))
(;, TokenType.SPECIAL_SYMBOL, (54, 36))
(}, TokenType.SPECIAL_SYMBOL, (55, 5))
(}, TokenType.SPECIAL_SYMBOL, (56, 1))

[Dec1mo]

Sorry @xTrimy, you must have done the fixing on your own 😬. Thank you, we really appreciate that.

[Dec1mo]

And @tranvien98, please kindly give this issue a look when you have time as you were the one who built the PhpTokenizer module.

[xTrimy]

@Dec1mo The same fix can be applied to the PhpTokenizer.
The same else condition should be added to the if cur == '/': condition.

I may fix it as well when you verify that the CppTokenizer fix is working fine.

Thanks in advance.

[Dec1mo]

It's very kind of you, @xTrimy. @tranvien98 used to be in charge of this module so I left this for him. But he didn't respond somehow.

As you pointed out, this might be an issue for all Tokenizers. However, there's still a bug with your previous pull request, which I mentioned here #9. Please give it a look whenever you can.

Thanks in advance.

[klenze]

I still had this issue (for c++) for the version which got installed with pip3. Uninstalling the package, cloning the repository from github and running pip install -e . in it fixed the issue for me.

[xTrimy]

@klenze I think the package itself is not updated yet.

[ngocbh]

Thank you for your comments. I will update the package soon.