Comments are not handled at all?

[KvanTTT]

Consider the following javascript code:

// c

I'm trying to get all comments with the following code:

var parserOptions = new ParserOptions(source) { Comment = true, Tokens = true };
var scanner = new Scanner(source, parserOptions);
var comments = scanner.ScanComments();

But getting an empty comments collection.

The following code also does not work (from Esprima.Sample):

do
{
    scanner.ScanComments();
    token = scanner.Lex();
    tokens.Add(token);
} while (token.Type != TokenType.EOF);

Returns only a single empty token:

[
  {
    "Type": 1,
    "Start": 4,
    "End": 4,
    "LineNumber": 1,
    "Location": {}
  }
]

How can I handle them?

[KvanTTT]

Finally, I found a way to extract all comments:

var comments = new List<Comment>();
do
{
    comments.AddRange(scanner.ScanComments());
    token = scanner.Lex();
    tokens.Add(token);
} while (token.Type != TokenType.EOF);

But:

• All comment values Comment.Value are null, but should be filled or this property should be removed.
• Lexer goes into infinite loop on some files with custom ErrorHandler that does not throw an exception on the first error. For example, angular-1.2.5.js file from this repository. Actually, I don't understand why errors are appeared during lexing this file, not parsing (the first error on 44 line and 44 column).

[sebastienros]

I think the issue is related to this PR that I seem to have skipped https://github.com/jquery/esprima/pull/1439/files

Each node is passed to proxyDelegate such that external components can react to each parsed node. The a composite one will also plug commentHandler to it to attach comments to their statement/expression. As a general rule I think we should have that in place. It might make hoisting easier to implement.

[adams85]

Checked and the following code works just as expected:

var scanner = new Scanner(source, new ScannerOptions { Comments = true });
var comments = scanner.ScanComments();

For extracting all the comments, you can do what you invented here but please be aware that tokenization is not always possible. (For more details, see #44 (comment))

A reliable way to extract comments is the following:

var parser = new JavaScriptParser(new ParserOptions { Comments = true });
var root = parser.ParseScript(source);
var comments = root.Comments;

All the comments contained in the parsed code will be listed in the root.Comments collection (along with their location and range info). Please note though that you can only get the comments "in bulk" currently, there are no plans to port commentHandler from Esprima to attach them to the corresponding syntax nodes because it is impossible to do this correctly with the current AST model. (More about this: #39 (comment))