6

Is it a lexer's job to undo any escaping done to a string literal? For example:

"Me: \"Hello World!\""

Becomes:

Me: "Hello World!"

Should this conversion be done inside the lexer? I am guessing it should, because it'd allow for a more abstract and modular design. You could add ways to represent strings and you won't have to update every component.

Jeroen
  • 613
  • 1
  • 7
  • 13

1 Answers1

4

If you are implementing something close to string literals in C, then yes. This is because at the level of the parser, you are only concerned about something being a string literal and not how they are implemented.

But if you have some additional requirement such as the double quotes appearing inside the string literal must be matched (i.e., "\"" is invalid). Then this can only be captured only using a context free grammar and can only be handled by a parser.

Balagopal Komarath
  • 424
  • 3
  • 7
  • 2
    The second paragraph is incorrect. Escapes can be handled easily by the lexer, as shown in this Lex/Flex example: http://stackoverflow.com/questions/5418181/flex-lex-encoding-strings-with-escaped-characters – Blrfl May 03 '14 at 12:24
  • @Blrfl: Please re-read that second paragraph. It isn't about handling escapes, but about the fact that a lexer can't deal with a balanced quotes/parentheses problem, even within a string-literal. – Bart van Ingen Schenau May 03 '14 at 12:52
  • 1
    What kind of language permits arbitrarily nested balanced quotes? I only know of `"using "" to signify a single quote"`. –  May 03 '14 at 13:05
  • @BartvanIngenSchenau: You're right that I misinterpreted the paragraph, but it's still bogus. You couldn't write a (correct) parser for a requirement like that because `"foo" bar "baz"` would be valid but ambiguous. – Blrfl May 03 '14 at 17:29
  • @BartvanIngenSchenau Lexers can handle balanced constructs using the [starting condition stack](http://flex.sourceforge.net/manual/Start-Conditions.html#index-stacks_002c-routines-for-manipulating-182), i.e. yy_push_state() and yy_pop_state(), etc. – Hai Zhang Aug 02 '15 at 04:11