javacc
  1. javacc
  2. JAVACC-255

Add to ability to have the token's offset

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 5.0
    • Fix Version/s: None
    • Component/s: parser generator
    • Labels:
      None
    • Environment:

      All

      Description

      Using JavaCC in environments like Eclipse which are more based for tokens on offsets than lines / columns is difficult ; it is possible to get it with the current JavaCC version but very uggly :
      one must :

      • create a NewToken.java with an int member (offset)
      • set the 'TOKEN_EXTENDS="NewToken";' option
      • generate the ParserTokenManager and the Parser with 'USER_TOKEN_MANAGER=false;'
      • create a NewTokenManager.java extending the ParserTokenManager implementing the TokenManager, and overriding the jjFillToken() and ReInit(...) methods
      • alter the grammar to set the 'USER_TOKEN_MANAGER = true;' option, insert code copied from the Parser (member, constructors, ReInit methods), and adapt it to the new classes (changes and casts)

      However it looks like it would be quite straightforward to incoporate it in the generated classes (something like adding a new member in Token and a line 't.offset = input_stream.bufpos;' in jjFillToken())

        Activity

        Hide
        sreeni added a comment -

        We would have done it long time ago if it were that simple with unicode escape processing, backup etc.

        That said, there is a really easy way to track offsets if you have simple stringbuffer. The whole circular buffer thing is unnecessary for more recent systems where 2G ram is the new norm. So you can maintain a very simple stringbuffer and offset into that can be stored in the token. See the C++ version of the streams if you want to do this.

        Show
        sreeni added a comment - We would have done it long time ago if it were that simple with unicode escape processing, backup etc. That said, there is a really easy way to track offsets if you have simple stringbuffer. The whole circular buffer thing is unnecessary for more recent systems where 2G ram is the new norm. So you can maintain a very simple stringbuffer and offset into that can be stored in the token. See the C++ version of the streams if you want to do this.

          People

          • Assignee:
            Unassigned
            Reporter:
            marc_mazas
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: