lexer_builder 0.1.3

SDKdart
Platformwindowslinuxmacos

Generate lexer code from regular expression rules. Arbitrary custom tokens are supported with generics.

lexer_builder

A lexer generator for dart.

Features

  • Match tokens using RegExp syntax.
  • Rules dependent on lexer state.
  • Generates lexer code automatically.

Caveats

  • Generated lexers using regex are likely slower than handwritten ones.

Getting started

Include lexer_builder and build_runner as dev_dependencies and lexer_builder_runtime in your pubspec.yml.

Usage

Annotate a class with Lexer() and methods in it with Rule() to define a lexer. See the example for detailed instructions.

TODO

  • Eventually generate custom code for the rules instead of using RegExp internally.
  • Support async lexing with that by accurately measuring partly-matched subgroups, dispatching a rule if the next character doesn't make any match longer.

Additional information

For more information about lexers in general, see the flex lexer generator and its documentation.