lexer

Although you can circumvent that by processing indentation in the lexer or inbetween lexer and parser.

The grammar is basically useless without the lexer, which is very complex, full of weird corner cases, and changes version by version.

And much of it, in both the lexer and the grammar rules, is very heavily tied into MRI internals.

There's really no need to have either section on lexer or parser when working with s-expr.

Also note that ia and aa are text objects for shell arguments using zsh's internal lexer.

Writing a lexer in C, while simple, is non-trivial mainly due to the tasks of pointer arithmetic and string manipulation.

So, just to clarify...is this a port of the venerable Python lexer generator called Plex?

If 1:1 binary representation for the data from the lexer doesn't yield much benefit, then a representation for the AST might.

I think it's due to the fact that the human brain does not work linearly, like a lexer.

Honest question, wouldn't you just write a lexer with yacc/bison for a DSL?

Malformed XML documents are trivial with a good lexer and we need tons of validation too on our side.

You write the structure, yacc generates the parser (and flex the lexer).

I suspect most people that think "parsing text" is difficult are confusing the parser with lexer; the latter is the only part that changes when you switch between text and binary formats.

It's a very simple design: a simple lexer with a "pull" API (next()) feeds a precedence-climbing expression parser that spits out bytecode for a simple stack machine.

In github it states "The CLR importer, lexer, parser, interpreter, type checker, language runtime, JavaScript generator, and other components of Wasabi are missing."