Struct Lexer 

pub struct Lexer<'t> {
    path: Symbol,
    text: &'t str,
    iter: Peekable<CharIndices<'t>>,
    head: u32,
    tail: u32,
}

Fields

path: Symbol

text: &'t str

The source text

iter: Peekable<CharIndices<'t>>

A peekable iterator over the source text

head: u32

The start of the current token

tail: u32

The end of the current token

Implementations

impl<'t> Lexer<'t>

pub fn new(path: Symbol, text: &'t str) -> Self

Constructs a new Lexer from some text

pub const fn span(&self) -> Span

Gets the Span of the current token.

When called from outside Lexer::scan, this will return a zero-length Span marking the current lexer location.

pub fn peek(&mut self) -> Option<char>

Peeks the next character without advancing the lexer

fn advance_tail(&mut self)

Advances the tail to the current character index

fn take(&mut self) -> Option<char>

Takes the last character

fn next_if(&mut self, expected: char) -> Option<char>

Takes the next character if it matches expected, else returns None.

fn consume(&mut self) -> &mut Self

Consumes the last-peeked character, advancing the tail

const fn error(&self, res: LexFailure) -> LexError

Produces a LexError at the start of the current token

fn as_str(&self) -> (&'t str, Span)

Gets the Lexer’s current &str lexeme and Span

fn produce(&mut self, kind: TKind) -> Token

Produces a Token

fn produce_with_lexeme(&mut self, kind: TKind, lexeme: Lexeme) -> Token

fn skip_whitespace(&mut self) -> &mut Self

Consumes 0 or more whitespace

const fn start_token(&mut self) -> &mut Self

Marks the start of a new Token’s Span

pub fn scan(&mut self) -> Result<Token, LexError>

Scans forward until it finds the next Token in the input

pub fn line_comment(&mut self) -> Result<Token, LexError>

Consumes characters until the lexer reaches a newline '\n'

pub fn block_comment(&mut self) -> Result<&mut Self, LexError>

Consumes characters until the lexer reaches the end of a nested block comment. This allows you to arbitrarily comment out code, even if that code has a block comment.

pub fn identifier(&mut self) -> Result<Token, LexError>

Consumes characters until it reaches a character not in is_xid_continue.

Always consumes the first character.

Maps the result to either a TKind::Identifier or a TKind keyword.

pub fn character(&mut self, as_int: bool) -> Result<Token, LexError>

Eagerly parses a character literal starting at the current lexer position.

pub fn string(&mut self) -> Result<Token, LexError>

pub fn escape(&mut self) -> Result<char, LexError>

Parses a single escape sequence into its resulting char value.

pub fn hex_escape(&mut self) -> Result<char, LexError>

Parses two hex-digits and constructs a char out of them.

pub fn unicode_escape(&mut self) -> Result<char, LexError>

Parses a sequence of {}-bracketed hex-digits and constructs a char out of them.

pub fn digits<const BASE: u32>(&mut self) -> Result<Token, LexError>

Parses a sequence of digits (and underscores) in base BASE, where 2 <= BASE <= 36.

If the sequence of digits exceeds the bounds of a u128, the resulting number will wrap around 2^128.

pub fn digit<const BASE: u32>(&mut self) -> Result<u32, LexError>

Parses a single digit in base BASE as a u32, where 2 <= BASE <= 36.

Trait Implementations

impl<'t> Clone for Lexer<'t>

fn clone(&self) -> Lexer<'t>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'t> Debug for Lexer<'t>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.