Tokenizer

MMO\core\helpers · 1 symbol · generated 30 Sep 2026

class

Tokenizer

A generic, hand-rollable lexer: split text into a stream of Token instances, driven by a stack of

class MMO\core\helpers\Tokenizer implements Iterator, MMO\core\interfaces\iSerializable

Namespace: MMO\core\helpers
Implements: Iterator, iSerializable
See: TokenDomain "grammars" (e.g. one for the top level, another opened while inside a quoted string or bracketed group). Implements \Iterator, so foreach ($tokenizer as $token) walks the whole token stream. Used by JsonParser / XmlParser.

$tokenizer = new Tokenizer(new TokenDomain([',', ':'])); $tokenizer->setText('a,b:c'); foreach ($tokenizer as $token) {echo $token->text . ' ';} // 'a , b : c '

Index

	CHUNK_SIZE
	USER_TOKEN_ID
	col	{int} general column offset
	domains	{array<TokenDomain>} stack of domains in use
	filters	{array<fn(string $text): string>} list of filters to apply to incoming text
	line	{int} general line offset
	offset	{int} general offset
	stream
	text	{string} current text being treated.
	token	{?Token} current token
	tokens	{array<Token>} tokens found
	addFilter($fn, $priority)	Register a text filter (applied by `addText` to every chunk of incoming text, e.g.
	addText($text, $first)	Append more text to the not-yet-tokenized buffer (running it through every registered `addFilter` first).
	clear()	Reset all state (text, tokens read so far, open domains) but keep the root domain, so this instance is ready to tokenize a new text via `setText`.
	closeDomain()	Pop the current domain, returning to its parent.
	current()	\Iterator::current -- the current `Token`
	currentDomain()
	getCol()
	getDomains()
	getLastTokenIndex($token, $offset)	Same as `getTokenIndex`, but searches backward from $offset.
	getLine()
	getOffset()
	getStream()
	getText($length, $offset)	Peek at the not-yet-tokenized text buffer, without consuming it.
	getToken($index)
	getTokenIndex($token, $offset)
	getTokens()
	goto($search)	Advance tokenizer to the searched token.
	ignore($search)	Advance tokenizer until it does not find the searched token.
	isEscaped($char, $offsetFromEnd)	Tells if current token is escaped (by backslash).
	isInsideDomain($domain)
	key()	\Iterator::key -- the number of tokens already consumed before this one
	next()	\Iterator::next -- read and consume the next token from the buffer (pulling more from `getStream` if needed), delegating to the current domain's own token list, and to `TokenDomain::open` / :…
	nextToken($count)	Advance $count tokens forward.
	openDomain($domain)	Push a domain onto the stack (e.g.
	parentDomain()
	rewind()	WARNING: this function won't rewind stream.
	rootDomain()
	serialize($path, $serializer)	\MMO\core\interfaces\iSerializable::serialize -- only the root domain is persisted (grammar, not read progress)
	setRootDomain($domain)
	setText($stream)	Set what to tokenize: a plain string (read whole, immediately), a stream resource (read in
	setToken($token)	Consume $token: record it as current, advance offset/line/col past it, and remove its text from the not-yet-tokenized buffer.
	unserialize($data, $path, $instances)	\MMO\core\interfaces\iSerializable::unserialize -- rebuilds a fresh instance with the stored root domain
	valid()	\Iterator::valid -- false once tokenizing has reached the end of the text
	__construct($domain)

Constants2

CHUNK_SIZE

const CHUNK_SIZE = 8192

USER_TOKEN_ID

const USER_TOKEN_ID = 10000

Properties9

col

{int} general column offset

protected int $col = 0

Type: int

domains

{array<TokenDomain>} stack of domains in use

protected array $domains = []

Type: array

filters

{array<fn(string $text): string>} list of filters to apply to incoming text

protected array $filters = []

Type: array

line

{int} general line offset

protected int $line = 0

Type: int

offset

{int} general offset

protected int $offset = 0

Type: int

stream

protected $stream = null

text

{string} current text being treated. it is shrunk every time a token is created

protected string $text = ''

Type: string

token

{?Token} current token

protected ?MMO\core\helpers\Token $token

Type: null | Token

tokens

{array<Token>} tokens found

protected array $tokens = []

Type: array

Methods34

addFilter($fn, $priority)

Tokenizer.php:210

function addFilter(callable $fn, int $priority = 0): void

Filters run in descending priority order (the highest-priority filter first); among filters with equal priority, the one added first runs first.

$fn	callable	function(string $text): string
$priority = 0	int
returns	void	NOTE: the original insertion logic was genuinely broken -- it only ever compared the new filter against the single element it happened to stop scanning at, not the array as a whole, so filters routinely ended up in neither ascending nor descending order (verified: adding priorities 1, 2, 3 in that order produced [2, 3, 1] instead of the correct [3, 2, 1]). No code in the framework currently calls this, so this replaces the insertion with a correct, verified stable descending-priority insert instead of patching the old one.

$tokenizer->addFilter($fn, 5); $tokenizer->addFilter($fn2, 10); // runs BEFORE $fn, since it has a higher priority

addText($text, $first)

Tokenizer.php:68

Append more text to the not-yet-tokenized buffer (running it through every registered addFilter first). Used internally to read more from setText's stream as tokenizing progresses, but can also be called directly to feed text incrementally.

function addText(string $text, bool $first = false)

$text	string
$first = false	bool	if true, prepend instead of append
returns

clear()

Tokenizer.php:52

Reset all state (text, tokens read so far, open domains) but keep the root domain, so this instance is ready to tokenize a new text via setText.

function clear(): void

returns

void

closeDomain()

Tokenizer.php:277

Pop the current domain, returning to its parent.

function closeDomain(): ?MMO\core\helpers\TokenDomain

returns

null | TokenDomain

the closed domain, or null if already at the root (the root is never popped)

current()

Tokenizer.php:368

\Iterator::current -- the current Token

function current(): mixed

returns

mixed

currentDomain()

Tokenizer.php:282

function currentDomain(): MMO\core\helpers\TokenDomain

returns

TokenDomain

the innermost (currently active) domain

getCol()

Tokenizer.php:117

function getCol(): int

returns

int

the current column

getDomains()

Tokenizer.php:287

function getDomains(): array

returns

array

the whole domain stack, root first

getLastTokenIndex($token, $offset)

Tokenizer.php:162

Same as getTokenIndex, but searches backward from $offset.

function getLastTokenIndex(MMO\core\helpers\Token $token, int $offset = 0): int|bool

$token	Token
$offset = 0	int
returns	int \| bool

getLine()

Tokenizer.php:112

function getLine(): int

returns

int

the 1-based line of the token currently being read (updated as tokens containing '\n' are consumed)

getOffset()

Tokenizer.php:122

function getOffset(): int

returns

int

how many characters have been consumed (tokenized) so far

getStream()

Tokenizer.php:96

function getStream()

returns the stream passed to setText, if any

getText($length, $offset)

Tokenizer.php:107

Peek at the not-yet-tokenized text buffer, without consuming it.

function getText(?int $length = null, int $offset = 0): string

$length = null	null \| int	null reads to the end
$offset = 0	int
returns	string

$tokenizer->getText(1); // the single next character to be tokenized

getToken($index)

Tokenizer.php:130

function getToken(int $index): ?MMO\core\helpers\Token

$index	int	negative counts from the end (-1 is the last read token)
returns	null \| Token	null if $index is out of range

getTokenIndex($token, $offset)

Tokenizer.php:145

function getTokenIndex(MMO\core\helpers\Token $token, int $offset = 0): int|bool

$token	Token	the exact instance to look for (identity, not equality)
$offset = 0	int	where to start searching forward from (negative counts from the end)
returns	int \| bool	the token's index, or false if not found

getTokens()

Tokenizer.php:136

function getTokens(): array

returns

array

every token consumed so far

goto($search)

Tokenizer.php:320

Advance tokenizer to the searched token.

function goto(...$search): array

...$search		the token's id or text to look for
returns	array

$tokenizer->goto(','); // every token up to (not including) the next comma

ignore($search)

Tokenizer.php:336

Advance tokenizer until it does not find the searched token.

function ignore(...$search): ?MMO\core\helpers\Token

...$search		the token's id or text to look for
returns	null \| Token	the current token

$tokenizer->ignore('x'); // skips every leading 'x' token, returns the first non-'x' one

isEscaped($char, $offsetFromEnd)

Tokenizer.php:228

Tells if current token is escaped (by backslash).

function isEscaped(string $char = '\\', int $offsetFromEnd = 0): bool

To be escaped, there must be an odd number of escape char.

$char = '\\'	string	the escape char
$offsetFromEnd = 0	int	the offset from current token
returns	bool

// after tokenizing "\\'" (an escaped quote): isEscaped() === true // after tokenizing "\\\\'" (an escaped backslash, then an unescaped quote): isEscaped() === false

isInsideDomain($domain)

Tokenizer.php:298

function isInsideDomain(MMO\core\helpers\TokenDomain $domain): bool

$domain	TokenDomain
returns	bool	true if $domain is anywhere in the current domain stack

key()

Tokenizer.php:373

\Iterator::key -- the number of tokens already consumed before this one

function key(): mixed

returns

mixed

next()

Tokenizer.php:382

\Iterator::next -- read and consume the next token from the buffer (pulling more from getStream if needed), delegating to the current domain's own token list, and to TokenDomain::open / ::close for nested domains.

function next(): void

returns

void

nextToken($count)

Tokenizer.php:309

Advance $count tokens forward.

function nextToken(int $count = 1): ?MMO\core\helpers\Token

$count = 1	int
returns	null \| Token	the resulting current token (`current`), or null once exhausted

openDomain($domain)

Tokenizer.php:269

Push a domain onto the stack (e.g. entering a quoted string or bracketed group) -- it becomes

function openDomain(MMO\core\helpers\TokenDomain $domain): void

See: currentDomain until closeDomain pops it.

$domain	TokenDomain
returns	void

parentDomain()

Tokenizer.php:292

function parentDomain(): ?MMO\core\helpers\TokenDomain

returns null | TokenDomain the domain enclosing currentDomain, or null if currently at the root

rewind()

Tokenizer.php:349

WARNING: this function won't rewind stream. You need to do it manually

function rewind(): void

returns

void

rootDomain()

Tokenizer.php:259

function rootDomain(): MMO\core\helpers\TokenDomain

returns

TokenDomain

the bottom of the domain stack

serialize($path, $serializer)

Tokenizer.php:436

\MMO\core\interfaces\iSerializable::serialize -- only the root domain is persisted (grammar, not read progress)

function serialize(string $path, callable $serializer): array

$path	string
$serializer	callable
returns	array

setRootDomain($domain)

Tokenizer.php:254

function setRootDomain(MMO\core\helpers\TokenDomain $domain): void

$domain	TokenDomain	replaces the whole domain stack with just this one
returns	void

setText($stream)

Tokenizer.php:85

Set what to tokenize: a plain string (read whole, immediately), a stream resource (read in

function setText($stream)

See: CHUNK_SIZE chunks as tokenizing progresses), or null (nothing -- e.g. to tokenize only via manual, addText calls).

$stream
returns
throws		for any other type

$tokenizer->setText('a,b,c');

setToken($token)

Tokenizer.php:179

Consume $token: record it as current, advance offset/line/col past it, and remove its text from the not-yet-tokenized buffer.

protected function setToken(MMO\core\helpers\Token $token)

$token	Token
returns

unserialize($data, $path, $instances)

Tokenizer.php:441

\MMO\core\interfaces\iSerializable::unserialize -- rebuilds a fresh instance with the stored root domain

function unserialize( array $data, string $path, Psr\Container\ContainerInterface|array &$instances ): void

$data	array
$path	string
$instances	`ContainerInterface` \| array
returns	void

valid()

Tokenizer.php:363

\Iterator::valid -- false once tokenizing has reached the end of the text

function valid(): bool

returns

bool

__construct($domain)

Tokenizer.php:43

function __construct(MMO\core\helpers\TokenDomain $domain)

$domain TokenDomain the top-level grammar (setRootDomain)