Skip to content

Tokenizer & Parser

The tokenizer converts template strings into AST node lists (Phase 1). The parser converts individual expression text into structured NLTExpression dataclasses.

python
from pyrx_nlt import tokenize, parse_nlt_expression, NLTExpression
from pyrx_nlt.parser import NLTParseError
from pyrx_nlt.condition_parser import NLTConditionParseError

tokenize()

python
def tokenize(template: str) -> list[ASTNode]

Parse a template string into a list of AST nodes. This is Phase 1 of the two-phase pipeline.

Parameters

ParameterTypeDescription
templatestrThe full template string containing NLT expressions, control flow, and literal text.

Returns

list[ASTNode] -- a list of AST nodes ready for rendering. See AST Nodes for all node types.

Processing Steps

  1. Normalize apostrophes -- convert Unicode apostrophe-like characters (smart quotes \u2018 \u2019, backticks, acute accents, etc.) to ASCII '. This ensures templates pasted from Google Docs, macOS, or rich text editors work correctly.
  2. Extract {raw}...{end raw} content to placeholders (preserves literal content)
  3. Strip comments ({# ... #})
  4. Clean HTML tags from inside {...} expressions (handles rich text editor artifacts)
  5. Extract all {...} blocks into a flat token list
  6. Build nested AST from if/else/end-if, for/end-for, and other block structures

Graceful Degradation

Malformed blocks (e.g., {if} without {end if}) degrade to LiteralNode rather than raising errors. The tokenizer always produces a valid AST.

Example

python
from pyrx_nlt import tokenize
from pyrx_nlt.nodes import LiteralNode, ExpressionNode, IfNode
 
nodes = tokenize("Hello, `{the user's Name}`!")
 
# nodes[0] = LiteralNode(text="Hello, ")
# nodes[1] = ExpressionNode(raw="the user's Name", raw_block="`{the user's Name}`")
# nodes[2] = LiteralNode(text="!")
python
nodes = tokenize(
"`{if the user's Plan is premium}`VIP!`{else}`Welcome!`{end if}`"
)
 
# nodes[0] is an IfNode with:
# condition = ConditionClause(source="user", attribute_path="Plan", operator="eq", value="premium")
# then_nodes = [LiteralNode(text="VIP!")]
# else_nodes = [LiteralNode(text="Welcome!")]

parse_nlt_expression()

python
def parse_nlt_expression(raw: str) -> NLTExpression

Parse the inner text of a {...} block into a structured expression. This function handles source identification, math operator extraction, and modifier/filter parsing.

Parameters

ParameterTypeDescription
rawstrInner text of the expression, e.g. the user's First Name, or "Customer"

Returns

NLTExpression -- a frozen dataclass with the parsed source, attribute path, and all modifiers.

Raises

NLTParseError -- if the source keyword cannot be identified (i.e., the expression does not match the user's ..., the ... from the trigger event, or the ... from the ... event).

Example

python
from pyrx_nlt import parse_nlt_expression
 
expr = parse_nlt_expression('the user\'s First Name, or "Customer"')
# NLTExpression(
# source="user",
# attribute_path="First Name",
# fallback="Customer",
# required=False,
# ...
# )
 
expr = parse_nlt_expression("the Amount from the trigger event, as \"currency\"")
# NLTExpression(
# source="trigger_event",
# attribute_path="Amount",
# format_spec="currency",
# ...
# )
 
expr = parse_nlt_expression("the user's Subtotal times 1.08, as \"currency\"")
# NLTExpression(
# source="user",
# attribute_path="Subtotal",
# math_op="multiply",
# math_operand=1.08,
# format_spec="currency",
# ...
# )

NLTExpression

python
from pyrx_nlt import NLTExpression
 
@dataclass(frozen=True)
class NLTExpression:
source: str
attribute_path: str
event_name: str | None = None
fallback: str | None = None
required: bool = False
format_spec: str | None = None
transform: str | None = None
pluralize: bool = False
pluralize_word: str | None = None
math_op: str | None = None
math_operand: float | None = None
filters: tuple[tuple[str, str | None], ...] = ()

A frozen (immutable) dataclass representing a parsed NLT variable expression.

Fields

FieldTypeDescription
sourcestrData source: "user", "trigger_event", or "named_event" (also "latest_event" for legacy compat).
attribute_pathstrThe attribute name, e.g. "First Name", "Reference Number". Supports dot notation.
event_namestr | NoneEvent name for named_event source, e.g. "payment". None for user and trigger sources.
fallbackstr | NoneDefault value from the or "..." modifier.
requiredboolTrue if the required modifier is present. Suppresses the entire email if the value is missing.
format_specstr | NoneFormat string from the as "..." modifier. E.g. "currency", "currency EUR", "short date", "DD MMM YYYY".
transformstr | NoneText transform: "uppercase", "lowercase", "titlecase", or "capitalize".
pluralizeboolLegacy pluralize flag (backward compatibility).
pluralize_wordstr | NoneWord to pluralize, from pluralize "item".
math_opstr | NoneMath operation: "add", "subtract", "multiply", "divide", or "modulo".
math_operandfloat | NoneNumeric operand for the math operation.
filterstuple[tuple[str, str | None], ...]Chain of (filter_name, argument) pairs. Argument is None for no-argument filters. Multi-argument filters encode args with | separators.
Note

The filters field uses a tuple of tuples (not a list) because NLTExpression is frozen. Multi-argument filters like replace "a" with "b" are stored as ("replace", "a|b") and where "key" is "value" as ("where", "key|value").

NLTParseError

python
from pyrx_nlt.parser import NLTParseError
 
class NLTParseError(Exception):
"""Failed to parse an NLT expression."""

Raised by parse_nlt_expression() when the source keyword cannot be identified. The renderer catches this error internally and falls back to emitting the original {...} text unchanged.

NLTConditionParseError

python
from pyrx_nlt.condition_parser import NLTConditionParseError
 
class NLTConditionParseError(Exception):
"""Failed to parse a condition expression."""

Raised by the condition parser when an {if ...} condition cannot be parsed. The tokenizer catches this and degrades the block to literal text.