Beyond Word Order: Giving Transformers a Sense of Syntax

Haris Riaz default photo
Haris Riaz
PhD student
University of Arizona

Research Topic

Can injecting syntactic structure into a Transformer’s positional pathway improve language understanding — without the inference overhead of full syntactic language models?

Project Summary

Haris Riaz is a PhD student in the Computational Language Understanding (CLU) Lab at the University of Arizona, working with Professor Mihai Surdeanu on the intersection of formal linguistics and deep learning. The project addresses a structural gap in how Transformer-based language models process text: positional embeddings tell a model where tokens appear in a sequence, but not how they relate to one another. In the sentence “Please move this large file to another folder,” the word folder sits eight tokens from move, a distance at which attention already starts to fade, even though a single dependency arc connects the two. Positional encodings of every variety measure where tokens sit. None of them captures how tokens relate.

The team developed SiPE (Syntax-Informed Positional Embeddings), a method that learns a lightweight syntactic prior from dependency parses and injects it into the model’s positional pathway, leaving self-attention and the rest of the architecture untouched. The prior is derived from a single parse per sentence and requires approximately 7·D parameters, well under 0.01% of model size. This design avoids the two extremes in the field: joint syntactic language models, which must evaluate hundreds of candidate parse trees per sentence at inference time, and parser-free methods that discard syntax entirely after training. SiPE conditions on exactly one parse at both training and inference, placing it on the Pareto frontier between syntactic supervision and inference cost.

The team trained and evaluated four model architectures spanning all three dominant positional-encoding families, RoBERTa (absolute), DeBERTa-v3 and Transformer-XL (relative), and ModernBERT (rotary), using WikiText and BLLIP corpora as pretraining data. The work involved a neural architecture search to determine how best to combine the syntax-informed embeddings with each model’s native positional scheme, amounting to hundreds of training runs and approximately 10,000 GPU-hours of compute across pretraining and downstream fine-tuning. The resulting model weights and optimizer states were initially stored on Jetstream2 instance volumes; as storage filled up, the team used Globus to move several terabytes of checkpoints back to persistent institutional storage, with centralized transfer tracking and completion notifications reducing the management overhead between experiments. The team used Open OnDemand on Jetstream2 to monitor pretraining and fine-tuning runs across model configurations and ablations, reviewing training loss curves, tracking perplexity on held-out data, and comparing GLUE evaluation results after fine-tuning to assess convergence and whether different injection strategies improved downstream performance. “We used NSF ACCESS allocations on Jetstream2 to apply the approach across three major families of positional embeddings — absolute, relative, and rotary — and tested it in both encoder and decoder models,” Riaz said. “And we were a bit surprised by the results.” Without Jetstream2, the scale of pretraining, architecture search, and fine-tuning would not have been feasible within the team’s academic compute budget.

A central finding was that where the syntactic information enters the model matters much more than how much syntax is injected, and the best entry point differs by architecture. For autoregressive decoders using relative positional encodings, the strongest approach was to couple the syntactic prior multiplicatively with the relative-position component of the attention score. For encoders, the best approach was to add the syntactic information directly to the input embeddings alongside the model’s existing positional mechanism. “Our results suggest that explicitly teaching models something about sentence structure in the positional embedding layer can improve their language capabilities without the performance trade-offs associated with many earlier syntax-focused approaches,” Riaz said.

Models pretrained with SiPE improved SyntaxGym scores, a benchmark built from minimal sentence pairs, exactly one of which is grammatical, by up to 10.3% over matched baselines with no syntactic supervision, while simultaneously reducing perplexity by 9.0%, a metric that nearly every prior syntax-injection method instead makes worse. Real-world language understanding also improved: GLUE scores rose by up to 8.2%, with the decoder improving on all eight tasks. “We feel that our findings point toward a practical research direction: improving language models by making better use of sentence structure within components they already have,” Riaz said. The paper was accepted as an oral presentation at EMNLP 2026. Code, hexatagged training data, and pretrained checkpoints are available at github.com/hriaz17/SiPE.

Overview of the approach. Left: dependency arcs → binary head tree → hexatags (

Access Tools

Allocation Information

ACCESS Resource
Jetstream2 GPU
Allocation Type
Explore
Field of Science
Computer Science

Software