Explaining Transformers Using Model-Based Stochastic Signal Processing

Carey Bunks will presenting their ongoing work on analysing Transformers

Abstract

This talk presents a novel analysis of Transformer components based on techniques drawn from model-based stochastic signal processing.  The proposed framework offers a theoretical foundation useful for mechanistic interpretability, and also suggests practical strategies for improving the performance of Transformer architectures.  Some preliminary experimental results will be presented.