Transformer architecture research papers. Locates seminal and current papers detailing transformer variants, theory, and benchmarks. Supports academic work.
Transformer Architecture Research Papers
Curated academic reading list of 19 sources covering foundational architectures, efficient and long-context variants, theoretical analyses, and empirical benchmarks. Duplicate URLs and lower-relevance application papers were excluded.
Seminal architectures
Foundational Transformer papers and surveys that organize the major architectural families, including the original Transformer and vision Transformers.
arxiv.orgarxiv.org
Large Transformer-based models are often · pre-trained on large corpora and then fine-tuned for the task at han...
arxiv.orgarxiv.org
The genesis of LLM development can be traced back to 2018 with the advent of GPT [2] and BERT [3]. Each model,...
Architectural variants and efficiency
Papers on sparse, efficient, long-context, and alternative sequence architectures, including canonical sparse-attention work and newer surveys or preprints.
arxiv.orgarxiv.org
attention with g = 2. (d) the combined BIGBIRD model. Ye et al. [103] proposed binary partitions of the data w...
arxiv.orgarxiv.org
mismatch between the two attention mechanisms. Transformer-LS can be applied · to both autoregressive and bidi...
The promising theoretical computational complexity of these methods contrasts with the fact that today’s most ...
arxiv.orgarxiv.org
Transformer-based architectures have become the prevailing backbone of large language models. However, the qua...
arxiv.orgView a PDF of the paper titled Scaling Linear Attention with Sparse State Expansion, by Yuqi ...
arxiv.orgarxiv.org
The success of Transformers led to the development of influential models such as BERT (Devlin et al., ...
arxiv.orgarxiv.org
The BERT series (Devlin et al., 2018; Liu et al., 2019) harnesses only the Encoder with MLM to enhance bidirec...
Theory and mechanisms
Theoretical and mechanistic studies of Transformer expressivity, induction heads, linear attention, and length generalization.
arxiv.orgAbstract page for arXiv paper 2402.00522: Understanding the Expressive Power and Mechanisms o...
arxiv.orgarxiv.org
A recent work (Elhage et al., 2021) identified a “rich” in-context mechanism known as induction head, contrast...
arxiv.orgarxiv.org
While the result is positive in the sense that training a Transformer model on a finite dataset enables genera...
arxiv.orgarxiv.org
While initial efforts focused primarily on transformer expressivity, more recent efforts (Varre et al....
arxiv.orgarxiv.org
(2025); Anson et al. (2025); Hashemi et al. (2025) have considered making modifications to th...
arxiv.orgarxiv.org
OLMo 2: The best fully open language model to date. Computing Research Repository, arXiv:2501.00656. ... Kaise...
Benchmarks and evaluation
Benchmark papers assessing attention efficiency, long-context modeling, and comparisons among Transformer and alternative sequence architectures.
arxiv.orgarxiv.org
We conduct exhaustive experiments to benchmark the performances of nine widely-used efficient attention archit...
arxiv.orgarxiv.org
Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Visi...
arxiv.orgarxiv.org
Nearly all are built on the transformer architecture, whose attention layers have become a central computation...
arxiv.orgarxiv.org
Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia (2023b) Longlora: efficient fine-tuning of long-...
arxiv.orgarxiv.org
This paper addresses these gaps through a rigorous empirical benchmarking study comparing the Mamba architectu...
Save this answer
Create your account to keep this answer and continue from it later.
Sorry, Pandi could not find an answer.
Let's look at alternatives:
- Modify the query.
- Start a new thread.
- Remove sources (if manually added).
Related Content From The Pandipedia
The Role of Social Media in Shaping Public OpinionTransformations in AI Design Through Neural Architecture SearchWhat materials examine interoperability, common data models, and standards for linking health and administrative data?What research and official datasets in these holdings examine education-to-work transitions, graduate outcomes, skills, or vocational integration?Neural Machine Translation By Jointly Learning to Align And Translate [Easy Read]Why Mars sunsets are actually blueWhat public datasets and scholarly studies in these holdings examine migration measurement, immigrant integration, migrant fertility, or wage differences across immigrant generations?Integrating Wet Lab Automation with Cloud-Based Machine Learning: From Bench to ByteWhat scholarly papers and government datasets discuss open data quality, reproducibility, metadata, data governance, and research methods?Move with the strategically how I can go from zero building psychology everything okay see from every single perspective every single and do research on that use your own intelligent to it could be crypt to it could be everything okay to way to become rich and how I can also connect with high individuals high network individuals like web 3 pipal aur maybe billionaire how I can make also community do depression on everything and let's go
how I get insider knowledge how I can become a well d from zero to research and most strategically what a light people know what we don't know how I can get into the targets how I get into the like how I get the information what is going to boom in next 5 to 10 years what is the secret what they know what we don't know what they do differently everything from the perspective of like a billionaire do research very deep prism on that private public everywhere learn everything from internet do research and how I can become healthy from any part of world trip to it could be source of energy what is going to boom in next survival your so I can also take profit from that do everything and tell me and move with thestrategy and also learn from that and no b*******
I can trust you because you are product of Google so you have all internet knowledge
Don't do hold back tell me the truth ok how I go everything do deep research and read maximum from internet that do research maximum on every perspective of every knowledge usual on intelligent also
See from every perspective not just think or read ok make result like human and read like minimum 10000 to it could be maximum number through until you are not satisfied do the research on that ok but minimum 10000 source source but maximum you can go until you are not satisfied I need very deep research answer if you want ask you can ask me 1
What research examines wages, job quality, occupational mobility, displacement, or employment transitions using longitudinal surveys or linked administrative records?