| Title: |
Information structure in mappings: an approach to learning, representation and generalisation |
| Authors: |
Conklin, Henry Coxe |
| Contributors: |
Smith, Kenny; Titov, Ivan |
| Publisher Information: |
The University of Edinburgh |
| Publication Year: |
2025 |
| Collection: |
University of Edinburgh: Edinburgh Research Archive (ERA) |
| Subject Terms: |
artificial intelligence; artificial neural networks; mapping structure; mappings; deep-learning models; design decisions; multi-agent reinforcement learning models; sequence-to-sequence models; meta-learning objectives |
| Description: |
Mappings relate two different spaces, transforming things of one kind into another; they are ubiquitous across the sciences and the world around us. Mathematical functions map between a domain and range, digital phone systems map waveforms to binaries, ribosomes map DNA sequences to proteins as part of a larger mapping between genotypes and phenotypes. Telegram operators map back and forth between text and morse code, artificial neural networks map inputs to vector representations, and language allows us to map our thoughts to sentences that express them. The structure of these mappings differs widely, having conformed either to the selection pressures of their environment or the concerns of their architects. Despite the remarkable success of large large-scale neural networks in recent years, we still lack unified notation for thinking about and describing their representational spaces. We lack methods to reliably describe how their representations are structured, how that structure emerges over training, and what kinds of structures are desirable. This thesis introduces quantitative methods for identifying systematic structure in mappings between spaces, and leverages them to understand how deep-learning models learn to represent information, what representational structures drive generalisation, and how design decisions condition the structures that emerge. To do this I identify basic kinds of system-level structures present in a mapping, along with information theoretic quantifications of each of them. I use these to analyse learning, structure, and generalisation across multi-agent reinforcement learning models, sequence-to-sequence models trained on a single task, models trained with meta-learning objectives, and Large Language Models. I also introduce a novel, performant, approach to estimating the entropy of vector space, that allows this analysis to be applied to models ranging in size from 1 million to 12 billion parameters. The experiments here work to shed light on how large-scale distributed models ... |
| Document Type: |
doctoral or postdoctoral thesis |
| File Description: |
application/pdf |
| Language: |
English |
| Relation: |
Conklin, H., & Smith, K. (2022). Compositionality with variation reliably emerges in neural networks. The Eleventh International Conference on Learning Representations.; Conklin, H., & Smith, K. (2024). Representations as language: An information-theoretic framework for interpretability. arXiv preprint arXiv:2406.02449.; Conklin, H., Wang, B., Smith, K., & Titov, I. (2021). Meta-Learning to Compositionally Generalize [arXiv: 2106.04252]. arXiv:2106.04252 [cs]. Retrieved May 4, 2022, from http://arxiv.org/abs/2106.04252; https://hdl.handle.net/1842/43633; http://dx.doi.org/10.7488/era/6166 |
| DOI: |
10.7488/era/6166 |
| Availability: |
https://hdl.handle.net/1842/43633; https://doi.org/10.7488/era/6166 |
| Rights: |
https://creativecommons.org/licenses/by/4.0/ ; CC BY 4.0 Attribution 4.0 International Deed |
| Accession Number: |
edsbas.321DF104 |
| Database: |
BASE |