| Title: |
On the Interaction of Noise, Compression Role, and Adaptivity under $(L_0, L_1)$-Smoothness: An SDE-based Approach |
| Authors: |
Compagnoni, Enea Monzio; Islamov, Rustem; Orvieto, Antonio; Gorbunov, Eduard |
| Publication Year: |
2025 |
| Collection: |
ArXiv.org (Cornell University Library) |
| Subject Terms: |
Machine Learning |
| Description: |
Using stochastic differential equation (SDE) approximations, we study the dynamics of Distributed SGD, Distributed Compressed SGD, and Distributed SignSGD under $(L_0,L_1)$-smoothness and flexible noise assumptions. Our analysis provides insights -- which we validate through simulation -- into the intricate interactions between batch noise, stochastic gradient compression, and adaptivity in this modern theoretical setup. For instance, we show that \textit{adaptive} methods such as Distributed SignSGD can successfully converge under standard assumptions on the learning rate scheduler, even under heavy-tailed noise. On the contrary, Distributed (Compressed) SGD with pre-scheduled decaying learning rate fails to achieve convergence, unless such a schedule also accounts for an inverse dependency on the gradient norm -- de facto falling back into an adaptive method. ; This manuscript is a work in progress: We welcome comments |
| Document Type: |
text |
| Language: |
unknown |
| Relation: |
http://arxiv.org/abs/2506.00181 |
| Availability: |
http://arxiv.org/abs/2506.00181 |
| Accession Number: |
edsbas.401FAD26 |
| Database: |
BASE |