| Title: |
Enabling Peta-Scale Federated Repositories through Cloud-Native Formats: Lessons from a fast-paced challenge in the bioimaging community |
| Authors: |
Moore, Josh; Rzepka, Norman; Ratamero, Erick; Pape, Constantin; Wetzker, Cornelia; Moore, William; Lindner, Dominik; Wong, Frances; Mohamed, Khaled; Ho, Kenneth H.L.; Kyoda, Koji; Sherwood, Francois; Yoldas, Aybuke Kupcu; de Folter, Joost; Zobel, Thomas; Jones, Martin L.; Wendt, Jens; Perlman, Eric; Stoeter, Torsten; Zoccoler, Marcelo Leomil; Thoennissen, Julia; Burel, Jean-Marie; Besson, Sebastien; Swedlow, Jason R.; Onami, Shuichi; Hartley, Matthew; Huisken, Jan; Weidtkamp-Peters, Stefanie |
| Contributors: |
Sure-Vetter, York; Groth, Paul |
| Publisher Information: |
Zenodo |
| Publication Year: |
2025 |
| Collection: |
Zenodo |
| Subject Terms: |
Bioimaging; Cloud; N-Dimensional; Next-generation file formats (NGFF); OME; RDM; Tensors; Zarr |
| Description: |
As research disciplines increasingly generate large-scale imaging data, the need for robust, scalable, and interoperable data infrastructure has become paramount. Cloud-native data formats — specifically Zarr — are emerging as critical enablers for the creation of distributed, federated repositories that adhere to FAIR data principles. This proposal presents the outcomes of the OME2024 NGFF Challenge, an international community effort that demonstrated the viability of constructing such infrastructure for bioimaging data using OME-Zarr. The Open Microscopy Environment (OME) is an open-source, community-driven initiative that develops interoperable data formats, tools, and standards for biological imaging. As part of its commitment to open and FAIR research data, NFDI4BIOIMAGE actively contributes to OME, particularly the specification of OME-Zarr for cloud-native image storage. The challenge launched at the 2024 OME Annual Meeting in Dundee, Scotland and was designed to advance the maturity of the OME-Zarr format, particularly in conjunction with the new major version of the specification, Zarr v3, which improves the scalability through the use of sharding. Coordinated by NFDI4BIOIMAGE, international participants contributed converted datasets hosted on their own infrastructure to the challenge. Submissions were indexed using a lightweight CSV-based mechanism, with each row corresponding to a Zarr-formatted dataset at participating institutions. Participants agreed to complete the Challenge in time for the next major bioimaging community convening, the 2024 Global BioImaging Meeting, in Okazaki, Japan. During the four months of the Challenge, the community accumulated over 500TB of OME-Zarr data spanning multiple imaging modalities, all publicly accessible via HTTP. Importantly, these data were not centrally stored or managed; rather, each participating institution hosted its own data, forming a nascent federated repository. A centralized viewer was developed to aggregate and present the metadata from all ... |
| Document Type: |
conference object |
| Language: |
unknown |
| Relation: |
https://zenodo.org/communities/cordi-2025/; https://zenodo.org/records/16735915; oai:zenodo.org:16735915; https://doi.org/10.5281/zenodo.16735915 |
| DOI: |
10.5281/zenodo.16735915 |
| Availability: |
https://doi.org/10.5281/zenodo.16735915; https://zenodo.org/records/16735915 |
| Rights: |
Creative Commons Attribution 4.0 International ; cc-by-4.0 ; https://creativecommons.org/licenses/by/4.0/legalcode |
| Accession Number: |
edsbas.3DD7FE75 |
| Database: |
BASE |