Scalable processing of ozone data from the Odin satellite

The Odin satellite has measured ozone and related gases in the stratosphere and mesosphere for more than 15 years. This is one of the longest data series with global coverage that exists in atmospheric science. To make sure we can provide the best possible reference data set to the science community ESA has funded a total overhaul of the data set. This includes everything from review of the calibration algorithm of the sensor, to a complete review of the algorithms that provide concentrations of the different species at different heights and locations in the atmosphere.

The instrument is a passive sub millimeter radiometer, measuring emitted energy from different molecules through the limb of the atmosphere. The atmosphere is scanned over different heights around 60 times an orbit and the satellite makes about 15 orbits a day. That way measurements quickly cover most of the Earth’s atmosphere. As an example we have been able follow the development of the ozone depletion over the polar regions. We can also follow the circulation of air masses globally.

The REST API is the interface for all communication between the different components in the system. As an example the number crunching “processors” are not aware of the technology behind the REST API - they only ask for data and deliver results back to the API. This means it is easy for us to scale, move or change the underlying data storage technology.
The REST API is the interface for all communication between the different components in the system. As an example the number crunching “processors” are not aware of the technology behind the REST API – they only ask for data and deliver results back to the API. This means it is easy for us to scale, move or change the underlying data storage technology.

Technology

The Odin science community is using the SNIC Science Cloud. This enables scalable processing of Odin data both for testing of alternative algorithms and standard products.

To be able to process the dataset as quickly as possible we have packaged a set of “processors” in docker images ready to be deployed at any docker enabled computer. These docker images are self-contained with auxiliary datasets and code ready to be fed with measurements from the Odin satellite. Once the docker images are deployed the container asks the Odin-API, a REST service, for the next available measurement and starts immediately to crunch the data. When the process is done it delivers the results back to the Odin-API and starts over with a new measurement.

Users can browse and download data from the user interface http://odin.rss.chalmers.se and power-users can communicate directly with the REST-API to analyse data programmatically or start new processing campaigns.