The NASA Space Weather Science Operations Center (SWxSOC)

An open-source, multi-mission Science Operations Center supporting NASA missions.

Summary

The NASA Space Weather Science Operations Center (SWxSOC) is a project to deliver and operate an open-source, multi-mission science operations center for the community, with an initial focus on NASA space weather missions that require consistent and timely delivery of data products. SWxSOC provides a reusable solution for transitioning data management from on-premises systems to the cloud, processing data files, monitoring operations, and providing a cloud-based analysis environment.
By using Amazon Web Services (AWS) and making its open-source tools available on GitHub, SWxSOC demonstrates NASA’s commitment to open-source standards and collaboration. It also demonstrates the practical benefits of cloud technology for data processing, analysis, and observability. This approach improves current operations and enables faster, more consistent deployments for future missions.

What SWxSOC Provides

A open source multi-mission Science Operations Center supporting NASA missions aligned with NASA's Open Science initiative (OSSI).

Services we provide:

  • Data Processing — Pipeline for science file calibration
  • Data Monitoring and Observability — Graphical interfaces for pipeline and telemetry monitoring and alerting
  • Payload Operations Support - Support the instrument teams as needed

Supported Missions

  • HERMES — Heliophysics Environmental and Radiation Measurement Experiment Suite, four in-situ instruments to be deployed on TBD.
  • PADRE — solar PolArization and Directivity X-ray Experiment, two solar-observing X-ray instruments (launched June 2025).
  • REACH - Responsive Environmental Assessment Commercially Hosted constellation of 32 dosimeter payloads hosted on a commercial LEO satellite constellation.
  • IMPAX - Imaging Microburst Precipitation with Atmospheric X-ray emissions to be launched in 2027.
  • GDC Trailblazer - Geospace Dynamics Constellation (GDC) Trailblazer to launch in December 2027.
IMPAX Logo HERMES Logo PADRE Logo GDC Trailblazer Logo

A High-Level Overview of the Open Source Data Processing Pipeline

  • Multi-mission cloud-based architecture enables easy scaling.
  • Infrastructure as Code — all cloud resource configurations available on GitHub (Terraform).
  • Hybrid Cloud — supports both on-premises and cloud environments for flexibility and redundancy.
  • Event-Driven Architecture — utilizes AWS EventBridge and Lambda functions for efficient task execution.
  • Modern design — Emphasizes scalability, maintainability, and ease of integration.
Pipeline architecture diagram

Managing Data Across On-Premises and Cloud

  • Supports data received from any source, including on-premises and cloud-based systems.
  • All data is mirrored between on-premises and cloud environments.
  • Cloud environments act as high-availability backups for on-premises data.
  • On-premises servers are linked to NASA archives for long-term storage.
Cloud data flow diagram

Utilizing the Cloud for Data Processing and Analysis

Sorting Lambda Function

  • Triggers on file events and moves files to relevant instrument buckets for processing.

Processing Lambda Function

  • Triggers on file events and processes files by using each mission's instrument packages to calibrate data to higher levels.
  • Pushes summary data products to databases for visualization.

Artifact Lambda Function

  • Tracks pipeline progress using the MetaTracker database.
  • Manages Slack notifications for efficient communication.
Lambda processing diagram

Running Auxiliary Tasks via the Executor Lambda Function

  • Runs auxiliary but regular tasks outside of the main processing pipeline.
  • Uses AWS EventBridge to schedule individual tasks.

Current Executor Activities:

  • Fetching ancillary data products (e.g. GOES XRS from NOAA APIs)
  • Creating Grafana annotations

Data and Pipeline Monitoring and Observability

Grafana, an open-source tool, enables the integration of multiple data sources to create insightful dashboards. Each mission is provided with its own dedicated Grafana server, allowing team members to log in with their GitHub account if they belong to the mission’s github organization.

Dashboards:

  • Housekeeping Dashboard — at-a-glance view of housekeeping data from science products.
  • MetaTracker Dashboard — metadata for all science products in the pipeline.
  • Processing Pipeline Dashboard — Lambda function logs and pipeline activity.
Grafana dashboard screenshot

Codebase Management and CI/CD Workflows

  • Organized Repositories — codebase split across multiple GitHub repos including Python packages for AWS Lambda functions.
  • Automated Rebuilds and Deployments — changes trigger automatic rebuilds via AWS CodeBuild CI/CD pipeline.
  • GitHub Actions Integration — linting, testing, and documentation updates on pull requests.
  • End-to-End Workflow Testing in Lambda Containers — changes tested directly within the Lambda container before deployment.

Standardized and Open-Source Instrument Processing Packages

Supporting NASA's Open-Source Science Initiative (OSSI) to make publicly funded scientific research transparent, accessible, and reproducible

  • Python Package Structure — Maintained and organized according to best practices for reusable and modular code.
  • Documentation next to code — documentation is maintained alongside the code/calibration products to ensure consistency and reproducibility.
  • Metadata Templating — templates for NASA-approved metadata (e.g. ISTP/CDF, Solarnet/FITS) to ensure FAIR principles.
  • Continuous Integration and Testing — ensures that changes are automatically tested and validated to maintain code quality and reliability.

Reprocessing Capabilities

A streamlined process for reprocessing data sets at different data levels.

  1. Request Submission — submit a pull request using the provided schema to the main Git repository, maintaining a full version history.
  2. Validation and Archiving — each request is validated and organized into the appropriate archive.
  3. Processing and Monitoring — valid requests from organization members are allowed to merge. Requesters monitor status through CI/CD or Grafana.
Reprocessing workflow diagram

Additional Tools and Packages

  • Mission Core & Instrument Packages — structured framework for the primary processing pipeline, adaptable for other pipelines.
  • SAMMI — Python package for templating for ISTP metadata for CDF and some NetCDF files.
  • SOLARNET — Python package for templating and validating common solar metadata for FITS files.
  • MetaTracker — Python package for monitoring raw binary and CDF files within a relational database.
  • FSWatcher & S3Watcher — Containerized tools that monitor directories and upload files to S3, and monitor S3 buckets and download new files to on-premises systems.

AI and Machine Learning

  • AI-supported code development and maintenance — leveraging artificial intelligence to assist in writing, reviewing, and maintaining code efficiently.
  • Natural language data search — leveraging Large Language Models to enable users to query, retrieve, and analyze relevant scientific and housekeeping data using natural language queries, improving accessibility and usability of scientific datasets.
  • Anomaly Detection — Identify anomalies, outlier detection, forecasting, and predictive maintenance using machine learning techniques.

Space Weather Science Alerts

  • In collaboration with the NASA Goddard Astrophysics Global Coordinate Network (GCN), SWxSOC can generate, subscribe to, and receive space weather science alerts.
    • The Astrophysics Gamma-ray Coordinates Network (GCN) enabled seminal breakthroughs in astrophysics by coordinating worldwide follow-up observations that revealed the nature of gamma-ray bursts.
    • Provides infrastructure to send streaming, real-time, machine-readable, and standardized science alert messages
    • Publisher-subscriber model -- users choose which topics to subscribe to such as gamma-ray bursts and gravity-waves
  • Leveraging this existing capability, heliophysics missions can share science alerts based on their data, and subscribe to alerts from other missions or external sources, enabling timely notifications of space weather science events.

Any Questions?

Feel free to contact us for more information!

Email: steven.christe@nasa.gov

References

SWxSOC logo NASA Goddard logo
SWxSOC GitHub QR code