High Throughput Processing of Space Debris Data

High-Throughput Processing of
Space Debris Data
Andreas Schreiber, Michael Meinel
German Aerospace Center (DLR)
Simulation and Software Technology,
Berlin / Cologne, Germany
PyData Seattle
July 26, 2015
> PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015DLR.de • Chart 1

• Space Debris
• BACARDI
• Skynet
Outline

Major research areas of DLR
• Aeronautics
• Space
• Transportation
• Energy
• Security
Software research and dev.
• Software engineering, HPC,
distributed systems, …
DLR
Aerospace Research Center, Space Agency

Space Debris

Lower Orbit
300 – 2000 km
Geostationary Orbit
≈36,000 km

Space Flight

• Almost 5,000 rockets launched
• More than 6,000 satellites placed in orbit
• About 1,000 active satellites today
(that’s 7% of known objects in orbit)
About 50 Years of Space Flight

Defunct objects in space
• Disused satellites & rocket stages
• Fragments from explosions &
collisions
• Released object (tools etc.)
• Slag of solid-fueled rockets
• … and other sources of small
particles
Space Debris

Space Debris
January 2007:
FengYun-1C
February 2009:
Collision
Iridium-33 &
Kosmos-2251

Space Debris
Iridium-33 & Kosmos-2251:
Debris fields after 50 minutes
Fengyun-1C debris one month
after its disintegration

16,300 Objects in Catalogue

29,000 Objects Larger than 10 cm

750,000 Objects Larger than 1 cm

150M Objects Larger than 1 mm

Impact
Laboratory Impact Aluminum Sphere / Block

Impact
Real Debris Object on a Satellite Solar Panel

Distribution of Space Debris
Source: NASA Orbital Debris Quarterly News 2/2012
ISS
Many Earth Observation
Satellites (e.g., Envisat)
TerraSAR-X

Collision Prediction
International Space Station
DLR.de • Chart 18 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015

http://rbth.com/news/2015/07/26/iss_successfully_ducks_from_space_debris_48023.html

Detecting Space Debris
Sources: FhG FHR, AIUB, ESA
Optical Radar Laser

SMARTnet
• Theory: complete coverage
with 3 locations
• Northern / southern hemisphere
for compensating seasonal
variations (6 locations)
• Telerobotical operation
• Optimized scheduler for all
telescopes
Operational Collision Avoidance
Global network for monitoring the geostationary ring

Operational Collision Avoidance
Global network for monitoring the geostationary ring

Object Identification with Optical Measurements
DLR.de • Chart 24 > PyData Seattle > A. Schreiber • High-Throughput Processing of Space Debris Data > 26.07.2015

• Separation of real / false tracklets above threshold of loss
function (chi-squared distribution)
• Filter rate depends on accuracy, time difference, survey
strategy, …
Object Correlation

BACARDI

Objective
• Database with preferably high completeness and high
accuracy
• Primary source: sensor data and operator data
• Secondary source: externally generated ephemerides
BACARDI
Backbone Catalogue of Relational Debris Information

Science and Research
• Database of more than 1M objects
• High-performance computational…
• object correlation
• orbit determination
• propagation
• object identification
• detection of maneuvers and
fragmentations
BACARDI

Mission Support
• Orbit information
• Collision prediction
• Re-entry prediction
Real-time operation
BACARDI

BACARDI Overview

BACARDI Overview
Sensor network
BACARDI
Sensor
scheduler
User
Groups
External orbital /
object information
Optical Radar Laser
Database
User Interface
Processes

BACARDI

External Data Sources
Example: space-track.org

BACARDI
System Components
RDBMS
Cache
Import
Processing
Python
FORTRAN
Middleware
Security
Traceability
• Simple
• Parallel
• Fast
• Huge number
Export

Skynet

Specialized middleware with following objectives
• Decentralized message queues
• Scalable, self-organizing network
• Minimal network overhead
• Platform independent
• Operation: Linux
• Development: Windows, Mac OSX
• Automatic provenance recording
Skynet
Network for surveillance of the sky

ZeroMQ
• Decentralized network infrastructure and messaging
Protocol Buffers
• Highly efficient serialization of data
SQLAlchemy
• Database mapping
ZeroMQ and Protocol Buffers are available for a wide range
of platforms and programming languages.
Skynet
Technology Decisions

Skynet
Modules
skynet
skynet.network Messaging
skynet.model Data Model
skynet.process Processes
skynet.record Provenance

• Socket library for messaging
• Multi-platform, multi-language
• Fast and small
• Many connection patterns
ZeroMQ
www.zeromq.org

• Network layer based on ZeroMQ
• „Device“ as abstract endpoint
• XML configuration: ZDCF (ZeroMQ Device Configuration File)
• Automatically connected to other matching „devices“
(Peer-to-Peer)
• Compression of large data packets
• (De-)serialization via Protocol Buffer messages
• Prepared for encryption and signing
• Synchronous or asynchronous
skynet.network

skynet.network

• Specified by Google (https://github.com/google/protobuf)
• Serializing (encoding) structured data
• Efficient and extensible
• Language-neutral, platform-neutral
• Interface definition language to describe data structure
Protocol Buffers (“ProtoBuf”)
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}

• All models are defined as Protocol Buffer messages
• Provides basic data type as ProtoBuf (e.g., 3D vector, 6D
vector, orbital data, …)
• Interfaces for caching
• ORM for ProtoBuf messages with SQLAlchemy (database-
neutral)
• Very few additional annotation
• Compiled to Python module using protoc
• skynet.network device for database connection
skynet.model

If a model should be mapped to a database table, it should
have the db_table message option extension set
skynet.model
Model Definition
// Commonly used message types.
package bacardi.model.common;
import "skynet/model/proto/options.proto";
// A spatial vector with x, y and z double components
message vec3d {
required double x = 1;
required double y = 2;
required double z = 3;
}

skynet.model
BACARDI Network Message
HEADER BLOCK 1 BLOCK 2 BLOCK N NULL…
DEST NULL SOURCE NULL… …
TYPE COMMAND NULL [SIG]…[KEY]

// Orbits and orbital parameters.
package bacardi.model.orbit;
import "skynet/model/proto/options.proto";
import "bacardi/model/common.proto";
// One entry of a propagation error dataset.
message properr_item {
required double days_prop = 1;
required double altitude = 2;
required double mean_f107 = 3;
required common.vec6d std_state_rtn = 4;
}
// Propagation error dataset.
message properr {
option (skynet.model.db_table) = {
name: "properr";
};
Model Example
Orbits and Orbital Parameters

skynet.model
Initialize a Database with All the Models
# Import skynet ORM and database.
from skynet.model import mapping, db
# Search for models in 'bacardi.model'.
# Alternatively, you could also pass a Python module.
mapping.Mapper.scan_module('bacardi.model’)
# Connect to SQLite database
store = db.DataStore('sqlite:///bacardi.db')
# Create all tables (SQL schemas) for
# the registered models.
store.create_tables()

Two ways of accessing data
• Sending a Query to the DataStore
• Using the db extension
• More compact code and hence better readability
• To retrieve the ISS data, one could simply:
skynet.model
Data access
# Import data model for objects.
from bacardi.model import object_pb2
# Get the obj with NORAD ID 25544 (ISS Zarya).
iss = object_pb2.obj.get(norad_id=25544)

Abstraction layer for processes
• Decoupling of processes from network layer
• Automatic caching, if needed
Easy integration of processes
• as sub process (data via disk I/O or named pipes)
• as FORTRAN or C module with Python wrapper
• as pure Python module
Worker device for multiple processes
• One device per CPU / Core
skynet.process

Provenance is defined as a
record that describes
the people, institutions,
entities, and activities
involved in producing,
influencing, or delivering
a piece of data or a thing.
(W3C Provenance Working Group,
http://www.w3.org/2011/prov)
Provenance
Agent
Entity Activity
used
wasGeneratedBy
wasDerivedFrom
wasStartedBy
wasEndedBy
wasAssociatedWith
actedOnBehalfOf

Provenance recording
• Recording of all activities during runtime
• Multiple Provenance stores
• Provenance information stored as graph in Neo4j
Traceability and provability of all processed data
• Backtracking of each produced product (ephemerides,
state vectors, correlated objects, …)
• Reproducibility of products and data generated
Work-in-progress
skynet.record

Technical
• Graphical User Interfaces
• Web (Django) and desktop (PyQt)
• Workflow
Organizational
• Operational for simple workflows end of 2015 at DLR
• Extended SSA versions for DLR and German Army
• Other use cases for Skynet (e.g., data acquisition for
rocket motors)
BACARDI and Skynet
Current and Future Work

Thank You!
Questions?
Andreas.Schreiber@dlr.de
www.DLR.de/sc | @onyame

High Throughput Processing of Space Debris Data

More Related Content

What's hot

Similar to High Throughput Processing of Space Debris Data

More from Andreas Schreiber

Recently uploaded

High Throughput Processing of Space Debris Data

Editor's Notes