Data Processing#

Data are a first-class citizen of Constellation. They are transmitted as messages with one or more frames, and mechanisms are in place which ensure that no data message is lost in transmission. This document provides an overview of the concepts of data handling in Constellation and details some of the features available.

Transmitters & Receivers#

Data in Constellation are handled exclusively by satellites. Two types of satellites exits, transmitter satellites which generate data and send it off through the Constellation network, and receiver satellites which receive data from one or more transmitter satellites and process it, e.g. by storing it to disk.

See also

Details about how to implement data transmission in a satellite can be found in the application development guide.

Structure#

Data consist of individual messages that are grouped to so-called runs. A run represents a set of data recorded in known conditions and with a fixed set of parameters. A new run starts upon entering the RUN state of the satellite finite state machine and ends with the departure from this state. Each run is labeled by its run identifier, which is provided along with the start command sent to the satellite, and consists of individual data messages. The first message of a run is the so-called begin-of-run (BOR) message, the last one the end-of-run (EOR) message. This structure enables checks on sending and receiving side, e.g. that the connection is up and running or that all data have been transmitted correctly.

Messages#

Messages transmitted over the data communication channel of Constellation consist of a header with auxiliary information, and a payload. The content of header as well as payload differs for the different message types and is detailed below.

Data Messages#

Regular messages are data messages. Their header contains the sender name, the timestamp of message creation or dispatch, and a dictionary of message tags. In addition, data message headers contain a sequence number which is automatically incremented for each generated data message. This sequence is continuously checked at receiving end and therefore allows to directly detect missing messages.

Message tags may be added to the header by the sending code. If such information is available at the time of creating the message, it is recommended to encode the begin and end of the data block contained in the respective message, as integer values counting picoseconds since the start of the measurement run, in the tags timestamp_begin and timestamp_end.

The payload of data messages can be an arbitrary number of frames containing binary data. This data is not interpreted by the Constellation framework but e.g. directly written to file without further processing. Whether single frames or multiple frames per message are used is left to the sending code, but especially for small frame data blocks, multi-frame messages should be considered to optimize the network throughput.

Begin-of-Run & End-of-Run Messages#

The header tags of the BOR and EOR messages can be added by the satellite implementations in their respective starting and stopping functions. This can be used e.g. to record additional information such as a firmware version read from attached instrument hardware at the time of starting, or to indicate an accumulated metric at the time of stopping.

The payload of the BOR message contains the configuration of the transmitting satellite at the time of entering the RUN state. Depending on the actual implementation of the satellite itself, this allows to unequivocally infer the setup, state and configuration of the satellite and its attached hardware from the recorded data.

The payload of the EOR message contains a dictionary with run metadata generated by the Constellation framework. Typical run metadata entries are:

  • run_id: the run identifier of this run

  • condition: the run condition flags at the end of the run

  • condition_code: the run condition flags code

  • time_start: the starting time of the run, i.e. the time of sending the BOR message

  • time_end: the ending time of the run, i.e. the time of sending the EOR message

The run condition flags allows to assess the condition during the RUN state of the Constellation and thereby the quality of the data recorded. Runs are marked with any of the following run condition flags:

  • GOOD (code: 0x00, no flag set): The run has concluded normally, no other information has been provided by the sender

  • TAINTED (code: 0x01): The data has been marked as tainted by the sender

  • INCOMPLETE (code: 0x02): The receiver has noticed missing messages in the sequence

  • INTERRUPTED (code: 0x04): The run has been interrupted by this sender because of a failure condition elsewhere in the Constellation

  • ABORTED (code: 0x08): The run has been aborted by the sender and the EOR message has been appended by the receiver