Space Science Satellite Data Processing Framework Research and System Implementation

doi:10.16182/j.issn1004731x.joss.21-0133

Abstract

Abstract:

Based on the needs of China's space science strategy and series of on-orbit and forthcoming satellite missions in China's Strategic Priority Program on space science, data processing framework and key technologies of the satellite ground segment are studied. A general technical framework SDPF (space science satellite data processing framework) is proposed with two-layer scheduling engine, including mission-level and resource-level. The design and implementation of an automatic, efficient, real-time and standard space science satellite data processing system has been established. In this way, complicated processing procedures on large-scale data from multi-satellite missions and multi-payload can be completed quickly in parallel. It shows good application effects in terms of data processing capabilities, computing resource planning, reliability in response to failures, flexibility and scalability, andcan also provide some ideas for system design in other fields.

Key words: space science satellite, data processing framework, mission scheduling, resource scheduling, data processing workflow

CLC Number:

TP391

Wenzhen Ma, Ziming Zou, Jianhui Li, Qinsi Yu, Jizhou Tong, Jingjing Li. Space Science Satellite Data Processing Framework Research and System Implementation[J]. Journal of System Simulation, 2022, 34(7): 1605-1618.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Table 1

Space science satellite data product levels in SDPF

数据级别	处理程度	主要处理过程描述
0级	原始数据	卫星经过数传站或测控站下行，地面站经接收、解调、解码后还原得到的原始数据传输帧，以及返回式卫星任务通过回收舱获取的原始数据、通过地面实验或者地面联合观测等获取的地面原始数据等。
1A级	单轨源包编辑	对单次获取的原始数据进行源包级的编辑处理。对于地面站一次数传接收的原始数据，处理过程包括虚拟信道分离、传输误码剔除、解AOS帧、APID分路、源包提取、源包拼接、源包验证、打时间戳、时差校正、源包去重复、错误数据剔除、源包排序、格式化处理、质量标注等。对于来自地面比对实验、卫星回收舱存储器的数据，在数据组织方面比卫星下行的数传原始数据少了传输帧结构，数据处理过程直接从源包提取开始。
1B级	源包融合拼接与标准切分	对多次卫星过站（或多个地面实验、多段回收舱数据）生成的多个1A级产品按照APID进行融合拼接处理，处理过程包括数据融合、数据拼接、数据排重、时间排序、按照固定时间(如观测日期)或实验号或实验工况进行数据切分、格式化处理、质量标注等，生成标准化的源包数据产品。
1C级	单轨物理量处理	在1A级产品的基础上，进行解析源包、载荷数据内容提取、解压缩、数据内容验证、观测模式区分、实验号区分、数值类数据的物理量转换、图像和视频类数据的拼接与截取、格式转换、空间定位、质量标注等处理，生成单轨的物理量数据产品。
1D级	物理量级的融合拼接与标准切分	在1B级产品的基础上，进行解源包、载荷数据内容提取、解压缩、数据内容验证、观测模式区分、实验号区分、数值类数据的物理量转换、图像和视频类数据的拼接与截取、格式转换、空间定位、质量标注等处理，生成按照固定时间（如观测日期）或实验号或实验工况切分的标准物理量数据产品。
1Q级	科学快视处理	在1C或1D级产品的基础上，进行快速标定、数据统计分析、科学数据与工程数据匹配、多通道图像的时间匹配、图片转视频格式等处理后生成的具有直观可视化科学意义的数据产品。
2A级	标定处理	利用地面标定数据，对编辑级数据进行标定、交叉校正、时频或数值科学变换、观测质量分析等专业处理，根据观测天区、观测目标、空间事件、实验过程等科学研究需要进行再组织处理，生成标定数据产品。

Table 1

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Table 2

References 18

1	孙小涓, 石涛, 胡玉新, 等. 基于流式计算的空间科学卫星数据实时处理[J]. 计算机应用, 2019, 39(6): 1563-1568.
	Sun Xiaojuan, Shi Tao, Hu Yuxin, et al. Real-time Processing of Space Science Satellite Data Based on Stream Computing[J]. Journal of Computer Applications, 2019, 39(6): 1563-1568.
2	邹自明, 马文臻. 空间科学数据应用环境研究[J]. 中国计算机学会通讯, 2013, 9(9): 12-15.
	Zou Ziming, Ma Wenzhen. Research on Space Science Data Application Environment[J]. Communication of China Computer Federation, 2013, 9(9): 12-15.
3	黎建辉, 李跃鹏, 王华进, 等. 科学大数据管理技术与系统[J]. 中国科学院院刊, 2018, 33(8): 796-803.
	Li Jianhui, Li Yuepeng, Wang Huajin, et al. Scientific Big Data Management Technique and System[J]. Bulletin of Chinese Academy of Sciences 2018, 33(8): 796-803.
4	徐业帷. 科学工作流在空间信息处理领域的应用研究[D]. 北京: 中国科学院大学, 2016.
	Xu Yewei. Application of Scientific Workflow in Spatial Information Processing[D]. Beijing:University of Chinese Academy of Sciences, 2016.
5	孙小雅. 科学工作流建模方法研究[D]. 吉林: 吉林大学, 2019.
	Sun Xiaoya. Research on Modeling Method of Scientific Workflow[D]. Jilin: Jinlin University, 2019.
6	陈岳亭. 分布式流处理系统中的任务调度[D]. 济南: 山东大学, 2017.
	Chen Yueting. Task Scheduling in Distributed Stream Processing Systems[D]. Jinan:Shandong University, 2017.
7	戚红雨. 流式处理框架发展综述[J]. 信息化研究, 2019, 45(6): 1-8.
	Qi Hongyu. Survey of Distributed Data Flow Processing Framework[J]. Informatization Research, 2019, 45(6): 1-8.
8	张云峰. 基于策略的分布式实时计算系统框架的设计和实现[D]. 济南: 山东大学, 2019.
	Zhang Yunfeng. Design and Implementation of Distributed Real-time Computation System Framework Based on Strategy[D]. Jinan: Shandong University, 2019.
9	张智, 江果, 蒋鸣远. 面向军用网格的广域分布式数据处理框架[J]. 中国电子科学研究院学报, 2019, 14(1): 20-25.
	Zhang Zhi, Jiang Guo, Jiang Mingyuan. Wide Area Distributed Data Processing Framework for Military Grid[J]. Journal of China Academy of Electronics and Information Technology, 2019, 14(1): 20-25.
10	赵海升, 李承奎, 贾淑梅, 等. 天文卫星下传数据处理系统的规划[J]. 天文研究与技术, 2019, 16(3): 366-371.
	Zhao Haisheng, Li Chengkui, Jia Shumei, et al. Planning of Data Processing System for Astronomical Satellite[J]. Astronomical Research & Technology, 2019, 16(3): 366-371.
11	Casti M, Fineschi S, Messinew R, et al. Data Integration of Remote Sensing and in Situ Data from Several Solar Space Missions for Space Weather Services[C]//2017 Conference on Big Data from Space (BiDS'17). Toulouse, France: Publications Office of the European Union, 2017.
12	Montmory A, Chaumat L. Internet Major Actors Technologies Applied to Meteorological Satellite: Stream Data Applied to Data Processing[C]//2017 Conference on Big Data from Space (BiDS'17). Toulouse, France: Publications Office of the European Union, 2017.
13	Eynard-Bontemps G, Melet O, Palacin H, et al. From Hadoop Map/Reduce to Spark: GAIA and IMAGE Processing Use Cases[C]// 2017 Conference on Big Data from Space (BiDS'17). Toulouse, France: Publications Office of the European Union, 2017.
14	Edwards K, Shipman R F, Kester D, et al. The Data Processing Pipeline for the Herschel-HIFI Instrument[J]. Astronomy and Computing (S2213-1337), 2019, 27: 156-170.
15	Alkhanak E N, Lee S P. A Hyper-heuristic Cost Optimisation Approach for Scientific Workflow Scheduling in Cloud Computing[J]. Future Generation Computer Systems (S0167-739X), 2018, 86(9): 480-506.
16	Casas I, Taheri J, Ranjan R, et al. PSO-DS: A Scheduling Engine for Scientific Workflow Managers[J]. The Journal of Supercomputing (S0920-8542), 2017, 73(9): 3924-3947.
17	王甫棣, 赵希鹏, 王帅. 基于SOA的任务调度框架设计与实现[J]. 气象科技, 2020, 48(3): 362-367.
	Wang Fudi, Zhao Xipeng, Wang Shuai. Design and Implementation of a Task Scheduling Architecture Based on SOA[J]. Meteorological Science and Technology, 2020, 48(3): 362-367.
18	范程斌. 基于Mesos的分布式计算资源调度研究与应用[D]. 昆明: 昆明理工大学, 2017.
	Fan Chengbin. Distributed Computing Resources Scheduling Research and Practice Based on Mesos[D].Kunming:Kunming University of Science and Technology, 2017.

卫星	数据/GB	各级产品的处理时间/生成产品数量/生成产品体量
卫星	数据/GB	1A	1B	1C	1D
DAMPE	4.6	1min1s/14个/4.6GB	5min40s/14个/12.1GB	6min17s/4个/4.6GB	58s/4个/12.1GB
QUESS	0.13	15s/6个/149MB	30s/8个/256MB	7min46s/109个/15MB	无
SJ-10	9.9	3min19s/48个/8.8GB	4min56s/48个/12.9GB	5min56s/82个/116.3GB	3min56s/82个/125.1GB
HXMT	9.8	5min40s/36个/5.2GB	4min43s/118个/7.7GB	17s/29个/23MB	无

[1]	Zhang Xiao, Gao Yuan, Wang Xiaoliang, Ge Yiyong, Yang Haixiang, Wan Shupeng. Data Management of Data Processing Framework in Green Data Center [J]. Journal of System Simulation, 2016, 28(3): 592-599.
[2]	Chen Zhijia, Zhu Yuanchang, Di Yanqiang, Feng Shaochong. Design of IaaS Mode “Cloud Training” System [J]. Journal of System Simulation, 2015, 27(5): 1095-1104.