Contents
Preface xi
1 Remote Sensing and Cloud Computing 1
1.1 Remote Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Remote sensing de_nition . . . . . . . . . . . . . . . . . 1
1.1.2 Remote sensing big data . . . . . . . . . . . . . . . . . 2
1.1.3 Applications of remote sensing big data . . . . . . . . 3
1.1.4 Challenges of remote sensing big data . . . . . . . . . 5
1.1.4.1 Data integration challenges . . . . . . . . . . 5
1.1.4.2 Data processing challenges . . . . . . . . . . 5
1.2 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Cloud service models . . . . . . . . . . . . . . . . . . . 6
1.2.2 Cloud deployment models . . . . . . . . . . . . . . . . . 7
1.2.3 Security in the Cloud . . . . . . . . . . . . . . . . . . . 7
1.2.4 Open-source Cloud frameworks . . . . . . . . . . . . . 8
1.2.4.1 OpenStack . . . . . . . . . . . . . . . . . . . 8
1.2.4.2 Apache CloudStack . . . . . . . . . . . . . . 10
1.2.4.3 OpenNebula . . . . . . . . . . . . . . . . . . 10
1.2.5 Big data in the Cloud . . . . . . . . . . . . . . . . . . 12
1.2.5.1 Big data management in the Cloud . . . . . 12
1.2.5.2 Big data analytics in the Cloud . . . . . . . 12
1.3 Cloud Computing in Remote Sensing . . . . . . . . . . . . . 14
2 Remote Sensing Data Integration in a Cloud Computing
Environment 17
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Background on Architectures for Remote Sensing Data Integration 18
2.2.1 Distributed integration of remote sensing data . . . . 18
2.2.2 OODT: a data integration framework . . . . . . . . . 19
2.3 Distributed Integration of Multi-Source Remote Sensing Data 20
2.3.1 The ISO 19115-basedc metadata transformation . . . . 20
2.3.2 Distributed multi-source remote sensing data integration 22
2.4 Experiment and Analysis . . . . . . . . . . . . . . . . . . . . 24
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 Remote Sensing Data Organization and Management in a
Cloud Computing Environment 29
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Preliminaries and Related Techniques . . . . . . . . . . . . . . 31
3.2.1 Spatial organization of remote sensing data . . . . . . . 31
3.2.2 MapReduce and Hadoop . . . . . . . . . . . . . . . . . 32
3.2.3 HBase . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.4 Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 LSI Organization Model of Multi-Source Remote Sensing Data 35
3.4 Remote Sensing Big Data Management in a Parallel File System 38
3.4.1 Full-text index of multi-source remote sensing metadata 38
3.4.2 Distributed data retrieval . . . . . . . . . . . . . . . . 40
3.5 Remote Sensing Big Data Management in the Hadoop Ecosystem . . . 42
3.5.1 Data organization and storage component . . . . . . . 42
3.5.2 Data index and search component . . . . . . . . . . . 43
3.6 Metadata Retrieval Experiments in a Parallel File System . . 45
3.6.1 LSI model-based metadata retrieval experiments in a parallel _le system . . . . . 45
3.6.2 Comparative experiments and analysis . . . . . . . . . 48
3.6.2.1 Comparative experiments . . . . . . . . . . . 48
3.6.2.2 Results analysis . . . . . . . . . . . . . . . . 49
3.7 Metadata Retrieval Experiments in the Hadoop Ecosystem . . 51
3.7.1 Time comparisons of storing metadata in HBase . . . 52
3.7.2 Time comparisons of loading metadata from HBase to
Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 52
3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4 High Performance Remote Sensing Data Processing in a
Cloud Computing Environment 55
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2 High Performance Computing for RS Big Data: State of the Art 58
4.2.1 Cluster computing for RS data processing . . . . . . . 58
4.2.2 Cloud computing for RS data processing . . . . . . . . 59
4.2.2.1 Programming models for big data . . . . . . 60
4.2.2.2 Resource management and provisioning . . . 60
4.3 Requirements and Challenges: RSCloud for RS Big Data . . . 61
4.4 pipsCloud: High Performance Remote Sensing Clouds . . . . 62
4.4.1 The system architecture of pipsCloud . . . . . . . . . 63
4.4.2 RS data management and sharing . . . . . . . . . . . 65
4.4.2.1 HPGFS: distributed RS data storage with
application-aware data layouts and copies . . . 67
4.4.2.2 RS metadata management with NoSQL
database . . . . . . . . . . . . . . . . . . . . 68
4.4.2.3 RS data index with Hilbert R+tree . . . . . 69
4.4.2.4 RS data subscription and distribution . . . . . 71
4.4.3 VE-RS: RS-speci_c HPC environment as a service . . 72
4.4.3.1 On-demand HPC cluster platforms with
bare-metal provisioning . . . . . . . . . . . . 73
4.4.3.2 Skeletal programming for RS big data processing
. . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.4 VS-RS: Cloud-enabled RS data processing system . . . 77
4.4.4.1 Dynamic workow processing for RS applications
in the Cloud . . . . . . . . . . . . . . . 78
4.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 82
4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5 Programming Technologies for High Performance Remote
Sensing Data Processing in a Cloud Computing Environment 89
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.3 Problem De_nition . . . . . . . . . . . . . . . . . . . . . . . 92
5.3.1 Massive RS data . . . . . . . . . . . . . . . . . . . . . 92
5.3.2 Parallel programmability . . . . . . . . . . . . . . . . 93
5.3.3 Data processing speed . . . . . . . . . . . . . . . . . . 94
5.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 94
5.4.1 Generic algorithm skeletons for remote sensing applications
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4.1.1 Categories of remote sensing algorithms . . . 98
5.4.1.2 Generic RS farm-pipeline skeleton . . . . . . 98
5.4.1.3 Generic RS image-wrapper skeleton . . . . . 102
5.4.1.4 Generic feature abstract skeleton . . . . . . . 105
5.4.2 Distributed RS data templates . . . . . . . . . . . . . 108
5.4.2.1 RSData templates . . . . . . . . . . . . . . . 108
5.4.2.2 Dist RSData templates . . . . . . . . . . . . . 111
5.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 115
5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6 Construction and Management of Remote Sensing Produc-
tion Infrastructures across Multiple Satellite Data Centers 121
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.3 Infrastructures Overview . . . . . . . . . . . . . . . . . . . . 124
6.3.1 Target environment . . . . . . . . . . . . . . . . . . . 124
6.3.2 MDCPS infrastructures overview . . . . . . . . . . . . 125
6.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 128
6.4.1 Data management . . . . . . . . . . . . . . . . . . . . 128
6.4.1.1 Spatial metadata management for
co-processing . . . . . . . . . . . . . . . . . . 130
6.4.1.2 Distributed _le management . . . . . . . . . . 131
6.4.2 Workow management . . . . . . . . . . . . . . . . . . 133
6.4.2.1 Workow construction . . . . . . . . . . . . . 136
6.4.2.2 Task scheduling . . . . . . . . . . . . . . . . . 137
6.4.2.3 Workow fault-tolerance . . . . . . . . . . . . 141
6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.5.1 Related experiments on dynamic data management . . 142
6.5.2 Related experiments on workow management . . . . 146
6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.6.1 System architecture . . . . . . . . . . . . . . . . . . . . 147
6.6.2 System feasibility . . . . . . . . . . . . . . . . . . . . . 148
6.6.3 System scalability . . . . . . . . . . . . . . . . . . . . 148
6.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . 148
7 Remote Sensing Product Production in an OpenStack-Based
Cloud Computing Environment 151
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.2 Background and Related Work . . . . . . . . . . . . . . . . . 153
7.2.1 Remote sensing products . . . . . . . . . . . . . . . . 153
7.2.1.1 Fine processing products . . . . . . . . . . . 154
7.2.1.2 Inversion index products . . . . . . . . . . . 154
7.2.1.3 Thematic products . . . . . . . . . . . . . . . 154
7.2.2 Remote sensing production system . . . . . . . . . . . 155
7.3 Cloud-Based Remote Sensing Production System . . . . . . 156
7.3.1 Program framework . . . . . . . . . . . . . . . . . . . 156
7.3.2 System architecture . . . . . . . . . . . . . . . . . . . . 157
7.3.3 Knowledge base and inference rules . . . . . . . . . . . 159
7.3.3.1 The upper and lower hierarchical relationship
database . . . . . . . . . . . . . . . . . . . . 159
7.3.3.2 Input/output database of every kind of remote
sensing product . . . . . . . . . . . . . . . . 160
7.3.3.3 Inference rules for production demand data
selection . . . . . . . . . . . . . . . . . . . . . 161
7.3.3.4 Inference rules for workow organization . . . 161
7.3.4 Business logic . . . . . . . . . . . . . . . . . . . . . . . 162
7.3.5 Active service patterns . . . . . . . . . . . . . . . . . . 165
7.4 Experiment and Case Study . . . . . . . . . . . . . . . . . . . 167
7.4.1 Global scale remote sensing production . . . . . . . . . 167
7.4.2 Regional scale mosaic production . . . . . . . . . . . . 168
7.4.3 Local scale change detection . . . . . . . . . . . . . . . 170
7.4.3.1 Remote sensing data cube . . . . . . . . . . . . 171
7.4.3.2 Local scale time-series production . . . . . . . 171
7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8 Knowledge Discovery and Information Analysis from Remote
Sensing Big Data 175
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.2 Preliminaries and Related Work . . . . . . . . . . . . . . . . 176
8.2.1 Knowledge discovery categories . . . . . . . . . . . . . 176
8.2.2 Knowledge discovery methods . . . . . . . . . . . . . . 178
8.2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . 179
8.3 Architecture Overview . . . . . . . . . . . . . . . . . . . . . 180
8.3.1 Target data and environment . . . . . . . . . . . . . . 180
8.3.2 FRSDC architecture overview . . . . . . . . . . . . . . . 181
8.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 182
8.4.1 Feature data cube . . . . . . . . . . . . . . . . . . . . 182
8.4.1.1 Spatial feature object in FRSDC . . . . . . . 182
8.4.1.2 Data management . . . . . . . . . . . . . . . 182
8.4.2 Distributed executed engine . . . . . . . . . . . . . . . 184
8.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9 Automatic Construction of Cloud Computing Infrastructures
in Remote Sensing 191
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
9.2 De_nition of the Remote Sensing Oriented Cloud Computing
Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.2.1 Generally used cloud computing infrastructure . . . . 193
9.2.2 Remote sensing theme oriented cloud computing infrastructure . . . . 193
9.3 Design and Implementation of Remote Sensing Oriented Cloud
Computing Infrastructure . . . . . . . . . . . . . . . . . . . . 195
9.3.1 System architecture design . . . . . . . . . . . . . . . 195
9.3.2 System workow design . . . . . . . . . . . . . . . . . 196
9.3.3 System module design . . . . . . . . . . . . . . . . . . 198
9.4 Key Technologies of Remote Sensing Oriented Cloud Infrastructure
Automatic Construction . . . . . . . . . . . . . . . . . . 200
9.4.1 Automatic deployment based on OpenStack and Salt- Stack . . . . . 200
9.4.2 Resource monitoring based on Ganglia . . . . . . . . . 203
9.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
10 Security Management in a Remote-Sensing-Oriented Cloud
Computing Environment 207
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
10.2 User Behavior Authentication Scheme . . . . . . . . . . . . . 209
10.2.1 User behavior authentication set . . . . . . . . . . . . 209
10.2.2 User behavior authentication process . . . . . . . . . . 210
10.3 The Method for User Behavior Trust Level Prediction . . . . 213
10.3.1 Bayesian network model for user behavior trust prediction . . 213
10.3.2 The calculation method of user behavior prediction . . 214
10.3.2.1 Prior probability calculation of user behavior attribute level . . . 214
10.3.2.2 Conditional probability of behavioral authentication set . . . 215
10.3.2.3 Method of calculating behavioral trust level . 216
10.3.3 User behavior trust level prediction example and analysis 216
10.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
11 A Cloud-Based Remote Sensing Information Service System
Design and Implementation 221
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
11.2 Remote Sensing Information Service Mode Design . . . . . . 223
11.2.1 Overall process of remote sensing information service mode . . . 223
11.2.2 Service mode design of RSDaaS . . . . . . . . . . . . . 224
11.2.3 Service mode design of RSDPaaS . . . . . . . . . . . . 225
11.2.4 Service mode design of RSPPaaS . . . . . . . . . . . . 226
11.2.5 Service mode design of RSCPaaS . . . . . . . . . . . . 228
11.3 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . 229
11.4 Functional Module Design . . . . . . . . . . . . . . . . . . . 233
11.4.1 Function module design of RSDaaS . . . . . . . . . . . 233
11.4.2 Function module design of RSDPaaS . . . . . . . . . . 234
11.4.3 Function module design of RSPPaaS . . . . . . . . . . 235
11.4.4 Function module design of RSCPaaS . . . . . . . . . . . 237
11.5 Prototype System Design and Implementation . . . . . . . . 238
11.5.1 RSDaaS subsystem . . . . . . . . . . . . . . . . . . . . 240
11.5.2 RSDPaaS subsystem . . . . . . . . . . . . . . . . . . . 242
11.5.3 RSPPaaS subsystem . . . . . . . . . . . . . . . . . . . 243
11.5.4 RSCPaaS subsystem . . . . . . . . . . . . . . . . . . . 244
11.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Bibliography 247
Index 279