PDS3 is a data standard used extensively by NASA for archiving data from science missions, maintained by JPL. While being replaced by PDS4, PDS4 covers all currently active missions and those covering the history of US space exploration.
The R pds3 package provides tools for parsing PDS3 data, particularly the ODL label format which describes all the metadata of data collection. Want to plot a heatmap of Mars of all the images taken? This is the package for you!
The package is not yet on CRAN, so to install you’ll want to use devtools to get the development version from Github.
# Install devtools if you don't already have it
install.packages("devtools")
# Install the development version of pds3
devtools::install_github("mwaldstein/pds3")If you’re exploring this package, chances are you already have a collection of PDS files, but if you’re interested in exploring NASA data, the place to get started is The Planetary Data System.
Once you find a mission you’re interested in exploring, finding the data explorer is typically straight forward. The metadata this package processes is typically stored in .lbl files. For instance, we’ll look at the labels from the Mars Reconnaissance Orbiter’s HiRise experiment, particularly the data associated with this image.
We’ll start by grabbing the label (LBL) file.
href <- 'http://hirise.lpl.arizona.edu/PDS/RDR/ESP/ORB_011700_011799/ESP_011707_1440/ESP_011707_1440_COLOR.LBL'
req <- curl::curl_fetch_memory(href)
dat <- rawToChar(req$content)Now we have the data, we’ll process it using parse_pds3.
library(pds3)
res <- pds3_read(dat)
str(res$odl)
#> List of 26
#> $ PDS_VERSION_ID : chr "PDS3"
#> $ NOT_APPLICABLE_CONSTANT : int -9998
#> $ DATA_SET_ID : chr "MRO-M-HIRISE-3-RDR-V1.1"
#> $ DATA_SET_NAME : chr "MRO MARS HIGH RESOLUTION IMAGING SCIENCE\r\n EXPERIMENT RDR V1.1"
#> $ PRODUCER_INSTITUTION_NAME : chr "UNIVERSITY OF ARIZONA"
#> $ PRODUCER_ID : chr "UA"
#> $ PRODUCER_FULL_NAME : chr "ALFRED MCEWEN"
#> $ OBSERVATION_ID : chr "ESP_011707_1440"
#> $ PRODUCT_ID : chr "ESP_011707_1440_COLOR"
#> $ PRODUCT_VERSION_ID : chr "2.0"
#> $ INSTRUMENT_HOST_NAME : chr "MARS RECONNAISSANCE ORBITER"
#> $ INSTRUMENT_HOST_ID : chr "MRO"
#> $ INSTRUMENT_NAME : chr "HIGH RESOLUTION IMAGING SCIENCE EXPERIMENT"
#> $ INSTRUMENT_ID : chr "HIRISE"
#> $ TARGET_NAME : chr "MARS"
#> $ MISSION_PHASE_NAME : chr "EXTENDED SCIENCE PHASE"
#> $ ORBIT_NUMBER : int 11707
#> $ SOURCE_PRODUCT_ID : chr [1:12] "ESP_011707_1440_BG12_0" "ESP_011707_1440_BG12_1" "ESP_011707_1440_RED4_0" "ESP_011707_1440_RED4_1" ...
#> $ RATIONALE_DESC : chr "Channels on crater rim"
#> $ SOFTWARE_NAME : chr "PDS_to_JP2 v3.15.5 (1.49 2008/07/12 04:09:51)"
#> $ IMAGE_MAP_PROJECTION :List of 24
#> ..$ ^DATA_SET_MAP_PROJECTION :List of 2
#> .. ..$ value : chr "DSMAP.CAT"
#> .. ..$ offset: num -1
#> ..$ MAP_PROJECTION_TYPE : chr "EQUIRECTANGULAR"
#> ..$ PROJECTION_LATITUDE_TYPE : chr "PLANETOCENTRIC"
#> ..$ A_AXIS_RADIUS :List of 2
#> .. ..$ value: num 3390
#> .. ..$ unit : chr "KM"
#> ..$ B_AXIS_RADIUS :List of 2
#> .. ..$ value: num 3390
#> .. ..$ unit : chr "KM"
#> ..$ C_AXIS_RADIUS :List of 2
#> .. ..$ value: num 3390
#> .. ..$ unit : chr "KM"
#> ..$ COORDINATE_SYSTEM_NAME : chr "PLANETOCENTRIC"
#> ..$ POSITIVE_LONGITUDE_DIRECTION: chr "EAST"
#> ..$ KEYWORD_LATITUDE_TYPE : chr "PLANETOCENTRIC"
#> ..$ CENTER_LATITUDE :List of 2
#> .. ..$ value: num -35
#> .. ..$ unit : chr "DEG"
#> ..$ CENTER_LONGITUDE :List of 2
#> .. ..$ value: num 180
#> .. ..$ unit : chr "DEG"
#> ..$ LINE_FIRST_PIXEL : int 1
#> ..$ LINE_LAST_PIXEL : int 55636
#> ..$ SAMPLE_FIRST_PIXEL : int 1
#> ..$ SAMPLE_LAST_PIXEL : int 10633
#> ..$ MAP_PROJECTION_ROTATION :List of 2
#> .. ..$ value: num 0
#> .. ..$ unit : chr "DEG"
#> ..$ MAP_RESOLUTION :List of 2
#> .. ..$ value: num 236637
#> .. ..$ unit : chr "PIX/DEG"
#> ..$ MAP_SCALE :List of 2
#> .. ..$ value: num 0.25
#> .. ..$ unit : chr "METERS/PIXEL"
#> ..$ MAXIMUM_LATITUDE :List of 2
#> .. ..$ value: num -35.4
#> .. ..$ unit : chr "DEG"
#> ..$ MINIMUM_LATITUDE :List of 2
#> .. ..$ value: num -35.6
#> .. ..$ unit : chr "DEG"
#> ..$ LINE_PROJECTION_OFFSET :List of 2
#> .. ..$ value: num -8377212
#> .. ..$ unit : chr "PIXEL"
#> ..$ SAMPLE_PROJECTION_OFFSET :List of 2
#> .. ..$ value: num 7151126
#> .. ..$ unit : chr "PIXEL"
#> ..$ EASTERNMOST_LONGITUDE :List of 2
#> .. ..$ value: num 143
#> .. ..$ unit : chr "DEG"
#> ..$ WESTERNMOST_LONGITUDE :List of 2
#> .. ..$ value: num 143
#> .. ..$ unit : chr "DEG"
#> $ TIME_PARAMETERS :List of 6
#> ..$ MRO:OBSERVATION_START_TIME : POSIXlt[1:1], format: "2009-01-24 17:07:14"
#> ..$ START_TIME : POSIXlt[1:1], format: "2009-01-24 17:07:14"
#> ..$ SPACECRAFT_CLOCK_START_COUNT: chr "917284056:11103"
#> ..$ STOP_TIME : POSIXlt[1:1], format: "2009-01-24 17:07:19"
#> ..$ SPACECRAFT_CLOCK_STOP_COUNT : chr "917284060:44707"
#> ..$ PRODUCT_CREATION_TIME : POSIXlt[1:1], format: "2009-09-12 04:31:46"
#> $ INSTRUMENT_SETTING_PARAMETERS:List of 4
#> ..$ MRO:CCD_FLAG : chr [1:14] "ON" "ON" "ON" "ON" ...
#> ..$ MRO:BINNING : int [1:14] -9998 -9998 -9998 -9998 1 1 -9998 -9998 -9998 -9998 ...
#> ..$ MRO:TDI : int [1:14] -9998 -9998 -9998 -9998 128 128 -9998 -9998 -9998 -9998 ...
#> ..$ MRO:SPECIAL_PROCESSING_FLAG: chr [1:14] "NULL" "NULL" "NULL" "NULL" ...
#> $ VIEWING_PARAMETERS :List of 7
#> ..$ INCIDENCE_ANGLE :List of 2
#> .. ..$ value: num 60.9
#> .. ..$ unit : chr "DEG"
#> ..$ EMISSION_ANGLE :List of 2
#> .. ..$ value: num 0.27
#> .. ..$ unit : chr "DEG"
#> ..$ PHASE_ANGLE :List of 2
#> .. ..$ value: num 61
#> .. ..$ unit : chr "DEG"
#> ..$ LOCAL_TIME :List of 2
#> .. ..$ value: num 16
#> .. ..$ unit : chr "LOCALDAY/24"
#> ..$ SOLAR_LONGITUDE :List of 2
#> .. ..$ value: num 197
#> .. ..$ unit : chr "DEG"
#> ..$ SUB_SOLAR_AZIMUTH:List of 2
#> .. ..$ value: num 193
#> .. ..$ unit : chr "DEG"
#> ..$ NORTH_AZIMUTH :List of 2
#> .. ..$ value: num 270
#> .. ..$ unit : chr "DEG"
#> $ COMPRESSED_FILE :List of 8
#> ..$ FILE_NAME : chr "ESP_011707_1440_COLOR.JP2"
#> ..$ RECORD_TYPE : chr "UNDEFINED"
#> ..$ ENCODING_TYPE : chr "JP2"
#> ..$ ENCODING_TYPE_VERSION_NAME: chr "ISO/IEC15444-1:2004"
#> ..$ INTERCHANGE_FORMAT : chr "BINARY"
#> ..$ UNCOMPRESSED_FILE_NAME : chr "ESP_011707_1440_COLOR.IMG"
#> ..$ REQUIRED_STORAGE_BYTES :List of 2
#> .. ..$ value: int NA
#> .. ..$ unit : chr "BYTES"
#> ..$ ^DESCRIPTION :List of 2
#> .. ..$ value : chr "JP2INFO.TXT"
#> .. ..$ offset: num -1
#> $ UNCOMPRESSED_FILE :List of 6
#> ..$ FILE_NAME : chr "ESP_011707_1440_COLOR.IMG"
#> ..$ RECORD_TYPE : chr "FIXED_LENGTH"
#> ..$ RECORD_BYTES:List of 2
#> .. ..$ value: int 21266
#> .. ..$ unit : chr "BYTES"
#> ..$ FILE_RECORDS: int 166908
#> ..$ ^IMAGE :List of 2
#> .. ..$ value : chr "ESP_011707_1440_COLOR.IMG"
#> .. ..$ offset: num -1
#> ..$ IMAGE :List of 19
#> .. ..$ DESCRIPTION : chr "HiRISE projected and mosaicked product"
#> .. ..$ LINES : int 55636
#> .. ..$ LINE_SAMPLES : int 10633
#> .. ..$ BANDS : int 3
#> .. ..$ SAMPLE_TYPE : chr "MSB_UNSIGNED_INTEGER"
#> .. ..$ SAMPLE_BITS : int 16
#> .. ..$ SAMPLE_BIT_MASK : int 1023
#> .. ..$ SCALING_FACTOR : num 0.000136
#> .. ..$ OFFSET : num 0.0332
#> .. ..$ BAND_STORAGE_TYPE : chr "BAND_SEQUENTIAL"
#> .. ..$ CORE_NULL : int 0
#> .. ..$ CORE_LOW_REPR_SATURATION : int 1
#> .. ..$ CORE_LOW_INSTR_SATURATION : int 2
#> .. ..$ CORE_HIGH_REPR_SATURATION : int 1023
#> .. ..$ CORE_HIGH_INSTR_SATURATION: int 1022
#> .. ..$ CENTER_FILTER_WAVELENGTH :List of 3
#> .. .. ..$ :List of 2
#> .. .. .. ..$ value: int 900
#> .. .. .. ..$ unit : chr "NM"
#> .. .. ..$ :List of 2
#> .. .. .. ..$ value: int 700
#> .. .. .. ..$ unit : chr "NM"
#> .. .. ..$ :List of 2
#> .. .. .. ..$ value: int 500
#> .. .. .. ..$ unit : chr "NM"
#> .. ..$ MRO:MINIMUM_STRETCH : int [1:3] 3 3 3
#> .. ..$ MRO:MAXIMUM_STRETCH : int [1:3] 1021 1021 1021
#> .. ..$ FILTER_NAME : chr [1:3] "NEAR-INFRARED" "RED" "BLUE-GREEN"All of the data you can access from NASA missions is public, but be aware that if you are going to download a large amount, there are almost always better ways than web scraping - be it either a catalog-only download or via FTP, investigate and be sure to be considerate.
Also of note that the data is public and free to use, it is the product of a lot of hard work and it is appropriate to cite the data source and the principle investigators of the instrument whose data you are using. NASA provides a guide to Citing PDS3 Data
One cool thing is that many PDS catalogs include citation information in a PDS3 formatted file we can use this package to extract!
cit_href <- "https://hirise.lpl.arizona.edu/PDS/CATALOG/RDR_DS.CAT"
cit_req <- curl::curl_fetch_memory(cit_href)
cit_dat <- rawToChar(cit_req$content)
cit_res <- pds3_read(cit_dat)
cit_res$odl$DATA_SET$DATA_SET_INFORMATION$CITATION_DESC
#> [1] "McEwen, A., Mars Reconnaissance \r\n Orbiter High Resolution Imaging Science Experiment, Reduced \r\n Data Record, MRO-M-HIRISE-3-RDR-V1.0, NASA Planetary Data\r\n System, 2007."