# Author: Maria Yli-Heikkilä
# Date : 10.6.2020

This project's Allas repository can stand as an ecosystem for Sentinel research. The idea is that 
collaborators can all add SAFE folders here for shared use.


########## sentinel-readme bucket contains:
- files containing list of uploaded SAFE folders with the email of a contact person, in case of errata and  authorship contribution. 
- files with short description of projects where this data was used.

You can see the 'directory listing' of sentinel-readme bucket here:
https://a3s.fi/sentinel-readme/index.html

All currently available Sentinel SAFE files are listed here:
https://a3s.fi/sentinel-readme/uploadedByMariaYliHeikkila.txt

Includes: Sentinel2, L2A product, AOI approx. Finland's territory, vegetation period (ca 10.5.-1.9.) in 2016-2020. 
From 2017 onwards ESA scihub has been providing Sentinel-2 products on level L2A. L2A products here are downloaded from scihub 
(unless stated otherwise). However, in 2016 only L1C level products are available. Thus, L2A products here are further processed from L1C downloads. L1C 
products were processed into L2A with FORCE software tool with command: L2A_Process filename.SAFE

The file contains rows like this:
s3://Sentinel2-MSIL2A-cloud-0-95-2016-T34VDM/S2A_MSIL2A_20160516T100032_N0202_R122_T34VDM_20160516T100032.SAFE/ uploaded by maria.yli-heikkila@luke.fi

The folder includes also short project descriptions who has used this data:
https://a3s.fi/sentinel-readme/projectSummary1.txt

There is a legal notice from ESA on the use or distribution of Sentinel data:
https://a3s.fi/sentinel-readme/Sentinel_Data_Legal_Notice.pdf


########## Sentinel buckets:
At the moment all Sentinel buckets are public. It is up to the person uploading data to decide whether to publish the bucket or not.

SAFE files are under granule (tile) wise buckets and the contents of SAFE-file have been unzipped, so single files can be accessed separately. 
This enables also accessing the .jpeg2000 files (bands) straight with GDAL VSICURL without prior manual download. 
This is possible with several tools based on GDAL, inc Python, R and QGIS: 
https://docs.csc.fi/apps/gdal/#using-files-directly-from-allas

For some users it is easier to download files via www, then it might be helpful to open the bucket of the granule, for example:
https://a3s.fi/Sentinel2-MSIL2A-cloud-0-95-2016-T34VDM/index.html

If there are errors or anything else, please notify the person who uploaded that SAFE folder. 


########## Acknowledgement:
Contact person is responsible for a specific part of the research work. But as acquiring data is not really creative input for research, 
which I consider as a merit for authorship, I see this work as the inevitable contribution of a technician, which merits to be cited in an 
acknowledgements section. Please acknowledge also oGIIR (urn:nbn:fi:research-infras-2016072513) and CSC (urn:nbn:fi:research-infras-2016072531) for providing the infra.

Please, also fill up a short description file of your project, if you use this repo. The description file follows the same structure of 
initiating a project at my.csc.fi. See e.g.:
https://a3s.fi/sentinel-readme/projectSummary1.txt
These files can be sent to giscoord@csc.fi, so that CSC can upload them to Allas.


######### Contributing (Sentinel) data to Allas
If you store significant amounts of data to Allas for own research project and the data is open and could be valuable for others, you are welcome to share your data with others.
The data could also be some other data than Sentinel, discuss with CSC first.

In case of Sentinel data the workflow could be following:
1) Download the data and save it as unzipped to Allas to your own project.

# to use s3cmd commands (more options than with a-commands):
module load allas
allas-conf -k --mode s3cmd

# upload a unzipped new SAFE dir to Allas:
s3cmd put -r $safedir s3://Sentinel2-MSIL2A-cloud-0-95-2016-$TILE/

# if bucket does not yet exist, make it by:
s3cmd mb s3://my_bucket

Please, use similar nomenclature as before when creating new buckets:
[satellite]-[product level]-cloud-[cloud threshold minimum]-[cloud threshold maximum]-[year]-[tilename]
e.g. Sentinel2-MSIL2A-cloud-0-95-2016-T34VDM

2) Make the data public
a-publish -b my_bucket

3) Create a list of available SAFE folders uploadedByXXX.txt that contains rows like:
s3://Sentinel2-MSIL2A-cloud-0-95-2016-T34VDM/S2A_MSIL2A_20160516T100032_N0202_R122_T34VDM_20160516T100032.SAFE/ uploaded by x.x@xx.fi

Ask Maria Yli-Heikkilä to be added to the Sentinel project to be able to modify files in sentinel-readme yourself: 
a-publish uploadedByMariaYliHeikkila.txt -b sentinel-readme

OR send the list of files to giscoord@csc.fi to be uploaded by CSC.

4) Notify CSC (giscoord@csc.fi) about new data, so that https://research.csc.fi/gis_data_in_csc_computing_env 
page could be updated.

Contact person is kindly asked to either delete a corrupted SAFE or redownload it from ESA. Errata file will also follow, if there is need for 
it. My idea is that contact person is responsible for documenting errata and also possibly fixing things.