Dataflows
All dataflows can be edited at aspbo_flows.drawio
Conditional. This workflow is triggered whenever files in the folders ./data/output/UAT_direct/ and/or ./data/output/UAT_processing/ are changed as a result of a pull request. Merging a pull request with the uat - branch triggers an upload to the UAT - bucket while merging the uat - branch with the main - branch triggers an upload to the PROD - bucket. The .yaml files of this workflow contain the assume role step, which determines the destination bucket and relevant credentials.
**CAUTION: ONLY THE "UAT" - BRANCH SHOULD BE MERGED (after thorough review) WITH THE "MAIN" - BRANCH !!! **
- [automated] condition: a CRON schedule was triggered -> a data update workflow is run using github actions.
- [automated] condition: files on the workflow branch have changed -> the data update workflow creates a pull request to the
uat- branch - [manual] the pull request is reviewed, approved & merged with the
uat- branch - [automated] condition: files in
./data/output/UAT_direct/or./data/output/UAT_processing/are changed on the theuat- branch -> the resp. upload workflow(s) are triggered to upload files to theUAT - bucket. - [manual] exoten-uat.inbo.be is thoroughly inspected.
- [manual] a pull request is created from the
uat- branch to themain- branch - [manual] the pull request is reviewed, approved & merged with the
main- branch - [automated] condition: files in
./data/output/UAT_direct/or./data/output/UAT_processing/on themain- branch have changed -> the resp. upload workflow(s) are triggered to upload files to thePROD - bucket.
After completion of the last step the data is LIVE and can be consulted on exoten.inbo.be
Scheduled. This workflow is triggered at 6am on every Monday. Specifically, Vespa velutina data are imported anew from GBIF and iAsset, after which they are processed to yield the required outputs. Part of these files are then stored in the folder ./data/output/UAT_processing/Vespa_velutina_shape/, which, in turn, triggers a pull request to merge these files with the uat - branch. Upon manual approval and subsequent merging of this pull request with the uat - branch, a conditonal workflow is triggered that uploads the files from the ./data/output/UAT_processing/ folder (including those in the Vespa_velutina_shape folder) to the UAT-bucket. In case the uat - branch is merged with the main - branch an upload to the PROD - bucket is triggered.
Scheduled. This workflow is triggered at 00h00 on the first day of each month.
A sampling event dataset on the invasive species Oxyura jamaicensis (ruddy duck) in Flanders is imported from GBIF, containing information on a number of sampling events in Flanders that validated the occurrence of ruddy duck. These data are further processed to create a dataset containing, among others, for each sampling event (indicated with a unique gbifID) the number of observed adults, the gender of observed adults (for younger individuals, juvenile or pulli is indicated in this column), and a count related to the gender column, indicating the number of males, females, juveniles etc. that were observed during each sampling event. This dataset is stored as Oxyura_jamaicensis.csv in the folder ./data/output/UAT_direct/. This triggers a pull request to merge this folder with the uat - branch. Upon manual approval and subsequent merging of this pull request with the uat - branch, a conditonal workflow is triggered that uploads all files from the ./data/output/UAT_direct/ folder (including the Oxyura_jamaicensis.csv file) to the UAT-bucket. In case the uat - branch is merged with the main - branch an upload to the PROD - bucket is triggered.
Scheduled. This workflow is triggered at 00h00 on the first day of each month.
Lithobates catesbeianus data are imported anew from GBIF, after which they are processed to yield the required outputs. A file containing occurrences without known coordinates is constructed and stored as locations_missing_coords.csv in the folder ./data/interim/. The occurrences with known coordinates are processed and stored in the file Litobathes_catesbeianus.csv in the folder ./data/output/UAT_direct/. This triggers a pull request to merge this folder with the uat - branch. Upon manual approval and subsequent merging of this pull request with the uat - branch, a conditonal workflow is triggered that uploads all files from the ./data/output/UAT_direct/ folder (including the Litobathes_catesbeianus.csv file) to the UAT-bucket. In case the uat - branch is merged with the main - branch an upload to the PROD - bucket is triggered.
Scheduled. This workflow is triggered at 00h00 on every Monday.
Ondatra zibethicus data are imported anew from GBIF, after which they are processed to yield the required outputs. A file containing these processed occurrence data is constructed and stored as Ondatra_zibethicus.csv in the folder ./data/output/UAT_direct/. This triggers a pull request to merge this folder with the uat - branch. Upon manual approval and subsequent merging of this pull request with the uat - branch, a conditonal workflow is triggered that uploads all files from the ./data/output/UAT_direct/ folder (including the Ondatra_zibethicus.csv file) to the UAT-bucket. In case the uat - branch is merged with the main - branch an upload to the PROD - bucket is triggered.
Scheduled. This workflow is triggered at 00h00 on the first day of each month.
The eu_concern_species.tsv file is downloaded from the UAT - bucket after which it is compared to the List of Invasive Alien Species of Union Concern, downloaded from GBIF. When differences exist between these two files (i.e., the online list was expanded or changed), the online list will be stored as eu_concern_species.tsv in the folder ./data/output/UAT_processing/. This triggers a pull request to merge this folder with the uat - branch. Upon manual approval and subsequent merging of this pull request with the uat - branch, a conditonal workflow is triggered that uploads all files from the ./data/output/UAT_processing/ folder (including the eu_concern_species.tsv file) to the UAT - bucket, overwriting the old eu_concern_species.tsv file. In case the uat - branch is merged with the main - branch an upload to the PROD - bucket is triggered.
Scheduled. This workflow is triggered at 00h00 on the first day of each month.
The 01_get_data_input_checklist_indicators.Rmd from the indicators repo of the trias-project is sourced and run. This script imports the Global Register of Introduced and Invasive Species - Belgium from GBIF and processes it to create the data_input_checklist_indicators.tsv file in the ./data/interim/ folder. This file is subsequently copied to the ./data/output/UAT_processing/ folder, upon which it is deleted in the ./data/interim/ folder. Moving this file to the ./data/output/UAT_processing/ folder triggers a pull request to merge this folder with the uat - branch. Upon manual approval and subsequent merging of this pull request with the uat - branch, TWO conditonal workflows are triggered:
- the
upload_files_processing workflowuploads all files from the./data/output/UAT_processing/folder (including the data_input_checklist_indicators.tsv file) to theUAT - bucket. In case theuat- branch is merged with themain- branch an upload to the PROD - bucket is triggered. - the get_occ_cube workflow (see below).
Scheduled This workflow triggers every 1st Monday of the month, OR
Conditional whenever ./data/interim/data_input_checklist_indicators.tsv was changed on the UAT or MAIN branch due to pull request.
This workflow consists of 3 separate jobs, which start upon completion of the previous one, namely:
- Build queries
This job uses a template query for each rank (subspecies, forms & varieties, species, genus or family) and fills it with the corresponding gbif keys (taxon-, species-, genus- or familykey). The resulting queries are uploaded as artifacts for further use.
- Download cubes
This job uses the queries populated in the previous job to download the cubes for each rank. Downloads are handled by rgbif::occ_download_sql(). These downloads are triggered in parallel (staggered to not exceed 3 active downloads). The resulting rank cubes are uploaded as artifacts for further use.
- Compile cubes
This job uses the rank cubes downloaded in the previous job and combines them into the be_alientaxa_cube. The resulting file is uploaded as an artifact for use by the cube_preprocessing - flow (see below)
Workflow run This workflow is triggered when the get_occ_cube flow succeeds.
cube_preprocessing.Rmd is sourced and run.
The be_alientaxa_cube is joined to gridcells which contains information about, whether the grid cell intersects any of the Natura2000 Belgian protected areas and which commune, province and region contains most of the gridcell. This file is immediately stored on the S3 bucket, for this the script calls upon Connect_to_Bucket.R. The be_alientaxa_cube is processed further and coordinates with corrupted eea_cell_code entries are stored as corrupt_bl_eea_cell_codes.csv in the ./data/interim/ folder. Next, the taxa in the data_input_checklist_indicators.tsv file which are not present in the be_alientaxa_cube dataset are stored in the file alien_taxa_without_occs.tsv in the ./data/output/ folder while those that have a last observation date before 1950 are stored in the filetaxa_last_observed_in_BE_before_1950.tsv in the ./data/output folder. The be_alientaxa_cube is finally reshaped into a continuous time series grouped by taxa, gridcell & year.