diff --git a/how-we-work/genome-dk.qmd b/how-we-work/genome-dk.qmd index 980dde3..a600760 100644 --- a/how-we-work/genome-dk.qmd +++ b/how-we-work/genome-dk.qmd @@ -4,151 +4,169 @@ title: "Working on GenomeDK" ## GenomeDK -[GenomeDK](https://genome.au.dk) is the name of Aarhus University's -super computer cluster. It is a high-performance computing (HPC) system. -It is ISO 27001 certified and approved to store sensitive data in -compliance with GDPR and the Danish Data Protection Act. +[GenomeDK](https://genome.au.dk) is the name of Aarhus University's super +computer cluster. It is a high-performance computing (HPC) system. It is ISO +27001 certified and approved to store sensitive data in compliance with GDPR and +the Danish Data Protection Act. -Within a Seedcase context we are going to use it as an example of how -Sprout can be used to import, organise and store data on a remote -cluster. It will also be used by the two projects that we are currently -working with, [DP-Next](https://dp-next.github.io) and -[ON-LiMiT](https://onlimit.org/). +Within a Seedcase context we are going to use it as an example of how Sprout can +be used to import, organise and store data on a remote cluster. It will also be +used by the two projects that we are currently working with, +[DP-Next](https://dp-next.github.io) and [ON-LiMiT](https://onlimit.org/). ## Access -People associated with a university, the public sector (government -agencies, regions, and municipalities), or small and medium-sized -enterprises (SMEs) can ask for an account on GenomeDK, which is done via -the [request -account](https://console.genome.au.dk/user-requests/create/) portal. For -now, we will need to select the "Open" zone. You need to state your -preferred username and password even though the form seems to say to -only do it if you want to re-activate an old account. +People associated with a university, the public sector (government agencies, +regions, and municipalities), or small and medium-sized enterprises (SMEs) can +ask for an account on GenomeDK, which is done via the [request +account](https://console.genome.au.dk/user-requests/create/) portal. For now, we +will need to select the "Open" zone. You need to state your preferred username +and password even though the form seems to say to only do it if you want to +re-activate an old account. ## Setting up Once you get approved for access you can run through the -[documentation](https://genome.au.dk/docs/), which is extensive and -quite detailed. Because GenomeDK is a Linux-based server, you setup, -access, and do many actions through a shell terminal. You will have to -set up two-factor authentication using an authenticator app (they don't -stipulate any in particular). +[documentation](https://genome.au.dk/docs/), which is extensive and quite +detailed. Because GenomeDK is a Linux-based server, you setup, access, and do +many actions through a shell terminal. You will have to set up two-factor +authentication using an authenticator app (they don't stipulate any in +particular). ## Connecting ### SSH {#ssh} -You can connect to your GenomeDK environment from the terminal on your -local machine. Follow the -[instructions](https://genome.au.dk/docs/getting-started/#ssh) on -GenomeDK's website. +You can connect to your GenomeDK environment from the terminal on your local +machine. Follow the +[instructions](https://genome.au.dk/docs/getting-started/#ssh) on GenomeDK's +website. ### Desktop {#desktop} -If you prefer a graphical interface, you can also access GenomeDK -through their [virtual desktop](https://desktop.genome.au.dk). Follow -the [instructions](https://genome.au.dk/docs/getting-started/#desktop) -on GenomeDK's website. +If you prefer a graphical interface, you can also access GenomeDK through their +[virtual desktop](https://desktop.genome.au.dk). Follow the +[instructions](https://genome.au.dk/docs/getting-started/#desktop) on GenomeDK's +website. ### Mounting files -If you want to view and edit the files stored on GenomeDK using -applications installed on your local machine, you can mount your -GenomeDK file system locally. Follow the +If you want to view and edit the files stored on GenomeDK using applications +installed on your local machine, you can mount your GenomeDK file system +locally. Follow the [instructions](https://genome.au.dk/docs/working-with-data/#mounting) on GenomeDK's website. ## Installing software tools -To develop Seedcase projects on GenomeDK, you will need to install uv. -If you want to render markdown documents, you will also need to install -Quarto. Finally, if you want to run recipes in the -[`justfile`](workflow.qmd#workflow-helper), you will need to install -`just`. +To develop Seedcase projects on GenomeDK, you will need to install uv. If you +want to render markdown documents, you will also need to install Quarto. +Finally, if you want to run recipes in the +[`justfile`](workflow.qmd#workflow-helper), you will need to install `just`. ### Installing uv {#installing-uv} -Log in to GenomeDK (either through [SSH](#ssh) or the -[desktop interface](#desktop)) and open a terminal window. Following -uv's -[installation guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer), +Log in to GenomeDK (either through [SSH](#ssh) or the [desktop +interface](#desktop)) and open a terminal window. Following uv's [installation +guide](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer), run: -``` {.bash filename="Terminal"} +```bash {filename="Terminal"} curl -LsSf https://astral.sh/uv/install.sh | sh ``` ### Installing Quarto -Log in to GenomeDK (either through [SSH](#ssh) or the -[desktop interface](#desktop)) and open a terminal window. Follow the -instructions on Quarto's +Log in to GenomeDK (either through [SSH](#ssh) or the [desktop +interface](#desktop)) and open a terminal window. Follow the instructions on +Quarto's [website](https://quarto.org/docs/download/tarball.html?version=1.7.32&idPrefix=download). Notes: -- Step 3: if `~/.local/bin` exists, you don't need to run - `mkdir ~/.local/bin` +- Step 3: if `~/.local/bin` exists, you don't need to run `mkdir ~/.local/bin` - Step 4: as the instructions say, you can probably skip this step - If VS Code is open, you'll need to restart it to detect the Quarto CLI ### Installing `just` -Log in to GenomeDK (either through [SSH](#ssh) or the -[desktop interface](#desktop)) and open a terminal window. Following -`just`'s -[installation guide](https://just.systems/man/en/pre-built-binaries.html#pre-built-binaries), +Log in to GenomeDK (either through [SSH](#ssh) or the [desktop +interface](#desktop)) and open a terminal window. Following `just`'s +[installation +guide](https://just.systems/man/en/pre-built-binaries.html#pre-built-binaries), run: -``` {.bash filename="Terminal"} +```bash {filename="Terminal"} curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to ~/bin ``` -## Developing with VS Code - -The best way to work on Seedcase projects in VS Code is to use the VS -Code application on your local machine and connect to GenomeDK via SSH. -This allows you to edit files and run code on GenomeDK as if you were -working locally. This setup uses the -[Remote SSH VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh), -which works by adding a `.vscode-server` folder in your home directory -on GenomeDK. This is a lightweight backend that allows VS Code to run -terminals, extensions, debuggers, etc., remotely on GenomeDK. If at any -point you want to remove the backend, you can simply delete this folder. +### Installing Git LFS -1. Open VS Code on your local machine and install the - [Remote SSH VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) - -2. Select "Remote-SSH: Connect to Host..." from the Command Palette - (`F1`, `Ctrl+Shift+P`) and enter `@login.genome.au.dk`. - Follow the prompts given to update or create your config file. In the - "Remotes Explorer" panel, you should now see the GenomeDK host listed - as `login.genome.au.dk`. - -3. Click the GenomeDK SSH entry and enter your GenomeDK password when - prompted. Now, you should be connected to GenomeDK and see the remote - file system if you open up the VS Code terminal. +Log in to GenomeDK (either through [SSH](#ssh) or the [desktop +interface](#desktop)). Install Git LFS in your GenomeDK user space either from +the +[tarball](https://gist.github.com/pourmand1376/bc48a407f781d6decae316a5cfa7d8ab) +(note that this guide uses an older version of Git LFS, but the current version +can be gotten from [the official site](https://git-lfs.com) by copying the link +from the "Download" button) or via +[conda](https://anaconda.org/channels/conda-forge/packages/git-lfs/overview). +Then run: -4. Clone your repository using the VS Code terminal, if you haven’t - already: +```bash +git lfs install +``` - ``` {.bash filename="Terminal"} - cd <.../my/projects> - git clone - cd - ``` +If you'll work in a repository with an existing LFS store, you also need to tell +Git that the LFS store is safe: -5. Open your project folder in VS Code by clicking the "Open Folder" - button in the panel on the left-hand side +```bash +git config --global --add safe.directory +``` -6. Install your project dependencies (you'll need - [uv installed](#installing-uv)): +E.g., the path for the ON-LiMiT feasibility data LFS store is +`/faststorage/project/sdca-onlimit-study/feasibility-data-lfs-store`. - ``` {.bash filename="Terminal"} - uv sync - ``` +## Developing with VS Code +The best way to work on Seedcase projects in VS Code is to use the VS Code +application on your local machine and connect to GenomeDK via SSH. This allows +you to edit files and run code on GenomeDK as if you were working locally. This +setup uses the [Remote SSH VS Code +extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh), +which works by adding a `.vscode-server` folder in your home directory on +GenomeDK. This is a lightweight backend that allows VS Code to run terminals, +extensions, debuggers, etc., remotely on GenomeDK. If at any point you want to +remove the backend, you can simply delete this folder. + +1. Open VS Code on your local machine and install the [Remote SSH VS Code + extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) + +2. Select "Remote-SSH: Connect to Host..." from the Command Palette (`F1`, + `Ctrl+Shift+P`) and enter `@login.genome.au.dk`. Follow the prompts + given to update or create your config file. In the "Remotes Explorer" panel, + you should now see the GenomeDK host listed as `login.genome.au.dk`. + +3. Click the GenomeDK SSH entry and enter your GenomeDK password when prompted. + Now, you should be connected to GenomeDK and see the remote file system if + you open up the VS Code terminal. + +4. Clone your repository using the VS Code terminal, if you haven't already: + + ```bash {filename="Terminal"} + cd <.../my/projects> + git clone + cd + ``` + +5. Open your project folder in VS Code, named after you e.g, `signekb`, by + clicking the "Open Folder" button in the panel on the left-hand side. + +6. Install your project dependencies (you'll need [uv + installed](#installing-uv)): + + ```bash {filename="Terminal"} + uv sync + ``` ::: callout-note If you're working in a shared folder, you might get a "permission denied" error about a "dubious ownership" in the repository after running `uv sync`. After the @@ -161,33 +179,32 @@ point you want to remove the backend, you can simply delete this folder. panel and type `@recommended` into the search bar. You can install extensions from here. -8. To be able to commit to GitHub, you will need to set up your Git - credentials. You can do this by running the following commands in the - terminal: +8. To be able to commit to GitHub, you will need to set up your Git credentials. + You can do this by running the following commands in the terminal: - ``` {.bash filename="Terminal"} - git config --global user.name "Your Name" - git config --global user.email "you@example.com" - ``` + ```bash {filename="Terminal"} + git config --global user.name "Your Name" + git config --global user.email "you@example.com" + ``` -Now, you should be able to run scripts in your project using either the -VS Code terminal or the Run/Debug UI, as well as push changes to GitHub. +Now, you should be able to run scripts in your project using either the VS Code +terminal or the Run/Debug UI, as well as push changes to GitHub. ## Using Git LFS on GenomeDK -In Data Packages on GenomeDK, all Parquet data files and data kept in -the `raw/` folder are tracked using [Git LFS](https://git-lfs.com/) with -a local LFS store on GenomeDK itself. This means that the contents of -data files are stored on GenomeDK and only pointers to these files are -uploaded to GitHub. On GenomeDK you can see the contents of the data -files in two places: in your working copy of the Data Package and in a -folder outside the Data Package directory called the LFS store. +In data packages on GenomeDK, all Parquet data files and data kept in the `raw/` +folder are tracked using [Git LFS](https://git-lfs.com/) with a local LFS store +on GenomeDK itself. This means that the contents of data files are stored on +GenomeDK and only pointers to these files are uploaded to GitHub. On GenomeDK +you can see the contents of the data files in two places: in your working copy +of the data package and in a folder outside the data package directory called +the LFS store. -When working with data or adding new data to the Data Package, it is -important to take care that the Git LFS configuration is not changed and -that sensitive data is always tracked by Git LFS. Any data that is not -tracked by Git LFS runs the risk of being uploaded to GitHub -accidentally, which would constitute a serious data breach. +When working with data or adding new data to the data package, it is important +to take care that the Git LFS configuration is not changed and that sensitive +data is always tracked by Git LFS. Any data that is not tracked by Git LFS runs +the risk of being uploaded to GitHub accidentally, which would constitute a +serious data breach. ### Setting up Git LFS @@ -198,73 +215,68 @@ or via Then run `git lfs install`. ::: callout-important -If you don't initialise Git LFS with `git lfs install`, Git LFS will -not manage data files. Instead, files that should be tracked by LFS -will be handled by Git as normal and uploaded to GitHub on push. +If you don't initialise Git LFS with `git lfs install`, Git LFS will not manage +data files. Instead, files that should be tracked by LFS will be handled by Git +as normal and uploaded to GitHub on push. ::: -Afterwards, create a folder on GenomeDK that everyone working on the -Data Package can access and edit: this will be the LFS store where all -the data will be saved. This folder should be **outside** the Data -Package folder. +Afterwards, create a folder on GenomeDK that everyone working on the Data +Package can access and edit: this will be the LFS store where all the data will +be saved. This folder should be **outside** the data package folder. -Set the LFS store folder up as a -[bare repository](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-barerepository) +Set the LFS store folder up as a [bare +repository](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-barerepository) by opening it in the Terminal and running `git init --bare`. -Then, create an `.lfsconfig` file in the Data Package root folder and -add the absolute path to the LFS store: +Then, create an `.lfsconfig` file in the data package root folder and add the +absolute path to the LFS store: -``` {.txt filename=".lfsconfig"} +```txt {filename=".lfsconfig"} [lfs] url = file:///path/to/.../lfs-store ``` ::: callout-tip -Be sure to get the real filesystem path to this folder, not the one -pointing to your user space. You can find this by running -`readlink -f .` from the the LFS store folder. +Be sure to get the real filesystem path to this folder, not the one pointing to +your user space. You can find this by running `readlink -f .` from the the LFS +store folder. ::: -Define a rule for tracking all Parquet files and all data in the `raw/` -folder by running the following commands from the root of the Data -Package: +Define a rule for tracking all Parquet files and all data in the `raw/` folder +by running the following commands from the root of the data package: -``` {.bash filename="Terminal"} +```bash {filename="Terminal"} git lfs track "*.parquet" git lfs track "raw/**" ``` -This will create a `.gitattributes` file listing the tracked paths. -Tracked files shouldn't be listed in `.gitignore`. +This will create a `.gitattributes` file listing the tracked paths. Tracked +files shouldn't be listed in `.gitignore`. -To get an overview of the LFS settings in the repo, run -`git lfs env` or `git lfs ls-files` from the root of the Data -Package. +To get an overview of the LFS settings in the repo, run `git lfs env` or +`git lfs ls-files` from the root of the data package. ### Adding data to LFS -1. To add data, create a new Parquet file or move data file(s) into the - `raw/` folder, which is tracked by LFS. +1. To add data, create a new Parquet file or move data file(s) into the `raw/` + folder, which is tracked by LFS. -2. Stage the new files (using `git add`). They should now be tracked by - LFS. +2. Stage the new files (using `git add`). They should now be tracked by LFS. 3. Double-check that Git LFS is configured correctly: - The output of `git lfs ls-files` should include the new files. - - `git lfs env` should show that `Endpoint` is set to a local path - pointing to the LFS store on GenomeDK. + - `git lfs env` should show that `Endpoint` is set to a local path pointing + to the LFS store on GenomeDK. -4. Commit and push the new data. If you look at the new data file on - GitHub, you should see a pointer file rather than the actual data. +4. Commit and push the new data. If you look at the new data file on GitHub, you + should see a pointer file rather than the actual data. -### Working on a Data Package outside GenomeDK +### Working on a data package outside GenomeDK {#outside-genomedk} -If you want to work on a Data Package that uses Git LFS on your own -machine, you can check out the repo without downloading the data files. -In fact, it is not possible to download the data files via Git LFS, -because the LFS store path points to a local folder on GenomeDK that -your machine cannot access. +If you want to work on a data package that uses Git LFS on your own machine, you +can check out the repo without downloading the data files. In fact, it is not +possible to download the data files via Git LFS, because the LFS store path +points to a local folder on GenomeDK that your machine cannot access. You don't need Git LFS installed locally, but you do have to tell Git not to attempt to fetch data files tracked by Git LFS, as this would lead to an error. @@ -280,9 +292,9 @@ the Git LFS smudge filter, which reads the Git LFS pointer files and tries to write the contents of the tracked files to standard output. This would fail because the LFS store is only accessible on GenomeDK. -After cloning the repo, run the following from the root of the Data Package: +After cloning the repo, run the following from the root of the data package: -``` {.bash filename="Terminal"} +```bash {filename="Terminal"} # Show LFS-tracked files as pointers in the working tree git config --local filter.lfs.smudge "cat" # Do not require Git LFS to be installed @@ -293,6 +305,77 @@ git config --local filter.lfs.process "" git checkout ``` -With this setup, you won't have access to the actual data or be able to -run scripts that depend on it. However, you can still run other parts of -the code and work on data-dependent scripts without executing them. +With this setup, you won't have access to the actual data or be able to run +scripts that depend on it. However, you can still run other parts of the code +and work on data-dependent scripts without executing them. + +## Collaborating + +When collaborating on a data package on GenomeDK, we work in individual +repository clones in our own user spaces. A shared folder holds the repository +clone, Git LFS store, and secrets (like API keys) needed by the project. The +shared clone isn't meant for day-to-day editing: any change made there (e.g., +editing files, or checking out a different branch) is visible to everyone +working on the project. Instead, each of us uses our own clones to do our actual +development work. + +For example, for a repository named `feasibility-data`, the folder structure in +the shared folder looks like this: + +``` +/ +├── env/ +│ └── .feasibility-data-env +├── feasibility-data/ +└── feasibility-data-lfs-store/ +``` + +The `env/` folder contains the [`.env` file](https://env.dev/guides/dotenv)(s) +of the project. We name it after the repository in case we need to work on other +repositories in the same shared folder. + +::: callout-tip +If you're setting up a new shared repository, see the [setting up Git +LFS](#setting-up-git-lfs) section. +::: + +To set up your own repository, follow these steps: + +1. Log in to GenomeDK, either via an [SSH connection](#ssh), e.g., with [VS + Code](#developing-with-vs-code), or the [GenomeDK desktop](#desktop). + +2. Ensure you have Git LFS [installed](#installing-git-lfs). If you've installed + it with conda, make sure to activate your conda environment before + continuing, so the Git LFS file pointers get resolved correctly. + +3. Mark the existing Git LFS store as safe using the command from the + [installing Git LFS](#installing-git-lfs) section. + +4. Now, clone the Git repository to your own user space, the exact location is + up to you: + + ```bash + cd /your/wanted/location/ + git clone + ``` + +This sets up your individual clone of the repo while still having access to the +shared Git LFS store, since your clone's `.lfsconfig` (see [setting up Git +LFS](#setting-up-git-lfs)) already points there. + +::: callout-note +When working like this, the data will be duplicated in our individual clones. +For now, it's not an issue, but we might need to find another solution in the +future. +::: + +Lastly, to be able to use the shared `.env` file in your clone, the repository +needs to be set up to load the correct file. This can be achieved by pointing to +the shared folder path in the +[`load_dotenv()`](https://pypi.org/project/python-dotenv/) call. For an example, +see how we've set it up in the ON-LiMiT +[feasibility-data](https://github.com/onlimit-study/feasibility-data) +repository. + +With this setup, we can collaborate on a data package by working with the actual +data in our own clones, without interfering with each other's work.