Make build infra‑agnostic: update README, add Packer examples, improve build playbook and docs #1
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Credentials / secrets | ||
| *.pem | ||
| *-openrc.sh | ||
| openrc.sh | ||
|
|
||
| # Logs | ||
| *.log | ||
| packerlog.txt | ||
|
|
||
| # Virtual environments / local tooling | ||
| openstack_cli/ | ||
| .venv/ | ||
| venv/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| packer { | ||
| required_plugins { | ||
| openstack = { | ||
| version = ">= 1.1.2" | ||
| source = "github.com/hashicorp/openstack" | ||
| } | ||
|
|
||
| ansible = { | ||
| version = ">= 1.1.1" | ||
| source = "github.com/hashicorp/ansible" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| source "openstack" "ubuntu" { | ||
| image_name = "bioimage" | ||
| flavor = "c3.1c2m10d" | ||
| networks = [""] # update via: openstack network list - select network associated with your project | ||
| availability_zone = "CloudV3" | ||
| source_image = "68b8635c-9ae8-457a-afd6-b8609a36bf66" | ||
| ssh_username = "ubuntu" | ||
| volume_size = 20 | ||
| } | ||
|
|
||
| build { | ||
| sources = ["source.openstack.ubuntu"] | ||
|
|
||
| provisioner "ansible" { | ||
| playbook_file = "./build-bioimage.yml" | ||
|
|
||
| extra_arguments = [ | ||
| "--extra-vars", "ansible_user=ubuntu" | ||
| ] | ||
|
|
||
| ansible_env_vars = [ | ||
| "ANSIBLE_HOST_KEY_CHECKING=False", | ||
| "ANSIBLE_SCP_IF_SSH=True" | ||
| ] | ||
|
|
||
| use_proxy = false | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| packer { | ||
| required_plugins { | ||
| openstack = { | ||
| version = ">= 1.1.2" | ||
| source = "github.com/hashicorp/openstack" | ||
| } | ||
|
|
||
| ansible = { | ||
| version = ">= 1.1.1" | ||
| source = "github.com/hashicorp/ansible" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| source "openstack" "ubuntu" { | ||
| image_name = "bioimage" | ||
| flavor = "" # update via: openstack flavor list | ||
| networks = [""] # update via: openstack network list | ||
| availability_zone = "" # update via: openstack availability zone list | ||
| source_image = "" # update via: openstack image list | ||
| ssh_username = "ubuntu" | ||
| volume_size = 20 | ||
| } | ||
|
|
||
| build { | ||
| sources = ["source.openstack.ubuntu"] | ||
|
|
||
| provisioner "ansible" { | ||
| playbook_file = "./build-bioimage.yml" | ||
|
|
||
| extra_arguments = [ | ||
| "--extra-vars", "ansible_user=ubuntu" | ||
| ] | ||
|
|
||
| ansible_env_vars = [ | ||
| "ANSIBLE_HOST_KEY_CHECKING=False", | ||
| "ANSIBLE_SCP_IF_SSH=True" | ||
| ] | ||
|
|
||
| use_proxy = false | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
|
|
||
| ### ⚠️ **DEPRECATED — UNDER REVIEW** | ||
| This document is currently outdated and should not be used. It may be updated in the future. | ||
| please refer to the main [README.md](../README.md) for instructions on building an image. | ||
|
|
||
|
|
||
| ## Instance Management | ||
|
|
||
| ### Create Instances and Boot Image | ||
| You may follow the [Nirin Quick Start Guide](https://opus.nci.org.au/spaces/Help/pages/152207474/Nirin+-+Quick+Start+Guide) , or use the script to create instances | ||
| ``` | ||
| cd bioimage/manage | ||
| ./openstack/create-instances-with-image.sh | ||
| ``` | ||
|
|
||
| If instance creation fails using an image, create **bootable volumes** instead: | ||
|
Member
There was a problem hiding this comment. Why would the build fail? Is this common or would it only happen if i did something wrong? |
||
|
|
||
| 1. In the dashboard, create a volume with "Image" as the source. | ||
|  | ||
|
|
||
|
|
||
| Once the volumes are created, you can launch instances from them.Use the NCI Cloud Dashboard to create volumes and assign them a consistent prefix (e.g., `training-VM-1`, `training-VM-2`, etc.). | ||
|
|
||
| 2. Launch instances using existing volumes. | ||
| To launch instances via dashboard, fill out the required sections: `Details`, `Source`, `Falvour`, and `Key pair`. In the **Source** section, select the option to boot from an existing volume. | ||
|  | ||
|
|
||
|
|
||
| For bulk operations: | ||
| ``` | ||
| cd bioimage/manage | ||
| ./openstack/create-bootable-volumes.sh | ||
| ./openstack/create-instances-with-volume.sh <key-pair> <VM-prefix> | ||
| ``` | ||
|
|
||
| ### Shut Down and Restart Instances | ||
| Stop instances when they are not in use and restart them as needed. | ||
| ``` | ||
| ./openstack/instances-start.sh <VM-prefix> | ||
| ./openstack/instances-stop.sh <VM-prefix> | ||
| ``` | ||
|
|
||
| ### Tools available in Bioimage | ||
|
|
||
| #### modules | ||
|
|
||
| The image should include the following applications: | ||
| - Singularity | ||
| - SHPC | ||
| - Spack | ||
| - Ansible | ||
| - Jupyter Notebook | ||
| - RStudio | ||
| - Nextflow | ||
| - Snakemake | ||
| - CernVM-FS client | ||
|
|
||
| Check available modules with: | ||
| ``` | ||
| module avail | ||
| ``` | ||
|
|
||
| To use an application, load it with: | ||
| ``` | ||
| module load <app> | ||
| ``` | ||
| #### CernVM-FS (CVMFS) | ||
|
|
||
| This image uses CernVM-FS (CVMFS) to provide access to shared bioinformatics software and datasets without installing them locally on the VM. | ||
|
|
||
| Access CVMFS repositories: | ||
| ``` | ||
| ls /cvmfs/data.biocommons.aarnet.edu.au | ||
| ls /cvmfs/data.galaxyproject.org | ||
| ls /cvmfs/singularity.galaxyproject.org | ||
|
|
||
| ``` | ||
| For an explanation of what CVMFS is, how it works, and how it is used in BioImage, see [CVMFS documentation](docs/cvmfs.md). | ||
|
|
||
|
|
||
| ## User Access | ||
|
|
||
| ### Single User per Instance | ||
|
|
||
| #### Step 1: Generate passwords and Update IP Information | ||
| Generate passwords for each user and update the inventory file with the password and instance IPs. | ||
| ``` | ||
| cd bioimage/manage | ||
| python3 python/update-IP.py <VM-prefix> <project-id> | ||
| python3 python/update-keys.py <VM-prefix> | ||
| ``` | ||
|
|
||
| #### Step 2: Create Users and Enable Password Access | ||
| Add the `training` user with the generated passwords and enable password access. | ||
| ``` | ||
| ansible-playbook ./ansible/users-create-1-1.yml | ||
| ansible-playbook ./ansible/ssh-password-enable.yml | ||
| ``` | ||
|
|
||
| #### Step 3: Associate Floating IPs and Save VM Information | ||
| Create and associate floating IPs with the instances for public access. Save the username, password and public IP information in the `VMs` folder. | ||
| ``` | ||
| ./openstack/floating-IP-create.sh <VM-prefix> | ||
| python3 python/list-VM-info.py <VM-prefix> # Save each VM details to a txt file | ||
| python3 python/save-VM-info-csv.py <VM-prefix> # Or, save all VMs details to a csv file | ||
| ``` | ||
|
|
||
| #### Step 4: Delete Users, Disable Password and Public IP Access | ||
| After the training session, delete the `training` user accounts, disable password access and disacciate public IP. | ||
| ``` | ||
| ansible-playbook ./ansible/users-delete.yml | ||
| ansible-playbook ./ansible/ssh-password-disable.yml | ||
| ./openstack/floating-IP-delete.sh <VM-prefix> | ||
| ``` | ||
|
|
||
| #### Step 5: Optional – Shut Down the Instances | ||
| Shut down the instances when they are not in use. | ||
| ``` | ||
| ./openstack/instances-stop.sh <VM-prefix> | ||
| ``` | ||
|
|
||
| ### Multiple Users per Instance | ||
| When multiple users need to share the same instance, follow these steps: | ||
|
|
||
| #### Step 1: Step 1: Generate passwords and Update IP Information | ||
| Manually specify the number of users and generate their passwords. | ||
|
|
||
| 1. Generate password keys: | ||
| ``` | ||
| cd bioimage/manage | ||
| python3 python/generate-keys.py | ||
| ``` | ||
|
|
||
| Update the file `host_vars/[VM-name].yml` with the generated passwords. Sample files are provided. | ||
|
|
||
| 2. Rename the inventory file and update IPs: | ||
| Rename the `inventory.n` file to `invenroty` and run the command: | ||
| ``` | ||
| python3 python/update-IP.py <VM-prefix> <project-id> | ||
| ``` | ||
|
|
||
| #### Step 2: Create Users and Enable Password Access | ||
| Create all the users and enable password access for them: | ||
| ``` | ||
| ansible-playbook ./ansible/users-create-n-1.yml | ||
| ansible-playbook ./ansible/ssh-passwords-enable.yml | ||
| ``` | ||
|
|
||
| #### Step 3: Associate Floating IPs and Save VM Information | ||
| Create and associate floating IPs with the instances for public access. | ||
| ``` | ||
| ./openstack/floating-IP-create.sh <VM-prefix> | ||
| ``` | ||
|
|
||
| #### Step 4: Delete Users and Disable Password and Public IP Access | ||
| After the training session, remove user accounts, disable password access. | ||
| ``` | ||
| ansible-playbook ./ansible/users-delete-all.yml | ||
| ansible-playbook ./ansible/ssh-password-disable.yml | ||
| ``` | ||
|
|
||
| #### Step 5: Optional – Shut Down the Instances | ||
| Shut down the instances when they are not in use. | ||
| ``` | ||
| ./openstack/instances-stop.sh <VM-prefix> | ||
| ``` | ||
|
|
||
| ### Install New Tools | ||
| `ansible/install-tools.yml` is an example script to install `tree`. Modify it to install more tools for all the VMs. | ||
| ``` | ||
| ansible-playbook ./ansible/install-tools.yml | ||
| ``` | ||
|
|
||
| ### Set Up Home Directory | ||
| Use `ansible/set-home-dir.yml` as an example to set up a custom Home directory for training users. Modify it as needed. | ||
| ``` | ||
| ansible-playbook ./ansible/set-home-dir.yml | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its not really clear what you're expecting me to do here. Am I spinning up a vm first and then running the script? If so, can you explain what flavour etc i need to spin up.
Also provide git clone instruction for this repo before cd bioimage/manage.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is old documentation that I am replacing in the main README.md. I have added instructions to direct users to the main readme and labelled the doc deprecated for now