Make build infra‑agnostic: update README, add Packer examples, improve build playbook and docs by Mitchob · Pull Request #1 · AustralianBioCommons/bioimage

Mitchob · 2026-02-02T05:04:55Z

Summary

This PR updates documentation and build tooling to make BioImage infrastructure‑agnostic and easier to build from scratch.
Primary aim: verify building the bioimage end‑to‑end using the available scripts and document the process for different OpenStack providers (tested on Nectar and Nirin).

What I changed:

README.md: Rewrote and expanded the README to be infra‑agnostic. Added a clear intro to BioImage, step‑by‑step instructions for creating a head VM, obtaining/sourcing OpenStack RC files, activating the environment, and building images. Included example references for Nectar and Nirin (tested on 2026-02-02).
.gitignore: added patterns for credentials, logs, and local virtualenvs.
- build/build-bioimage.yml: reliability fixes and small functional changes:
- Wait for apt/dpkg locks before operations.
- Retry apt updates with retries/delay.
- Install Java 21 and set it as the default java (required for Nextflow).
- Set Nextflow-related env vars for deterministic checks.
- Default CVMFS_HTTP_PROXY to 'DIRECT' (increase portability) and removed aggressive probes.
Packer configs:
- Added a generic template: build/openstack-bioimage.pkr.hcl (template requiring user to fill flavor, networks, availability_zone, source_image).
- Moved/added provider examples:
  - build/examples/openstack-bioimage-nectar.pkr.hcl (example config used for Nectar)
  - build/examples/openstack-bioimage-nirin.pkr.hcl (example config used for Nirin)
- Provisioner ansible blocks improved to set ansible env vars and avoid host key checking in automated builds.
Docs:
- docs/cvmfs.md: new doc explaining CVMFS usage in BioImage.
- docs/nirin.md: provider‑specific guidance for Nirin (instance creation, scripts, common tasks).
Renames/relocations in build folder to separate examples from template and make it clear which values must be filled per provider.

Testing performed

Built the image using the example Packer configs against Nectar and Nirin environments (example configs included).
Verified the build playbook handles apt locks and apt update retry behavior and that Nextflow can be interrogated via nextflow -version in the image build step.
Confirmed that CVMFS is configured as DIRECT by default (portable across clouds), with instructions for setting cloud-specific proxies if available.

Next steps

see if we can move the tasks out of the playbook as they are, and into a separate bioimage role
refactor to ensure that anything specifically needed for NCI/Nectar/AWS/anywhere else is templated out into variables
Intoduce tests to make sure updates arent breaking things

…va tasks

… to false

… added an introductory explanation of the Bioimage

Duplicated file

…n a provider-agnostic OpenStack cloud? Nirin details moved to a nirin document

…ctar

…ur, network and source

…revent init issues

…nectar

…VMFS proxy to Direct for portability

…y CVMFS Configuration for proxies

georgiesamaha

Need some clarification before I can do the review. Can't yet follow your instructions for setting up on Nirin.

georgiesamaha · 2026-02-02T22:45:16Z

docs/DEPRECATED_old_guid_nirin.md

+### Create Instances and Boot Image
+You may follow the [Nirin Quick Start Guide](https://opus.nci.org.au/spaces/Help/pages/152207474/Nirin+-+Quick+Start+Guide) , or use the script to create instances
+```
+cd bioimage/manage
+./openstack/create-instances-with-image.sh
+```


Its not really clear what you're expecting me to do here. Am I spinning up a vm first and then running the script? If so, can you explain what flavour etc i need to spin up.

Also provide git clone instruction for this repo before cd bioimage/manage.

This is old documentation that I am replacing in the main README.md. I have added instructions to direct users to the main readme and labelled the doc deprecated for now

georgiesamaha · 2026-02-02T22:45:57Z

docs/DEPRECATED_old_guid_nirin.md

+./openstack/create-instances-with-image.sh
+```
+
+If instance creation fails using an image, create  **bootable volumes** instead:  


Why would the build fail? Is this common or would it only happen if i did something wrong?

georgiesamaha · 2026-02-02T22:46:20Z

docs/DEPRECATED_old_guid_nirin.md

+
+2. Launch instances using existing volumes.
+    To launch instances via dashboard, fill out the required sections: `Details`, `Source`, `Falvour`, and `Key pair`. In the **Source** section, select the option to boot from an existing volume.
+    ![Launch an instance](screenshots/nirin-2.jpg)


This image doesnt render

georgiesamaha · 2026-02-02T22:46:29Z

docs/DEPRECATED_old_guid_nirin.md

+If instance creation fails using an image, create  **bootable volumes** instead:  
+
+1. In the dashboard, create a volume with "Image" as the source.
+    ![Create a Volume](screenshots/nirin-1.jpg)


This image doesnt render

…to fix file

Mitchob and others added 25 commits December 9, 2025 02:09

WIP: local testing edits to build-bioimage.yml update nextflow and ja…

df93e3c

…va tasks

updated ansible_env_vars to force packer to enable aftp and use_proxy…

72d486d

… to false

Fix http and port in CVMFS config

f735cd4

Document how to download the OpenStack RC file from the dashboard and…

1bf195b

… added an introductory explanation of the Bioimage

docs: add docs/ and ignore local OpenStack credentials

d6afb66

Delete build/openstack-bioimage.pkr.hcl

5f4fe00

Duplicated file

move bioimage tool descriptions to logical position in readme

5c9a6f8

fix github address

f74e5b2

Added instructions for spinning up and logging into a VM

f1b9da4

update activation step in image build to include sanity check

098241e

move activation sanity check earlier in instructions

53b484c

remove repitition from introduction to bioimage

47a5d24

Editing readme to be about what the bioimage is and how to build it o…

8983747

…n a provider-agnostic OpenStack cloud? Nirin details moved to a nirin document

added instuctions of variables that require updating in build step

366b883

adding clarifications in Build bioimage step based on learnings on Ne…

b2be710

…ctar

Template OpenStack Packer config by removing environment-specific values

439d663

added and example nirin OpenStack Packer config - preconfigures flavo…

cbb62fb

…ur, network and source

Added examples folder for nirin and nectar tested packer configs to p…

abdfb90

…revent init issues

moved to examples folder

c49ac73

update build section, with reference to tested examples on nirin and …

bf2ebcf

…nectar

update Nirin and Nectar example build configs

6c8b155

fix(ansible): add retry logic and lock wait for apt update. Updated C…

ba87135

…VMFS proxy to Direct for portability

update build instructions to point to nirin/nectar examples and clrif…

45a446d

…y CVMFS Configuration for proxies

update build instructions to point to nirin/nectar examples and clrif…

c70527b

…y CVMFS Configuration for proxies

remove redundant statement

ac04bf8

Mitchob requested a review from georgiesamaha February 2, 2026 05:04

georgiesamaha requested changes Feb 2, 2026

View reviewed changes

Mitchob added 2 commits February 2, 2026 23:14

adding instructions to use main readme for deprecated nirin docs

5bc47d6

update build image step. split step 2, added removed markdown syntax …

81b4391

…to fix file

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make build infra‑agnostic: update README, add Packer examples, improve build playbook and docs#1

Make build infra‑agnostic: update README, add Packer examples, improve build playbook and docs#1
Mitchob wants to merge 27 commits intoAustralianBioCommons:mainfrom
Sydney-Informatics-Hub:testing-MO

Mitchob commented Feb 2, 2026

Uh oh!

georgiesamaha left a comment

Uh oh!

georgiesamaha Feb 2, 2026

Uh oh!

Mitchob Feb 2, 2026 •

edited

Loading

Uh oh!

georgiesamaha Feb 2, 2026

Uh oh!

georgiesamaha Feb 2, 2026

Uh oh!

georgiesamaha Feb 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Mitchob commented Feb 2, 2026

Summary

What I changed:

Testing performed

Next steps

Uh oh!

georgiesamaha left a comment

Choose a reason for hiding this comment

Uh oh!

georgiesamaha Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Mitchob Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgiesamaha Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

georgiesamaha Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

georgiesamaha Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mitchob Feb 2, 2026 •

edited

Loading