Observation Priority Manager (OPM)

The Observation Priority Manager (OPM) for Galactic Science in the TVSSC.

Observational Planning/Priority Manager (OPM) - A Target and Observation Manager Instance for the TVS galactic science community

The OPM is an instance of the Target and Observation Manager (TOM Toolkit) designed to track galactic science alerts over time, ingest targets with their alert priority, display corresponding light curves, and communicate with other TOM systems connected to observatories and proposals. This repository provides a system that can be installed locally by members of the TVS galactic science community, for instance the Microlensing Subgroup, and eventually deployed in a suitable cloud environment.

Usage

Once installed, you can access and interact with the OPM through its web interface. Here are some key features of the system:

• Target Management: Create, view, and update targets associated with microlensing alerts, subscribe to broker services
• Light Curve Visualization: View and analyze light curves associated with each target to monitor microlensing events over time.

Planned Features

• Communication with Other TOM Systems: Interact with other TOM systems connected to observatories and proposals, enabling easy requesting of observations directly from the OPM interface.
• Requesting and interacting with HPC systems to fit events: Interact safely with HPC centers to fit complicated microlensing events, i.e. binary and triple lens events

Quick start (local docker-based dev setup)

Requirements

• docker or other compatible solution such as podman
• node(NodeJS) and npm

Both node and npm can be installed using nvm, which is a Version Manager for NodeJS (similar to pyenv). Once nvm is installed, run these two commands:

nvm install --lts
nvm use --lts

This will install node and npm in the current LTS version. You will not have to run these commands again, unless you want to use a different version of node or npm.

Steps

1. Clone the repository and change your directory in to the cloned repo.

2. Start the docker containers using the following command:

   docker compose -f compose.base.yaml -f compose.local.yaml up -d
   # or the following, if there have been significant changes or it has been a while
   docker compose -f compose.base.yaml -f compose.local.yaml up -d --build

   This will, according to the compose.*.yaml files, build a docker image, run it in a container, using a Postgresql database running in another container.

3. Change into the directory frontend (cd frontend) and run:

   Make sure you installed node and npm as described in Requirements

   npm install
   npx vite build

   This builds the frontend assets.

4. Point your browser to http://127.0.0.1:8000 for the Galactic Science OPM home page running locally.

5. When the database is empty, you may want to create an admin user for your TOM. You can do so by running the Django createsuperuser management command in the the container. First 'exec' into the container:

   docker exec -it galactic-science-opm-galactic-science-opm-1 /bin/bash

   In the container's bash shell (that you just opened):

   ./manage.py createsuperuser

   Ctrl-d to leave the container. Log into the OPM with the credentials you just created.

   To stop the containers:

   docker compose -f compose.base.yaml -f compose.local.yaml down

Setting up for local development without docker

Requirements

• a compatible version of Python
• the package manager poetry
• nodejs and npm: See Requirements for information on how to install these.

Steps

After cloning the repository and changing your directory (cd-ing) into the cloned repo:

1. Create a virtual environment

Always work in a virtual environment. To create and activate one:

python -m venv .venv
source .venv/bin/activate

2. Install the dependencies

There are more than one way to do this. The pyproject.toml project description file is common to all of them. Here, we use poetry:

poetry install

3. Work in a branch for the development that you'll be doing

If you've just cloned the repo, you'll be in the dev branch. You'll want to do your development in a branch that can later be merged into the dev branch (and ultimately into the main branch for deployment). With the -b switch, git checkout creates a new branch.If the branch you want already exists, you don't need the -b switch.

git checkout -b <name-of-your-branch>

4. Spin up a database for your development OPM to use

We quote from settings.py:

# Here is how to start a dockerized postgresql container compatible with the default
# values in the 'default' DATABASEs configuration below:

# 1. create a postgres docker image named 'opm-tom-postgres':
# docker create --name opm-tom-postgres -e POSTGRES_DB=galactic_science_opm -e POSTGRES_PASSWORD=opm -e POSTGRES_USER=opm -p 5432:5432 postgres

# 2. start the container from that image
# docker start opm-tom-postgres

# 3.(optional -- this is for completeness) If you want to shut down the dockerized postgres server started in step 2:
# docker stop  opm-tom-postgres

# Also, NOTE: the values in the configuration dictionary below are also referenced in the compose.yaml file!!

5. Build the frontend assets

Change into the directory frontend (cd frontend) and run:

Make sure you installed node and npm as described in Requirements!

npm install
npx vite build

This builds the frontend assets.

6. Start the Django development server

./manage.py runserver

Point your browser to http://127.0.0.1:8000 for the Galactic Science OPM home page running locally.

Additional info for starting up

Local setup

Uses django dev server for faster iterations.

docker compose -f compose.base.yaml -f compose.local.yaml up -d
docker compose -f compose.base.yaml -f compose.local.yaml down

Prod like setup

docker compose -f compose.base.yaml -f compose.prod.yaml up -d
docker compose -f compose.base.yaml -f compose.prod.yaml down

E2E debugging setup

This can be used to make debugging easier when writing end to end tests. It does not run with the actual e2e setup (which would use nginx) but with Django's dev server.

docker compose -f compose.base.yaml -f compose.local.yaml -f compose.e2e.yaml up -d
docker compose -f compose.base.yaml -f compose.local.yaml -f compose.e2e.yaml down

Note that this will not automatically run the e2e tests without overriding the BASE_URL, because it wants to run against the frontend-proxy as defined in compose.e2e.yaml but it does not exist here. If you want to run the tests immediately, use this:

BASE_URL="http://galactic-science-opm:8000" \
docker-compose -f compose.base.yaml -f compose.local.yaml -f compose.e2e.yaml up -d

Notes on testing

Unit test coverage report for the current branch

If you are working on a new feature or fixing a bug, you should make sure that there are tests that make sure that your feature works and that it keeps working in the future or the bug won't go unnoticed again.

To check whether you have sufficiently tested your code, you can generate a coverage report for just the difference of your branch in relation to the dev branch.

Currently this works for unit tests only, but this will be expanded, once we introduce more integration tests.

To generate a HTML report, use the just command just show-branch-coverage-as-html. For this to work, you have to first run the unit test suite by e.g. running the just command just run-unittest. If you don't use just, you can look into justfile and copy the corresponding command (if you happen to use podman, replacing docker with podman should be enough to make it work).

A coverage of 100% should not always be the main goal. Rather you should make sure that all the important aspects of your feature are tested in a way, so that you immediately notice, if a feature no longer works or no longer works as intended.

There is also a just command for generating a markdown report. This can be used to insert the coverage report for your feature into your PR description, so the reviewers can see, if the required coverage goals have been met.

Running unit tests

Can also be run in your local directory, because nothing django related should be used in these tests: python manage.py test custom_code.tests.unit --settings=galactic_science_opm.settings_test or pytest custom_code/tests/unit

If the commands above lead to an error like the following one...

Got an error creating the test database: permission denied to create database

... try creating a new container for the database, as described under "Spin up a database for your development OPM to use"

You can also run the tests directly in the container using this command: docker compose exec galactic-science-opm coverage run -m pytest custom_code/tests/unit/ Doing it this way also generates the information for coverage reporting.

You can generate and open a HTML coverage report using the following command:

docker compose exec galactic-science-opm coverage html
open ./htmlcov/index.html

Running integration tests

Since these might interact with the database, they should/must be run with docker (docker exec), since e.g. hosts for databases and so on are set via envs and also have no meaning outside the docker network: python manage.py test custom_code.tests.integration --settings=galactic_science_opm.settings_test or pytest custom_code/tests/integration

Running e2e tests

To run E2E tests, you have to use the compose.e2e.yaml, so first use: docker compose -f compose.base.yaml -f compose.prod.yaml -f compose.e2e.yaml up -d

This will also run the tests in the container e2e.

The start up will also run the management command seed_e2e_data, which clears the test-database and adds testing data. For this to work, you have to put the file db_out_full.json in the folder specified in the env variable DJANGO_TEST_DATA_DIR. As this file might at some point contain data that is proprietary, this file is not committed for now. This might change in the future. To get this file, reach out to another dev.

Then you can run the tests on your machine, using pytest custom_code/tests/e2e -rs --headed --slowmo=200. This has the advantage that you can see what the browser is doing and debug errors more easily.

The tests can also be run headless on your machine, as they would in a CI setup by using this command (locally or from docker): pytest custom_code/tests/e2e -rs

Beware that running the E2E tests does not flush the test DB afterwards. So any data that was created during the test, will remain in the database.

If you want to reset the test data right away, just visit /flush_and_seed/ when running the e2e setup and the test data will be reapplied.

When you run the E2E tests again, though, the database is reset and all the test data is applied to the test database.

Or (but this has not been tested) to run this in some kind of CI setting:

docker compose -f compose.base.yaml -f compose.prod.yaml -f compose.e2e.yaml up \
--abort-on-container-exit --exit-code-from e2e

For help in creating tests, use playwright codegen http://localhost:8000.

Automation tooling

There is a justfile in this repo with various commands to make common tasks more easy, but this is completely optional.

After you installed just, you can do things like just run-unittest and the unit tests will run.

This assumes that you can use docker. If you happen to use podman, replacing docker with podman should be enough to make the commands inside the justfile work, if you want to run the commands manually. If you use podman and want to use the justfile, consider setting up an alias, so docker points to podman.

If you do not want to use justfile or use a non-docker workflow i.e. running only the database in docker and the django dev server locally, you have to prefix the commands that use external packages (e.g. coverage or diff-cover) with poetry run. So, if you want to use the justfile command show-branch-coverage-as-html locally, docker compose exec galactic-science-opm coverage xml will have to be changed to poetry run coverage xml.

pre-commit

There is a pre-commit hook which lints and formats code with ruff on every commit. To use this, run poetry run pre-commit install. This only needs to happen once. Make sure to run poetry install first so that pre-commit is installed.