Feat/alert system #2599

sandeshit · 2025-11-28T10:58:11Z

Changes

Alert-system app
Polling, ETL

Checklist

Things that should succeed before merging.

Updated/ran unit tests
Updated CHANGELOG.md

Release

If there is a version update, make sure to tag the repository with the latest version.

main/sentry.py

deploy/helm/ifrcgo-helm/values.yaml

alert_system/admin.py

alert_system/models.py

alert_system/management/commands/poll_gdacs_flood.py

susilnem

Minor changes...

alert_system/tasks.py

alert_system/etl/base/extraction.py

alert_system/models.py

susilnem

Small changes!

alert_system/models.py

alert_system/tasks.py

alert_system/etl/base/extraction.py

sudip-khanal

Minor changes

alert_system/management/commands/poll_gdacs_cyclone.py

alert_system/management/commands/poll_gdacs_flood.py

alert_system/management/commands/poll_usgs_earthquake.py

susilnem

Do we have any analytics on usage and memory consumption?
Has any profiling been done so far?

alert_system/tasks.py

susilnem · 2025-12-30T05:16:49Z

alert_system/tasks.py

+        go_event_ids = list(
+            Event.objects.filter(field_reports__num_affected__gte=item.total_people_exposed, dtype=connector.dtype)
+            .values_list("id", flat=True)
+            .distinct()
+        )


thenav56 · 2025-12-30T09:18:35Z

alert_system/tasks.py

+            .distinct()
+        )
+
+        item.related_go_events = go_event_ids


Check if we have instead use M2M

Use bulk insert to add M2M (Use ignore existing)

alert_system/tasks.py

alert_system/models.py

alert_system/management/commands/poll_usgs_earthquake.py

thenav56 · 2025-12-30T09:40:41Z

alert_system/etl/base/loader.py

+        )
+
+        action = "Created" if created else "Updated"
+        logger.info(f"{action} Event for correlation_id={correlation_id}")


Suggested change

logger.info(f"{action} Event for correlation_id={correlation_id}")

logger.info(f"{action} Event for {correlation_id=}")

alert_system/etl/gdacs_flood/extraction.py

alert_system/etl/base/extraction.py

thenav56 · 2025-12-30T09:49:34Z

alert_system/etl/Gdacs_flood/extraction.py

+    event_collection_type = gdacs_flood_config.event_collection_type
+    hazard_collection_type = getattr(gdacs_flood_config, "hazard_collection_type", None)
+    impact_collection_type = getattr(gdacs_flood_config, "impact_collection_type", None)
+    filter_event = getattr(gdacs_flood_config, "filter_event", None)
+    transformer_class = GdacsTransformer
+    loader_class = GdacsLoader


We can replace all this with a single typed dict...

- Nested extraction with correlation id as fliter. - Self retry with exponential backoff. - Different models for Hazards and Impacts.

This commit splits the data extraction, transform and load part. The filtration part is also included in the load part.

ranjan-stha · 2026-01-08T04:41:56Z

alert_system/etl/base/extraction.py

+    def _hazard_filter(self, unit: str, value: int) -> str:
+        return f"monty:hazard_detail.severity_unit = '{unit}' AND " f"monty:hazard_detail.severity_value >= {value}"


@sandeshit To verify: unit and value mayn't exist in all the source items. Handle it if it is None or NaN.

ranjan-stha · 2026-01-08T04:47:48Z

alert_system/etl/base/extraction.py

+        current_payload = filters.copy() if filters else None
+
+        while current_url:
+            response = httpx.get(current_url, params=current_payload, timeout=30)


@sandeshit instead of hard code timeout value, would be nice to pass it in the arg with default val of 30.

ranjan-stha · 2026-01-08T04:54:31Z

alert_system/etl/gdacs_flood/config.py

+    "event_collection_type": "gdacs-events",
+    "hazard_collection_type": "gdacs-hazards",
+    "impact_collection_type": "gdacs-impacts",
+    "filter_event": {"hazard_codes": ["FL", "MH0600", "nat-hyd-flo-flo"]},


@sandeshit can we handle multiple set of same/similar hazards? Example: FL is a general one for Flood but there are other types of Flood as well like Flash Flood etc. What if we want to handle different sets of similar hazard types. Might need to discuss.

ranjan-stha · 2026-01-08T04:56:33Z

alert_system/etl/gdacs_flood/transform.py

+    # NOTE: This logic might change in future
+    def compute_people_exposed(self, metadata_list) -> int:
+        for data in metadata_list:
+            if data["category"] == "people" and data["type"] == "affected_total":


@sandeshit instead of hard coding the types, can we make use of Enums ?

ranjan-stha · 2026-01-08T04:58:29Z

alert_system/etl/gdacs_flood/transform.py

+            return {
+                "severity_unit": "",
+                "severity_label": "",
+                "severity_value": 0,


@sandeshit The severity_value: 0 carries a meaning. If the hazard_item is not present, can we use None or do it differently.

ranjan-stha · 2026-01-08T04:58:51Z

alert_system/etl/gdacs_flood/transform.py

+        return {
+            "severity_unit": detail.get("severity_unit", ""),
+            "severity_label": detail.get("severity_label", ""),
+            "severity_value": detail.get("severity_value", 0),


@sandeshit The severity_value: 0 carries a meaning. If the hazard_item is not present, can we use None or do it differently.

ranjan-stha · 2026-01-08T05:00:19Z

alert_system/etl/usgs_earthquake/transform.py

+            return {
+                "severity_unit": "",
+                "severity_label": "",
+                "severity_value": 0,


@sandeshit check the comment above

- Added category and type enum. - Alter people and building exposed fallback to None. - Add polling start date and lookback weeks field in Connector. - Squash Migrations.

sandeshit mentioned this pull request Nov 28, 2025

Alert-system polling #2584

Closed

2 tasks

sandeshit requested review from susilnem and thenav56 December 2, 2025 04:46

susilnem requested changes Dec 5, 2025

View reviewed changes

sandeshit force-pushed the feat/alert-system branch from 3fa8ab2 to 9fd5372 Compare December 7, 2025 17:49

sandeshit changed the base branch from project/alert-system-workflow to develop December 7, 2025 18:22

sandeshit changed the base branch from develop to project/alert-system-workflow December 7, 2025 18:23

sandeshit marked this pull request as draft December 7, 2025 18:24

sandeshit requested a review from susilnem December 8, 2025 09:09

susilnem requested changes Dec 9, 2025

View reviewed changes

susilnem requested changes Dec 11, 2025

View reviewed changes

sudip-khanal requested changes Dec 11, 2025

View reviewed changes

alert_system/management/commands/poll_gdacs_cyclone.py Outdated Show resolved Hide resolved

alert_system/management/commands/poll_gdacs_flood.py Outdated Show resolved Hide resolved

alert_system/management/commands/poll_usgs_earthquake.py Outdated Show resolved Hide resolved

sandeshit requested a review from susilnem December 12, 2025 09:02

susilnem requested changes Dec 30, 2025

View reviewed changes

thenav56 requested changes Dec 30, 2025

View reviewed changes

sandeshit added 13 commits January 2, 2026 14:13

chore(app): base setup for alert system app

c2f60f3

feat(polling): extraction logic for polling

2dcdd6d

chore(sentry): add cron monitor for tracking

a6011ac

feat(commands): management commands to run different polling tasks

1813a7c

refactor(extraction): refactor extraction logic for hazards and impacts.

6185bee

- Nested extraction with correlation id as fliter. - Self retry with exponential backoff. - Different models for Hazards and Impacts.

feat(extraction): use different classes for different sources

b469ece

feat(filter): add filtration classes and cronjob

aa10976

feat(etl): refactor existing extraction into ETL.

77560f3

This commit splits the data extraction, transform and load part. The filtration part is also included in the load part.

feat(etl): add past events fetcher

d839541

chore(alert-system): clean up naming and cronjobs

c8a8e1d

feat(etl): add past events from GO event table

0746f17

feat(etl): add usgs source

dcae4c6

chore(etl): separate past event extraction into different class

fcc9bdd

sandeshit force-pushed the feat/alert-system branch from c4d7b89 to fcc9bdd Compare January 2, 2026 08:42

fixup! chore(etl): separate past event extraction into different class

356e806

ranjan-stha reviewed Jan 8, 2026

View reviewed changes

feat(etl): change transform logic.

ad0aa8c

- Added category and type enum. - Alter people and building exposed fallback to None. - Add polling start date and lookback weeks field in Connector. - Squash Migrations.

	logger.info(f"{action} Event for correlation_id={correlation_id}")
	logger.info(f"{action} Event for {correlation_id=}")

		def _hazard_filter(self, unit: str, value: int) -> str:
		return f"monty:hazard_detail.severity_unit = '{unit}' AND " f"monty:hazard_detail.severity_value >= {value}"

Feat/alert system #2599

Are you sure you want to change the base?

Feat/alert system #2599

Conversation

sandeshit commented Nov 28, 2025

Changes

Checklist

Release

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susilnem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susilnem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sudip-khanal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susilnem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers