Aggie Experts

Aggie Experts is a joint project between the Office of the Provost and the UC Davis Library. Its purpose is to create a central registry of UC Davis faculty, researchers, experts, and creators and showcase the scholarship created at the university. Aggie Experts can be used as an expertise discovery platform for finding collaborators, mentors, and expert opinions.

flowchart LR
    subgraph Sources ["Data Sources"]
        CDL["CDL Symplectic Elements\noapolicy.universityofcalifornia.edu"]
        IAM["UC Davis IAM API"]
        AE_ENT["Aggie Enterprise\ngrant data"]
    end

    subgraph Anduin ["Aggie Experts Harvest Platform (powered by Anduin)"]
        GW["Auth Gateway\nKeycloak OIDC"]
        DAG["Dagster\norchestration"]
        WORKERS["Celery Workers"]
        CASK["CaskFS\nartifact store"]
        SUP["Superset\ndashboards"]
    end

    subgraph AEApp ["Aggie Experts Application"]
        ES[("Elasticsearch")]
        WEBAPP["Webapp\nSPA + API"]
    end

    subgraph Campus ["Campus Integrations"]
        SF["SiteFarm"]
        MIV["MyInfoVault"]
    end

    AE_ENT -->|"grant feed"| CDL
    CDL -->|"Symplectic API"| WORKERS
    IAM -->|"IAM API"| WORKERS
    DAG --> WORKERS
    WORKERS <-->|"harvest artifacts"| CASK
    WORKERS -->|"load"| ES
    WORKERS -->|"reporting"| SUP
    GW --> DAG & CASK & WEBAPP
    WEBAPP --> ES
    WEBAPP -->|"APIs"| SF & MIV

Loading

Components

• CDL — The California Digital Library hosts an instance of Symplectic Elements to facilitate UC open access policies. Aggie Experts pulls researcher profiles, publications, and grants from CDL using the Symplectic API.

• Aggie Experts Harvest Platform — The ETL and harvest infrastructure, powered by project-anduin. Hosts Dagster (workflow orchestration), CaskFS (artifact storage), Apache Superset (ETL reporting dashboards), and a Keycloak OIDC auth gateway. Celery workers execute the harvest and write reporting data directly to Superset's backing database.

• CaskFS — Content-addressed file storage with a built-in RDF/linked-data layer. Stores all intermediate harvest artifacts (raw CDL data, AE standard JSON-LD, Elasticsearch-ready JSON-LD) on a shared persistent disk volume.

• Elasticsearch — Aggie Experts creates expert, work, and grant JSON-LD document indexes to support query and presentation of data.

• Webapp — SPA and REST API serving the public-facing site and APIs. Powered by Lit web components and Express.

• Grant Processor — Grants originate in Aggie Enterprise and are imported into CDL Elements via the Grant Feed Process, making them available to the harvest.

• IAM — The Campus Identity Management API supplements researcher profiles during harvest.

• SiteFarm — Campus department websites can enhance faculty pages by embedding profiles and publications from Aggie Experts.

• MyInfoVault (integration notes) — UCD academics can import their publications from Aggie Experts, reducing duplicate entry.

ETL Reporting and Dashboards

Each weekly harvest writes run statistics, command outcomes, and scholarly output counts to a PostgreSQL reporting schema. These are surfaced in an Apache Superset dashboard accessible through the Anduin auth gateway at /superset.

The dashboard is the primary tool for understanding week-to-week changes in the harvest: which users had errors, who joined or left the expert group, how publication and grant counts shifted, and whether error rates are trending up or down. The reporting schema views — such as user_scholarly_output_weekly_changes, user_left_this_week, and this_week_harvest_errors — power these comparisons across weeks and are queryable directly in Superset for ad-hoc investigation.

See Reporting Database ERD and the Dagster Harvest Workflow — Reporting for schema details.

Development and Operation How To's

• Process Descriptions
  • Harvest Process — how data flows from CDL/IAM through CaskFS to Elasticsearch
  • Anduin Integration — ETL platform topology, auth, and custom gateway
  • Dagster Harvest Workflow — asset/job/schedule reference
  • Code and Data Deployment — how to build, deploy, and run the ETL
  • Process Weekly Grants
  • Update Archived Grants
• APIs
  • FIN Data Models
• Data Standards and Ontologies
  • Digital Objects
• Docker Images and Deployment
  • Deployment repo — GitOps repo for dev (Kubernetes) and prod (Docker Compose) environments
• Platform Docs
  • project-anduin
  • CaskFS