feat: v12.0 extensible platform — custom skills, user tools, multi-agent pipelines#1
Open
feat: v12.0 extensible platform — custom skills, user tools, multi-agent pipelines#1
Conversation
…ent pipelines Add self-service extension capabilities for end users: - Custom Skills frontend CRUD in CapabilitiesView (create/edit/delete agents) - User-defined declarative tools (http_call, sql_query, file_transform, chain) with UserToolset exposing dynamic FunctionTool instances to ADK agents - Multi-agent pipeline composition via WorkflowEditor Skill Agent nodes (pipeline_type: "custom_skill" in DAG execution engine) - Capabilities browser tab (13th DataPanel tab) with filter/search - Knowledge Base frontend UI (14th tab: CRUD, doc management, semantic search) - Resizable three-panel layout with drag handles (240-700px range) - DataPanel tab horizontal scrolling for overflow Security fixes: - SEC-1: Remove hardcoded admin/admin123 DB fallback — require DB for auth - SEC-2: Add brute-force protection (5 consecutive failures → 15min lockout) Architecture improvements: - S-1: Extract intent_router.py (153 lines) and pipeline_helpers.py (284 lines) from app.py, reducing it from 3563 to 3267 lines - T-4: Track router token consumption separately (pipeline_type="router") - F-4: Add React ErrorBoundary wrapping all three panels - Upgrade ADK v1.26.0 → v1.27.2 (Session Rewind, CredentialManager, OTel) Bug fixes: - Fix arcpy_tools.py syntax error (duplicate import json + try block) - Fix test_knowledge_agent.py (wrong prompts path, broken imports) - Install APScheduler for workflow cron scheduling - Create chainlit_zh-CN.md for Chinese localization - Improve MCP Hub health check (distinguish disabled vs failed servers) Documentation: sync CLAUDE.md, technical-guide.md, roadmap.md, 7 DITA sources, 2 preview HTMLs with accurate metrics (92 endpoints, 23 toolsets, 18 skills, 2100+ tests, 17 DB tables). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update both README.md (Chinese) and README_en.md (English) to reflect: - 23 toolsets (was 22), 18 ADK Skills (was 16), 92 REST endpoints (was 85) - 2121 tests (was 2104), ADK v1.27.2 (was v1.26) - New v12.0 self-service extension section (Custom Skills CRUD, User Tools, multi-agent pipeline composition, capabilities/KB tabs) - Updated project structure with new modules (intent_router, pipeline_helpers, user_tools, user_tool_engines, capabilities, user_tools_toolset) - Frontend: 13 tabs (was 7), resizable panels, Error Boundaries - WorkflowEditor: 4 node types (added Skill Agent) - Security: brute-force protection, DB auth required - Updated GitHub repo description via gh repo edit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the three-layer architecture of the multi-agent system: - Tool: atomic FunctionTool, grouped in 23 BaseToolset subclasses - Skill: dual identity (instruction template + callable AgentTool), 3 sources (built-in ADK Skills, Custom Skills, Prompt YAML) - Agent: LlmAgent with model/instruction/tools/output_key, composed via SequentialAgent/ParallelAgent/LoopAgent Includes execution flow diagram, model tiering strategy, state passing mechanism, and user self-service extension loop (Tool → Skill → Pipeline). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document TUI vs Web UI differences, 6 core usage scenarios (batch processing, headless servers, CI/CD, cron automation, large data, Unix pipe integration), multi-channel unified architecture, file path handling (zero-copy local access), visualization degradation strategy, command design draft, and infrastructure readiness assessment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the complete memory system: - L1 Immediate: output_key state passing + ContextVar propagation - L2 Session: last_context multi-turn injection - L3 Cross-session: PostgresMemoryService + Memory ETL auto-extraction - L4 Long-term: spatial memory (6 types) + failure learning - L5 Knowledge: Knowledge Base (RAG + pgvector) + Knowledge Graph Includes injection flow diagram, comparison table, tool inventory, and design principles (layered isolation, auto+manual, dedup+quota). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all 34 agents across 4 pipelines: - 22 LlmAgent (reasoning entities) - 6 SequentialAgent, 3 ParallelAgent, 3 LoopAgent (orchestrators) - Plus unlimited user-defined CustomSkill agents Includes per-pipeline hierarchy diagrams, model tier allocation (9 Fast + 14 Standard + 2 Premium), factory function mapping, and orchestration pattern summary. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all 18 built-in ADK Skills across 8 domains: GIS (6), Governance (3), Database (2), Visualization (2), Analysis (2), Fusion (1), General (1), Collaboration (1). Includes trigger keywords, three-level incremental loading, SKILL.md structure, Custom Skills CRUD, and Skill Bundles. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document all tools grouped by 23 BaseToolset subclasses: GeoProcessing (26 incl. 8 ArcPy), Visualization (11), DataLake (10), SemanticLayer (10), KnowledgeBase (10), Location (9), Team (9), AdvancedAnalysis (8), RemoteSensing (8), Exploration (7), Admin (6), Database (6), SpatialT2 (6), Streaming (6), Fusion (5), Memory (5), SpatialStats (4), KnowledgeGraph (4), Watershed (4), Analysis (3), File (3), plus dynamic MCP and UserToolset. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…shboard Backend: - GET /api/system/status — aggregated health (DB, MCP, bots, A2A, features, models) - GET /api/bots/status — per-platform bot config (configured keys, missing env vars) Frontend (AdminDashboard.tsx): - "系统状态" tab: DB/MCP/ArcPy/Cloud status cards, model tier config table, feature flags - "Bot 管理" tab: WeChat/DingTalk/Feishu cards with config status, missing env hints - "A2A" tab: Agent Card display, service status, exposed skills list All three sections surface backend capabilities that previously had no frontend UI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Security: - SEC-3: Fix symlink bypass in sandbox path validation — use os.path.realpath() instead of os.path.abspath() in user_context.py and sharing.py - SEC-5: Change default ContextVar role from 'analyst' to 'anonymous', forcing explicit role assignment on every request Model configuration: - Make MODEL_FAST/STANDARD/PREMIUM configurable via env vars (agent.py) - Add get_model_config() API for frontend exposure - GET /api/config/models — returns tier config with provider detection - AdminDashboard "模型配置" tab: tier table, router model, env var guide, example configs for Gemini/Anthropic/OpenAI Route count: 92 → 95 (system/status, bots/status, config/models) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skill Bundles frontend (Roadmap: Skill Bundles 前端 UI ✅): - CapabilitiesView: new "技能包" filter tab with full CRUD - Bundle form: name, description, toolset multi-select, skill multi-select, intent triggers, shared toggle - Fetches /api/bundles and /api/bundles/available-tools on mount API refactoring (S-4 — incremental): - Create data_agent/api/ package with helpers.py (shared auth) and bundle_routes.py (extracted from frontend_api.py) - frontend_api.py bundle handlers now delegate to api.bundle_routes - Pattern established for future domain module extraction Code quality: - T-3: Evaluation thresholds now configurable via env vars (EVAL_THRESHOLD_GENERAL, EVAL_THRESHOLD_OPTIMIZATION, etc.) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Knowledge Base GraphRAG UI (Roadmap ✅):
- GraphRAGSection component in KB detail view
- Build graph button → POST /api/kb/{id}/build-graph
- Entity/relation list with type badges and counts
- Graph search → POST /api/kb/{id}/graph-search
Thread safety (S-2 ✅):
- _mcp_started: double-checked locking with threading.Lock
- _a2a_started_at: threading.Lock guard in mark_started()
Frontend quality (F-2 ✅):
- Replace window.__resolveAnnotation / window.__deleteAnnotation
with document.dispatchEvent(CustomEvent) pattern
- Proper cleanup via removeEventListener
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
User Tools Phase 2 — Python Sandbox: - validate_python_code(): AST analysis with import whitelist (19 modules), forbidden builtins (exec/eval/__import__/open), forbidden attrs, must define tool_function() - python_sandbox.py: subprocess execution with sanitized env, restricted builtins, timeout (30s default, 60s max), 100KB output cap - API: create/update endpoints validate Python code via AST before persisting - Frontend: "Python 沙箱" template type in tool creation form React Context API (F-1 ✅): - contexts.ts: MapContext (layers/center/zoom/layerControl) + AppContext (userRole/dataFile/onDataUpdate) - App.tsx: MapContext.Provider + AppContext.Provider wrapping workspace - Components can use useMapContext()/useAppContext() instead of props Thread safety (S-2) + Global callbacks (F-2): - _mcp_started: double-checked locking with threading.Lock - _a2a_started_at: threading.Lock guard - MapPanel annotations: window.__* → CustomEvent dispatch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WorkflowEditor — live execution status (Roadmap ✅):
- handleExecute captures run_id from execute response
- Polls /api/workflows/{id}/runs/{run_id}/status every 2s
- Live status panel shows per-node status (running/completed/failed)
with colored dots, duration, and overall progress
- Auto-stops polling on completion/failure
SEC-4 — Prompt injection hardening (Roadmap ✅):
- Expand FORBIDDEN_PATTERNS from 7 to 24 patterns covering:
role hijacking, prompt boundary markers, instruction override,
injection delimiters, data exfiltration attempts
- build_custom_agent(): wrap user instruction with safety boundary
+ explicit refusal directive for prompt leaking requests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
API refactoring (S-4 continued): - Extract 10 MCP handlers (275 lines) to api/mcp_routes.py - frontend_api.py: 2473 → 2180 lines (-293, delegate pattern) - Total extracted: bundle_routes.py + mcp_routes.py + helpers.py ADK v1.27 feature adoption: - capabilities.py: Replace manual SKILL.md YAML parsing with google.adk.skills.list_skills_in_dir() (Roadmap: list_skills_in_dir ✅) - Removes yaml dependency from capabilities module Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
S-4 API splitting (continued): - Extract 11 KB handlers (186 lines) to api/kb_routes.py - frontend_api.py: 2180 → 1996 lines (delegate pattern for KB) - Cumulative extraction: helpers + bundles + kb = 3 domain modules ADK v1.27 adoption: - capabilities.py: use google.adk.skills.list_skills_in_dir() instead of manual SKILL.md YAML parsing (Roadmap ✅) Note: 6 pre-existing TestMcpServerCrudAPI failures unrelated to this change (were broken before this session). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document 80+ security controls across 6 defense layers: - L1 Authentication: PBKDF2-SHA256, JWT, OAuth2, brute-force lockout - L2 Authorization: RBAC 3-tier, ContextVar isolation, file sandbox - L3 Input validation: SQL injection, prompt injection (24 patterns), SSRF, path traversal, AST code validation, MCP command whitelist - L4 Execution isolation: Python sandbox, env sanitization, timeout - L5 Output security: API key/password/token redaction, hallucination - L6 Audit: 30+ event types, 90-day retention, Prometheus metrics Includes encryption inventory, bot security, known limitations, and per-area strength ratings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The MapContext.Provider + AppContext.Provider wrapping caused React error #310 (too many re-renders) because useMemo dependencies (mapLayers, userRole) produced new references on each render cycle, triggering infinite context value updates. Fix: remove Provider wrapping from App.tsx, keep contexts.ts for future migration with stable refs (useRef-based values). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Restore original MCP handlers in frontend_api.py (mcp_routes.py was deleted but delegates still referenced it, causing ModuleNotFoundError) - Add ensure_workflow_template_tables() to startup init - Add ensure_skill_bundles_table() to startup init - Fixes "relation agent_workflow_templates/agent_skill_bundles does not exist" All 69 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…zation, UI redesign BP-3: Automatic analysis lineage recording - Add pipeline_run_id ContextVar to pipeline_helpers.py, set at pipeline start - Fix tool_params passthrough in sync_tool_output_to_obs → register_tool_output - Add pipeline_run_id column to agent_data_catalog with migration - Enhance lineage queries to return pipeline_run_id in ancestors/descendants - Add derives_from/feeds_into edge types + add_lineage_edge() to knowledge_graph BP-5: Industry analysis templates (first batch) - Add 3 industry templates: urban heat island, vegetation change, land use optimization - TemplatesView: add Chinese industry category filter buttons - CapabilitiesView: add 'template' filter type with /api/templates integration S-4: API route extraction (18% → 42%) - Extract mcp_routes.py (10 endpoints), workflow_routes.py (8), skills_routes.py (5) - Delegate from frontend_api.py, total route count unchanged at 95 UI: Cartographic Precision design system - Space Grotesk + JetBrains Mono fonts - Teal (#0d9488) / Amber (#d97706) color palette, warm Stone backgrounds - Topographic contour login page, underline-style tabs, upgraded shadows/radii - Lineage DAG visualization with SVG arrows in asset detail view Tests: 2123 passed (50 catalog + 69 API), 0 failures Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sc fixes - Add 10 new DITA technical guide topics (tg-*): database architecture, DRL optimization, evaluation CI/CD, fusion engine, knowledge graph, map rendering pipeline, multi-pipeline orchestration, multi-tenancy, observability, semantic intent router - Add A2A capabilities documentation (docs/a2a-capabilities.md) - Planner prompt v7.1.3: enforce auto-visualization after analysis, improve PDF report image embedding logic - Fix report_generator PNG regex to handle relative paths - Add watch_ignore for uploads/downloads/logs/db in chainlit config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…t graph, metrics Semantic hybrid search (BP-2): - Add embedding JSONB column to agent_data_catalog with migration - Generate text-embedding-004 vectors on asset registration - Hybrid search: 60% fuzzy n-gram + 40% vector cosine similarity - Graceful degradation when embedding API unavailable Knowledge graph asset integration: - register_catalog_assets(): create data_asset nodes with belongs_to_domain edges - discover_related_assets(): traverse lineage + domain edges for related assets - Asset type → domain mapping (vector→GIS, raster→遥感, tabular→统计) Planner data discovery priority (v7.2.0): - New prompt section: search catalog before requesting data upload - Extract keywords from user request → search_data_assets → confirm → execute Semantic metrics (v12.2): - New table agent_semantic_metrics with migration 011 - register_metric/resolve_metric/list_metrics tool functions - seed_builtin_metrics: 5 presets (植被覆盖率, 建筑密度, 碎片化指数, 人口密度, 坡度均值) - Fuzzy + alias matching for natural language → SQL definition resolution Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… system, roadmap v14.x
v13.0 Virtual Data Layer:
- virtual_sources.py: CRUD + Fernet encryption + 4 connectors (WFS/STAC/OGC API/custom)
- VirtualSourceToolset: 5 ADK tools wired to General + Planner pipelines (24 toolsets)
- REST API: 6 endpoints /api/virtual-sources/* (virtual_routes.py)
- Frontend: DataPanel "数据源" tab with list/create/edit/delete/test UI
- Schema semantic mapping: text-embedding-004 cosine similarity + 35 canonical fields
- 64 unit tests (test_virtual_sources.py)
v13.1 MCP Server v2.0:
- 6 high-level metadata tools: search_catalog, get_data_lineage, list_skills,
list_toolsets, list_virtual_sources, run_analysis_pipeline
- MCP Server upgraded to v2.0 (36+ tools total)
v14.0 Rating & Clone System:
- Skills/Tools: rating_sum, rating_count, clone_count columns + migration
- REST endpoints: POST /api/skills/{id}/rate, /clone; POST /api/user-tools/{id}/rate, /clone
- 105 total REST API endpoints
Roadmap: v14.x four-version plan across 5 directions (NLP, marketplace,
DRL, SPA, multi-agent) written to docs/roadmap.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…kpoints
Marketplace:
- GET /api/marketplace — aggregated shared skills/tools/templates/bundles
- DataPanel "市场" tab with search, type filter, sort (rating/usage/recent)
- Star rating + clone buttons per item
Rating & Clone:
- Skills/Tools: rating_sum/rating_count/clone_count columns (migration 013)
- POST /api/skills/{id}/rate, /clone; POST /api/user-tools/{id}/rate, /clone
- rate_skill(), clone_skill(), rate_tool(), clone_tool() functions
Workflow Checkpoints (v14.0):
- node_checkpoints JSONB column on workflow_runs (migration 014)
- Per-layer checkpoint save during DAG execution
- get_run_checkpoint() + retry_workflow_node() for single-node retry
- POST /api/workflows/{id}/runs/{run_id}/retry
- GET /api/workflows/{id}/runs/{run_id}/checkpoint
- 108 total REST API endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Heatmap (2D + 3D): - MapPanel: leaflet.heat integration for point/polygon data with intensity - Map3DView: density-colored ScatterplotLayer for 3D heatmap rendering - Both support value_column for intensity weighting Measurement Tools: - MapPanel: distance + area measurement via click-to-add points - Polyline distance (meters/km) + shoelace area (m²/km²) - Measurement overlay with clear button 3D Layer Control: - Map3DView: layer panel with show/hide toggles per layer - Layer visibility state integrated into deckLayers useMemo Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
….0 milestone DRL Scenario Templates: - DRLScenario config class with 3 built-in scenarios: farmland_optimization, urban_green_space, facility_siting - Each scenario defines source/target types, reward weights, max_conversions - list_scenarios() API + GET /api/drl/scenarios endpoint Memory Search: - GET /api/memory/search?q=keyword&type=region — search user spatial memories - Reuses existing recall_memories() from memory.py v14.0 Complete: 110 REST API endpoints total. - Intent disambiguation (already existed) - Rating + clone system (skills/tools) - Marketplace gallery - Workflow checkpoints + node retry - Heatmap + measurement + 3D layer control - DRL scenario templates - Memory search API Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…g engine, 3D sync NLP Interaction: - generate_followup_questions() in pipeline_helpers.py — Gemini Flash generates 3 recommended follow-up analyses after pipeline completion - Sent as clickable action cards in ChatPanel User Extension: - Skills/Tools: version, category, tags[], use_count columns (migration 015) - agent_skill_versions / agent_tool_versions history tables (last 10 versions) - rollback_skill(), get_skill_versions(), increment_skill_use_count() - increment_tool_use_count() for usage tracking Multi-Agent: - agent_registry.py: PostgreSQL-backed service registry with register/discover/heartbeat - invoke_remote_agent() for bidirectional A2A RPC via HTTP - AgentMessageBus upgraded to PostgreSQL persistence (agent_messages table) - ensure_registry_table() called at startup DRL: - LandUseOptEnv accepts DRLScenario config — configurable types and reward weights - Instance-level weights (self.slope_w etc.) instead of module globals in reward calc Frontend: - Map3DView: basemap prop synced from MapPanel selection (3 GL styles) - DataPanel: GeoJSON editor tab — paste/edit/validate/format/save - 110 REST API endpoints, frontend builds clean Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…raining, annotation export
Analysis Chains (v14.2):
- analysis_chains.py: conditional follow-up automation (if metric > threshold, execute Y)
- Chain evaluation after each pipeline completion, triggered as action cards
- CRUD endpoints: GET/POST /api/chains, DELETE /api/chains/{id}
Circuit Breaker:
- circuit_breaker.py: failure tracking with closed/open/half-open states
- Auto-disable after 5 failures within 5min window, 2min cooldown recovery
- Singleton get_circuit_breaker() for tool/agent reliability
Skill Approval Workflow:
- publish_status column (draft→pending_approval→approved→rejected)
- request_publish(), review_publish(), list_pending_approvals()
- REST: POST /api/skills/{id}/publish, /review; GET /api/skills/pending
DRL Training API:
- train_drl_model() tool — user data + scenario + epochs → MaskablePPO training
- list_drl_scenarios() tool exposed to ADK agents
- DRL engine uses instance-level reward weights from DRLScenario config
DAG Crash Recovery:
- find_incomplete_runs() + mark_run_failed() + recover_incomplete_runs()
- Auto-recovery at startup for stuck 'running' state runs
Annotation Export:
- GET /api/annotations/export?format=geojson|csv
117 REST API endpoints total. Frontend builds clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update core metrics: 34 toolsets, 21 skills, 170+ APIs, 2550+ tests. Add AgentOps maturity score (3.9/5, Google P2P 78% compliance). Add StorageManager data lake, cost management, HITL, Feature Flags. Remove duplicate metric rows from v14.x era. Co-Authored-By: Claude <noreply@anthropic.com>
… GeoFM Embeddings LaTeX manuscript targeting IJGIS, covering: - JEPA architecture with frozen AlphaEarth + LatentDynamicsNet (459K params) - 17 study areas with train/val/test/OOD splits - Three innovations: L2 manifold preservation, dilated conv, multi-step loss - Change-pixel evaluation protocol - Complete results tables and references (22 citations) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- world_model_paper_en.docx (43K) - English version - world_model_paper_cn.docx (44K) - Chinese version - generate_paper_docx.py - generation script Format matches reference paper_manuscript_cn.docx: title/author/affiliation headers, numbered sections, tables (Light Grid Accent 1), references. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…x, LaTeX corrections
Reviewer response (4 major + 3 minor issues):
R1: Added formal ablation study (3 experiments):
- w/o L2 norm: advantage drops 62%, norm drifts to 1.11
- w/o dilated conv: change-pixel advantage drops 32%
- w/o unrolled loss: validation flips negative, multi-step degrades
R2: Fixed scenario conditioning paradox — now explicitly stated as
"architectural placeholder trained on baseline only", removed
counterfactual claims from abstract/conclusions
R3: Fixed LaTeX — concat notation (||), removed phantom Fig reference,
added \label{sec:future} for cross-ref, updated scenario section
R4: Regenerated both Word documents (en + cn) with ablation table
and corrected scenario language
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…set, map timeline World Model integration debugging & optimization: - Fix intent router: "世界模型" → GENERAL (was OPTIMIZATION), add confirmation word routing - Fix planner: add WorldModelToolset + NL2SQLToolset to planner agent tools - Add world model shortcut path in app.py: skip LLM planner, direct tool call (1 API call vs 6) - Fix n_years regex: exclude 4-digit years from matching - Fix GeoJSON token explosion: strip geojson_layers from LLM response - Fix layer type: "geojson" → "categorized" for LULC color rendering - Fix sampleRectangle scale: setDefaultProjection(atScale) for proper grid resolution - Fix auto-scale logic: allow 256px grid for small areas (<0.1°) - Retrain dynamics model + LULC decoder with correct resolution data (83.5% CV accuracy) Frontend & map enhancements: - Add timeline slider with play/pause animation for temporal LULC layers - Add satellite basemaps: Gaode Satellite + ESRI World Imagery - Add style_map support for categorized layer rendering - Add layer visible property for initial visibility control - Expose handleMapUpdate on window for DataPanel → MapPanel communication New tools & capabilities: - NL2SQLToolset: discover_database_schema, execute_safe_sql, execute_spatial_query - load_admin_boundary: dedicated admin boundary tool with fuzzy matching - Safety guard: xiangzhen table requires sql_filter (prevents 890MB full-table download) Paper updates: - Add Section 6.3: Interpretability and Geographic Theory Alignment (Tobler, Hagerstrand, linear decodability) - Add TikZ architecture diagram (Fig. 1) + embedded PNG in Word documents - Remove Mean Reversion ghost baseline from Section 5.2 and abstract - Add Tobler (1970) reference Other fixes: - 429 retry: all agents + router use Gemini instances with retry_options (2s backoff, 3 attempts) - Session history: backfill Thread.userIdentifier + on_chat_start auto-update - CORE_TOOLS: add world_model_predict, load_admin_boundary (always available) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SQLAlchemy 2.x requires text() wrapping for raw SQL strings passed to read_postgis(). Without it, the connection object treats the SQL as parameterized query and fails with "immutabledict is not a sequence". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…l limit AlphaEarth has 64 bands, so max spatial pixels = 262144/64 = 4096 (64x64). Previous 256px grid caused 2.9M total pixels, exceeding GEE limit and causing sampleRectangle to hang indefinitely. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Paper documents (LaTeX, Word, figures, ablation scripts) are private and should not be shared on GitHub. Files remain on local disk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…emaps - Version bump to v15.2 - Add World Model Tech Preview section (JEPA, AlphaEarth, LatentDynamicsNet) - Add NL2SQL dynamic query section (schema discovery, spatial query, fuzzy matching) - Update core metrics: 35 toolsets, 210+ tools - Add v15.2 to version roadmap - Update GitHub repo description/about Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New module: embedding_store.py — pgvector storage layer for 64-dim embeddings - store_grid_embeddings(): batch INSERT with PostGIS point geometry - load_grid_embeddings(): cache lookup by bbox+year - find_similar_embeddings(): cosine similarity search with spatial filter - get_temporal_trajectory(): multi-year embedding history for a location - import_npy_cache(): bulk migrate from .npy files to pgvector - get_coverage(): summary of cached areas/years - Cache hierarchy: pgvector (ms) → .npy files (ms, backward compat) → GEE (s) - Auto-store: GEE downloads automatically written to pgvector for future use - Auto-migrate: .npy file reads trigger lazy migration to pgvector - New API endpoints: - GET /api/world-model/embeddings/coverage - POST /api/world-model/embeddings/search - POST /api/world-model/embeddings/import - New tools: world_model_embedding_coverage, world_model_find_similar - Migration 036: agent_geo_embeddings table + ivfflat index + gist spatial index - docker-db-init.sql: +pgvector extension Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… Model) Implement complete causal inference framework for paper support: Angle A — GeoFM-based statistical causal inference (6 tools): PSM, ERF, DiD, Granger, GCCM, Causal Forest with optional AlphaEarth 64-dim embedding augmentation (use_geofm_embedding). Angle B — LLM causal reasoning via Gemini (4 tools): Causal DAG construction, counterfactual reasoning, causal mechanism explanation, and structured what-if scenario generation. Angle C — Causal world model (4 tools): Spatial intervention prediction with spillover analysis, counterfactual comparison between parallel scenarios, embedding-space treatment effect measurement, and statistical prior integration (ATT calibration). Also includes: 8 REST API endpoints, CausalReasoningTab frontend, WorldModelTab intervention/counterfactual mode, data catalog semantic search enhancement, and 82 new tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Version bump to v15.3 with causal inference system overview - Add project origin vision image (2023-09 architecture concept) - New section: Angle A (GeoFM 6 tools), B (LLM 4 tools), C (World Model 4 tools) - Update metrics: 37 toolsets, 220+ tools, 178+ APIs, 22 tabs, 82 causal tests - Update GitHub repo description/about Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update all metrics: 321 Python files (99,684 lines), 36 frontend files (11,861 lines), 37 toolsets, 14 API route files, 113 test files. Add v14.5→v15.3 growth comparison table and new module highlights (causal inference A/B/C, world model, embedding store, NL2SQL). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…g triage - Add v15.2 section: World Model (AlphaEarth JEPA), NL2SQL, pgvector cache, map timeline, satellite basemaps - Add v15.3 section: three-angle causal inference (Angle A/B/C, 14 tools, 82 tests, 8 REST APIs, CausalReasoningTab, WorldModelTab intervention mode) - Triage historical backlog: 3 priority items, 7 deferred, 18 frozen - Update benchmark table with v15.3 column (+3 new capabilities tracked) - Update governance assessment: 73%→75% - Refresh "持续强化" table with v15.3 completion status Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…CP guide
Item 1 — DRL reward weight UI:
- drl_model() accepts optional weight override params (scenario_id,
slope_weight, contiguity_weight, balance_weight, pair_bonus)
- POST /api/drl/run-custom endpoint with range validation
- OptimizationTab.tsx: scenario selector + 4 weight sliders + run button
- DataPanel: 23 tabs (new: optimization in orchestration group)
Item 2 — Field mapping visual editor:
- 3 new API endpoints: preview-columns, infer-mapping, update schema-mapping
- FieldMappingEditor.tsx: modal with column table + target dropdowns +
auto-infer button + save workflow
- VirtualSourcesTab: "映射" button per source row
Item 3 — MCP external agent integration:
- mcp_server.py: --test self-check flag + --transport stdio/sse CLI
- docs/mcp-integration-guide.md: Claude Desktop + Cursor setup,
tool catalog, env vars, troubleshooting (250 lines)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…l SDK Phase 1 — Quick wins: - Auto-memory extraction notification after pipeline completion - Adaptive mobile layout with bottom tab bar (≤1024px breakpoint) Phase 2 — Platform maturity: - Intent disambiguation v2: multi-step query decomposition via task_decomposer - Message bus PostgreSQL persistence with delivery tracking and replay - Skill dependency graph: cycle detection, topological sort, 3 API endpoints Phase 3 — DRL enhancements + quality: - DRL explainability: permutation feature importance + chart generation - DRL run history + A/B comparison panel with metrics diff table - Pydantic v2 output schema validation for Generator/Reviewer skills Phase 4 — Productization: - Helm chart packaging (9 templates from existing K8s manifests) - E2E happy-path tests + world model and causal inference demo scripts - Benchmark data suite (100/1K/10K parcel generation + spatial ops benchmarks) Phase 5 — Skill SDK: - gis-skill-sdk standalone Python package with CLI (validate/list/new/info) 12 items, ~3500 lines new code, 77 new tests all passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rld Model Dreamer integration Three deliverables: - docs/causal_inference_paper.tex: three-angle causal inference framework paper (~520 lines LaTeX, target IJGIS) - docs/world_model_paper_response_r2.md: formal reviewer response letter (both R2 issues are reviewer misreadings) - data_agent/dreamer_env.py: Dreamer-style DRL environment with world model look-ahead (ParcelEmbeddingMapper, ActionToScenarioEncoder, DreamerEnv wrapper, 24 tests) - data_agent/toolsets/dreamer_tools.py: DreamerToolset (dreamer_optimize + dreamer_status) - Fix stale count-based test assertions across 6 test files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…dated metrics - Update both README.md and README_en.md to v15.5 - Add thesis topics image showing alignment with graduation research directions - Update metrics: 38 toolsets, 191+ APIs, 2650+ tests, 113 test files - Add DRL + World Model Dreamer integration to spatial optimization section - Add dreamer_env.py and dreamer_tools.py to project structure - Update GitHub repo About description Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nt critic) + causal paper figures Dreamer integration deepened with 3 new components: - EmbeddingAugmentedEnv: per-parcel coherence + change magnitude in obs space - DreamPlanner: imagination-based trajectory rollout + action candidate scoring - LatentValueEstimator: lightweight MLP V(z) trained on dream trajectories - 34 tests (11 new), all passing Causal inference paper enhanced: - Table 3: quantitative summary of 6 synthetic scenarios (all sub-6% error) - Figure 2: TikZ causal DAG example (5-node urban green space scenario) - Figure 3: TikZ intervention comparison schematic (baseline vs intervention vs effect) - All 6 scenario descriptions strengthened with specific numerical results Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ws, 4 subsystems) Full implementation of the Surveying QC Agent across 3 tracks: Track 1 — DA Platform Enhancement: - Defect taxonomy (30 codes, 5 categories per GB/T 24356) with quality scoring - Workflow SLA/timeout enforcement with 3 QC templates (standard/quick/full) - GovernanceToolset expanded to 18 tools (logic consistency, temporal validity, naming, defect classification) - QC report engine with cover page, TOC, dynamic tables, chart embedding - MCP tool selection rule engine (task_type → tool matching with fallback) - Alert rule engine (configurable thresholds + webhook push) - Auto-fix engine (3 new DataCleaningToolset tools: defect-based fix, auto-classify, batch CRS) - Case library (structured QC experience records in knowledge base) - Overlay precision check (multi-source spatial alignment verification) - Human review workflow (review → mark → fix → approve cycle) - 5 DB migrations (039-043), API endpoints 191 → 202 Track 2 — 4 Independent Subsystems (subsystems/): - cv-service: FastAPI + YOLO visual detection + MCP wrapper - cad-parser: ezdxf + trimesh parsing + GeoJSON export + MCP wrapper - tool-mcp-servers: arcgis-mcp (subprocess→local arcpy), qgis-mcp, blender-mcp - reference-data: PostGIS control point service + ReferenceDataConnector Track 3 — Integration: - 14 integration tests (DA ↔ all subsystems + end-to-end QC workflow) - MCP server config for all subsystems - ReferenceDataConnector (BaseConnector subclass) 235 tests pass, 2650+ total test suite stable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- QcMonitorTab: workflow templates, defect taxonomy browser, human review panel - AlertsTab: alert rule CRUD, alert history timeline - DataPanel: registered both tabs under ops group (质检 + 告警) - Docker Compose: added cv-service, cad-parser, reference-data with qc profile - DB migrations 039-043 executed on production PostgreSQL - ArcPy 3.6.2 smoke test passed (subprocess chain verified) - Frontend: TypeScript zero errors, Vite build successful Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dual Python environment architecture: - Basic engine (ARCPY_PYTHON_EXE): topology, spatial join, buffer, map export - DL engine (ARCPY_DL_PYTHON_EXE): object detection (SingleShotDetector), pixel classification (UnetClassifier), change detection (ChangeDetector), image quality assessment (arcpy.ia), super-resolution (SuperResolution) DL scripts use sys.stdout.write + os._exit(0) to avoid ArcGIS Pro Python finalizer crash (known issue with arcgis.learn subprocess teardown). Verified: PyTorch 2.5.1, arcgis.learn 2.4.2, arcpy.ia, arcpy.sa all available via subprocess chain. 4 DL model classes import successfully. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…le taxonomy QcMonitorTab: summary stat bar (4 cards), collapsible defect categories with severity badges, expandable review rows with comment/fix detail. AlertsTab: metric dropdown (4 presets), conditional webhook URL input, per-rule enable toggle, dark theme matching GovernanceTab. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…resentation End-to-end QC test on real data (重庆市中心城区建筑物轮廓, 107,452 features): - Format validation: detected WGS84 (should be CGCS2000) - Topology: 417 null geoms, 416 duplicates, 0 invalid - Attributes: Floor 1-66, median 4, no nulls - Overlay: CRS mismatch auto-detected, 1,591 buildings in historic districts - Score: 79.5/100 (良), 3 defect types, full run in 7.98s Demo script (docs/surveying_qc_demo_script.md): - 15-minute presentation covering all 7 QC capabilities - Real test results embedded as evidence - Architecture diagram, ArcGIS DL integration, alerting - Coverage matrix against the original requirements doc Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
QC workflow templates (7 total, +4 new): - qc_dlg: DLG digital line map QC (7 steps, 90min SLA) — feature classification, topology, attribute encoding, edge matching - qc_dom: DOM orthophoto QC (6 steps, 60min SLA) — image quality (arcpy.ia), geometric accuracy, edge quality, color consistency - qc_dem: DEM elevation model QC (6 steps, 60min SLA) — elevation accuracy, slope/aspect validation, contour consistency, NoData check - qc_3dmodel: 3D model QC (6 steps, 120min SLA) — geometry quality (trimesh), texture quality (cv-service), positional accuracy, LOD consistency Dashboard API (GET /api/qc/dashboard): - Aggregated stats: templates count, review stats (total/pending/approved/rejected/fixed), workflow stats (total/running/completed/failed/sla_violated), alert stats (total_rules/enabled_rules/recent_alerts) - Recent reviews list (last 10) - Used by QcMonitorTab dashboard section for real-time monitoring Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…kflow progress Dashboard section (default view) with 3 panels: - Stat cards: templates count, pending reviews, running workflows, recent alerts - Recent Reviews table: file path, defect code, severity badge (A/B/C color-coded), status badge, formatted timestamp - Workflow Stats bars: visual progress bars for completed/running/failed/sla_violated with percentages Fetches from GET /api/qc/dashboard. Dark theme styling (#111827 bg, #1f2937 borders). Null-safe with dashboard?.field || 0 pattern. Section switcher: 概览 (dashboard) | 模板 | 缺陷分类 | 复核 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed from git tracking (files kept locally): - docs/surveying_qc_demo_script.md - docs/surveying_qc_demo_script.docx - docs/surveying_qc_agent_design.md - docs/surveying_qc_agent_design.docx These are client-facing documents not intended for public repository. Also sanitized local data paths in scripts/qc_e2e_test.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zhouning
force-pushed
the
feat/v12-extensible-platform
branch
from
4246db6 to
45dd67a
Compare
…bsystems Updated core metrics: - Tests: 2700+ (was 2650+) - Toolsets: 40+ (was 38), GovernanceToolset 18 tools, DataCleaningToolset 11, PrecisionToolset 5 - Skills: 22 (was 21, +surveying-qc) - REST APIs: 203+ (was 191+) - DataPanel: 24 tabs (was 22, +QcMonitor +Alerts) - Connectors: 9 (was 8, +ReferenceData) New v15.7 sections: - QC workflow templates (7: 3 generic + DLG/DOM/DEM/3D model) - Defect taxonomy (30 codes, 5 categories, GB/T 24356) - ArcGIS Pro dual-engine MCP (4 basic + 5 DL tools) - 4 independent subsystems (cv-service/cad-parser/arcgis-mcp/reference-data) - Alert rule engine + real-time monitoring dashboard - E2E validation: 107K features, 8s full QC pipeline Updated GitHub About description. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removed from tracking:
- docs/causal_inference_paper.tex
- docs/world_model_paper_response_r2.md
- docs/generate_fusion_paper_docx.py
- docs/technical_paper_fusion_engine.md
Added .gitignore rules for:
- docs/*paper*.{tex,docx,pdf,md}
- docs/*paper*_{cn,en}.*
- docs/surveying_qc_demo_script.*
- docs/surveying_qc_agent_design.*
Generated locally (not committed):
- docs/causal_inference_paper_en.docx (English, 50KB)
- docs/causal_inference_paper_cn.docx (Chinese, 50KB)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
将 Data Agent 从功能封闭的分析平台升级为高度可扩展的 Agent 平台，用户可自助扩展 Skills、Tools 和多 Agent 工作流。
新增能力
安全修复
架构改进
Bug 修复
文档
Test plan
🤖 Generated with Claude Code