From 52df61a3b51f73c56a6cadb79e1ad0cc84a0e094 Mon Sep 17 00:00:00 2001 From: Ryan Haney <25014745+thathaneydude@users.noreply.github.com> Date: Fri, 14 Aug 2026 09:24:45 -0500 Subject: [PATCH] docs: add AGENTS.md as canonical agent guidance Adopt AGENTS.md as the canonical agent instruction file, with CLAUDE.md reduced to a one-line `@AGENTS.md` import directive. Records the generating model in frontmatter so future regenerations do not have to infer provenance from git history. Cuts 362 lines to 125 and removes stale claims. The previous file listed sensor-single, fleet, and config/fleet as pending migrations when all three exist; asked for a LICENSE, gitignore, workflows, and examples/ that all already exist; and named a feature branch as the current branch. A stale next-steps list is worse than none, since an agent may try to create what is already there. Carried forward and verified: the v- tagging scheme driven by auto-tag.yml, the relative-paths-for-internal-references rule, the submodules layout, and the resource-type-per-file convention. Adds the terraform-docs gotcha: editing a module's variables.tf or outputs.tf without running `just docs` fails CI via docs-check. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 130 +++++++++++++++++++ CLAUDE.md | 363 +----------------------------------------------------- 2 files changed, 131 insertions(+), 362 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2a9d76c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,130 @@ +--- +generated_by: claude-opus-5[1m] +generated_at: 2026-08-13 +--- + +# Corelight Terraform Monorepo + +`github.com/corelight/terraform`. A module *library*, not a deployment root: it publishes +reusable Terraform modules for deploying Corelight sensors, Fleet, and cloud enrichment across +AWS, Azure, and GCP. The primary consumer is `enrichment-deploy` (a separate GitLab repo), which +pins modules via `source = "github.com/corelight/terraform//modules//?ref=..."`. + +This is a **public** repository. Never add internal-only detail: no GitLab project IDs, no +internal registry hostnames, no cloud account/subscription/project identifiers, no tokens, no +internal URLs. + +## Module inventory + +- `_shared/config/sensor` - cloud-agnostic sensor cloud-init generator (no `main.tf`; config is + built in `data.tf` via the `cloudinit_config` data source). Used by every cloud's sensor module. +- `_shared/config/fleet` - cloud-agnostic Fleet Manager cloud-init generator. +- `aws/sensor` - auto-scaling sensor via ASG + Gateway Load Balancer; ships a + `submodules/iam-lambda` submodule for the ASG lifecycle-hook Lambda's IAM role. +- `aws/sensor-single` - single EC2 instance sensor (no ASG/GWLB); submodules for `instance`, + `network_interface`, and `vpc_flow_assume_role`. +- `aws/fleet` - Fleet Manager on ALB + EC2 + Route53. +- `aws/enrichment` - Lambda + EventBridge enrichment; submodules under `submodules/iam/*` and + `submodules/secondary-event-rule`. +- `azure/sensor` - auto-scaling sensor via VM Scale Set with internal LB and NAT gateway. +- `azure/sensor-single` - single sensor VM; submodule `vnet_flow_storage_access`. +- `azure/enrichment` - Container App + Service Bus + Storage; submodules `iam`, `service_bus`, + `storage`. +- `gcp/sensor` - auto-scaling sensor via Managed Instance Group with internal LB and Packet + Mirroring. +- `gcp/enrichment` - Cloud Run + Pub/Sub + Cloud Scheduler; submodule `org-iam`. + +`sensor` modules are the fleet-managed, auto-scaling deployment topology per cloud; `sensor-single` +is a simpler, single-instance deployment without scaling infrastructure. Both consume +`_shared/config/sensor` for cloud-init. + +## Consumer hazard: `?ref=main` + +`auto-tag.yml` (below) publishes a new tag and GitHub Release on every merge to `main`. If a +consumer pins `?ref=main` instead of a release tag, it tracks unreleased HEAD rather than a +published release. State this as fact when it comes up; it is documented on the consumer side too. + +## Versioning + +Format: `v-` (e.g. `v29.0.5-5`). + +- `VERSION` (repo root, one line, e.g. `29.0.5`) is the Corelight sensor version. Bumped by hand + when a new sensor ships. +- `` is derived from existing `v-*` git tags: `(highest existing meta) + 1`, or `1` + if none exist for that sensor. It resets to `1` automatically when `VERSION` changes. +- `.github/workflows/auto-tag.yml` runs on every push to `main`: computes the next tag via + `scripts/release/compute-next-tag.sh`, pushes the tag, and runs `gh release create + --generate-notes`. A `concurrency` group serializes runs. +- Details and force-push recovery: `scripts/release/README.md`. +- Test the tag logic: `just test-release` (wraps `scripts/release/test_compute_next_tag.sh`). + +## Module source conventions (load-bearing) + +- **Internal references MUST use relative paths**, never GitHub URLs, e.g. + `source = "../../_shared/config/sensor"`. +- **External consumers** use `source = "github.com/corelight/terraform//modules//?ref="`. +- Nested modules live under `modules///submodules//` (e.g. + `modules/aws/sensor/submodules/iam-lambda`). +- Every module file is organized resource-type-per-file (`autoscaling_group.tf`, + `load_balancer.tf`, `security_groups.tf`, ...), not one giant `main.tf`. The + `_shared/config/sensor` module deliberately has no `main.tf`; its logic lives in `data.tf`. +- Required per-module files: `README.md`, `variables.tf`, `outputs.tf`, `versions.tf`. +- Provider version constraints live in each module's own `versions.tf`, not centrally. +- Naming: module directories `lowercase-with-hyphens`; files and variables `snake_case`. +- Tests live per-module/per-example at `modules/*/*/tests/*.tftest.hcl` and + `examples/*/*/tests/*.tftest.hcl`, not in a root `tests/` (none exists). Per-module `docs/` + directories exist for `azure/enrichment`, `azure/sensor`, `gcp/enrichment`; there is no root + `docs/`. + +## Docs generation gotcha (high value) + +Module README Inputs/Outputs tables are generated, not hand-written, via terraform-docs with +`--output-check` mode injecting between `` / `` markers +(`.terraform-docs.yml`). **If you edit any module's `variables.tf` or `outputs.tf`, run `just +docs` before committing**, or CI's `just lint` (which calls `just docs-check`) will fail. + +``` +just docs # regenerate Inputs/Outputs tables +just docs-check # fail if README tables are stale (what CI runs) +``` + +## justfile recipes (verbatim from `justfile`) + +`fmt`, `fmt-check`, `validate`, `lint` (`tflint --init && tflint --recursive && just docs-check`), +`docs`, `docs-check`, `test` (`test-unit test-aws test-release`), `test-unit`, `test-aws` (AWS +Lambda Python tests via pytest), `test-release`, `trivy-scan`, `clean`. Run `just --list` for the +current set. + +## CI workflows (`.github/workflows/`) + +- `terraform-lint.yml`: on push/PR to `main`, runs `just fmt-check`, then `just lint` (tflint + + docs-check) and `just test-release`. Pins Terraform `1.14.0`, terraform-docs `v0.24.0`, just + `1.51.0`. +- `terraform-test.yml`: on push/PR/dispatch, runs `just test-unit` (Terraform `~> 1.10`), `just + test-aws` (Python 3.12), and a separate Terraform Validate job. +- `scan-trivy.yml`: on PR to `main`, nightly cron, and dispatch; runs + `corelight/shared-actions/trivy-terraform-scan@main` against `.github/trivy/trivy.yml`, + opening issues on findings. +- `auto-tag.yml`: see Versioning above. + +## Environment notes for this session + +Local Terraform is `1.12.2` and local terraform-docs is `v0.17.0`; CI pins Terraform `1.14.0` and +terraform-docs `v0.24.0`. `v0.17.0` does not support the `--recursive-include-main` flag used in +`just docs`/`just docs-check`, so those recipes cannot be validated locally without upgrading +terraform-docs; treat failures there as version skew, not a repo defect. `tflint` is commonly not +installed locally, so `just lint` fails at its first step for that reason alone. + +Two modules currently fail `terraform validate` against recent provider releases (unrelated to any +docs change): `modules/azure/enrichment` (`service_bus_queue_endpoint_id` argument no longer valid +on `azurerm_eventgrid_system_topic_event_subscription` under azurerm provider `5.0.1`, though the +module's `versions.tf` only requires `>= 4.0`) and `modules/azure/sensor` +(`enable_accelerated_networking` unsupported on `azurerm_linux_virtual_machine_scale_set` under the +same provider version). This also fails the corresponding `azure/enrichment` unit test. If you hit +either error, it is a known provider-compatibility gap, not something introduced by an unrelated +change; consider pinning `azurerm` tighter in those modules' `versions.tf` if you're the one fixing +it. + +## Never do + +Never run `terraform apply` or `terraform destroy` in this repo. diff --git a/CLAUDE.md b/CLAUDE.md index daad1ef..43c994c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,362 +1 @@ -# Corelight Terraform Monorepo - Structure Guide - -This document provides a comprehensive guide for AI assistants and developers working with the Corelight Terraform monorepo. - -## Repository Overview - -**Repository**: `github.com/corelight/terraform` -**Current Sensor**: `29.0.5` (see `VERSION`; meta is tag-derived) -**Purpose**: Consolidated Terraform modules for deploying Corelight sensors, enrichment, and Fleet across AWS, GCP, and Azure. - -## Core Principles - -### 1. Monorepo Structure -All Terraform modules are organized in a single repository with the following top-level directories: -- `modules/` - All Terraform modules (cloud-specific and shared) -- `examples/` - Usage examples organized by cloud provider -- `tests/` - Unit, integration, and E2E tests -- `docs/` - Comprehensive documentation -- `scripts/` - Utility scripts for validation, testing, and release automation - -### 2. Versioning Strategy -**Format**: `v-` - -- Example: `v29.0.5-3` - - `29.0.5` = Corelight sensor version - - `-3` = Third Terraform module release for this sensor version - -**Version Rules**: -- The `VERSION` file at the repo root holds the sensor version (e.g. `29.0.5`) - and is the single source of truth for the sensor. It is bumped by hand. -- The `-` counter is derived from existing `v-*` git tags and - resets to `1` automatically when the sensor in `VERSION` changes. -- Every merge to `main` triggers `.github/workflows/auto-tag.yml`, which - computes the next tag, creates it, and publishes a GitHub Release. -- See `scripts/release/README.md` for details and recovery procedures. - -### 3. Module Organization - -``` -modules/ -├── _shared/ # Cloud-agnostic shared modules -│ └── config/ -│ ├── sensor/ # Sensor configuration (cloud-init) -│ └── fleet/ # Fleet configuration -├── aws/ # AWS-specific modules -│ ├── sensor/ # Auto-scaling sensor -│ ├── sensor-single/ # Single instance sensor -│ ├── enrichment/ # Enrichment service -│ └── fleet/ # Fleet manager -├── gcp/ # GCP-specific modules -│ ├── sensor/ # MIG sensor -│ └── enrichment/ # Enrichment service -└── azure/ # Azure-specific modules - ├── sensor/ # Scale set sensor - └── enrichment/ # Enrichment service -``` - -### 4. Internal Module References - -**Rule**: Internal modules MUST use relative paths, NOT GitHub URLs. - -**Examples**: -```terraform -# AWS Sensor referencing shared config -module "sensor_config" { - source = "../../_shared/config/sensor" - # ... -} - -# GCP Sensor referencing shared config -module "sensor_config" { - source = "../../_shared/config/sensor" - # ... -} - -# Azure Sensor referencing shared config -module "sensor_config" { - source = "../../_shared/config/sensor" - # ... -} -``` - -### 5. External User References - -**Rule**: Users reference modules with double-slash syntax and version tags. - -**Examples**: -```terraform -# AWS Sensor -module "sensor" { - source = "github.com/corelight/terraform//modules/aws/sensor?ref=v29.0.5-1" - # ... -} - -# Submodule (if needed directly) -module "iam_lambda" { - source = "github.com/corelight/terraform//modules/aws/sensor/submodules/iam-lambda?ref=v29.0.5-1" - # ... -} -``` - -### 6. Submodules Pattern - -**Rule**: Nested modules are placed in `submodules/` directories. - -**Pattern**: `modules/{cloud}/{module}/submodules/{submodule-name}/` - -**Example**: `modules/aws/sensor/submodules/iam-lambda/` - -## Module Structure Standards - -### Required Files for Each Module - -1. **Configuration Files**: - - `*.tf` files (specific to module purpose) - - `variables.tf` - Input variables - - `outputs.tf` - Output values - - `versions.tf` - Terraform and provider version constraints - - `README.md` - Module documentation - -2. **Optional Files** (based on module complexity): - - `data.tf` - Data sources - - `locals.tf` - Local values - - Resource-specific files (e.g., `autoscaling_group.tf`, `load_balancer.tf`) - -### AWS Sensor Module Structure - -The AWS sensor module is organized by resource type: - -``` -modules/aws/sensor/ -├── autoscaling_group.tf # ASG configuration -├── data.tf # Data sources -├── lambda.tf # Lifecycle hook Lambda -├── launch_template.tf # Launch template for ASG -├── load_balancer.tf # Gateway load balancer -├── outputs.tf # Module outputs -├── security_groups.tf # Security groups -├── sensor_config.tf # References ../../_shared/config/sensor -├── variables.tf # Input variables -├── versions.tf # Provider constraints -├── README.md # Module documentation -└── submodules/ - └── iam-lambda/ - ├── main.tf # IAM roles and policies - ├── outputs.tf - ├── variables.tf - ├── versions.tf - └── README.md -``` - -### Shared Config Sensor Module - -The shared sensor configuration module generates cloud-init configuration: - -``` -modules/_shared/config/sensor/ -├── data.tf # Cloud-init config generation -├── outputs.tf # Config output -├── variables.tf # Configuration parameters -├── versions.tf # Provider constraints (cloudinit) -├── README.md # Module documentation -└── templates/ - └── cloud-init.yaml.tpl # Cloud-init template -``` - -**Key Point**: This module does NOT have a `main.tf` file. Configuration is generated via `data.tf` using the `cloudinit_config` data source. - -## Provider Version Constraints - -### Root Modules -All root modules must define: -- `required_version` - Terraform version (>= 1.3.2) -- `required_providers` - Provider versions - -### AWS Sensor Module -```terraform -terraform { - required_version = ">= 1.3.2" - - required_providers { - aws = { - source = "hashicorp/aws" - version = ">= 5.0" - } - archive = { - source = "hashicorp/archive" - version = ">= 2.4.0" - } - } -} -``` - -### Shared Config Sensor Module -```terraform -terraform { - required_version = ">= 1.3.2" - - required_providers { - cloudinit = { - source = "hashicorp/cloudinit" - version = ">= 2.3.0" - } - } -} -``` - -## Development Workflow - -### Task Automation (justfile) - -The repository uses `justfile` for common development tasks: - -```bash -just fmt # Format all Terraform files -just lint # Run tflint recursively on all modules -just validate # Validate all Terraform modules -just test # Run tests -``` - -### Linting - -- Uses `tflint` with `--recursive` flag -- Auto-discovers modules by finding `versions.tf` files -- All modules must pass lint checks before committing - -### Git Workflow - -**Branch Strategy**: -- `main` - Protected branch for releases -- Feature branches: `feature/{description}` - -**Current Branch**: `feature/monorepo-structure-and-aws-sensor` - -**Commit Guidelines**: -- Descriptive commit messages -- Include footer: - ``` - 🤖 Generated with [Claude Code](https://claude.com/claude-code) - - Co-Authored-By: Claude - ``` - -## Migration Status - -### Completed Migrations - -1. **terraform-config-sensor** → `modules/_shared/config/sensor/` - - ✅ All files migrated - - ✅ Template path updated to `templates/cloud-init.yaml.tpl` - - ✅ versions.tf added with cloudinit provider - -2. **terraform-aws-sensor** → `modules/aws/sensor/` - - ✅ All files migrated - - ✅ Internal references updated to use `../../_shared/config/sensor` - - ✅ Submodules reorganized under `submodules/` - - ✅ All lint issues resolved - -3. **terraform-aws-enrichment** → `modules/aws/enrichment/` - - ✅ All files migrated - - ✅ Submodules reorganized under `submodules/iam/` - - ✅ Example created in `examples/aws/enrichment/` - - ✅ Tests added - -4. **terraform-gcp-sensor** → `modules/gcp/sensor/` - - ✅ All files migrated - - ✅ Example created in `examples/gcp/sensor-only/` - - ✅ Tests added - -5. **terraform-gcp-enrichment** → `modules/gcp/enrichment/` - - ✅ All files migrated - - ✅ Submodules reorganized under `submodules/org-iam/` - - ✅ Example created in `examples/gcp/enrichment/` - - ✅ Comprehensive test suite (12 scenarios, 344 lines) - - ✅ Documentation and diagrams included - - ✅ Verified byte-for-byte identical to original module - -6. **terraform-azure-sensor** → `modules/azure/sensor/` - - ✅ All files migrated - - ✅ Tests added - -7. **terraform-azure-enrichment** → `modules/azure/enrichment/` - - ✅ All files migrated - - ✅ Example created - - ✅ Tests added - -### Pending Migrations - -These modules still need to be migrated from their standalone repositories: - -1. **terraform-aws-single-sensor** → `modules/aws/sensor-single/` -2. **terraform-aws-fleet** → `modules/aws/fleet/` -3. **terraform-config-fleet** → `modules/_shared/config/fleet/` - -## Important Notes for AI Assistants - -### When Working with This Repository - -1. **Always verify module paths** - Use relative paths for internal references -2. **Check versions.tf** - Ensure all provider constraints are defined -3. **Run lint checks** - Use `just lint` after making changes -4. **Follow file organization** - AWS sensor uses separate files per resource type (not a single `main.tf`) -5. **No examples in modules** - Module-specific examples should be removed; use top-level `examples/` directory -6. **Version alignment** - Remember version format is `-` - -### Common Patterns - -**Shared Configuration Module Usage**: -```terraform -# In any sensor module (aws/gcp/azure) -module "sensor_config" { - source = "../../_shared/config/sensor" - - sensor_license = var.license_key - fleet_community_string = var.community_string - fleet_token = var.fleet_token - fleet_url = var.fleet_url - fleet_server_sslname = var.fleet_server_sslname - sensor_management_interface_name = "eth1" - sensor_monitoring_interface_name = "eth0" - base64_encode_config = true - sensor_health_check_http_port = "41080" -} -``` - -**Submodule Usage**: -```terraform -# In parent module or example -module "asg_lambda_role" { - source = "./submodules/iam-lambda" - - # Variables specific to IAM Lambda role -} -``` - -### Troubleshooting - -**Lint Errors**: -- Missing `versions.tf` - Add file with provider constraints -- Unused variables - Remove or mark as used -- Wrong module paths - Verify relative path is correct - -**Module Reference Issues**: -- Internal: Use relative paths (`../../_shared/config/sensor`) -- External (users): Use GitHub URL with `//` separator - -## Next Steps - -To complete the monorepo migration: - -1. Create missing root files (LICENSE, .gitignore, .pre-commit-config.yaml) -2. Implement additional GitHub Actions workflows -3. Migrate remaining modules (single-sensor, enrichment, fleet, GCP, Azure) -4. Create working examples in `examples/` directory -5. Write comprehensive documentation in `docs/` -6. Implement testing framework in `tests/` - -## Reference Documentation - -- Full structure specification: `/Users/jacobfiola/Downloads/MONOREPO_STRUCTURE.md` -- Repository: https://github.com/corelight/terraform -- Current working directory: `/Users/jacobfiola/work/terraform` +@AGENTS.md