From 855cec83741d19fada3c8f700029397525011098 Mon Sep 17 00:00:00 2001 From: scanossmining Date: Wed, 12 Aug 2026 16:15:24 +0300 Subject: [PATCH] docs: update install, deploy and usage documentation to match current code README.md: - Fix Makefile target names (build_amd64, build_import_amd64, lint, lint-fix, clean-testcache) and binary output paths (target/scanoss-folder-hashing-api-*) - Fix REST endpoint path: /v2/scanning/echo (no /api prefix) - Add Qdrant setup section (docker compose, ports, persistence) with Debian note (apparmor package, compose plugin availability) - Add scan request example documenting rank_threshold semantics (default 0 returns no results) and lang_extensions collection routing - Add production deployment section (make package_amd64 + env-setup.sh + systemd) - Correct import tool docs: make targets, dynamic worker count, 13-column CSV requirement, -top-purls JSON format example - Update Go requirement to 1.23 config.example.json: - Use localhost for QdrantHost (the API runs on the host; the docker network alias is not resolvable from there) scripts/README.md: - Fix build command (module lives in ./cmd/server), compose file name, packaging steps (make package_* targets), Go version --- README.md | 260 ++++++++++++++++++++++++++++++++++---------- config.example.json | 2 +- scripts/README.md | 38 ++++--- 3 files changed, 221 insertions(+), 79 deletions(-) diff --git a/README.md b/README.md index ba12c89..5825875 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,41 @@ # SCANOSS Folder Hashing API [![License](https://img.shields.io/badge/License-GPL%20v2%2B-blue.svg)](LICENSE) -[![Go Version](https://img.shields.io/badge/Go-1.22+-00ADD8.svg)](go.mod) +[![Go Version](https://img.shields.io/badge/Go-1.23+-00ADD8.svg)](go.mod) A high-performance REST and gRPC API service for component fingerprinting and similarity matching using Qdrant vector database. The SCANOSS Folder Hashing API enables efficient code component analysis and similarity detection for software composition analysis. ## Prerequisites -- **Go 1.22+**: For building and running the service -- **Qdrant**: Vector database (must be running and accessible) +- **Go 1.23+**: For building and running the service +- **Docker + Docker Compose**: For running the Qdrant vector database +- **jq**: Required by the Qdrant snapshot scripts (optional otherwise) + +## Setting up Qdrant + +The service requires a running Qdrant instance. The repository ships a ready-to-use Docker Compose file ([`docker-compose-qdrant.yml`](docker-compose-qdrant.yml)) tuned for large-scale imports. + +```bash +# 1. Start Qdrant (from the repository root) +docker compose -f docker-compose-qdrant.yml up -d + +# 2. Verify it's up +curl http://localhost:6333/collections +docker logs scanoss-qdrant +``` + +This starts a `scanoss-qdrant` container with: + +| Port | Protocol | Used by | +|------|----------|---------| +| 6333 | HTTP | Snapshot scripts, manual inspection (`curl http://localhost:6333/...`) | +| 6334 | gRPC | The API service and the import tool | + +Data is persisted on the host in `./qdrant_data` (bind mount), so the database survives container restarts. To stop Qdrant: `docker compose -f docker-compose-qdrant.yml down` (data is kept). + +> **Debian note:** Debian's own repositories ship neither the `docker compose` v2 plugin nor (on minimal installs) the AppArmor userspace tools that Docker needs. If containers fail to start with an `apparmor_parser: executable file not found` error, run `sudo apt-get install -y apparmor`. If `docker compose` is unavailable, either install Docker CE from Docker's official apt repository, or start Qdrant with a plain `docker run` using the same image, ports (`-p 6333:6333 -p 6334:6334`), volume (`-v $(pwd)/qdrant_data:/qdrant/storage`) and environment variables as the compose file. + +Once Qdrant is running, populate it with data — see [Importing Data](#importing-data). ## Quick Start @@ -17,18 +44,24 @@ A high-performance REST and gRPC API service for component fingerprinting and si git clone https://github.com/scanoss/folder-hashing-api.git cd folder-hashing-api -# 2. Set up configuration -cp config.example.json config/app-config.json -# Edit config/app-config.json as needed +# 2. Start Qdrant (see "Setting up Qdrant" above) +docker compose -f docker-compose-qdrant.yml up -d + +# 3. Set up configuration +cp config.example.json app-config.json +# Edit app-config.json as needed (Qdrant host/port, ports, etc.) + +# 4. Build the service (binary goes into ./target) +make build_amd64 # or: make build_arm64 -# 3. Build the service -make build_amd # or build_arm for ARM64 +# 5. Run the service +./target/scanoss-folder-hashing-api-linux-amd64 --json-config app-config.json -# 4. Run the service -./dist/scanoss-hfh-api --json-config config/app-config.json +# Alternatively, run directly from source during development: +make run -# 5. Verify it's running -curl -X POST -H "Content-Type: application/json" -d '{"message":"test"}' http://localhost:40061/api/v2/scanning/echo +# 6. Verify it's running +curl -X POST -H "Content-Type: application/json" -d '{"message":"test"}' http://localhost:40061/v2/scanning/echo ``` ## Service Endpoints @@ -37,14 +70,45 @@ Once running, the service provides: | Service | Default Endpoint | Description | |---------|----------|-------------| -| **REST API** | http://localhost:40061 | Main API interface | +| **REST API** | http://localhost:40061 | Main API interface (grpc-gateway) | | **gRPC API** | localhost:50061 | High-performance gRPC interface | | **Dynamic Logging** | localhost:60061 | Runtime log level control | +REST routes (from the `scanoss/papi` scanning v2 definition): + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/v2/scanning/echo` | Echo test endpoint | +| POST | `/v2/scanning/hfh/scan` | Folder hash scan (similarity search) | + +### Scan request example + +```bash +curl -s -X POST -H "Content-Type: application/json" -d '{ + "rank_threshold": 10, + "root": { + "path_id": "/", + "sim_hash_dir_names": "cfedb95e5bfdefab", + "sim_hash_names": "e8ab6d7fcbce5fe9", + "sim_hash_content": "827dc8fd1c6b0c57", + "lang_extensions": {"py": 10} + } +}' http://localhost:40061/v2/scanning/hfh/scan +``` + +Two parameters trip people up: + +- **`rank_threshold` is required in practice.** Only components with `rank <= rank_threshold` are returned, and it defaults to `0` — which filters out every component with rank 1 or higher, i.e. **everything**, so an omitted threshold produces empty results even on a fully populated database. Set it to cover the ranks you want (lower rank = higher priority component). +- **`lang_extensions` selects which collection is searched.** The dominant extension routes the query (e.g. `{"py": 10}` searches `python_collection`, `{"class": 156}` searches `java_collection`), so it must be consistent with the folder being scanned — a mismatch searches the wrong collection and returns weak or empty matches. Folders with no mapped extensions go to `misc_collection`. + ## Configuration +Configuration priority: **environment variables > JSON config file > `.env` file > built-in defaults**. If a `.env` file exists in the working directory it is picked up automatically. + ### JSON Configuration (Recommended) +See [`config.example.json`](config.example.json) for a full example: + ```json { "App": { @@ -69,11 +133,15 @@ Once running, the service provides: } ``` +> **Note:** `Hfh.QdrantPort` must be the **gRPC** port of Qdrant (default `6334`), not the HTTP port (`6333`). + ### Environment Variables +See [`.env.example`](.env.example) for the full list: + ```bash -export APP_PORT=50061 -export REST_PORT=40061 +export APP_PORT=50061 # gRPC port +export REST_PORT=40061 # REST port export QDRANT_HOST=localhost export QDRANT_PORT=6334 export APP_DEBUG=true @@ -83,79 +151,93 @@ export APP_DEBUG=true ```bash # Using JSON config -./dist/scanoss-hfh-api --json-config config/app-config.json +./target/scanoss-folder-hashing-api-linux-amd64 --json-config app-config.json # Using environment file -./dist/scanoss-hfh-api --env-config .env +./target/scanoss-folder-hashing-api-linux-amd64 --env-config .env # With debug flag -./dist/scanoss-hfh-api --debug --json-config config/app-config.json +./target/scanoss-folder-hashing-api-linux-amd64 --debug --json-config app-config.json + +# Display version +./target/scanoss-folder-hashing-api-linux-amd64 --version ``` ## Building +Binaries are written to the `./target` directory: + ```bash -# Build for AMD64 -make build_amd +# Build the API for AMD64 -> target/scanoss-folder-hashing-api-linux-amd64 +make build_amd64 + +# Build the API for ARM64 -> target/scanoss-folder-hashing-api-linux-arm64 +make build_arm64 + +# Build both architectures +make build -# Build for ARM64 -make build_arm +# Build the import tool -> target/scanoss-folder-hashing-import-linux-amd64 +make build_import_amd64 # or: make build_import_arm64 -# Run locally (development) -make run_local +# Run locally from source (development) +make run ``` ## Testing ```bash -# Run all tests +# Run all tests (race detector + coverage profile) make test -# Run with coverage -go test -v -cover ./... +# Open the HTML coverage report +make test-coverage # Run linting -make lint_local +make lint # Auto-fix linting issues -make lint_local_fix +make lint-fix + +# Clear the Go test cache +make clean-testcache ``` ## Importing Data There are two ways to populate the Qdrant vector database: -1. **Import from CSV files** — build the database from raw component data using the `cmd/import/main.go` tool. Use this to create or update the data from scratch. +1. **Import from CSV files** — build the database from raw component data using the import tool (`cmd/import`). Use this to create or update the data from scratch. 2. **Restore from snapshots** — recreate the database from previously generated Qdrant snapshots. This is much faster than a full CSV import and is the recommended way to provision a new environment from an existing dataset. See [Restoring from Snapshots](#restoring-from-snapshots). ### Basic Usage ```bash # Build the import tool -go build -o dist/import-tool cmd/import/main.go +make build_import_amd64 # Update database (adds/updates data in existing collections) # -top-purls is optional; when omitted, the rank from the CSV is used -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/directory # Update database with an optional PURL ranking file to prioritize results -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/directory \ -top-purls /path/to/top-purls.json # Recreate database (deletes existing collections and imports fresh) -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/directory \ -overwrite -# Specify Qdrant host and port -./dist/import-tool \ +# Specify Qdrant host and port (defaults: localhost:6334) +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/directory \ -top-purls /path/to/top-purls.json \ -overwrite \ -qdrant-host my-qdrant-host.example.com \ - -qdrant-port 5555 + -qdrant-port 6334 ``` ### Command Options @@ -165,8 +247,10 @@ go build -o dist/import-tool cmd/import/main.go | `-dir` | Yes | Directory containing CSV files to import | | `-top-purls` | **No (optional)** | JSON file with PURL rankings for search prioritization. **When omitted, the `rank` column from the CSV is used as-is.** | | `-overwrite` | No | Delete and recreate all collections (use for fresh start) | +| `-qdrant-host` | No | Qdrant server host (default `localhost`) | +| `-qdrant-port` | No | Qdrant server gRPC port (default `6334`) | -> **Note:** The `-top-purls` file is **optional**. It only overrides the `rank` of the matching PURLs to prioritize them in search results; if you don't provide it, the import relies entirely on the `rank` column already present in the CSV. +> **Note:** The `-top-purls` file is **optional**. It only overrides the `rank` of the matching PURLs to prioritize them in search results; if you don't provide it, the import relies entirely on the `rank` column already present in the CSV. The file is a JSON object mapping purl to rank, e.g. `{"pkg:github/torvalds/linux": 1, "pkg:npm/react": 1}`. ### CSV Format @@ -195,29 +279,30 @@ Example row: ``` Notes: -- Rows with fewer than 11 fields are skipped with a warning. +- Rows with fewer than 13 fields are skipped with a warning. - Empty `language_extensions` routes the record to `misc_collection`. - Invalid or empty `rank` defaults to `0`, which ranks higher than any positive value in the current sort — make sure the generator emits sanitized values. ### How It Works The import tool: -- Processes CSV files in parallel using 12 concurrent workers -- Groups components by programming language into separate collections (e.g., `py_collection`, `js_collection`) +- Processes CSV files in parallel; the number of workers (2–32) is calculated automatically from available CPU cores and memory +- Groups components by programming language into separate collections (e.g., `python_collection`, `javascript_collection`, `misc_collection`) - Creates optimized vector indexes with named vectors (`dirs`, `names`, `contents`) -- Handles large datasets with batching (2000 records per batch) +- Handles large datasets with batching (2000 records per batch, 1000 when running with more than 16 workers) +- Disables HNSW indexing during the bulk load and re-enables it at the end; Qdrant then builds the indexes in the background ### Example Workflow ```bash # 1. Ensure Qdrant is running -# (Start your Qdrant instance) +docker compose -f docker-compose-qdrant.yml up -d # 2. Build the tool -go build -o dist/import-tool cmd/import/main.go +make build_import_amd64 # 3. Import your data (-top-purls is optional) -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /data/csv/ \ -top-purls /data/top-purls.json @@ -258,6 +343,44 @@ Notes: - The restore uses `priority=snapshot`, so the uploaded snapshot wins over any existing data in the collection. - `jq` is required by both scripts. +## Production Deployment (systemd) + +The service is deployed as a systemd unit on Linux servers using the scripts in [`scripts/`](scripts/). See [`scripts/README.md`](scripts/README.md) for full details. + +```bash +# 1. Build the binary and package it with the deployment scripts +# (produces scanoss-folder-hashing-api_linux-amd64_-1.tgz; +# the binary itself is placed into scripts/) +make package_amd64 # or: make package_arm64 + +# 2. Copy the archive to the target server and extract it +tar xzvf scanoss-folder-hashing-api_linux-amd64_-1.tgz +cd scripts + +# 3. Create the runtime user (once per server) +sudo useradd --system scanoss + +# 4. Run the environment setup script +# (installs binary + startup script to /usr/local/bin, systemd unit to +# /etc/systemd/system, config to /usr/local/etc/scanoss/folder-hashing-api) +sudo ./env-setup.sh # interactive +sudo ./env-setup.sh --force # automated, no prompts + +# 5. Review the configuration (Qdrant host/port, ports, telemetry) +sudo vi /usr/local/etc/scanoss/folder-hashing-api/app-config-prod.json + +# 6. Start and enable the service +sudo systemctl start scanoss-folder-hashing-api +sudo systemctl enable scanoss-folder-hashing-api + +# 7. Check status and logs +sudo systemctl status scanoss-folder-hashing-api +sudo journalctl -u scanoss-folder-hashing-api -f +sudo tail -f /var/log/scanoss/folder-hashing-api/scanoss-folder-hashing-api-prod.log +``` + +The target server also needs a running Qdrant instance (see [Setting up Qdrant](#setting-up-qdrant)) populated via CSV import or snapshot restore. + ## Development ### Local Development Setup @@ -283,13 +406,24 @@ make run ```bash make help # Show all available commands -make build_amd64 # Build for AMD64 -make build_arm64 # Build for ARM64 -make run # Run the service locally +make run # Run the API locally from source make test # Run all unit tests +make test-coverage # Run tests and open HTML coverage report make lint # Run linting make lint-fix # Run linting with auto-fix -make clean_testcache # Clean Go test caches +make fmt # Format code (gofumpt + goimports) +make vet # Run go vet +make build # Build API binaries for all architectures +make build_amd64 # Build API for AMD64 +make build_arm64 # Build API for ARM64 +make build_import_amd64 # Build import tool for AMD64 +make build_import_arm64 # Build import tool for ARM64 +make package_amd64 # Build & package AMD64 binary + deploy scripts +make package_arm64 # Build & package ARM64 binary + deploy scripts +make clean # Clean build artifacts +make clean-testcache # Clean Go test caches +make tidy # Tidy and verify dependencies +make version # Display current version ``` ## Troubleshooting @@ -298,23 +432,29 @@ make clean_testcache # Clean Go test caches ```bash # Check if service is running -ps aux | grep scanoss-hfh-api +ps aux | grep scanoss-folder-hashing-api # Check configuration -cat config/app-config.json +cat app-config.json # Run with debug logging -./dist/scanoss-hfh-api --debug --json-config config/app-config.json +./target/scanoss-folder-hashing-api-linux-amd64 --debug --json-config app-config.json ``` ### Qdrant connection issues ```bash +# Verify the container is running +docker ps | grep qdrant + # Verify Qdrant is accessible curl http://localhost:6333/collections -# Check Qdrant host/port in config -grep -A 3 "Hfh" config/app-config.json +# Review Qdrant logs +docker logs scanoss-qdrant + +# Check Qdrant host/port in config (must be the gRPC port, default 6334) +grep -A 3 "Hfh" app-config.json ``` ### Import tool issues @@ -326,15 +466,15 @@ ls -la /path/to/csv/directory/ # Verify top-purls.json is valid JSON cat /path/to/top-purls.json | jq . -# Run with verbose output -./dist/import-tool -dir /path/to/csv/ -top-purls /path/to/top-purls.json +# Run the import again (progress and per-collection stats are printed) +./target/scanoss-folder-hashing-import-linux-amd64 -dir /path/to/csv/ -top-purls /path/to/top-purls.json ``` ## Documentation -- **API Documentation**: Available at REST endpoints when service is running -- **Configuration Reference**: See `config.example.json` for all available options -- **Scripts**: Check `scripts/` directory for additional utilities +- **API Definition**: [scanoss/papi](https://github.com/scanoss/papi) (scanning v2 service) +- **Configuration Reference**: See `config.example.json` and `.env.example` for all available options +- **Deployment**: See [`scripts/README.md`](scripts/README.md) for deployment and management scripts ## Contributing @@ -342,7 +482,7 @@ cat /path/to/top-purls.json | jq . 2. Create a feature branch (`git checkout -b feature/amazing-feature`) 3. Make your changes 4. Run tests (`make test`) -5. Run linting (`make lint_local`) +5. Run linting (`make lint`) 6. Commit your changes (`git commit -m 'Add amazing feature'`) 7. Push to the branch (`git push origin feature/amazing-feature`) 8. Open a Pull Request diff --git a/config.example.json b/config.example.json index f56b552..76684d0 100644 --- a/config.example.json +++ b/config.example.json @@ -16,7 +16,7 @@ "OltpExporter": "0.0.0.0:4317" }, "Hfh": { - "QdrantHost": "scanoss-qdrant", + "QdrantHost": "localhost", "QdrantPort": 6334 } } diff --git a/scripts/README.md b/scripts/README.md index 8420120..32798e1 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -17,11 +17,11 @@ This directory contains deployment and management scripts for the SCANOSS Folder ### Supporting Infrastructure -The project includes `docker-compose.qdrant.yml` in the root directory for running the Qdrant vector database: +The project includes `docker-compose-qdrant.yml` in the root directory for running the Qdrant vector database: ```bash # Start Qdrant vector database -docker-compose -f docker-compose-qdrant.yml up -d +docker compose -f docker-compose-qdrant.yml up -d ``` ## 🚀 Installation @@ -29,7 +29,7 @@ docker-compose -f docker-compose-qdrant.yml up -d ### Prerequisites - Linux server with systemd -- Go 1.22+ for building the binary +- Go 1.23+ for building the binary - Docker for Qdrant database - `scanoss` system user @@ -40,18 +40,20 @@ docker-compose -f docker-compose-qdrant.yml up -d sudo useradd --system scanoss ``` -2. **Build the API binary**: +2. **Build the API binary** (from the repository root; this places the binary into `scripts/`): ```bash - go build -o scanoss-folder-hashing-api + make package_amd64 # or: make package_arm64 ``` -3. **Run the setup script**: +3. **Run the setup script** (from the `scripts/` directory, so it finds the binary and startup script): ```bash + cd scripts + # Interactive mode (prompts for confirmations) - sudo ./scripts/env-setup.sh + sudo ./env-setup.sh # Force mode (automated, no prompts) - sudo ./scripts/env-setup.sh --force + sudo ./env-setup.sh --force ``` 4. **Configure the service**: @@ -156,7 +158,7 @@ sudo systemctl enable scanoss-folder-hashing-api Start Qdrant using Docker Compose: ```bash -docker-compose -f docker-compose-qdrant.yml up -d +docker compose -f docker-compose-qdrant.yml up -d ``` This starts Qdrant with: @@ -173,20 +175,20 @@ There are two ways to populate the vector database: import raw data from CSV fil Use the `cmd/import` tool to populate the vector database with component data: ```bash -# Build the import tool -go build -o dist/import-tool cmd/import/main.go +# Build the import tool (from the repository root) +make build_import_amd64 # Import CSV data (-top-purls is optional) -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/files # Import CSV data with an optional PURL ranking file to prioritize results -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/files \ -top-purls /path/to/top-purls.json # Recreate database from scratch -./dist/import-tool \ +./target/scanoss-folder-hashing-import-linux-amd64 \ -dir /path/to/csv/files \ -overwrite ``` @@ -223,17 +225,17 @@ For more details, see the [main README](../README.md#restoring-from-snapshots). ### Creating a Distribution Package -Use the `package-scripts.sh` in the root directory: +Use the Make targets in the root directory (they build the binary into `scripts/` and then archive the folder via `package-scripts.sh`): ```bash # Create package for AMD64 -./package-scripts.sh linux_amd64 1.0.0 +make package_amd64 # Create package for ARM64 -./package-scripts.sh linux_arm64 1.0.0 +make package_arm64 ``` -This creates a tar archive containing all scripts for deployment on target servers. +This creates a tar archive (`scanoss-folder-hashing-api_linux-amd64_-1.tgz`) containing the binary and all scripts for deployment on target servers. ## ⚙️ Configuration