Add openvino_hggd module: HGGD OpenVINO iGPU enablement by deepaks2 · Pull Request #1069 · openvinotoolkit/openvino_contrib

deepaks2 · 2026-06-10T05:16:48Z

This pull request adds a new module, openvino_hggd, to the contrib repository, providing OpenVINO support for the HGGD (Hybrid Grasp Detection and Generation) model, a state-of-the-art Efficient Heatmap-Guided 6-Dof Grasp Detection in Cluttered Scenes. The changes include documentation updates, usage guides, code for HGGD (Hybrid Grasp Detection and Generation) with OpenVINO custom extensions, and supporting files for building and running the module.

Major additions and updates:

New HGGD (Hybrid Grasp Detection and Generation) module integration

Added the openvino_hggd module
Implemented OpenVino extensions for pointcloud, written optimized kernels for farthest point sampling, knn and ball query.

Documentation and Usage Guides
Added EXPORT_AND_INFERENCE_GUIDE.md with step-by-step instructions for exporting HGGD models to OpenVINO, building required OpenVINO extensions, and running inference/evaluation.

Added a .gitignore to the module directory to exclude build artifacts, logs, and virtual environments.

AI Usage: Yes, reviewed and edited manually

- Intel iGPU inference for HGGD (Hybrid Grasp Detection and Generation) via OpenVINO with custom C++/OpenCL point cloud extensions - Includes: export_models.py, infer.py, run.sh, setup.sh, ov_gpu_extensions/ - Users must copy customgraspnetAPI, dataset, models from upstream https://github.com/THU-VCLab/HGGD (see README) - Add 15 missing third-party entries to third-party-programs.txt (HGGD, graspnetAPI, dexnet, scipy, scikit-image, matplotlib, pandas, tensorboardX, torchsummary, transforms3d, trimesh, autolab_core, cvxopt, grasp-nms, Pillow)

allnes

Code review of the new openvino_hggd module (Python/ML + C++/OpenCL custom extension for OpenVINO GPU) plus the third-party-programs.txt additions. Read the affected files in full; didn't build (needs an Intel GPU, a torch+xpu conda env, the built .so, and the copied upstream GraspNet dataset).

Verdict: changes needed, with a few items for maintainer/legal review. The blockers are on the licensing side, not the code:

grasp-nms is listed as MIT but the pinned package declares no license (see inline).
dexnet is a UC Berkeley non-commercial license, incompatible with Apache-2.0 (see inline).
cvxopt is GPLv3 (see inline).

Two items without a line to attach to:

The module isn't listed in the root README.md where every other module is — please add an entry. The PR description also mentions an EXPORT_AND_INFERENCE_GUIDE.md, but there's no such file in the diff; either add it or drop the claim.
The feature commit has no Signed-off-by. If this repo enforces DCO, rebase with git commit -s so the check passes.

No tests are wired into CI (no job, no labeler.yml/CODEOWNERS entry), and export_models.py prints max_diff(PT vs OV) without asserting a threshold — worth adding a numeric gate. Detailed findings inline.

- Remove hard 8192-point cap from fps_single_optimized.cl and fps_with_lengths_optimized.cl; recompute distances per iteration so large padded clouds (e.g., 32768 points) are handled correctly - Add PyTorch3D BSD-3 license/copyright notices to the affected OpenCL kernels and to pointcloud_ops.cpp - Sync pointcloud_ops_gpu_v2.xml for the updated kernels

Zero out invalid idx slots instead of clamping them to 0 Clamps were returning features[0] for invalid positions; masked assignment now matches MaskedGather::evaluate behavior

Remove -march=native to avoid ISA lock-in on the build machine Drop -ffast-math to preserve IEEE FP semantics needed for distance comparisons

Align export_models.py docstring with setup.sh and README.md so copy-paste setup instructions work

Replace lingering hggd_xpu references with hggd_intel for consistency

Signed-off-by: Deepak Soma Reddy <deepak.s@intel.com>

FP16 silently rounds integer indices above 2048, but HGGD point clouds reach 25000+ points and KNN/BallQuery/FPS indices are packed as floats. Switch shim default to f32; keep f16 opt-in only for small N.

Add NODE_VALIDATION_CHECK so partial_sort does not walk past dist_idx.end() when K > N2, which is UB and would read out of bounds in the output loop.

github-actions · 2026-08-19T02:01:15Z

This pull request has been automatically marked as stale because it has not had any activity for the last 2 weeks. It will be closed in 7 days if no further activity occurs. Please push new commits or leave a comment to keep it open. Thank you for your contributions!

allnes · 2026-08-19T13:51:09Z

@deepaks2 Is this PR still relevant?

deepaks2 · 2026-08-19T14:32:07Z

@deepaks2 Is this PR still relevant?

Apologies for the delay. I was working on the demo setup for an event because of that i did not push commit with new changes. I will address the comments and push the PR

Add it for consistency and license compliance.

When lengths=None, fps() zero-pads points up to a power-of-two bucket and dispatches FPSSingle, which has no notion of valid length. The all-zero pad rows become real FPS candidates; for clouds not centred at the origin (0,0,0) is frequently selected as the farthest point, and the resulting out-of-range index was silently collapsed to N-1 by np.clip, producing a wrong but valid-looking index with no warning. Fix: when padding is needed (N_pad > N) and no lengths are provided, synthesise a lengths array filled with the true N and route through FPSWithLengths, which already handles this correctly. FPSSingle is only dispatched when N_pad == N (no padding, so no spurious rows exist).

cvxopt==1.3.3 carries a GPL-3.0 license which is incompatible with the Apache-2.0 distribution. Replace both cvxopt.solvers.qp() call sites in quality.py with a _solve_qp() helper backed by scipy.optimize.minimize (SLSQP), which solves the identical QP formulation: minimize 0.5 x'Px + q'x subject to Gx <= h, Ax = b scipy is already listed in setup.sh (scipy==1.15.3), so no new dependency is introduced. Remove cvxopt==1.3.3 from setup.sh and its GPL-3.0 license entry from third-party-programs.txt.

- pointcloud_ops.cpp: add NODE_VALIDATION_CHECK(k <= MAX_K=64) to KNNPoints, KNNPointsSingle, BallQuery and BallQuerySingle. GPU kernels use fixed private arrays best_dists/best_idx[MAX_K=64]; any k > 64 is an out-of-bounds write on GPU. - fps_single_optimized.cl, fps_with_lengths_optimized.cl: guard the FPS argmax with 'md > 0.0f' to prevent re-selecting already-chosen points. When md == 0 the candidate is coincident with a selected point; the old code could re-emit a duplicate index on degenerate or zero-padded clouds (diverging from the CPU path which sets min_dist[selected] = -1). - Remove dead kernel files fps_single_v2.cl, fps_with_lengths.cl and knn_points_single_v2.cl: none are referenced by pointcloud_ops_gpu_v2.xml or installed by CMakeLists. fps_single_v2.cl also documents a MAX_N=8192 truncation that the shipped kernels were rewritten to remove. knn_points_single_v2.cl lacked PyTorch3D attribution. - ov_shim_gpu.py: add 'API-compatible with pytorch3d.transforms' provenance note to euler_angles_to_matrix and matrix_to_quaternion, consistent with attribution in other ported files. - __init__.py: fix stale module names in docstring (ov_shim_v3 -> ov_shim_gpu, pointcloud_ops_native -> pointcloud_ops_native_gpu). - README.md: add OpenVINO HGGD to the main module overview list, where every other module is listed. It was only in 'Additional build instructions'.

KNNPoints, BallQuery, FPS, MaskedGather, GatherMaxPool and PointGather were fully implemented in pointcloud_ops.cpp and registered in ov_extension.cpp but had zero callers in any .py, .xml or calling .cpp. They are development leftovers from before the GPU-compatible *Single / FPSWithLengths variants were built. Remove their class definitions from pointcloud_ops.hpp, all method implementations from pointcloud_ops.cpp, and both OpExtension + frontend::OpExtension registrations per op (12 lines) from ov_extension.cpp. The four active ops are unchanged: KNNPointsSingle, BallQuerySingle, FPSSingle, FPSWithLengths.

Added new Collision-Free AP (NMS, object assignment, gripper-box collision/empty test, AP): standalone numpy/open3d port of the generic GraspNet eval stages, bit-identical to the official pipeline on the non-scoring stages authored-by: Deepak Soma Reddy <deepak.s@intel.com>

allnes · 2026-09-14T19:16:14Z

+
+```bash
+git clone https://github.com/THU-VCLab/HGGD /tmp/HGGD_upstream
+cp -r /tmp/HGGD_upstream/customgraspnetAPI path_to_openvino_hggd/


This copies the complete customgraspnetAPI tree, including Dex-Net code whose license permits only educational, research, and not-for-profit use. That restriction is incompatible with presenting the resulting module as unrestricted Apache-2.0 software. Copy only the required permissively licensed files, or obtain and document commercial-compatible permission; pin the exact HGGD revision and retain its notices.

allnes · 2026-09-14T19:16:14Z

+
+MIT License
+
+Copyright (c) 2019-2023 GraspNet Group


This MIT notice is not traceable to graspnetAPI==1.2.11. Its 2023 sdist contains no license file or license metadata, and the repository added an MIT license only in 2025 with Copyright (c) 2025 GraspNet. Pin a revision carrying a valid grant, confirm that it covers the ported code, and reproduce the actual notice instead of the unsupported 2019-2023 attribution.

allnes · 2026-09-14T19:16:14Z

+bash run.sh \
+  /path/to/HGGD_realsense_checkpoint \
+  /path/to/dataset/6dto2drefine_realsense \
+  /path/to/graspnet \


The documented workflow requires GraspNet data and an HGGD checkpoint, but does not state their terms. GraspNet's official site restricts its data, labels, code, and models to non-commercial use under CC BY-NC-SA, while no explicit license was found for the referenced HGGD checkpoint. Document exact artifact URLs and terms and obtain legal approval before treating this as a generally usable Apache-2.0 module.

allnes · 2026-09-14T19:16:14Z

+conda run -n "$ENV_NAME" pip install \
+    openvino==2026.1.0 \
+    numpy==2.0.2 \
+    scipy==1.15.3 \


The documented copy step still fetches upstream quality.py, which optionally imports cvxopt and then unconditionally accesses cvx.solvers during package import. setup.sh no longer installs cvxopt, so from customgraspnetAPI import Grasp fails before inference starts. The SciPy rewrite mentioned in the commit history is not part of this PR. Include that patch in a pinned source tree or remove the import path.

allnes · 2026-09-14T19:16:14Z

+        output[out_base + 2 * OUTPUT0_PITCHES[2]] = (OUTPUT0_TYPE)pz;
+        output[out_base + 3 * OUTPUT0_PITCHES[2]] = (OUTPUT0_TYPE)0;
+    }
+    barrier(CLK_LOCAL_MEM_FENCE);


Work-item 0 writes the selected index to global output, then every work-item reads it in the next iteration. A local-memory fence does not make that global write visible. Use barrier(CLK_LOCAL_MEM_FENCE | CLK_GLOBAL_MEM_FENCE) after initialization and after each selected-point write here and in fps_with_lengths_optimized.cl:104,171.

allnes · 2026-09-14T19:16:15Z

+        pt_outputs,
+    )):
+        max_diff = np.abs(pt_out.numpy() - ov_result[i]).max()
+        log.info(f"  {name}: shape={pt_out.shape}, max_diff(PT vs OV@{ov_device})={max_diff:.6f}")


This validation cannot fail when conversion produces incorrect values. There are also no automated tests for the custom operations or kernels. Add tolerance assertions and CPU-reference tests covering non-power-of-two sizes, K boundaries, empty lengths, duplicate points, FP32/FP16, and repeated execution; wire at least the CPU-capable subset into CI.

allnes · 2026-09-14T19:16:15Z

+ * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+/*
+ * Farthest Point Sampling - Recomputing O(N*K*K) kernel


The kernel rescans every previously selected point on every FPS iteration, changing the normal O(NK) algorithm to O(NK^2). run.sh uses K=512 for up to 48 groups, so this adds 130,816 selected-point distance checks per candidate. Preserve per-point minimum distances in scratch/global storage or provide measurements showing that this cost is acceptable.

allnes · 2026-09-14T19:16:15Z

+
+# ── Collision-Free evaluation ─────────────────────────────────────────────
+# The stage 0 (grasp NMS + object assignment + top-K selection) and stage 1
+# (gripper-box collision / empty test) below are verbatim ports of the


evaluate.py explicitly contains verbatim GraspNetAPI ports, and infer.py substantially adapts HGGD inference and collision code, but both carry only the Intel/Apache header. Record the exact source revisions and retain the applicable upstream copyright and license notices. Resolve the GraspNetAPI license grant before assigning Apache-2.0 to its port.

allnes · 2026-09-14T19:16:15Z

+
+# Core Intel XPU runtime libraries used by hggd_intel.
+conda run -n "$ENV_NAME" pip install \
+    intel-cmplr-lib-rt==2025.0.2 \


Four pinned runtime packages declare the proprietary Intel End User License Agreement, and none of the seven packages in this block is recorded in third-party-programs.txt. They are downloaded rather than vendored, but their terms and necessity still need review, especially because Torch computation is patched to CPU and OpenVINO owns the GPU path. Document all seven and remove those that are not required.

allnes · 2026-09-14T19:16:15Z

+
+MIT License
+
+Copyright (c) 2021 graspnet contributors


github-actions · 2026-09-29T01:51:15Z

This pull request has been automatically marked as stale because it has not had any activity for the last 2 weeks. It will be closed in 7 days if no further activity occurs. Please push new commits or leave a comment to keep it open. Thank you for your contributions!

deepaks2 requested a review from a team as a code owner June 10, 2026 05:16

github-actions Bot added the category: build OpenVINO cmake script / infra label Jun 10, 2026

deepaks2 and others added 4 commits June 10, 2026 10:54

Merge branch 'master' into hggd_ov

50d3278

Merge branch 'master' into hggd_ov

727b3ef

Merge branch 'master' into hggd_ov

8f24be7

Merge branch 'master' into hggd_ov

b616135

allnes reviewed Jul 21, 2026

View reviewed changes

allnes self-assigned this Jul 21, 2026

deepaks2 added 10 commits July 29, 2026 17:52

Merge branch 'master' into hggd_ov

8b00fba

Fix masked_gather -1 index handling in GPU shim

389568e

Zero out invalid idx slots instead of clamping them to 0 Clamps were returning features[0] for invalid positions; masked assignment now matches MaskedGather::evaluate behavior

Make compiler flags portable

c46855f

Remove -march=native to avoid ISA lock-in on the build machine Drop -ffast-math to preserve IEEE FP semantics needed for distance comparisons

Normalize conda env name to hggd_intel

a739643

Align export_models.py docstring with setup.sh and README.md so copy-paste setup instructions work

Normalize conda env name in setup.sh comments

bb5ddcd

Replace lingering hggd_xpu references with hggd_intel for consistency

Adding hggd to README

68750d8

Signed-off-by: Deepak Soma Reddy <deepak.s@intel.com>

Force FP32 for index-carrying GPU point-cloud ops

fa257ee

FP16 silently rounds integer indices above 2048, but HGGD point clouds reach 25000+ points and KNN/BallQuery/FPS indices are packed as floats. Switch shim default to f32; keep f16 opt-in only for small N.

Guard KNN K <= N2 in validate_and_infer_types

f15e408

Add NODE_VALIDATION_CHECK so partial_sort does not walk past dist_idx.end() when K > N2, which is UB and would read out of bounds in the output loop.

Fix install file list to real kernel/config names

05c7ff4

deepaks2 requested a review from allnes July 30, 2026 11:41

allnes requested changes Aug 4, 2026

View reviewed changes

github-actions Bot added the stale No activity for 2 weeks; will be closed in 7 days label Aug 19, 2026

allnes removed the stale No activity for 2 weeks; will be closed in 7 days label Aug 19, 2026

deepaks2 added 4 commits August 19, 2026 20:59

AAdd missing PyTorch3D BSD-3 attribution to ball_query_single_v2.cl

d22a75d

Add it for consistency and license compliance.

github-actions Bot added the category: CI OpenVINO public CI label Aug 19, 2026

deepaks2 requested a review from allnes August 19, 2026 15:33

Merge branch 'master' into hggd_ov

f996307

deepaks2 force-pushed the hggd_ov branch from d5b15e3 to 8c38ccb Compare August 30, 2026 02:57

deepaks2 added 2 commits August 30, 2026 08:27

Merge branch 'master' into hggd_ov

ea3e99f

deepaks2 requested a review from a team as a code owner September 3, 2026 11:06

deepaks2 force-pushed the hggd_ov branch 3 times, most recently from e4aef47 to 6a7a619 Compare September 3, 2026 12:44

Update .gitignore, README.md, labeler.yml, labels.yml

6a7a619

allnes requested changes Sep 14, 2026

View reviewed changes

github-actions Bot added the stale No activity for 2 weeks; will be closed in 7 days label Sep 29, 2026

Merge branch 'master' into hggd_ov

6bff275

allnes removed the stale No activity for 2 weeks; will be closed in 7 days label Oct 2, 2026

Conversation

deepaks2 commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

allnes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

allnes commented Aug 19, 2026

Uh oh!

deepaks2 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

deepaks2 commented Jun 10, 2026 •

edited

Loading

deepaks2 commented Aug 19, 2026 •

edited

Loading