feat(objectscript): InterSystems IRIS ObjectScript language support

[isc-tdyar]

What does this PR do?

Adds ObjectScript (InterSystems IRIS / Caché) as a supported language, per the discussion in #462.

ObjectScript powers large healthcare, finance, and enterprise systems and has no support in CBM (or most code-graph tools). This PR makes those codebases indexable and resolves the call-dispatch patterns that are structurally invisible to text search.

Refs #462

⚠️ Read first — grammar is a separate dependency

Per your note on #462, this PR contains only the CBM source changes — not the vendored grammar. The two tree-sitter grammars come from intersystems/tree-sitter-objectscript (MIT licensed, maintained by the language vendor): objectscript_udl (.cls) and objectscript_routine (.mac/.inc/.rtn/.int).

Consequence: the build will not link until the grammar is vendored at internal/cbm/vendored/grammars/objectscript_udl/ and …/objectscript_routine/ — it fails on the missing tree_sitter_objectscript_udl() / _routine() symbols. CI will be red until then. That is intentional, matching your plan to audit and vendor the grammar independently. The grammar shims (grammar_objectscript_*.c) that declare those factories are included — only the generated parser.c/scanner.c are omitted.

What's in the PR

Definition extraction — Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, Query members → graph nodes; base classes from the Extends clause.

Four call-dispatch patterns (all resolved statically at index time):

Pattern	Example	Notes
Explicit cross-class	##class(Pkg.Class).Method()	from AST
Relative-dot self-call	..Method()	the dominant intra-class form; large impact on CALLS completeness
Macro expansion	$$$Macro	resolved via a per-project table built from .inc files
Type inference	Set x = ##class(P).%New() … x.Save()	from %New/%OpenId + declared return types

Ensemble production topology (pass_ensemble_routing.c) — EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData; plus WorkMgr .Queue("##class(X).method") dispatch. All static — no live IRIS instance required.

Two design points for your review

1. Public API unchanged. ObjectScript needs two per-project tables (a $$$macro table and a method-return-type table) that single-file extraction can't build alone. Rather than widen the public cbm_extract_file() signature (which would ripple NULL, NULL through every call site), I added an internal cbm_extract_file_ex() that carries the tables; cbm_extract_file() is a thin wrapper that delegates with NULL, NULL. Only the pipeline passes that build the tables call _ex.

2. Extension collisions with Apex and BitBake (both added since I started this work). .mac/.int/.rtn map to ObjectScript routine directly. The two collisions are resolved by content sniffing, following the existing cbm_disambiguate_m() (.m MATLAB-vs-ObjC) pattern, and default to the existing language on any doubt so neither Apex nor BitBake regresses:

   • .cls (vs Apex): a Class <Uppercase…> header line → ObjectScript UDL, else Apex. Edge case: a .cls whose Class line sits beyond the first 4 KB (e.g. a very large license banner) would fall through to Apex.
   • .inc (vs BitBake): a ROUTINE <Uppercase> header or an ObjectScript preprocessor directive (#define / #def1arg / #;) → ObjectScript routine, else BitBake. (#define/#def1arg never collide with BitBake, which uses # only for # comment.)

   These are the spots most likely to need your input — happy to adjust the heuristics or the generalization however you prefer.

EnsembleItem label (your Q2 on #462)

I used a domain-specific EnsembleItem node label and ROUTES_TO edge type. If you'd prefer a generic label (ServiceComponent / WorkflowNode) with ensemble_item as a property, I'm glad to rename — just let me know before merge.

Tests

tests/test_extraction.c gains the ObjectScript suite: UDL class/method extraction, all four dispatch patterns, Ensemble topology parsing, macro expansion, trigger body text, and Export-XML transcoding. The Export-XML transcoder tests are grammar-independent and pass today; the grammar-dependent tests pass once the grammar is vendored. No other test files are touched.

Scope / roadmap

This PR is the foundation. If it's well received, two separate follow-up PRs would complete the story (each with its own issue): (a) cross-version version_tag + diff_versions, and (b) ObjectScript-tuned semantic embeddings. They're deliberately excluded here to keep this reviewable.

Checklist

• Every commit is signed off (git commit -s) — DCO
• Tests pass locally (make -f Makefile.cbm test) — passes once the grammar is vendored; see note above
• Lint passes (make -f Makefile.cbm lint-ci)
• New behavior is covered by tests

[isc-tdyar]

@DeusData - pinging you as I think this is ready for review :) If not, please lmk what you think needs to happen.

[DeusData]

Thanks @isc-tdyar will check ASAP

[DeusData]

Good news, @isc-tdyar — the ObjectScript grammars are now vendored on main (PR #590): internal/cbm/vendored/grammars/objectscript_udl/ and objectscript_routine/, from intersystems/tree-sitter-objectscript @ a7ffcdf (MIT, ABI 15).

Please rebase this PR onto latest main — your grammar_objectscript_{udl,routine}.c shims will now resolve the #include "vendored/grammars/objectscript_{udl,routine}/{parser,scanner}.c", so the build will link and CI should go green (the grammar-dependent extraction tests included).

One vendoring note (no action needed on your side): each scanner.c's upstream #include "../../common/scanner.h" was repointed to a per-directory objectscript_common.h, because this repo's shared vendored/common/scanner.h already belongs to the cfml/fsharp grammars. It's self-contained, so your shims don't need to change.

Once it's rebased and green, I'll review the extraction code and we can land it. Thanks for your patience on the grammar-vendoring step — that was on us to do.

[isc-tdyar]

This PR for ObjectScript language support is fully green across all platforms and ready for maintainer review @DeusData :)

[DeusData]

Thanks for the ObjectScript work. Since this PR is expected not to link until the grammars are vendored, we should not merge it as-is. Please either split this into a source-only design/review PR that does not wire the build, or include the independently reviewed vendored grammar provenance so CI can be green. The Makefile and vendored scanner changes will need maintainer review before merge.

[tom-dyar]

Hi @DeusData — just to clarify the current state: the grammars are already vendored, in your PR #590 on main — that's exactly what your Jun 23 comment asked us to wait for. We rebased onto that main on Jun 24, and CI has been fully green since then (all 13 checks pass across Ubuntu/ARM/macOS/Windows).

The commit message on this PR was written before #590 landed and says "grammars not vendored" — that's now out of date. The build links, the grammar-dependent extraction tests run, and the scanners are the vendored copies from internal/cbm/vendored/grammars/objectscript_{udl,routine}/.

So no split needed — the concern in your latest comment was the pre-rebase state. Happy to update the commit message to reflect that the grammars are now present if that would help clarify things.

[DeusData]

Thanks for clarifying — you’re right that my last comment was based on the stale pre-rebase state/commit text. I verified the current CI is green and the grammar-vendoring concern no longer applies.

The current repo-level blocker I see is GitHub marking this PR as conflicting. Please resolve the merge conflicts, and if convenient update the stale commit/PR wording so reviewers don’t trip over the old “grammars not vendored” state. After that this can move back to extraction-code review.

[isc-tdyar]

@DeusData - I just rebased to get the latest from upstream, so this should be in good shape now to merge, no conflicts showing for me. Thanks, kindly!

[DeusData]

Thank you for the comprehensive ObjectScript support — IRIS coverage is a genuinely novel addition. The PR is currently CONFLICTING against main (the branch is ~4 weeks old; Makefile.cbm, extract_defs.c, extract_unified.c and internal/cbm/cbm.c have all moved substantially since — grammar refresh, unified-dispatch, coverage signal). Please rebase, and while you're at it:

1. Vendored grammar policy (mandatory): the new grammar_objectscript_* sources need a license check (permissive only), the grammar's LICENSE vendored alongside, an entry in internal/cbm/vendored/grammars/MANIFEST.md, and THIRD_PARTY.md/SBOM updated in the same change. If the grammar is your own original work, state that explicitly in the PR + MANIFEST.
2. If any LSP-tier resolution is included, the new-LSP checklist applies: scripts/check-lsp-originality.sh output posted, plus the quadratic-pattern self-check (cursor-based child iteration, no per-registration finalize, no registry tail-scans).
3. After rebase, expect the extraction guard suites (grammar labels/imports/calls-breadth + the linearity guard) to exercise the new language — please make sure they're green on all legs including ARM before requesting re-review.

CI was otherwise green pre-conflict, which is promising. Looking forward to landing this once refreshed.

[DeusData]

Thanks for the substantial work here — ObjectScript (UDL + routine + Studio Export XML + $$$macro tables) is a language we've wanted, and the disambiguators for .cls/.inc collisions with Apex/BitBake show real care. CI is red on all four Unix legs though, and I tracked it to a concrete bug in the PR, plus a handful of things to fix before this can merge.

1. CI failure root cause (blocking): cbm_index_mark_done is bypassed

index_recovery_parallel_quarantines_crasher fails deterministically on all Unix legs (FAIL tests/test_mcp.c:4797: code == 65 = an innocent file was quarantined — the logs show index.file_quarantined path=idxpar_good_a.py).

Mechanism: on main, cbm_extract_file is a wrapper that calls the impl and then emits the crash-attribution "done" marker (cbm_index_mark_done(rel_path), internal/cbm/cbm.c:821). The "start" marker fires inside the impl (cbm.c:851). Your rename of cbm_extract_file_impl → public cbm_extract_file_ex keeps the start marker, but extract_worker in pass_parallel.c now calls cbm_extract_file_ex directly, so the done marker never fires in the parallel path. Every extracted file therefore stays "in-flight" in the supervisor's marker journal forever; the parallel-recovery contract ("quarantine a file only when it is in-flight across two consecutive failed runs") then deterministically quarantines the first good file. The Windows leg is green only because that guard is POSIX-only.

Fix: emit cbm_index_mark_done at every exit path of cbm_extract_file_ex (mirroring the entry-side cbm_index_mark_start), and remove it from the cbm_extract_file wrapper so it doesn't double-fire. Note your Export-XML inline loop in extract_worker also calls cbm_extract_file_ex per class — with the fix inside _ex that path ends "done" as well, which is what we want.

2. Raw fopen() on repo paths (3 new sites)

pp_build_macro_table (pass_parallel.c), cbm_build_macro_table_from_files (pipeline.c), and the Export-XML sniff in detect_file_language (discover.c) all use raw fopen() on repository file paths. Project rule: use cbm_fopen() (UTF-8 → _wfopen) — raw fopen breaks non-ASCII paths on Windows. The .cls/.inc disambiguators should use it too if they don't already.

3. strstr(path, ".inc") is not a suffix check

It matches foo.inc.bak and any directory named x.inc/. Please use a real suffix comparison (the strrchr(name, '.') + strcmp pattern used elsewhere in discover.c).

4. Macro-table arena ownership leak

Both macro-table builders create a local CBMArena, parse .inc content into it (the strings stored in the CBMMacroTable point into arena blocks), then return the table while the local arena handle goes out of scope. The blocks are unreachable — leaked — and there is no way to free them through the table. Please make the table own its arena (embed CBMArena in CBMMacroTable, destroy it in a cbm_macro_table_free). The ASan/LSan leg will eventually flag this, but it's an ownership design problem regardless.

5. Duplicated builders

pp_build_macro_table (pass_parallel.c) and cbm_build_macro_table_from_files (pipeline.c) are near-identical ~50-line functions. Keep one, share it between the parallel and sequential paths.

6. ensemble_routing pass runs unconditionally for every project — please split it out

cbm_pipeline_pass_ensemble_routing is wired into run_predump_passes with moderate_only=false, and its Pass A iterates all Method nodes of any language and re-reads each method's source file from disk. That's an unconditional extra O(all-method-files) repo re-read on every index for every user — including the overwhelming majority with zero ObjectScript. Two asks:

• Per our atomic-PR policy, please move the Ensemble production-routing pass to a follow-up PR. It's a meaty standalone feature (new pipeline pass + topology XML parsing + routing edges) and deserves its own review; this PR is already large. We'd genuinely welcome it as a follow-up.
• When it comes back: gate it on the project actually containing ObjectScript files (early-exit otherwise), and log when the MAX_ITEMS/MAX_SETTINGS caps truncate (silent truncation reads as full coverage).

7. Vendored-grammar policy items

• internal/cbm/vendored/grammars/MANIFEST.md must gain entries for both grammars (upstream repo, pinned commit a7ffcdf, license, local modifications) — this is mandatory for any vendored grammar change.
• The upstream MIT LICENSE file must be vendored alongside the code (one per grammar dir, or one shared with a note). THIRD_PARTY.md alone isn't sufficient; none is in the diff.
• Your THIRD_PARTY.md note says each scanner's include is repointed to a per-directory objectscript_common.h ("verbatim copy of upstream common/scanner.h") — but no such file exists in the PR. Since CI compiles, I assume the scanners were actually edited differently; please reconcile the note with what's really in the tree, and document any modification from upstream in the MANIFEST entry.

Happy to re-review quickly once these are in — the root-cause fix in item 1 is small, and the rest is mostly mechanical. Nice catch-rate on the disambiguation edge cases, and thanks again for the depth of this contribution.