fix(sharepoint): add compact site-group DLS mode to reduce ACL memory by Jan-Kazlouski-elastic · Pull Request #4396 · elastic/connectors

Jan-Kazlouski-elastic · 2026-08-28T17:06:46Z

Closes https://github.com/elastic/chat-program/issues/48

Part of https://github.com/elastic/chat-program/issues/47.
SharePoint Online DLS expands every site group member onto each document's _allow_access_control array. Large site groups can produce multi-megabyte ACLs per document and OOM Elasticsearch during _reindex.

This PR adds expand_site_group_members (default true to preserve existing behavior). When disabled (compact mode):

Content documents store a compact site_group:<site_id>:<group_id> token instead of every site group member
Direct user, Entra group, and site user permissions are unchanged on the document
ACL sync builds a site-group membership index and enriches identity docs so DLS term overlap still resolves access (including nested Entra groups and EEEU guest exclusion)
If a compact token cannot be written safely (e.g. missing site_id), the connector falls back to expanding site group members and logs a warning

Changing the setting requires a full content sync and access control sync.

Checklists

Pre-Review Checklist

Changes Requiring Extra Attention

Security-related changes (encryption, TLS, SSRF, etc)
New external service dependencies added.

Related Pull Requests

fix(servicenow): add compact role-based DLS to reduce ACL memory #4392 — ServiceNow compact role DLS (same tracking issue)
Kibana native_connectors.ts RCF for expand_site_group_members — to be opened separately

Release Note

SharePoint Online document-level security can store compact site-group tokens on documents instead of expanding every site group member onto each record, reducing memory use for large site groups. Disable Expand site group members to enable compact mode. The default preserves the previous behavior. Changing the setting requires a full content sync and access control sync.

Stop expanding large SharePoint site groups onto every document when expand_site_group_members is disabled; store site_group tokens on docs and resolve membership on identity docs during ACL sync instead.

…mpact DLS Resolve site group membership on identity docs using _access_control_for_member tokens (including nested Entra groups) and restore streaming ACL sync in legacy mode.

Build a token-to-site_group index before user enumeration so identity docs are yielded one at a time, and apply EEEU site groups only to non-guest users.

…ot be written Centralize site-group ACL resolution: use compact tokens when site_id is available, otherwise expand members so permissions are never silently dropped. Also handle role assignments that only include Member.Users without a group id.

erikcurrin-elastic · 2026-09-03T17:03:43Z

+        self._logger.info(
+            "Compact site-group DLS enabled: building site group membership index"
+        )
+


This may end up being expensive. The call to self.site_collections() means we get it again. We go through the list twice. Not sure if there are Sharepoint instances with lots of these. Same with self.sites()

Content and ACL sync run as separate jobs, so we can't share an in-memory cache across them. It's a deliberate tradeoff: one extra crawl per ACL sync vs expanding every site group member onto every document.

erikcurrin-elastic · 2026-09-03T17:04:55Z

+                if user_doc:
+                    yield user_doc
+            return
+


Any worries the acl_token map gets big?

5k-member group on 10k docs - expand mode puts ~50M ACL entries on documents, while compact mode puts 10k tokens on docs plus a ~5k-entry sync map. Much smaller.

erikcurrin-elastic · 2026-09-03T17:07:01Z

+                        continue
+
+                    site_group_token = _prefix_site_group(site_id, group_id)
+                    async for member in self.client.site_groups_users(


Should we call our self.site_group_users so we can use the cached version?

Switched the index builder to use self.site_group_users() so it goes through site_group_cache.

erikcurrin-elastic · 2026-09-03T17:08:24Z

+                group_id = role_assignment.get("PrincipalId")
+            if group_id is not None:
+                access_control.extend(
+                    await self._site_group_access_control(


None may be fragile here, since we use it downstream in self.site_group_users

site_web_url is now passed from all role assignment call sites, so fallback expansion has a real URL when needed.

…t DLS Use site_group_users in the compact ACL index builder and thread site_web_url through role assignment resolution for safe fallback.

## Closes elastic/chat-program#49 Part of elastic/chat-program#47. ServiceNow DLS expands every table read role into individual `user_id:` / `user:` / `email:` entries on each content document (and attachment). For large tenants — especially when a table ACL includes the `public` role — that can mean hundreds of thousands of ACL entries per document and OOM Elasticsearch during `_reindex`. This PR adds compact DLS, **opt-in** via `expand_role_members=false` (default `true` preserves existing behavior). When compact mode is enabled: - Content documents store compact `role_id:<sys_id>` tokens instead of every role member - Identity docs include role memberships from `sys_user_has_role` so DLS term overlap still works - Tables with a `public` read role omit `_allow_access_control` (world-readable via existing DLS `must_not exists`) - Tables where read roles cannot be resolved omit `_allow_access_control` (same as public tables; world-readable via existing DLS `must_not exists`) Also fixes advanced sync rules to resolve ACL per table instead of stamping an empty ACL on every batch. Changing the setting requires a full content sync and access control sync. ## Checklists #### Pre-Review Checklist - [x] this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check `config.yml.example`) - [x] this PR has a meaningful title - [x] this PR links to all relevant github issues that it fixes or partially addresses - [x] if there is no GH issue, please create it. Each PR should have a link to an issue - [x] this PR has a thorough description - [x] Covered the changes with automated tests - [x] Tested the changes locally - [ ] Added a label for each target release version (example: `v7.13.2`, `v7.14.0`, `v8.0.0`) - [ ] For bugfixes: backport safely to all minor branches still receiving patch releases - [ ] Considered corresponding documentation changes - [ ] Contributed any configuration settings changes to the configuration reference - [ ] if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in [Kibana](https://github.com/elastic/kibana/blob/main/packages/kbn-search-connectors/types/native_connectors.ts) #### Changes Requiring Extra Attention - [x] Security-related changes (encryption, TLS, SSRF, etc) - [ ] New external service dependencies added. ## Related Pull Requests * #4396 — SharePoint Online compact site-group DLS (same tracking issue) * Kibana `native_connectors.ts` RCF for `expand_role_members` — to be opened separately ## Release Note ServiceNow document-level security can store compact role tokens on content documents instead of expanding every role member onto each record, reducing memory use for large tenants. Disable **Expand role members** to enable compact mode. The default preserves the previous per-user expansion behavior. Changing the setting requires a full content sync and access control sync. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Jan-Kazlouski-elastic · 2026-09-09T14:50:55Z

E2E validation: SPO compact site-group DLS

Verdict: Pass. expand_site_group_members works as designed in both directions.

Setup

Live SPO tenant, local ES/Kibana stack, DLS enabled
Certificate auth with SharePoint Sites.FullControl.All
Custom site group + folder with 5 drive items
Full content + ACL sync after each config toggle

Results

Check	Compact	Legacy
`site_group:` on content	Yes (1 token/group)	None (members expanded)
`site_group:` on identities	Enriched at ACL sync	None
ACL payload (`acl-test` files)	6 tokens / ~575 chars	44 tokens / ~2 KB
Whole index ACL payload	~21 KB	~68 KB

DLS

Simulated via standard connector terms query on _allow_access_control.keyword.

Group members: visible in both modes ✅
Compact: access confirmed via site_group: token alone (identity enrichment path) ✅
Negative controls (invalid/empty tokens): 0 hits ✅

Gaps

ES query layer only (no Kibana Search UI test)
Single site, small fixture — not scale-tested

No blocking issues found.

…3645) (#4437) ## Summary With DLS enabled, unpublished SharePoint Online pages were still returned to users with view access. SharePoint does not update a page's ACLs when it is unpublished, so the connector kept the old view-only permissions on the indexed document. This detects draft vs published state from the page version string and restricts `_allow_access_control` on unpublished pages to owners/editors only. A `published` field is also exposed on `site_page` documents. Closes #3645 ## Changes - Detect published state from `OData__UIVersionString` (major version = published, minor version = draft) - For unpublished pages, restrict ACLs to owners/editors: - unique per-page permissions: only edit-or-higher role members - inherited site permissions: use a site editors ACL set instead of the full site ACL - Add `require_edit_access` to `_get_access_control_from_role_assignment` - Add `EDIT_ITEM_MASK` / `EDIT_ROLE_TYPES` constants - Add `published` field on `site_page` documents - Debug log when an unpublished page ACL is restricted ## Testing - Unit tests: version parsing, edit-only role filtering, unpublished/published ACL behavior (unique + inherited permissions). `209 passed` in `test_sharepoint_online.py`. - E2E: not run locally (requires live SharePoint Online tenant with DLS). ## Notes for reviewers - Published detection defaults to `published=True` when version info is missing, to avoid over-restricting access. - **Merge note with #4396:** both PRs touch `datasource.py` and `test_sharepoint_online.py`. A test merge shows one mechanical conflict in `_get_access_control_from_role_assignment` (this PR adds `require_edit_access`, #4396 adds `site_id`). Tests auto-merge. Resolution is to keep both optional parameters and pass both from `site_pages`. - Consider backport labels if needed. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Jan-Kazlouski-elastic added 3 commits August 27, 2026 16:29

Add compact SharePoint site-group DLS mode to reduce ACL memory

a0f86b9

Stop expanding large SharePoint site groups onto every document when expand_site_group_members is disabled; store site_group tokens on docs and resolve membership on identity docs during ACL sync instead.

fix(sharepoint): match nested site group members via ACL tokens in co…

b058860

…mpact DLS Resolve site group membership on identity docs using _access_control_for_member tokens (including nested Entra groups) and restore streaming ACL sync in legacy mode.

fix(sharepoint): stream compact ACL sync and exclude guests from EEEU

ad51f50

Build a token-to-site_group index before user enumeration so identity docs are yielded one at a time, and apply EEEU site groups only to non-guest users.

Jan-Kazlouski-elastic requested a review from a team as a code owner August 28, 2026 17:06

github-actions Bot added auto-backport v9.6.0 labels Aug 28, 2026

Update NOTICE.txt

bad996f

Jan-Kazlouski-elastic mentioned this pull request Aug 31, 2026

fix(servicenow): add compact role-based DLS to reduce ACL memory #4392

Merged

14 tasks

Jan-Kazlouski-elastic and others added 2 commits August 31, 2026 13:45

Update NOTICE.txt

364d6c8

Jan-Kazlouski-elastic mentioned this pull request Aug 31, 2026

Add SharePoint Online expand_site_group_members RCF elastic/kibana#288070

Open

10 tasks

Jan-Kazlouski-elastic self-assigned this Aug 31, 2026

Jan-Kazlouski-elastic mentioned this pull request Sep 3, 2026

fix: restrict unpublished SharePoint Online pages to owners/editors (#3645) #4437

Merged

Jan-Kazlouski-elastic requested a review from erikcurrin-elastic September 3, 2026 08:56

Jan-Kazlouski-elastic and others added 2 commits September 3, 2026 12:04

Merge branch 'main' into jan_kazlouski/spo-compact-site-group-dls

1f3056c

Update NOTICE.txt

0a2398d

erikcurrin-elastic reviewed Sep 3, 2026

View reviewed changes

Jan-Kazlouski-elastic and others added 2 commits September 5, 2026 02:25

fix(sharepoint): use site group cache and pass site_web_url in compac…

5c5aea1

…t DLS Use site_group_users in the compact ACL index builder and thread site_web_url through role assignment resolution for safe fallback.

Update NOTICE.txt

2ccdde3

Jan-Kazlouski-elastic requested a review from erikcurrin-elastic September 4, 2026 23:29

Merge branch 'main' into jan_kazlouski/spo-compact-site-group-dls

46ba624

Merge branch 'main' into jan_kazlouski/spo-compact-site-group-dls

afd32ad

Jan-Kazlouski-elastic requested a review from artem-shelkovnikov September 9, 2026 10:52

Jan-Kazlouski-elastic added v9.4.7 v8.19.22 v9.5.4 labels Sep 9, 2026

Merge branch 'main' into jan_kazlouski/spo-compact-site-group-dls

1304ea2

Merge branch 'main' into jan_kazlouski/spo-compact-site-group-dls

9cdaef2

Update NOTICE.txt

1bec3a8

This was referenced Sep 9, 2026

[Docs] Document SharePoint Online expand_site_group_members setting elastic/elasticsearch#158874

Open

[Docs] Document SharePoint Online expand_site_group_members setting (8.19) elastic/elasticsearch#158875

Open

Jan-Kazlouski-elastic added v9.4.8 v9.5.5 v8.19.23 and removed v9.4.7 v8.19.22 v9.5.4 labels Sep 14, 2026

Update NOTICE.txt

df11d91

erikcurrin-elastic approved these changes Sep 14, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(sharepoint): add compact site-group DLS mode to reduce ACL memory - #4396

fix(sharepoint): add compact site-group DLS mode to reduce ACL memory#4396
Jan-Kazlouski-elastic wants to merge 16 commits into
mainfrom
jan_kazlouski/spo-compact-site-group-dls

Jan-Kazlouski-elastic commented Aug 28, 2026 •

edited

Loading

Uh oh!

erikcurrin-elastic Sep 3, 2026

Uh oh!

Jan-Kazlouski-elastic Sep 4, 2026

Uh oh!

erikcurrin-elastic Sep 3, 2026

Uh oh!

Jan-Kazlouski-elastic Sep 4, 2026

Uh oh!

erikcurrin-elastic Sep 3, 2026

Uh oh!

Jan-Kazlouski-elastic Sep 4, 2026

Uh oh!

erikcurrin-elastic Sep 3, 2026

Uh oh!

Jan-Kazlouski-elastic Sep 4, 2026

Uh oh!

Jan-Kazlouski-elastic commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Jan-Kazlouski-elastic commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Closes https://github.com/elastic/chat-program/issues/48

Checklists

Pre-Review Checklist

Changes Requiring Extra Attention

Related Pull Requests

Release Note

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jan-Kazlouski-elastic commented Sep 9, 2026

E2E validation: SPO compact site-group DLS

Setup

Results

DLS

Gaps

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jan-Kazlouski-elastic commented Aug 28, 2026 •

edited

Loading