[Power BI] Address ARG payload size errors

docs-mslearn/toolkit/changelog.md

@@ -3,7 +3,7 @@ title: FinOps toolkit changelog
 description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more.
 author: MSBrett
 ms.author: brettwil
-ms.date: 07/30/2026
+ms.date: 08/02/2026
 ms.topic: reference
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -85,6 +85,7 @@ _Released June 2026_
 
 - **Fixed**
   - Fixed Power BI storage report refresh errors caused by ISO 8601 duration `x_SkuTerm` values (like `P3Y`) and empty strings in cost exports ([#2174](https://github.com/microsoft/finops-toolkit/issues/2174)).
+  - Paginate Azure Resource Graph queries by subscription to mitigate [payload size limit](help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) errors in the [Governance](power-bi/governance.md) and [Workload optimization](power-bi/workload-optimization.md) reports ([#1768](https://github.com/microsoft/finops-toolkit/issues/1768)). Queries still surface a payload size error (rather than silently returning truncated results) if a single batch of subscriptions exceeds the limit; see [Reduce the batch size](help/errors.md#option-1-reduce-the-batch-size) for the mitigation. As part of this change, the `AdvisorRecommendations` and `AdvisorReservationRecommendations` tables now return complete (non-truncated) results per batch, matching every other batched table; previously these two tables silently dropped rows beyond the payload limit instead of erroring.
 
 ### [Data dictionary](help/data-dictionary.md) updates
 

docs-mslearn/toolkit/help/errors.md

@@ -3,7 +3,7 @@ title: Troubleshoot common FinOps toolkit errors
 description: This article describes common FinOps toolkit errors and provides solutions to help you resolve issues you might encounter.
 author: flanakin
 ms.author: micflan
-ms.date: 06/03/2026
+ms.date: 08/02/2026
 ms.topic: troubleshooting
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -766,13 +766,59 @@ Azure Resource Graph queries in the Governance and Workload optimization Power B
 
 > _OLE DB or ODBC error: [Expression.Error] Please provide below info when asking for support: timestamp = {timestamp}, correlationId = {guid}. Details: Response payload size is {number}, and has exceeded the limit of 16777216. Please consider querying less data at a time and make paginated call if needed._
 
-This error means that you have more resources than are supported in an unfiltered Resource Graph query. This happens because FinOps toolkit reports are designed to show resource-level details and are not aggregated. They are designed for small- and medium-sized environments and not designed to support organizations with millions of resources.
+Azure Resource Graph enforces a 16 MB response payload limit per query. FinOps toolkit reports automatically paginate queries in batches of subscriptions to stay within this limit, so most environments should not encounter this error. The base batch size is 100 subscriptions (set by the `ftk_ARGBatchSize` function), but some queries multiply this value for tables that return less data per resource, so the effective batch size varies by table. If you still see this error, it means the resources in a single batch of subscriptions exceed the 16 MB limit.
 
-**Mitigation**: If you experience this error, there are several options:
+**Mitigation**: Try the following options in order:
 
-- Remove columns that are not necessary for your needs.
-- Filter the query to return fewer resources based on what's most important for you (e.g., subscriptions, tags).
-- Disable the query so it doesn't block other queries from running.
+### Option 1: Reduce the batch size
+
+Reduce the base number of subscriptions queried in each batch (some tables multiply this value, so the effective batch size for those tables will still be smaller, but proportionally reduced):
+
+1. Open Power BI Desktop and select **Transform data** from the ribbon.
+2. In the **Queries** pane on the left, expand the **Functions** folder.
+3. Select the **ftk_ARGBatchSize** function.
+4. Change the return value from `100` to a smaller number (e.g., `20` or `10`).
+5. Select **Close & Apply** to save changes.
+
+### Option 2: Filter by resource group or tags
+
+Add a filter clause to the failing query to reduce the number of resources returned:
+
+1. Open Power BI Desktop and select **Transform data** from the ribbon.
+2. In the **Queries** pane on the left, expand the **Resource Graph** folder.
+3. Select the query that's failing (e.g., **NetworkSecurityGroups**, **Resources**).
+4. In the query editor, find the `query = "` section in the formula bar.
+5. Add a filter clause after the `| where type` line and before any `| extend` clauses. For example:
+
+   ```kusto
+   | where resourceGroup in~ ('rg-production', 'rg-staging')
+   ```
+
+   Or filter by tags:
+
+   ```kusto
+   | where tags.Environment =~ 'Production'
+   ```
+
+6. Select **Close & Apply** to save changes.
+
+### Option 3: Remove unnecessary columns
+
+Reduce the payload size by removing columns you don't need:
+
+1. Open the query in Power Query Editor (steps 1-3 from Option 2).
+2. In the query text, remove column names from the `extend` or `project` statements that you don't need for your analysis.
+3. Be careful not to remove columns that are used in report visuals or relationships.
+
+### Option 4: Disable the failing query
+
+If a specific query consistently fails and isn't critical to your needs:
+
+1. In Power Query Editor, right-click the failing query in the **Queries** pane.
+2. Uncheck **Enable load** to prevent the query from loading data.
+3. The query will remain in the report but won't execute during refresh.
+
+For more information about Azure Resource Graph limits, see [Working with large Azure resource data sets](/azure/governance/resource-graph/concepts/work-with-data).
 
 <br>
 

docs-mslearn/toolkit/power-bi/governance.md

@@ -3,7 +3,7 @@ title: FinOps toolkit Governance report
 description: Summarize cloud governance posture including areas like compliance, security, operations, and resource management in Power BI.
 author: flanakin
 ms.author: micflan
-ms.date: 04/01/2026
+ms.date: 08/02/2026
 ms.topic: concept-article
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -119,6 +119,16 @@ The **Network security groups** page lists network security groups and network s
 
 <br>
 
+## Known limitations
+
+The Governance report uses Azure Resource Graph to query resource details. Azure Resource Graph has a response payload limit of 16 MB per query. The report automatically paginates queries in batches of subscriptions to stay within this limit, but may not work for large environments where a single batch exceeds the limit.
+
+If you experience a "Response payload size... exceeded the limit" error, open the report in Power BI Desktop, go to the Power Query editor, and reduce the value returned by the `ftk_ARGBatchSize` function (default: 100). For detailed steps, see [Response payload size exceeded the limit](../help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) in the error reference guide.
+
+Batching queries by subscription requires Reader access at the subscription level or above. If your account only has resource group-scoped access, some subscriptions may be silently excluded from the report even though you can see their resources elsewhere.
+
+<br>
+
 ## Looking for more?
 
 We'd love to hear about any reports, charts, or general reporting questions you're looking to answer. Create a new issue with the details that you'd like to see either included in existing or new reports.

docs-mslearn/toolkit/power-bi/workload-optimization.md

@@ -3,7 +3,7 @@ title: FinOps toolkit Workload optimization report
 description: Learn about the Workload optimization report, which identifies opportunities for rightsizing and removing unused resources to enhance efficiency.
 author: flanakin
 ms.author: micflan
-ms.date: 04/01/2026
+ms.date: 08/02/2026
 ms.topic: concept-article
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -79,6 +79,16 @@ The chart shows the cost of each disk over time. The table shows the disks with
 
 <br>
 
+## Known limitations
+
+The Workload optimization report uses Azure Resource Graph to query resource details. Azure Resource Graph has a response payload limit of 16 MB per query. The report automatically paginates queries in batches of subscriptions to stay within this limit, but may not work for large environments where a single batch exceeds the limit.
+
+If you experience a "Response payload size... exceeded the limit" error, open the report in Power BI Desktop, go to the Power Query editor, and reduce the value returned by the `ftk_ARGBatchSize` function (default: 100). For detailed steps, see [Response payload size exceeded the limit](../help/errors.md#response-payload-size-is-and-has-exceeded-the-limit) in the error reference guide.
+
+Batching queries by subscription requires Reader access at the subscription level or above. If your account only has resource group-scoped access, some subscriptions may be silently excluded from the report even though you can see their resources elsewhere.
+
+<br>
+
 ## See also
 
 - [Common terms](../help/terms.md)

src/power-bi/kql/Shared.Dataset/definition/expressions.tmdl

@@ -89,6 +89,72 @@ expression ftk_DemoFilter =
 
 	annotation PBI_ResultType = Function
 
+/// Number of subscriptions to include in each Azure Resource Graph query batch. Decrease if you experience payload limit errors. Default: 100.
+expression ftk_ARGBatchSize = () => 100
+	lineageTag: 8c0ab9c7-ff92-4d6f-9882-4ec18a0c041f
+	queryGroup: Functions
+
+	annotation PBI_NavigationStepName = Navigation
+
+	annotation PBI_ResultType = Function
+
+/// Queries Azure Resource Graph in batches of subscriptions and combines results to avoid the 16 MB payload limit.
+expression ftk_QueryARG =
+		(query as text, optional batchMultiplier as number, optional includeUnscoped as logical) =>
+		    let
+		      defaultBatchSize = 100,
+		      requestedBatchSize = try ftk_ARGBatchSize() * (if batchMultiplier <> null then batchMultiplier else 1) otherwise null,
+		      batchSize = if requestedBatchSize = null or requestedBatchSize < 1 then defaultBatchSize else Number.RoundDown(requestedBatchSize),
+		      // retry each Azure Resource Graph call with a short backoff (plus jitter so concurrently-throttled batches don't all retry in lockstep) to absorb throttling: batching multiplies ARG calls per table refresh, and Power BI evaluates table partitions concurrently
+		      retryDelays = {5, 10},
+		      QueryARG = (q as text, optional attempt as number) as any =>
+		        let
+		          attemptNumber = if attempt = null then 0 else attempt,
+		          attempted = try AzureResourceGraph.Query(q, "Tenant", null, null, [resultTruncated = false]),
+		          result =
+		            if attempted[HasError] and attemptNumber < List.Count(retryDelays) then
+		              Function.InvokeAfter(() => @QueryARG(q, attemptNumber + 1), #duration(0, 0, 0, retryDelays{attemptNumber} + Number.RandomBetween(0, 3)))
+		            else if attempted[HasError] then
+		              error attempted[Error]
+		            else
+		              attempted[Value]
+		        in
+		          result,
+		      // batch discovery requires Reader at subscription scope or above to enumerate resourcecontainers; a principal with only resource-group-scoped Reader won't see their subscription here even though they can read its resources, so their data is silently excluded from batching -- documented as a known limitation in the report docs
+		      subQuery = "resourcecontainers | where type == 'microsoft.resources/subscriptions' | project subscriptionId" & ftk_DemoFilter(),
+		      Subscriptions = try QueryARG(subQuery) otherwise null,
+		      // fail loudly instead of silently returning zero batches (indistinguishable from "refreshed successfully, no data" for every calling table) when the subscription list can't be determined at all; a well-formed empty result (zero subscriptions in scope) is treated as legitimate and passes through unchanged
+		      SubscriptionIds =
+		        if Subscriptions = null or not Table.HasColumns(Subscriptions, "subscriptionId") then
+		          error Error.Record("DataSource.Error", "Unable to determine the list of subscriptions to query from Azure Resource Graph. Verify Resource Graph access and connectivity, then refresh again.")
+		        else
+		          Subscriptions[subscriptionId],
+		      BatchCount = Number.RoundUp(List.Count(SubscriptionIds) / batchSize),
+		      Batches = if BatchCount <= 0 then {} else List.Transform({0..BatchCount-1}, each
+		        let
+		          batchStart = _ * batchSize,
+		          batchIds = List.Range(SubscriptionIds, batchStart, List.Min({batchSize, List.Count(SubscriptionIds) - batchStart})),
+		          // filter within the ARG query itself (rather than the request-level "subscriptions" scope parameter) so batching logic stays entirely in M and behaves the same for every caller regardless of query shape
+		          subFilter = " | where subscriptionId in (" & Text.Combine(List.Transform(batchIds, each "'" & _ & "'"), ",") & ")",
+		          fullQuery = query & subFilter,
+		          Source = QueryARG(fullQuery)
+		        in
+		          if Table.HasColumns(Source, "Results") and Table.RowCount(Source) = 1 then null else Source
+		      ),
+		      // some callers (for example tenant/management-group-scoped policy resources) can have a blank subscriptionId that never matches any subscription batch filter above; fetch those rows once, unbatched, when requested. ftk_DemoFilter() intentionally isn't applied here since these rows aren't tied to any specific subscription, so demo scoping can't cleanly narrow them
+		      UnscopedSource = if includeUnscoped = true then (try QueryARG(query & " | where isempty(subscriptionId)") otherwise null) else null,
+		      UnscopedFiltered = if UnscopedSource <> null and not (Table.HasColumns(UnscopedSource, "Results") and Table.RowCount(UnscopedSource) = 1) then UnscopedSource else null,
+		      Filtered = List.RemoveNulls(Batches & {UnscopedFiltered}),
+		      Combined = if List.Count(Filtered) > 0 then Table.Combine(Filtered) else null
+		    in
+		      Combined
+	lineageTag: b4bb963c-20fd-4e46-8c14-7ecc59559bf7
+	queryGroup: Functions
+
+	annotation PBI_NavigationStepName = Navigation
+
+	annotation PBI_ResultType = Function
+
 /// URI of the FinOps hub Azure Data Explorer cluster to pull data from. Copy from the "clusterUri" deployment output.
 /// 
 /// Pro tip: As a shortcut, you can alternatively use the "{name}.{region}" subset of the URI.
@@ -114,14 +180,15 @@ expression PolicyDefinitions =
 		  query = "
 		  policyResources
 		  | where type =='microsoft.authorization/policydefinitions'
-		  | extend displayName = properties.displayName,
+		  | extend
+		      subscriptionId = iff(tolower(tostring(split(id, '/')[1])) == 'subscriptions', tostring(split(id, '/')[2]), ''),
+		      displayName = properties.displayName,
 		      description = properties.description,
 		      version = properties.version
-		    | project subscriptionId, id, name, displayName, description, version"
-		  & ftk_DemoFilter(),
-		  Source = AzureResourceGraph.Query(query, "Tenant", null, null, [resultTruncated = false]),
+		    | project subscriptionId, id, name, displayName, description, version",
+		  Source = ftk_QueryARG(query, 5, true),
 		  NullHandling =
-		    if Table.HasColumns(Source, "Results") and Table.RowCount(Source) = 1 then
+		    if Source = null then
 					#table(
 						{ "subscriptionId", "id", "name", "displayName", "description", "version" },
 						{}

src/power-bi/kql/Shared.Dataset/definition/tables/AdvisorRecommendations.tmdl

@@ -475,10 +475,8 @@ table AdvisorRecommendations
 		queryGroup: 'Resource Graph'
 		source = ```
 				let
-				  Source = AzureResourceGraph.Query("
-
-
-				  advisorresources 
+				  query = "
+				  advisorresources
 				  | where type == 'microsoft.advisor/recommendations'
 				  | where properties.category == 'Cost'
 				  | extend
@@ -523,11 +521,10 @@ table AdvisorRecommendations
 				    ['properties.resourceMetadata.action'] = properties.resourceMetadata.action,
 				    ['properties.resourceMetadata.source'] = properties.resourceMetadata.source,
 				    SortOrder = case(properties.impact == 'High', 1, properties.impact == 'Medium', 2, properties.impact == 'Low', 3, 9)
-
-
-				  " & ftk_DemoFilter(), "Tenant", null, null, [resultTruncated = true]),
+				  ",
+				  Source = ftk_QueryARG(query, 2),
 				  NullHandling =
-				    if Table.HasColumns(Source, "Results") and Table.RowCount(Source) = 1 then
+				    if Source = null then
 							#table(
 								{ "id", "name", "type", "properties.recommendationTypeId", "properties.shortDescription", "properties.resourceMetadata", "properties.suppressionIds", "properties.impactedField", "properties.impactedValue", "properties.lastUpdated", "properties.category", "properties.metadata", "properties.impact", "properties.shortDescription.problem", "properties.shortDescription.solution", "properties.extendedProperties.ObservationPeriodStartDate", "properties.extendedProperties.ObservationPeriodEndDate", "properties.extendedProperties.annualSavingsAmount", "properties.extendedProperties.HasRecommendation", "properties.extendedProperties.savingsCurrency", "properties.extendedProperties.IsInReplication", "properties.extendedProperties.Recommended_DTU", "properties.extendedProperties.Recommended_SKU", "properties.extendedProperties.ResourceGroup", "properties.extendedProperties.savingsAmount", "properties.extendedProperties.DatabaseSize", "properties.extendedProperties.DatabaseName", "properties.extendedProperties.ServerName", "properties.extendedProperties.Region", "properties.extendedProperties.lookbackPeriod", "properties.extendedProperties.subId", "properties.extendedProperties.scope", "properties.extendedProperties.term", "properties.extendedProperties.sku", "properties.extendedProperties.commitment", "properties.extendedProperties.currentSku", "properties.extendedProperties.targetSku", "properties.extendedProperties.recommendationMessage", "properties.resourceMetadata.resourceId", "properties.resourceMetadata.singular", "properties.resourceMetadata.plural", "properties.resourceMetadata.action", "properties.resourceMetadata.source", "SortOrder" },
 								{}
@@ -540,9 +537,11 @@ table AdvisorRecommendations
 				      {"properties.extendedProperties.ObservationPeriodStartDate", type datetimezone},
 				      {"properties.extendedProperties.ObservationPeriodEndDate",   type datetimezone},
 				      {"properties.lastUpdated",                                   type datetimezone}
-				    }, "en-US")
+				    }, "en-US"),
+				  // Table.Combine concatenates per-batch results in batch order, so re-sort here to restore SortOrder ordering across the combined table
+				  Sorted = Table.Sort(Types, {{"SortOrder", Order.Ascending}})
 				in
-				  Types
+				  Sorted
 				```
 
 	annotation PBI_NavigationStepName = Navigation