next[dace]: `track_view` raises on a lowering produced View, and `reconfigure_dataflow_after_rerouting` asserts on an uninitialised Memlet

[havogt]

Two traps in runners/dace/transformations/utils.py that a caller cannot reasonably anticipate. Neither is reachable from the pipeline today, so this is about robustness of shared helpers rather than a live defect.

1. track_view raises on a View produced by the lowering

A View created by gtir_dataflow.LambdaToDataflow._construct_local_view lives inside a Map scope, is fed through a MapEntry, and carries no views connector (dst_conn=None). dace.sdfg.utils.get_view_edge cannot then determine the direction:

RuntimeError: Failed to determine the direction of the view 'view_gtir_tmp_8' |
  map_5_fieldop[...]:OUT_gtir_tmp_8 -(gtir_tmp_8[...])-> view_gtir_tmp_8:None.

track_view is the natural helper for "is this data viewed", and this is the shape the lowering actually emits — so the first thing a caller tries fails on the case that matters. Worse, catching the exception and treating it as "not a view" gives a false negative rather than an error.

_gt_modify_strides_of_views_non_recursive (strides.py:650+) calls track_view unguarded. In gt_auto_optimize that code runs after RemovePointwiseViews, so it is not reachable there at present — but the non-transient skip at strides.py:685-690 can consequently only ever apply to properly connected views, which is worth knowing when reasoning about that function.

2. reconfigure_dataflow_after_rerouting asserts on an uninitialised Memlet

utils.py:542, assert subset_to_adjust is not None. reroute_edge and reconfigure_dataflow_after_rerouting read the raw src_subset/dst_subset properties, which return None when Memlet._is_data_src was never set. add_mapped_tasklet sets it to False, which is why every producer built the normal way works, and why this only shows up on a hand constructed edge.

Not a defect in the helpers' intended use, but an unhelpful failure mode for anyone assembling an SDFG programmatically — an assertion deep inside a helper rather than a message about the input. It is equally reachable from GT4PyMapBufferElimination.

Suggestion

For (1), either make track_view return a definite "cannot determine" result instead of raising, or document that it only accepts properly connected views and provide something that answers the "is this data viewed" question for the lowering's own shapes. For (2), a clearer error, or try_initialize in the helper.

Found while adding a view rejection to GT4PyWriteBackBufferElimination (#2763) — track_view was the obvious tool and could not be used, so the check resolves the name from the view edge instead.

[philip-paul-mueller]

I agree with the proposed solution for (1), but think that the lowering also needs to change, Views are evil.

Regarding (2), this is indicate to my that the Memlet/edge was not constructed in a proper way.
I remember that this is also influenced by the order in which the edges are constructed.
In general this problem will go away if the edges are added from "outside to inside".
However, I agree that try_initialize() should be explicitly be called or Memlet.get_{src, dst}_subset() should be used, while I favour the first approach.

[havogt]

Correction to the second item: I filed it as reachable only from a hand constructed edge. It is reachable from icon4py, and it aborted a run.

File ".../transformations/splitting_tools.py", line 987, in _perform_node_split_with_bypass_impl
    gtx_transformations.utils.reconfigure_dataflow_after_rerouting(
File ".../transformations/utils.py", line 543, in reconfigure_dataflow_after_rerouting
    subset_to_adjust.offset(ss_offset, negative=False)
AttributeError: 'NoneType' object has no attribute 'offset'

The edge is created inside the split itself: _perform_node_split_with_bypass_impl() builds dace.Memlet(data=..., subset=..., other_subset=...) directly, so no side is associated with it and src_subset returns None. gt_split_access_nodes() runs try_initialize() over every edge before it starts, citing the same dace issue, but that happens before this edge exists.

Two things worth recording beyond the immediate cause.

The assert subset_to_adjust is not None on the line above did not fire — the ICON runs execute with assertions disabled, so what surfaced was an AttributeError deep in a helper rather than the message that was written to explain it. That is an argument for a real exception rather than an assert at that spot, independent of whether the callers are fixed.

And it stayed hidden because the unit tests build their Memlets through add_mapped_tasklet(), which sets _is_data_src. Every test passed both before and after the defect was introduced; only a full model run surfaced it.

I have fixed the caller by initializing the new edge where it is created. The helper side is still worth considering, since GT4PyMapBufferElimination reaches the same code.

[philip-paul-mueller]

Actually, your statement is strange.
As you can see the Memlet is directly constructed but it is an unnamed object (an rvalue in C speak) and it it is used as data argument for a add_edge() call, which calls try_initialize().
The fact that it is uninitialized means that the call failed, which possibly indicates the the order in which it was constructed is wrong.
You could also try to modify how _perform_node_split_with_bypass_impl() works, i.e. first deleting the old edge and only then add the new one.

See also: #2764 (comment)

[havogt]

Correction to my previous comment: the edge I blamed is not the one that is broken.

The Memlet without a src_subset is not the edge that _perform_node_split_with_bypass_impl() creates, it is a child of it in the Memlet tree — an edge from the MapEntry into a NestedSDFG that was already in the graph. It was corrupted earlier, by unsqueeze_memlet() during Memlet propagation:

actual_result = Memlet.from_memlet(internal_memlet)   # copies _is_data_src
actual_result.data = external_memlet.data
...
else:
    actual_result.subset = result.subset

from_memlet() copies _is_data_src from the Memlet inside the nested SDFG, where it describes a different data container, and then data is replaced by the external one. If the internal Memlet has no other_subset — a write expressed in terms of its destination, so _is_data_src is False, which is correct inside — the outer Memlet ends up claiming its data is on the destination side while carrying no other_subset. src_subset is None from then on.

Reproducer without gt4py, on dace 2.0.0a5:

import dace
from dace.sdfg import propagation

inner = dace.SDFG("inner")
inner.add_scalar("__arg", dace.float64)
inner.add_scalar("__output", dace.float64)
istate = inner.add_state()
istate.add_nedge(
    istate.add_access("__arg"),
    istate.add_access("__output"),
    dace.Memlet(data="__output", subset="0"),
)

sdfg = dace.SDFG("outer")
sdfg.add_array("A", [10], dace.float64)
sdfg.add_array("B", [10], dace.float64)
st = sdfg.add_state()
a_node, b_node = st.add_access("A"), st.add_access("B")

map_entry, map_exit = st.add_map("map", {"__i": "0:10"})
nsdfg = st.add_nested_sdfg(sdfg=inner, inputs={"__arg"}, outputs={"__output"})

map_entry.add_in_connector("IN_A")
map_entry.add_out_connector("OUT_A")
st.add_edge(a_node, None, map_entry, "IN_A", dace.Memlet(data="A", subset="0:10"))
st.add_edge(map_entry, "OUT_A", nsdfg, "__arg", dace.Memlet(data="A", subset="__i"))

map_exit.add_in_connector("IN_B")
map_exit.add_out_connector("OUT_B")
st.add_edge(nsdfg, "__output", map_exit, "IN_B", dace.Memlet(data="B", subset="__i"))
st.add_edge(map_exit, "OUT_B", b_node, None, dace.Memlet(data="B", subset="0:10"))
sdfg.validate()

edge = next(iter(st.in_edges(nsdfg)))
print("before:", edge.data, edge.data._is_data_src, edge.data.src_subset)
propagation.propagate_memlets_map_scope(sdfg, st, map_entry)
edge = next(iter(st.in_edges(nsdfg)))
print("after: ", edge.data, edge.data._is_data_src, edge.data.src_subset)

before: A[__i] True __i
after:  A[__i] False None

So your guidance holds — this is a Memlet that has gone bad, and try_initialize() does repair it, since the association is recoverable from the graph. What matters is when: the repair has to happen while the graph is still valid. reconfigure_dataflow_after_rerouting() cannot do it, it runs when the old and the new edge are both present.

It is also not enough to guard the propagation calls we make ourselves. PatternTransformation.apply_pattern() runs propagate_memlets_sdfg() after every applied transformation that does not annotates_memlets(), so the Memlets can go bad again between two applications.

Fixed in #2764 by re-initializing, at the entry of split_node(), the Memlet trees of the edges that are about to be rerouted, with a regression test that reproduces the assertion in ~3 s.

The first item stands as filed. The second one is really a DaCe defect: unsqueeze_memlet() should not carry _is_data_src across the SDFG boundary.

Independent of the cause, the assert subset_to_adjust is not None is worth turning into a real exception. The ICON runs execute with assertions disabled, so what surfaced there was AttributeError: 'NoneType' object has no attribute 'offset' from inside a helper, not the message that was written to explain it.

[philip-paul-mueller]

• Turning the assert into an exception is actually a good idea.
• We should also keep the get_src_subset() regardless since there are other places in DaCe where Memlets are not properly handled, although it increases runtime.
• I think this issue should be fixed in DaCe do you have any suggestions?