Avoid reference cycles in graphs

[SimonHeybrock]

networkx caches report views such as G.edges or G.out_degree in G.__dict__, and each of those views holds a reference back to the graph. Merely looking at a graph therefore places it in a reference cycle, so it can be freed only by the cyclic garbage collector. That collector is triggered by the number of allocated objects, so a handful of discarded graphs holding large node data can survive indefinitely. Cyclebane creates and discards graphs in most of its operations, and the graph returned by to_networkx becomes self-referential as soon as a consumer inspects it, so applications that repeatedly update a graph grow without bound. This OOM-killed a 32 GB host in 34 minutes, see scipp/esslivedata#1264 and scipp/esslivedata#1266.

Avoiding the view accesses in cyclebane itself does not work: nx.relabel_nodes and nx.compose_all materialise G.edges on their inputs internally, and the graph handed to the caller by to_networkx gets views taken on it by the caller. Instead, use a nx.DiGraph subclass in which the six back-referencing views are recreated on each access. Graphs derived by networkx operations are created via G.__class__(), so all intermediates, to_networkx results, subgraphs and copies inherit this, without changes at any of the call sites.

Note that Graph.__init__ now copies a graph that is not already of this type, with the same semantics as nx.DiGraph.copy.

Each of the added tests fails without the change.

Verification

With the reproducer from scipp/esslivedata#1264 (sciline + numpy only) and the cyclic collector disabled, 60 iterations of pipeline[Chunk] = <67 MB array>; pipeline.compute(Result) now stay flat at 126 MB instead of reaching 4.1 GB. With gc.DEBUG_SAVEALL, map/reduce/groupby/to_networkx produce no cyclic garbage at all.

Cost

Creating a view costs ~0.2 us, which is measurable only for code re-accessing a view per node, e.g., [G.degree(n) for n in G] is 2x slower. Complete networkx algorithms (topological_sort, dag_longest_path, pagerank, ancestors) are unaffected within noise, since they bind G.adj once and the views that do not hold a graph reference (adj, succ, pred, nodes) are still cached. to_networkx is unchanged, __setitem__ is slightly faster.

The views are no longer identical across accesses. They compare equal where networkx defines equality for them, but the degree views do not, i.e., G.degree == G.degree is now False.

[SimonHeybrock]

networkx caches report views such as G.edges or G.out_degree in G.__dict__, and each of those views holds a reference back to the graph. Merely looking at a graph therefore places it in a reference cycle, so it can be freed only by the cyclic garbage collector. That collector is triggered by the number of allocated objects, so a handful of discarded graphs holding large node data can survive indefinitely.

@MridulS Is networkx aware of this behavior?

[MridulS]

I remember this networkx/networkx#7697, and the hope was the CPython gc should take care of it. I'll look a bit more into this.