Support `ty` for filtering mutants

[ejfine]

I (and hopefully others) would like to follow this issue to know when it's available. Being able to use a very fast type checker to filter out mutants I think will dramatically increase the utility of mutmut

I saw

With pyright and ty, mutating a class method Foo.bar() can break the types of all methods of Foo, and therefore mutmut cannot match the type error with the mutant that caused the type error.

in the Readme. But I'm not sure I understand why that would prevent using ty. Wouldn't the logic just need to be:

1. mutmut makes a mutation
2. the type checker is run
3. if the type checker throws any error, then the mutation is considered invalid
4. if no type checker error, then proceed with unit test suite to see if mutation is caught by that

I'm not sure I understand why mutmut would need to match the type error with the mutant. Would that requirement only apply to codebases where there are existing errors detected by the type checker, and therefore the type checker exiting non-zero wouldn't be able to be clearly tied to the introduced mutation?

Or if there's more nuanced logic involved in the current mypy implementation, would it be easy (and useful to others) for this simpler flow to exist?

[Otto-AA]

Hi, the problem is that mutmut mutates the whole code base at once and later on tests individual mutants by setting an environment variable that enables this single mutation while all other mutations are dormant. This is useful for performance reasons (we can import the code with all mutations once, and then fork the process. So we don't need to import the code for each mutant).

For example take the following class:

class Foo:
  def __init__(self):
    self.x = 2

  def increment(self):
    self.x += 1

Mutmut mutates it to the following code:

from mutmut.mutation.trampoline import wrap_in_trampoline as _mutmut_mutated, MutantDict
mutants_xǁFooǁ__init____mutmut: MutantDict = {}  # type: ignore
mutants_xǁFooǁincrement__mutmut: MutantDict = {}  # type: ignore
class Foo:
  @_mutmut_mutated(mutants_xǁFooǁ__init____mutmut)
  def __init__(self):
    self.x = 2
  def xǁFooǁ__init____mutmut_orig(self):
    self.x = 2
  def xǁFooǁ__init____mutmut_1(self):
    self.x = None
  def xǁFooǁ__init____mutmut_2(self):
    self.x = 3

  @_mutmut_mutated(mutants_xǁFooǁincrement__mutmut)
  def increment(self):
    self.x += 1

  def xǁFooǁincrement__mutmut_orig(self):
    self.x += 1

  def xǁFooǁincrement__mutmut_1(self):
    self.x = 1

  def xǁFooǁincrement__mutmut_2(self):
    self.x -= 1

  def xǁFooǁincrement__mutmut_3(self):
    self.x += 2

mutants_xǁFooǁ__init____mutmut['_mutmut_orig'] = Foo.xǁFooǁ__init____mutmut_orig # type: ignore # mutmut generated
mutants_xǁFooǁ__init____mutmut['xǁFooǁ__init____mutmut_1'] = Foo.xǁFooǁ__init____mutmut_1 # type: ignore # mutmut generated
mutants_xǁFooǁ__init____mutmut['xǁFooǁ__init____mutmut_2'] = Foo.xǁFooǁ__init____mutmut_2 # type: ignore # mutmut generated

mutants_xǁFooǁincrement__mutmut['_mutmut_orig'] = Foo.xǁFooǁincrement__mutmut_orig # type: ignore # mutmut generated
mutants_xǁFooǁincrement__mutmut['xǁFooǁincrement__mutmut_1'] = Foo.xǁFooǁincrement__mutmut_1 # type: ignore # mutmut generated
mutants_xǁFooǁincrement__mutmut['xǁFooǁincrement__mutmut_2'] = Foo.xǁFooǁincrement__mutmut_2 # type: ignore # mutmut generated
mutants_xǁFooǁincrement__mutmut['xǁFooǁincrement__mutmut_3'] = Foo.xǁFooǁincrement__mutmut_3 # type: ignore # mutmut generated

Pyrefly reports the error in the xǁFooǁ__init____mutmut_1 method (where we mutated 2 to None, so the error is expected): "ERROR sandbox.py:11:14-18: None is not assignable to attribute x with type int [bad-assignment]". We use the reported line number to detect the invalid type in the __init___mutmut_1 mutant.

Ty reports errors in increment, xǁFooǁincrement__mutmut_2 and xǁFooǁincrement__mutmut_3, even though we didn't change anything type related there: "Operator += is not supported between objects of type None and Literal[1]". We would use the the reported line numbers and skip these increment mutations (or in fact directly error because increment has a type error, which is not even mutated code).

The difference is that pyrefly infers self.x as int because the first assignment assigns an integer, while Ty infers self.x as int | None because it looks at all assignments in the class, including the self.x = None assignment. So for Ty, the mutation of the __init__ method causes a type error in the increment methods.

Here are the playground links:

• Pyrefly
• Ty

I don't think this is fixable for Ty when we keep mutating multiple methods within a single class. So we'd need to create a copy of a class for each mutation, or think of another workaround.

Or if there's more nuanced logic involved in the current mypy implementation, would it be easy (and useful to others) for this simpler flow to exist?

We probably could do something like this: apply a single mutation, check if ty complains, undo the single mutation. Apply next mutation, check if ty complains, undo the mutation. etc.

I'm not sure if that's worth the effort tbh.

Being able to use a very fast type checker to filter out mutants I think will dramatically increase the utility of mutmut

Pyrefly is already a pretty fast type checker. If your code type checks against both, Ty and Pyrefly, you can still use Pyrefly for mutmut while using Ty as a LSP and main type checker in your CI.

[ejfine]

Thanks for the detailed explanation. Definitely something to chew on 🤔

pyrefly wasn't on my radar as a serious contender...I had just been eagerly awaiting ty being stable enough to switch to that from pyright (since ruff and uv from Astral turned out to be so awesome), but I will look into pyrefly in more detail

[ejfine]

I tried out pyrefly today, but started running into mutmut telling me to file github issues. so I did: #522