Trailing whitespace changes are made outside of --lines

[dmytrokyrychuk]

Description

Trailing whitespace is removed from the whole file when attempting to format a subset of a file.

Steps to reproduce

1. pip install yapf==0.23

2. Run the following script:

from yapf.yapflib.yapf_api import FormatCode

source = """\
def with_trailing_whitespace():  
    pass  

def without_trailing_whitespace(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, bbbbbbbbbb):
    return aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, bbbbbbbbbb
"""

reformatted_diff, changed = FormatCode(source, lines=[(4, 5)], print_diff=True)
print(reformatted_diff)

Expected output

--- <unknown>   (original)
+++ <unknown>   (reformatted)
@@ -1,5 +1,7 @@
 def with_trailing_whitespace():
     pass

-def without_trailing_whitespace(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, bbbbbbbbbb):
+
+def without_trailing_whitespace(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,
+                                bbbbbbbbbb):
     return aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, bbbbbbbbbb

Actual output

--- <unknown>   (original)
+++ <unknown>   (reformatted)
@@ -1,5 +1,7 @@
-def with_trailing_whitespace():
-    pass
+def with_trailing_whitespace():
+    pass

-def without_trailing_whitespace(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, bbbbbbbbbb):
+
+def without_trailing_whitespace(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa,
+                                bbbbbbbbbb):
     return aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, bbbbbbbbbb

[braineo]

I also ran into this issue.

[wwade]

It appears to be a problem in the PyTreeVisitor implmentation. When building the uwlines for the file, for example with_trailing_whitespace(): will lose the trailing whitespace as part of the Visit_suite method:

 def Visit_suite(self, node):  # pylint: disable=invalid-name
   # A 'suite' starts a new indentation level in Python.
   self._cur_depth += 1
   self._StartNewLine()
   self.DefaultNodeVisit(node)
   self._cur_depth -= 1

Here, node.children[ 0 ] will have a value of \n and a prefix of the trailing whitespace contents, but those trailing whitespace contents will not be added to the _cur_unwrapped_line before finalizing it in _StartNewLine().

[SkyeNygaard]

This is still an issue and it is preventing me from using yapf on my projects.

[limwz01]

@SkyeNygaard @braineo @dmytrokyrychuk

I've created a fork (PR 1102) that fixes this bug, but it needs extensive testing as making this right is quite difficult. It still wouldn't fix the test case in the initial post, as the whitespace before line 4 will be changed, but at least majority of the issue is fixed.

For convenience, here's the patch: yapf_restrict_lines.diff.txt