ListField.run_child_validation does not skip empty child fields

[realsuayip]

In:

django-rest-framework/rest_framework/fields.py

Line 1723 in b92edf5

result.append(self.child.run_validation(item))

When a child of a list filter raises SkipField, the exception is propagated and entire list is discarded, even if there is valid entries.

Following test does not pass because run_validation raises SkipField for entire list:

def test_listfield_skips_empty_child(self):
    from rest_framework.fields import empty

    f = serializers.ListField(child=serializers.CharField(required=False))
    t = f.run_validation(["hello", empty])
    assert t == ["hello"]

adding a SkipField check like so fixes this and does not break any tests:

diff --git a/rest_framework/fields.py b/rest_framework/fields.py
index 9b922ed9..a8c8895a 100644
--- a/rest_framework/fields.py
+++ b/rest_framework/fields.py
@@ -1725,6 +1725,8 @@ class ListField(Field):
                 errors[idx] = e.detail
             except DjangoValidationError as e:
                 errors[idx] = get_error_detail(e)
+            except SkipField:
+                continue
 
         if not errors:
             return result

A similar case probably exists for DictField too.

I wonder what are our options here. I'm designing an API around empty and SkipField interactions and this caught me off guard. I propose following:

1. Handle SkipField like the diff above. It does not break any tests but it might break some potential third party code.
2. Handle skip field conditionally by adding a new option to ListField.__init__ (something like allow_empty_child)
3. Do nothing and document this behavior.

[identity-wael]

There are two semantics to separate here: skipping an absent serializer field, and removing a supplied list element. empty represents missing input internally; it isn't a value a JSON client can send (null is None, which follows a different validation path).

Your example reaches child.get_default(), which raises SkipField when the non-required child has no default. The current ListField implementation catches validation exceptions but not SkipField, so it escapes to the containing serializer, where skipping the whole field is the normal interpretation. That explains the result without implying it is the best behaviour for your custom API.

For application code, I would make element omission explicit in a custom field or normalize your internal sentinel values before handing the list to DRF. If you implement a run_child_validation() override with the extra catch, retain the existing per-index ValidationError/DjangoValidationError aggregation. Catching SkipField around the entire super().run_child_validation() call is too late: it loses the accumulated valid elements.

For an upstream change, useful regression cases beyond the one above would be:

• All children skipped, including allow_empty=False and a nonzero min_length.
• A skipped child followed by an invalid child: does the error retain its original input index?
• A child with a default, and a list on a partial=True serializer.
• A custom child that deliberately raises SkipField, plus the analogous DictField case.

In particular, filtering inside run_child_validation() occurs after ListField's initial empty-input check, so “all skipped” can produce an empty result even when the original nonempty input passed that check. That policy needs defining alongside the exception catch. An opt-in field gives your API the behaviour now without depending on a change to DRF's global omission semantics.

[realsuayip]

Filtering out empty sentinel before using ListField is problematic, because
to retrieve 'required' errors empty sentinel needs to present on a field with
required=True.

Now, I could introspect the child serializer and only add empty values in case
it is required, but that would defeat the purpose for me since I want my
implementation to be generic around run_validation.

Additionally, allowing empty values for non-required fields keeps the index
of error message correct. For example if second element of this list were to be
invalid: [empty, "invalid-value"] you would expect error message to mention
index 1. If I had omitted empty, it would be 0.

Anyway, for now I'll probably export a subclass of ListSerializer that
works as I intended.

Besides, again, I think it's at least 'surprising' to discard whole list in case of
a skip. I feel like upstream impact will be limited because as you have said,
user input alone cannot trigger this type of skip. Some field implementation
needs to raise SkipField.

[mmaxjr]

I ran this against DRF master (3.18.1, commit 41764f1) and applied your diff to both ListField.run_child_validation and DictField.run_child_validation. No existing test changes behaviour: the 715 tests in test_fields.py, test_serializer.py, test_validators.py, test_model_serializer.py, test_generics.py, test_renderers.py, tests/authentication and tests/generic_relations pass identically before and after. What follows is what I'd want settled before merging, rather than just the patch.

The case your diff doesn't cover is allow_empty

The three options are about different things. Skipping a list item is exactly what Serializer.to_internal_value already does for its own fields, so the list loop treating SkipField differently from run_validation is the inconsistency, and I'd go with option 1. But filtering happens inside run_child_validation, which runs after the empty check:

def to_internal_value(self, data):
    if is_html_input(data):
        data = html.parse_html_list(data, prefix=self.field_name, default=empty)
    if not isinstance(data, list):
        self.fail('not_a_list', input_type=type(data).__name__)
    if not self.allow_empty and len(data) == 0:
        self.fail('empty')
    return self.run_child_validation(data)

allow_empty is only checked against the incoming list, so once items can disappear the invariant it protects is no longer what it validates:

case	result with the patch
[empty] with allow_empty=True	[]
outer allow_empty=False, inner ListField all skipped	[[]], outer check passed because the incoming list had one element
min_length=2, ["a", SkipField]	min_length error

min_length and max_length are registered as field validators in ListField.__init__ and run against the input length in Field.run_validation, so they see the pre-filter length while allow_empty sees nothing. Whichever way "all children skipped" is meant to resolve, the check belongs at the end of run_child_validation, after the loop:

if not self.allow_empty and not result:
    self.fail('empty')

with a DictField counterpart. I'd rather that decision be explicit in a test than be implied by allow_empty's current name.

On the in-tree producers of empty

Worth knowing before deciding whether this is reachable without custom code. SkipField from Field.run_validation only happens when the field itself is absent from the input: data is empty, or partial=True on the root. Neither applies to a list item. I checked each path:

• null is None, which goes through the allow_null branch and never reaches get_default(). ListField(child=CharField(required=False, allow_null=True)).run_validation(["hello", None]) returns ['hello', None].
• A required child raises ValidationError({1: 'This field is required.'}), not SkipField, so the patch doesn't weaken required-child validation.
• A child with an explicit default returns it: ListField(child=CharField(default='D')).run_validation(['hello', empty]) is ['hello', 'D'] either way.
• A read_only child already disappears silently, via validate_empty_values returning get_default(); with the patch ListField(child=CharField(read_only=True)).run_validation(['hello']) returns [] instead of raising. Nobody does that on purpose, but it shows the loop is already dropping results.
• HTML form input yields '' for a blank key, not empty: QueryDict('x=a&x=').getlist('x') is ['a', ''].
• partial=True on the root: ListField.get_value returns empty for the whole list, and validate_empty_values short-circuits before the loop.
• A nested Serializer as child= doesn't produce it either. Child(partial=True).run_validation({}) returns {}, not SkipField.

So with stock fields there is no in-tree way to get empty into a ListField item. It's your custom child raising SkipField, or a custom get_value returning empty per item. That doesn't argue against the fix, since Serializer.to_internal_value sets the precedent, but it does mean this is currently undefined behaviour rather than a bug a client can hit, which is why I'd lean towards adding the regression cases in the PR rather than treating it as urgent.

Regression coverage

class Skipping(serializers.CharField):
    def run_validation(self, data=empty):
        if data == "skipme":
            raise SkipField()
        if data == "boom":
            self.fail("invalid")
        return super().run_validation(data)


def test_listfield_skips_empty_child():
    f = serializers.ListField(child=serializers.CharField(required=False))
    assert f.run_validation(["hello", empty]) == ["hello"]


def test_listfield_error_index_is_preserved():
    f = serializers.ListField(child=Skipping())
    with pytest.raises(serializers.ValidationError) as exc:
        f.run_validation(["skipme", "skipme", "boom"])
    assert exc.value.detail == {2: [ErrorDetail("Not a valid string.", code="invalid")]}


def test_required_child_still_errors_on_empty():
    f = serializers.ListField(child=serializers.CharField())
    with pytest.raises(serializers.ValidationError) as exc:
        f.run_validation(["hello", empty])
    assert exc.value.detail == {1: [ErrorDetail("This field is required.", code="required")]}


def test_child_with_default_is_unaffected():
    f = serializers.ListField(child=serializers.CharField(default="D"))
    assert f.run_validation(["hello", empty]) == ["hello", "D"]


def test_dictfield_skips_empty_child():
    d = serializers.DictField(child=serializers.CharField(required=False))
    assert d.run_validation({"a": "x", "b": empty}) == {"a": "x"}

The index case is the one that matters most for correctness: errors[idx] is keyed by input position, so a skipped element must not shift the indices of errors that follow it. continue before errors[idx] = ... preserves that; using a separate list and zipping would not.

For the application side in the meantime

Overriding run_child_validation in a subclass gets the behaviour without depending on the patch, and lets you make the all-skipped policy explicit:

class SkippingListField(serializers.ListField):
    def run_child_validation(self, data):
        result, errors = [], {}
        for idx, item in enumerate(data):
            try:
                result.append(self.child.run_validation(item))
            except serializers.ValidationError as exc:
                errors[idx] = exc.detail
            except SkipField:
                continue
        if errors:
            raise serializers.ValidationError(errors)
        return result

For your use case I'd still normalise the sentinel before handing the list to DRF, so the API contract doesn't depend on how DRF resolves a missing field. empty inside a list element is a SkipField by accident of get_default() semantics, not a documented statement that a list element may be absent.

Environment: DRF master (3.18.1, commit 41764f1), Django 5.2, Python 3.11, SQLite. No database schema beyond Django's test setup.