Tentative testdriver methods

[jcsteh]

Rendered.
CC @zcorpan, @cookiecrook.

[jcscottiii]

I agree that the tests for these tentative methods should be marked with tentative. But I think we should treat these tentative methods similarly to how we treat tentative tests for the sake of consistency. Given tentative tests do not require consensus currently, I would remove this requirement from tentative methods.

[gsnedders]

I'd argue that tentative testdriver API pose a greater risk, because removing tests that rely on undocumented API is painful and hard — see #172.

[cookiecrook]

Both the tests and the method name will be clearly marked as tentative until in the spec with two implementations.

[cookiecrook]

Update in November 2025 from @jcsteh and @zcorpan... methods will not be named tentative, but they will throw an assertion until the tentative status is removed.

[gsnedders]

Most of my comments below are really "let's make this more explicit", rather than any real objection.

And mostly a question for the Core Team, but I do wonder what standard we thing we're holding things to based on #127: is an explainer enough for a testdriver API, or do we want a spec? Because if an explainer is enough, I'd question how often we'd actually add tentative API, because we probably do want something defining what the behaviour of the endpoint is, even if only at a high-level.

[gsnedders]

Let's be more explicit; we already have plenty of non-tentative things in testdriver which are not in either spec — #127 (final RFC) merely requires a WebDriver endpoint, not that it is in one of those specs.

[jcsteh]

Ah. I had assumed that the existence of a WebDriver endpoint implied it needed to be in the WebDriver spec. So how is WebDriver endpoint defined, given that Gecko uses Marionette for WPT and no other vendors implement this WebDriver endpoint yet?

[gsnedders]

For example, https://webbluetoothcg.github.io/web-bluetooth/#automated-testing defines a whole load of endpoints.

The policy is that it requires a spec for the endpoint, not that that spec be in any specific standard.

[jcsteh]

Thanks. Now I better understand your original question regarding whether an explainer is sufficient. I agree that at least having an explainer makes sense, so if the consensus is that an explainer is acceptable, we probably don't need this RFC. RFC 127 just says "an existing WebDriver endpoint", which isn't explicit about a formal spec, though I guess that could be inferred. So perhaps we still need an RFC making it clear that an explainer is sufficient.

That said, one difference between a spec and an explainer is that an explainer is far more likely to change than a spec. So that might still be one reason to mark these methods as tentative: to indicate that the details are still being ironed out.

[gsnedders]

For things like the Accessibility Investigation, is there any intention for there to be any, even high-level, explainer of what the proposed endpoints are? I'm mostly concerned about the risk of ending up somewhere where it is hard for anyone else to start passing the tests written using the tentative endpoint.

[jcsteh]

There is WICG/aom#203. However, that's an evolving issue, not a document. I could create an explainer document if that's what we need, though.

[cookiecrook]

Goal is to have:

• the original problem description and original proposal explainer
• this RFC
• two implementations (nearly there in Gecko and WebKit)
• tentative tests in WPT WebDriver and WPT ARIA
• and a TBD WebDriver spec update

I don't think we need another explainer... We just need to work out the remaining differences and make sure they are clearly documented in the relevant locations.

[jcsteh]

One open question here is whether we need this RFC at all or whether an explainer is sufficient. See #226 (comment). If an explainer is sufficient, is it sufficient for that explainer to be a GitHub issue (as is the case now) or does it need to be a document (to make it easier to follow)?

[gsnedders]

I'd argue that tentative testdriver API pose a greater risk, because removing tests that rely on undocumented API is painful and hard — see #172.

[Ms2ger]

I have no fundamental concerns with this, assuming we'd use this for experimentation, not things that stick around indefinitely.

[jcscottiii]

We discussed this today during the RFC meeting. I'm still opposed to the wording in Detail 1. But the rest of the RFC looks fine. And in the absence of better alternatives for that one point, I will approve this.

[zcorpan]

@gsnedders Can you take another look at this?

[zcorpan]

An alternative to having tentative_ in the method names is to have an assertion for these methods that checks that the test itself is tentative (by checking the test's URL).

[cookiecrook]

r+ with the addition of the new assertion expectation in tentative methods.

[cookiecrook]

Most of my comments below are really "let's make this more explicit", rather than any real objection.

I think it's okay to dismiss @gsnedders review since it's 5 months old, was self-described as not a "real objection", and I believe has fully addressed in the latest iteration.