docs: define applicability end states [skip-ci]

Author: yoshifuminakamura

docs/cx/ESTIMATE_JSON_SPEC.md

@@ -377,10 +377,25 @@ This field may include assumptions such as:
 
 フォールバックが行われた場合、`applicability` には、要求されたパッケージをそのまま適用できなかった理由を記録できることが望ましい。
 その場合、`estimate_metadata.requested_estimation_package` は最初に要求されたパッケージを、`estimate_metadata.estimation_package` は実際に適用されたパッケージを表す。
+また、`applicability.status` は Estimate JSON の最終状態を表す値として扱う。
+少なくとも以下の 4 値を標準値として扱う。
+
+- `applicable`
+  - 要求された推定パッケージで、そのまま推定が成立した
+- `partially_applicable`
+  - 推定全体は成立したが、一部の section / overlap / component でフォールバックが行われた
+- `fallback`
+  - 要求された top-level 推定パッケージでは成立せず、別の top-level パッケージへ切り替えて推定が成立した
+- `not_applicable`
+  - 推定は試みられたが、最終的に推定結果として成立しなかった
+
+`not_applicable` はパイプライン失敗を意味しない。
+BenchKit は、推定不成立であっても、その試行結果を Estimate JSON として保存・表示してよい。
+
 When fallback occurs, `applicability` should preferably record why the requested package could not be applied.
 In such a case, `estimate_metadata.requested_estimation_package` identifies the originally requested package, while `estimate_metadata.estimation_package` identifies the package actually applied.
 
-This field records whether the requested estimation method had sufficient inputs, whether fallback was used, and what was missing.
+This field records the final applicability state of the estimate, whether fallback was used, and what was missing.
 
 ### 6.6 confidence
 
@@ -580,7 +595,6 @@ incrementally over time.
 
 - `measurement` の具体キー集合
 - `model` の詳細体系
-- `applicability.status` の標準値集合
 - `confidence` の尺度
 - `assumptions` の標準辞書
 - section / overlap category の標準語彙
@@ -592,7 +606,6 @@ This document does not yet fix:
 
 - the complete key set of `measurement`
 - the detailed taxonomy of `model`
-- the standard value set of `applicability.status`
 - the scale for `confidence`
 - a standard dictionary for `assumptions`
 - a standard vocabulary for section / overlap categories

docs/cx/ESTIMATION_PACKAGE_SPEC.md

@@ -324,18 +324,24 @@ It should preferably be able to express at least:
 少なくとも以下を返せることが望ましい。
 
 - 適用可能
-- フォールバック適用可能
+- フォールバック
 - 不適用
-- 再計測または追加準備が必要
+- 具体的な不足入力または必要な準備
 
 The package must be able to evaluate whether it is applicable to the provided inputs.
 
 It should preferably be able to return at least:
 
 - applicable
-- applicable with fallback
+- fallback
 - not applicable
-- re-measurement or additional preparation required
+- a concrete list of missing inputs or required actions
+
+ここでの `fallback` は、当該 package 自身がより軽量な package または component に切り替えることで継続できることを意味する。
+最終的な Estimate JSON では、この package-level fallback は、部分区間だけで起きたなら `partially_applicable`、top-level package の切り替えとして起きたなら `fallback` として現れうる。
+
+At the package level, `fallback` means that the package itself can continue by switching to another lighter-weight package or component.
+At the final Estimate JSON level, a package-level fallback may appear either as `partially_applicable` or as top-level `fallback`, depending on whether the switch happened only inside one part of the estimate or at the top-level package boundary.
 
 ### 4.5 フォールバック方針 / Fallback Policy
 
@@ -344,6 +350,8 @@ It should preferably be able to return at least:
 
 The package should preferably be able to define which lighter-weight method may be used as fallback when required inputs are missing, or whether execution must stop without fallback.
 When fallback is used, the requested package identity should remain visible separately from the actually applied package identity.
+When no acceptable fallback exists, the package may return `not_applicable`.
+In that case, BenchKit may still store and present the resulting record as a `not_applicable` estimate attempt rather than treating it as a pipeline failure.
 
 ### 4.6 Estimate JSON への写像 / Mapping into Estimate JSON
 

docs/cx/ESTIMATION_SPEC.md

@@ -450,20 +450,27 @@ More detailed methods may instead use an explicit model, trace, counters, or int
 ### 4.5 適用可能性判定 / Applicability Evaluation
 
 推定実行前に、対象方式が必要とする入力が揃っているかを判定できることが望ましい。
-この判定は、少なくとも以下を返せることが望ましい。
+この判定は、少なくとも以下の最終状態を区別できることが望ましい。
 
 - 適用可能
-- フォールバック適用可能
+- 部分適用可能
+- フォールバック
 - 不適用
-- 再計測または追加準備が必要
 
 BenchKit should preferably be able to evaluate whether the required inputs for a chosen estimation method are present before execution.
-This evaluation should preferably be able to return at least:
+This evaluation should preferably be able to distinguish at least the following final states:
 
 - applicable
-- applicable with fallback
-- not applicable
-- re-measurement or additional preparation required
+- partially_applicable
+- fallback
+- not_applicable
+
+`partially_applicable` means that the estimate succeeded overall, but fallback was used for part of the section / overlap / component chain.
+`fallback` means that the requested top-level package did not succeed as requested, and the estimate succeeded only after switching to another top-level package.
+`not_applicable` means that estimation was attempted but did not succeed as an estimate result.
+
+`not_applicable` does not necessarily imply pipeline failure.
+BenchKit may still preserve and present a `not_applicable` estimate result as a record of an estimation attempt.
 
 applicability 評価が fallback を返す場合、少なくとも次を識別できることが望ましい。
 
@@ -477,6 +484,8 @@ When applicability evaluation returns fallback, it is desirable that the result
 - the actually applied fallback package
 - the concrete reason the requested package could not be applied
 
+When only part of the section / overlap / component chain falls back while the overall estimate still succeeds, it is desirable that this be surfaced as `partially_applicable` at the top level.
+
 ### 4.6 履歴と再推定 / History and Re-Estimation
 
 推定は一度きりの計算ではなく、モデルや仮定の更新に応じて再推定できることが重要である。

programs/qws/run.sh

@@ -59,7 +59,7 @@ print_results() {
     create_dummy_estimation_artifact "estimation_inputs/halo_exchange_trace.tgz" "dummy mpi trace archive for halo_exchange"
     create_dummy_estimation_artifact "estimation_inputs/allreduce_trace.tgz" "dummy collective trace archive for allreduce"
     create_dummy_estimation_artifact "estimation_inputs/write_result_interval.json" "{\"section\":\"write_result\",\"kind\":\"interval_time\"}"
-    # Temporary CI check for not_applicable: leave overlap artifact missing.
+    create_dummy_estimation_artifact "estimation_inputs/compute_halo_overlap.json" "{\"overlap\":[\"compute_hopping\",\"halo_exchange\"],\"kind\":\"overlap_time\"}"
 
     bk_emit_section prepare_rhs "$section_prepare_rhs" interval_time_simple results/estimation_inputs/prepare_rhs_interval.json
     bk_emit_section compute_hopping "$section_compute_hopping" counter_papi_detailed results/estimation_inputs/compute_hopping_papi.tgz