Skip to content

Commit c7497b8

Browse files
LiliDengLiliDeng
committed
enable perf_evaluation notifier
1 parent a5f099a commit c7497b8

File tree

6 files changed

+2483
-0
lines changed

6 files changed

+2483
-0
lines changed

‎docs/run_test/runbook.rst‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -959,6 +959,95 @@ Example of log_agent notifier:
959959
The AI analysis results are stored in the test result message's ``analysis["AI"]``
960960
field and can be consumed by other notifiers like HTML or custom reporting systems.
961961

962+
perfevaluation
963+
^^^^^^^^^^^^^^
964+
965+
Evaluates performance test results against predefined criteria and optionally fails tests when targets are not met.
966+
967+
**Basic Usage:**
968+
969+
.. code:: yaml
970+
971+
notifier:
972+
- type: perfevaluation
973+
criteria_file: "perf_criteria.yml"
974+
output_file: "results.json"
975+
fail_test_on_performance_failure: true
976+
977+
**Parameters:**
978+
979+
criteria_file
980+
'''''''''''''
981+
type: str, optional, default: "*_criteria.yml"
982+
983+
Path or glob pattern to YAML files containing performance criteria.
984+
985+
criteria
986+
''''''''
987+
type: dict, optional, default: None
988+
989+
Direct criteria definition in runbook. Takes priority over criteria_file.
990+
991+
output_file
992+
'''''''''''
993+
type: str, optional, default: None
994+
995+
Output path for detailed evaluation results in JSON format.
996+
997+
fail_test_on_performance_failure
998+
''''''''''''''''''''''''''''''''
999+
type: bool, optional, default: False
1000+
1001+
Mark tests as failed when performance criteria are not met.
1002+
1003+
**YAML Criteria Format:**
1004+
1005+
Hierarchical format with groups and conditions:
1006+
1007+
.. code:: yaml
1008+
1009+
# Global settings
1010+
statistics_times: 3
1011+
error_threshold: 0.1
1012+
statistics_type: average
1013+
1014+
groups:
1015+
- name: "Storage Performance"
1016+
conditions:
1017+
- name: "test_case"
1018+
type: "metadata"
1019+
value: "*fio*"
1020+
- name: "vm_size"
1021+
type: "information"
1022+
value: "Standard_D*"
1023+
1024+
metrics:
1025+
- name: "IOPS_Read"
1026+
min_value: 1000
1027+
target_value: 5000
1028+
error_threshold: 0.10
1029+
1030+
**Metric Properties:**
1031+
1032+
- ``min_value``: Minimum acceptable value
1033+
- ``max_value``: Maximum acceptable value
1034+
- ``target_value``: Expected target value
1035+
- ``error_threshold``: Acceptable deviation from target (as decimal, e.g., 0.10 = 10%)
1036+
1037+
**Pattern Matching:**
1038+
1039+
Uses fnmatch patterns:
1040+
1041+
- ``Standard_D*``: All D-series VMs
1042+
- ``*fio*``: Test cases containing "fio"
1043+
- ``Standard_L??s_v2``: L-series with specific patterns
1044+
1045+
**Condition Types:**
1046+
1047+
- ``test_case``: Match test case name
1048+
- ``vm_size``: Match VM size
1049+
- All conditions must match (AND logic)
1050+
9621051
environment
9631052
~~~~~~~~~~~
9641053

‎lisa/mixin_modules.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
import lisa.notifiers.file # noqa: F401
1616
import lisa.notifiers.junit # noqa: F401
1717
import lisa.notifiers.perfdump # noqa: F401
18+
import lisa.notifiers.perfevaluation.perfevaluation # noqa: F401
1819
import lisa.notifiers.text_result # noqa: F401
1920
import lisa.runners.lisa_runner # noqa: F401
2021
import lisa.sut_orchestrator.ready # noqa: F401

‎lisa/notifiers/perfevaluation/__init__.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT license.
3+
4+
"""
5+
Performance Evaluation Notifier Package
6+
7+
This package provides performance evaluation capabilities for LISA tests,
8+
including criteria validation and test result modification based on
9+
performance metrics.
10+
"""
11+
12+
__all__ = ["PerfEvaluation", "PerfEvaluationSchema", "MetricCriteria"]
13+
14+
from .perfevaluation import MetricCriteria, PerfEvaluation, PerfEvaluationSchema

‎lisa/notifiers/perfevaluation/perf_nvme_criteria.yml‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Performance evaluation criteria for NVMe storage tests
2+
# Global configuration
3+
statistics_times: 3 # Default: run each test 3 times to calculate statistics
4+
error_threshold: 0.1 # Default: 10% tolerance
5+
statistics_type: average # Default statistics method
6+
7+
# Test suite groups
8+
groups:
9+
- name: "NVMe Performance - L64s_v2 Specific"
10+
description: "Performance criteria for Standard_L64s_v2 VM"
11+
error_threshold: 0.20
12+
statistics_type: average
13+
statistics_times: 1
14+
15+
conditions:
16+
- name: "test_case"
17+
type: "metadata"
18+
value: "perf_nvme"
19+
- name: "vm_size"
20+
type: "information"
21+
value: "Standard_L64s_v2"
22+
23+
metrics:
24+
# 32 cores, 8 disks performance criteria for L64s_v2
25+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_read_iops"
26+
min_value: 800000.0
27+
target_value: 1033000.0
28+
error_threshold: 0.25
29+
30+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_write_iops"
31+
min_value: 750000.0
32+
target_value: 950000.0
33+
error_threshold: 0.25
34+
35+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randread_iops"
36+
min_value: 500000.0
37+
target_value: 1200000.0
38+
error_threshold: 0.30
39+
40+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randwrite_iops"
41+
min_value: 150000.0
42+
target_value: 650000.0
43+
error_threshold: 0.30
44+
45+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_read_latency"
46+
max_value: 50.0
47+
target_value: 10
48+
error_threshold: 0.30
49+
50+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_write_latency"
51+
max_value: 60.0
52+
target_value: 10
53+
error_threshold: 0.30
54+
55+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randread_latency"
56+
max_value: 80.0
57+
target_value: 25.0
58+
error_threshold: 0.30
59+
60+
- name: "qdepth_32_iodepth_1_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randwrite_latency"
61+
max_value: 100.0
62+
target_value: 80.0
63+
error_threshold: 0.30
64+
65+
# Higher queue depth tests
66+
- name: "qdepth_64_iodepth_2_numjob_32_setup_raw_bs_4k_cores_32_disks_8_read_iops"
67+
min_value: 1200000.0
68+
target_value: 1500000.0
69+
error_threshold: 0.25
70+
71+
- name: "qdepth_64_iodepth_2_numjob_32_setup_raw_bs_4k_cores_32_disks_8_write_iops"
72+
min_value: 1000000.0
73+
target_value: 1300000.0
74+
error_threshold: 0.25
75+
76+
- name: "qdepth_64_iodepth_2_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randread_iops"
77+
min_value: 800000.0
78+
target_value: 1100000.0
79+
error_threshold: 0.30
80+
81+
- name: "qdepth_64_iodepth_2_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randwrite_iops"
82+
min_value: 700000.0
83+
target_value: 1000000.0
84+
error_threshold: 0.30
85+
86+
# Highest queue depth tests
87+
- name: "qdepth_128_iodepth_4_numjob_32_setup_raw_bs_4k_cores_32_disks_8_read_iops"
88+
min_value: 1800000.0
89+
target_value: 2200000.0
90+
error_threshold: 0.25
91+
92+
- name: "qdepth_128_iodepth_4_numjob_32_setup_raw_bs_4k_cores_32_disks_8_write_iops"
93+
min_value: 1600000.0
94+
target_value: 2000000.0
95+
error_threshold: 0.25
96+
97+
- name: "qdepth_128_iodepth_4_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randread_iops"
98+
min_value: 1200000.0
99+
target_value: 1600000.0
100+
error_threshold: 0.30
101+
102+
- name: "qdepth_128_iodepth_4_numjob_32_setup_raw_bs_4k_cores_32_disks_8_randwrite_iops"
103+
min_value: 1000000.0
104+
target_value: 1400000.0
105+
error_threshold: 0.30
106+
107+
- name: "TCP NTTTCP SRIOV Performance - D2ads_v5 Specific"
108+
description: "Performance criteria for Standard_D2ads_v5 VM - TCP NTTTCP SRIOV"
109+
error_threshold: 0.20
110+
statistics_type: average
111+
statistics_times: 1
112+
113+
conditions:
114+
- name: "test_case"
115+
type: "metadata"
116+
value: "perf_tcp_ntttcp_sriov"
117+
- name: "vm_size"
118+
type: "information"
119+
value: "Standard_D2ads_v5"
120+
121+
metrics:
122+
- name: "buffer_size_conn_1"
123+
min_value: 0
124+
target_value: 65536
125+
error_threshold: 0.30
126+
127+
- name: "client_mtu_conn_1"
128+
min_value: 1400
129+
target_value: 1500
130+
error_threshold: 0.05
131+
132+
- name: "connections_created_time_conn_1"
133+
max_value: 5.0
134+
target_value: 1.0
135+
error_threshold: 0.50
136+
137+
- name: "latency_us_conn_1"
138+
max_value: 500.0
139+
target_value: 100.0
140+
error_threshold: 0.50
141+
142+
- name: "pkts_interrupts_conn_1"
143+
min_value: 0
144+
target_value: 100000
145+
error_threshold: 0.50
146+
147+
- name: "receiver_cycles_per_byte_conn_1"
148+
max_value: 100.0
149+
target_value: 20.0
150+
error_threshold: 0.50
151+
152+
- name: "retrans_segments_conn_1"
153+
max_value: 1000
154+
target_value: 0
155+
error_threshold: 2.0
156+
157+
- name: "rx_packets_conn_1"
158+
min_value: 100000
159+
target_value: 1000000
160+
error_threshold: 0.50
161+
162+
- name: "sender_cycles_per_byte_conn_1"
163+
max_value: 100.0
164+
target_value: 20.0
165+
error_threshold: 0.50
166+
167+
- name: "server_mtu_conn_1"
168+
min_value: 1400
169+
target_value: 1500
170+
error_threshold: 0.05
171+
172+
- name: "throughput_in_gbps_conn_1"
173+
min_value: 0.5
174+
target_value: 2.0
175+
error_threshold: 0.30
176+
177+
- name: "tx_packets_conn_1"
178+
min_value: 100000
179+
target_value: 1000000
180+
error_threshold: 0.50

0 commit comments

Comments
 (0)