time python ../vidur/profiling/mlp/main.py --models meta-llama/Llama-2-70b-hf --num_gpus 8 > mlp.txt && time python ../vidur/profiling/attention/main.py --models meta-llama/Llama-2-70b-hf --num_gpus 8 > att.txt
0%| | 0/1044 [00:00<?, ?it/s]2025-03-17 02:10:22,262 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
100%|████████████████████████████████████████████████████████████████████████| 1044/1044 [02:23<00:00, 7.26it/s]
real 2m31.669s
user 0m55.454s
sys 1m13.662s
0%| | 0/43888 [00:00<?, ?it/s]2025-03-17 02:12:55,162 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
0%| | 1/43888 [00:08<105:25:58, 8.65s/it](AttentionWrapper pid=149911) 2025-03-17 02:13:07,351 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
(AttentionWrapper pid=149911) /home/huday/vidur_test/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(AttentionWrapper pid=149911) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
(AttentionWrapper pid=149911) warnings.warn(
(AttentionWrapper pid=149913) 2025-03-17 02:13:07,363 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
Traceback (most recent call last):
File "/home/huday/vidur_test/vidur/data/../vidur/profiling/attention/main.py", line 240, in
main()
File "/home/huday/vidur_test/vidur/data/../vidur/profiling/attention/main.py", line 223, in main
profile_model(
File "/home/huday/vidur_test/vidur/data/../vidur/profiling/attention/main.py", line 157, in profile_model
results = ray.get(promises)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 2771, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 919, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TypeError): ray::AttentionWrapper.profile() (pid=149913, ip=172.17.0.2, actor_id=533c75b2787547df5b4810eb01000000, repr=<vidur.profiling.attention.attention_wrapper.AttentionWrapper object at 0x7f181e045870>)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/huday/vidur_test/vidur/vidur/profiling/attention/attention_wrapper.py", line 131, in profile
get_attention_wrapper().forward(query, key, value, kv_cache)
File "/home/huday/vidur_test/sarathi-serve/sarathi/model_executor/attention/flashinfer_attention_wrapper.py", line 224, in forward
append_paged_kv_cache(
TypeError: append_paged_kv_cache() missing 1 required positional argument: 'kv_last_page_len'
(AttentionWrapper pid=149914) 2025-03-17 02:13:07,832 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(AttentionWrapper pid=149915) /home/huday/vidur_test/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. [repeated 12x across cluster]
(AttentionWrapper pid=149915) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. [repeated 12x across cluster]
(AttentionWrapper pid=149915) warnings.warn( [repeated 12x across cluster]
(AttentionWrapper pid=149915) 2025-03-17 02:13:07,911 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False [repeated 5x across cluster]
0%| | 7/43888 [00:17<31:07:03, 2.55s/it]
real 0m23.165s
user 0m33.359s
sys 0m17.043s
time python ../vidur/profiling/mlp/main.py --models meta-llama/Llama-2-70b-hf --num_gpus 8 > mlp.txt && time python ../vidur/profiling/attention/main.py --models meta-llama/Llama-2-70b-hf --num_gpus 8 > att.txt
0%| | 0/1044 [00:00<?, ?it/s]2025-03-17 02:10:22,262 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
100%|████████████████████████████████████████████████████████████████████████| 1044/1044 [02:23<00:00, 7.26it/s]
real 2m31.669s
user 0m55.454s
sys 1m13.662s
0%| | 0/43888 [00:00<?, ?it/s]2025-03-17 02:12:55,162 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
0%| | 1/43888 [00:08<105:25:58, 8.65s/it](AttentionWrapper pid=149911) 2025-03-17 02:13:07,351 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
(AttentionWrapper pid=149911) /home/huday/vidur_test/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(AttentionWrapper pid=149911) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
(AttentionWrapper pid=149911) warnings.warn(
(AttentionWrapper pid=149913) 2025-03-17 02:13:07,363 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
Traceback (most recent call last):
File "/home/huday/vidur_test/vidur/data/../vidur/profiling/attention/main.py", line 240, in
main()
File "/home/huday/vidur_test/vidur/data/../vidur/profiling/attention/main.py", line 223, in main
profile_model(
File "/home/huday/vidur_test/vidur/data/../vidur/profiling/attention/main.py", line 157, in profile_model
results = ray.get(promises)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 2771, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 919, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TypeError): ray::AttentionWrapper.profile() (pid=149913, ip=172.17.0.2, actor_id=533c75b2787547df5b4810eb01000000, repr=<vidur.profiling.attention.attention_wrapper.AttentionWrapper object at 0x7f181e045870>)
File "/home/huday/vidur_test/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/huday/vidur_test/vidur/vidur/profiling/attention/attention_wrapper.py", line 131, in profile
get_attention_wrapper().forward(query, key, value, kv_cache)
File "/home/huday/vidur_test/sarathi-serve/sarathi/model_executor/attention/flashinfer_attention_wrapper.py", line 224, in forward
append_paged_kv_cache(
TypeError: append_paged_kv_cache() missing 1 required positional argument: 'kv_last_page_len'
(AttentionWrapper pid=149914) 2025-03-17 02:13:07,832 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(AttentionWrapper pid=149915) /home/huday/vidur_test/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. [repeated 12x across cluster]
(AttentionWrapper pid=149915) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. [repeated 12x across cluster]
(AttentionWrapper pid=149915) warnings.warn( [repeated 12x across cluster]
(AttentionWrapper pid=149915) 2025-03-17 02:13:07,911 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_f16_dtype_kv_f16_dtype_o_f16_dtype_idx_i32_head_dim_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False [repeated 5x across cluster]
0%| | 7/43888 [00:17<31:07:03, 2.55s/it]
real 0m23.165s
user 0m33.359s
sys 0m17.043s