[Windows] TorchRuntimeError from TabDPT

[umdsgy]

I have installed autugluon and tabarena following the github guidelines, and found the torch is the cpu version and can successfully run TabDPT regression. Then I uninstalled the original torch and installed the same version torch with cuda 128. Then I met this torchruntimeerror from TabDPT:

torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function (*(FakeTensor(..., device='cuda:0', size=(1, 8, 15340, 96)), FakeTensor(..., device='cuda:0', size=(1, 8, 1600, 96)), FakeTensor(..., device='cuda:0', size=(1, 8, 1600, 96), dtype=torch.bfloat16)), **{}): got RuntimeError('No available kernel. Aborting execution.')

from user code:
File "C:\Users\GS\PycharmProjects\SOC.venv\Lib\site-packages\tabdpt\utils.py", line 29, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\GS\PycharmProjects\SOC.venv\Lib\site-packages\tabdpt\model.py", line 74, in forward
src = layer(src, eval_pos)
File "C:\Users\GS\PycharmProjects\SOC.venv\Lib\site-packages\tabdpt\model.py", line 137, in forward
attn = F.scaled_dot_product_attention(q, k, v).transpose(1, 2)

[W302 00:58:03.000000000 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator ())

Here is my cuda info:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 591.86 Driver Version: 591.86 CUDA Version: 13.1 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5080 WDDM | 00000000:01:00.0 On | N/A |
| 0% 37C P0 47W / 360W | 843MiB / 16303MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

Is there anyone who knows how to address this issue?

[LennartPurucker]

Heyho, thanks for reporting the issue!

When you say "I uninstalled the original torch and installed the same version torch with cuda 128", did you just edit the existing Python environment? You might need to reinstall the entire environment, as other dependencies might also need to be changed.

[umdsgy]

I only changed the torch package as the original torch with following installation does not come with gpu.

pip install uv
uv init -p 3.11
uv sync
git clone https://github.com/autogluon/autogluon.git
./autogluon/full_install.sh
git clone https://github.com/autogluon/tabarena.git
cd tabarena
uv pip install --prerelease=allow -e ./tabarena[benchmark]

Then how should I use gpu to fit the model with the original installation?

[LennartPurucker]

So when you start from an empty environment and follow the above install descriptions, what do you get for the following code:

import torch
print(torch.cuda.is_available())  # Should be True
print(torch.cuda.device_count())  # Should be > 0

The default of Troch is the GPU version. When I run the install above, I get the GPU version of torch.
But I saw that you are using Windows, which might change behavior in ways I cannot judge.

Check out the install instructions from AutoGluon for Windows to see if this might help you: https://auto.gluon.ai/stable/install.html

[LennartPurucker]

Otherwise, it might be that TabDPT does not work for the newest version of Torch. In your current setup, you could try to install torch==2.7.1.

[umdsgy]

when I start an empty env and follow the original install. I do not have any available cuda.

import torch print(torch.cuda.is_available()) False print(torch.cuda.device_count()) 0

I will try the autogluon for windows or the torch==2.7.1 and let you know if any option works.

[umdsgy]

Hi there, I installed the autogluon following https://auto.gluon.ai/stable/install.html.

I saw gpu in the package:
print(torch.cuda.is_available()) True print(torch.cuda.device_count()) 1

But I just got the same error:

torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function <built-in function scaled_dot_product_attention>(*(FakeTensor(..., device='cuda:0', size=(1, 8, 1600, 96)), FakeTensor(..., device='cuda:0', size=(1, 8, 1280, 96)), FakeTensor(..., device='cuda:0', size=(1, 8, 1280, 96), dtype=torch.bfloat16)), **{}): got RuntimeError('No available kernel. Aborting execution.') from user code: File "C:\Users\GS\miniconda3\envs\ag\Lib\site-packages\tabdpt\utils.py", line 29, in wrapper return func(self, *args, **kwargs) File "C:\Users\GS\miniconda3\envs\ag\Lib\site-packages\tabdpt\model.py", line 74, in forward src = layer(src, eval_pos) File "C:\Users\GS\miniconda3\envs\ag\Lib\site-packages\tabdpt\model.py", line 137, in forward attn = F.scaled_dot_product_attention(q, k, v).transpose(1, 2) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

It is not an issue with autogluon or tabarena. It is from TabDPT itself.

[LennartPurucker]

Mhm, consider opening an issue in their repo: https://github.com/layer6ai-labs/TabDPT-inference/tree/main
Maybe @Innixma knows more from the AutoGluon side.

It might be that some of the functionality they use isn't supported by Windows (e.g., compiling the model).

You could consider using Windows Subsystem for Linux to get around such issues (for the most part): https://learn.microsoft.com/en-us/windows/wsl/install

[umdsgy]

Sure, I will report this issue to TabDPT repo first. Thanks anyway, Lennart.