Wrap the main cudnn functions

[WilliamTambellini]

cuew is wrapping most/all cuda and nvrtc api.
Woud it also make sens to wrap cudnn, at least its main api :

• cudnnCreate, cudnnDestroy
• cudnnSetTensorNdDescriptor, cudnnSetTensor4dDescriptor
• cudnnCreateTensorDescriptor, cudnnCreatePoolingDescriptor, cudnnCreateReduceTensorDescriptor
• cudnnSetReduceTensorDescriptor
• cudnnGetReductionWorkspaceSize
• cudnnSetPoolingNdDescriptor, cudnnSetPooling2dDescriptor
• cudnnPoolingForward, cudnnPoolingBackward
• cudnnReduceTensor
  ?
  Kind
  WT

[sergeyvfx]

Those functions are part of cuDNN SDK, and CUEW is generated from CUDA Toolkit SDK. It is probably not that much work to adopt code to also cover cuDNN functions, but that's probably would be better to happen as a separate wrangler project with own versioning and such (maybe within same CudaWrangler "umbrella").

Would need to have a closer look into that library, and also check on whether it causes issues with driver vs. runtime CUDA API (we can't really support wrangler for CUDA runtime API).

Anyway, will keep updated when have something new!

[WilliamTambellini]

Hello @Nazg-Gul
Thank you. FYI, cudnn support would be needed to integrate cuew in ngraph :
NervanaSystems/ngraph#765
I see cuew is already targeting cuda 9 so perhaps would it make sens to target cudnn 7 ?

[sergeyvfx]

Did some preliminary work to get wrangler generated. It is all a bit hacky (manual tweaks needed on top auto-generated header, don't have time to solve them tonight), so committed current work to own fork: https://github.com/Nazg-Gul/cuew/tree/cudnn_experiment

The test program displays:

CUDA found
NVCC path: /usr/local/cuda/bin/nvcc
NVCC version: 90
NVRTC found
Found runtime compilation library version 9.1
CUDNN found
Found Deep Neural Network library version 7.103

While it seems OK, i don't have any way to test thing further. I'm a bit worried about possible conflicts of static initializers which CUDA runtime API expects to run before main().

Anyway, give it a try and see if it works.

[WilliamTambellini]

Cool :

$ ./testcuew
CUDA found
NVCC path: /usr/local/cuda/bin/nvcc
NVCC version: 80
NVRTC found
Found runtime compilation library version 8.0
CUDNN found
Found Deep Neural Network library version 7.5

Sounds good but would be indeed safer to test through a minimum of cudnn unit tests. Let me check. Thank you.

[WilliamTambellini]

@Nazg-Gul Sounds pretty good, at least the "conv_sample" from the cudnn samples
https://docs.nvidia.com/deeplearning/sdk/cudnn-install/#verify
seems to compile and run correctly :
if building with cuew (and ofcourse cudart_static), then the exec has no more nvidia libs dependancies :

$ ldd conv_sample
linux-vdso.so.1 => (0x00007efd289d4000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007efd2842f000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efd2822b000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efd28023000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efd27e06000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007efd27bf0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efd27826000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efd2751d000)
/lib64/ld-linux-x86-64.so.2 (0x00007efd287b1000)

$ ./conv_sample
main:
CUDA found
NVCC path: /usr/local/cuda/bin/nvcc
NVCC version: 80
NVRTC found
Found runtime compilation library version 8.0
CUDNN found
Found Deep Neural Network library version 7.5
Testing single precision
Testing conv
^^^^ CUDA : elapsed = 7.20024e-05 sec,
Test PASSED
Testing half precision (math in single precision)
Testing conv
^^^^ CUDA : elapsed = 8.10623e-05 sec,
Test PASSED

Just a little warning that the exec is simply crashing without any printf/std::cerr message if cuew not initialized before trying to use any cudnn function.

[sergeyvfx]

Took a bit to fix issues with automated wrangler generator, and to make it possible to submit pull requests for review from own repo.

Submitted a pull-request #9 .