Issues with Proxmox and Zoraxy as reverse proxy

[tobychui]

For anyone using Proxmox and want to expose the proxmox management web-ui using Zoraxy, I need some inputs regarding the setup.
In simple words, proxmox user when using Zoraxy as reverse proxy might encounter 501 - Not Implemented error when trying to use certain feature of proxmox management UI. One of such example is "start" and "stop" vm.

I was previously trying to add proxmox support in Zoraxy but with no success. Here is an example of request header Zoraxy received from web browser when I press the start/stop button

{
  "Accept": [
    "*/*"
  ],
  "Accept-Encoding": [
    "gzip, deflate, br"
  ],
  "Accept-Language": [
    "en-US,en;q=0.5"
  ],
  "Cookie": [
    "PVEAuthCookie=PVE%3Aroot@pam%3A65DE9998%3A%3APu/u+vEo0CGCqrdvEBvPUs2OsJd6OqUvK8ycyTyErwY+c+udX4bi3RHqSd6qOeMa+aIRQv9TfpESctWjVjfvrWth9nDDzFnZv1zwotY/k3hYpOsKuvcm+QtVDbxIkSHHV+5f0UxQp0zZ1NqJO8AUJXsMMN7bbJFLvsvzHOBjkrTDMMw+XOpY7XNbCy2vKjQgRZ2/Kfnv5rZEGnpNIXnINoHjJ0FPpK16+9bWfI0nR0NOyCwlvO10DXZT6ywet5WQwJZ/HOa0jN8N1pvtNh8kW78kPHTDMrLq1eWfPBwFXohNGn/5zOAUqTRUV+YNWY+qI6PibKBJHmTsJaFS0PBHOw%3D%3D"
  ],
  "Csrfpreventiontoken": [
    "65DE9998:psURpq1uyJukNU9tepAAWPcZdwIn1Bsxq/V6x86uPB4"
  ],
  "Referer": [
    "https://vm.localhost/"
  ],
  "Sec-Fetch-Dest": [
    "empty"
  ],
  "Sec-Fetch-Mode": [
    "cors"
  ],
  "Sec-Fetch-Site": [
    "same-origin"
  ],
  "Te": [
    "trailers"
  ],
  "User-Agent": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
  ],
  "X-Forwarded-Host": [
    "vm.localhost"
  ],
  "X-Forwarded-Server": [
    "zoraxy-0dc57054-6e06-42b1-ac84-c6d29aa5208d"
  ],
  "X-Requested-With": [
    "XMLHttpRequest"
  ]
}

And the following is sent to the Proxmox server by Zoraxy (proxied request). Note that the forwarded host header and forwarded server was removed for testing purpose. However, I also tested with both header in-tact as well as keep / remove the Te header.

{
  "Accept": [
    "*/*"
  ],
  "Accept-Encoding": [
    "gzip, deflate, br"
  ],
  "Accept-Language": [
    "en-US,en;q=0.5"
  ],
  "Cookie": [
    "PVEAuthCookie=PVE%3Aroot@pam%3A65DE9998%3A%3APu/u+vEo0CGCqrdvEBvPUs2OsJd6OqUvK8ycyTyErwY+c+udX4bi3RHqSd6qOeMa+aIRQv9TfpESctWjVjfvrWth9nDDzFnZv1zwotY/k3hYpOsKuvcm+QtVDbxIkSHHV+5f0UxQp0zZ1NqJO8AUJXsMMN7bbJFLvsvzHOBjkrTDMMw+XOpY7XNbCy2vKjQgRZ2/Kfnv5rZEGnpNIXnINoHjJ0FPpK16+9bWfI0nR0NOyCwlvO10DXZT6ywet5WQwJZ/HOa0jN8N1pvtNh8kW78kPHTDMrLq1eWfPBwFXohNGn/5zOAUqTRUV+YNWY+qI6PibKBJHmTsJaFS0PBHOw%3D%3D"
  ],
  "Csrfpreventiontoken": [
    "65DE9998:psURpq1uyJukNU9tepAAWPcZdwIn1Bsxq/V6x86uPB4"
  ],
  "Referer": [
    "https://vm.localhost/"
  ],
  "Sec-Fetch-Dest": [
    "empty"
  ],
  "Sec-Fetch-Mode": [
    "cors"
  ],
  "Sec-Fetch-Site": [
    "same-origin"
  ],
  "User-Agent": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
  ],
  "X-Requested-With": [
    "XMLHttpRequest"
  ]
}

I highly suspect that there is something screwed up in Proxmox build-in API server which is written as perl module. However I am not expert in perl nor I can contribute anything to the proxmox repo, so I think I will just share my idea here and wait for someone who can contribute some idea what might be the issue.

If you are currently facing the same issue and in need for a quick (and dirty) solution, setup an Nginx vm inside your proxmox node and point Zoraxy http rule to your nginx proxy.

Any idea or insights will be welcomed.

[tugsi]

Hi,
I´m at the same problem.
Always used NPM and wanna give Zoraxy a try, but Proxmox is a must have for me.

Zoraxy is new for me and the documentation is where? ;)

In NPM I use this Advanced Configurations:
location / {
proxy_pass https://Proxmox-IP:8006;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Upgrade $http_upgrade;
}

Perhaps it could be implemented in the Custom Headers in Zoraxy?
The proxy_set_header Upgrade $http_upgrade; is a must for opening Shell/Console.

[tworth04]

Not sure if anyone has noticed but you can select a vm/lxc and once selected, you can click on the console dropdown menu to the right and novnc does work out of the box...the default however is xterm.js and it doesn't work at all.

[tobychui]

@tworth04 Interesting findings. From what I currently know, this should be an issue within the internal router use by Proxmox rejecting request forwarded by Zoraxy. However, websocket (usually) didn't check for origin, that is why novnc works, but xterm.js doesn't works is kind of weird as I thought both of them are based on websocket.

You got any error message when the xterm.js load / when typing into the terminal window?

[tworth04]

No nothing i see error wise so far...

looks like it works when launching new windows? so for instance..if you click on updates...and refresh..generally it launches a sub window to show it's process...that doesn't work..but if you hit upgrade, that pops open a new browser entirely and that works.

[tworth04]

i'm curious if this would be something similar that's happening here
gravitational/teleport#32039

[Nafalan]

Hi, I´m at the same problem. Always used NPM and wanna give Zoraxy a try, but Proxmox is a must have for me.

Zoraxy is new for me and the documentation is where? ;)

In NPM I use this Advanced Configurations: location / { proxy_pass https://Proxmox-IP:8006; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Host $host:$server_port; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Port $server_port; proxy_set_header Upgrade $http_upgrade; }

Perhaps it could be implemented in the Custom Headers in Zoraxy? The proxy_set_header Upgrade $http_upgrade; is a must for opening Shell/Console.

Hey thanks friend

I just set up nginx proxy manager and Pi-hole, and I was having this issue. However, your code, added to the advanced config in the GUI, sorted out my problem.

Thank you so much

[KmdKeen]

I'm in the same boat! Wanna switch vom NPM to Zoraxy, but xterm.js shows "Connection failed (Error 501: )". noVNC works, but I like xterm more. A solution for this would be awesome because I like Zoraxy a lot!

[franckleroy91]

Got the same issue, I migrated all my hosts to zoraxy and kept one Nginx just for Proxmox.
I follow this discussion to know when a solution will be found.
Thanks for your great job !

[KmdKeen]

Awesome! Updated to 3.1.0 and now Proxmox GUI is working like a charm! Thanks @tobychui for fixing this! Now zoraxy is my proxy of choice!

[Smiley-k]

Awesome! Updated to 3.1.0 and now Proxmox GUI is working like a charm! Thanks @tobychui for fixing this! Now zoraxy is my proxy of choice!

how? I can't get the PVE to work through zoraxy.

[tugsi]

how? I can't get the PVE to work through zoraxy.

What settings have you set for the upstream? Skip Websocket Origin check must be set.

[scroguard]

i have encountered the same issue. if i hit F12 in firefox and view the network logs when loading the page, i do see a 501 error regardig xterm.js:

i am also unable to make novnc work.

[scroguard]

i will try to test it this week. i'll respond as soon as i have had a chance to do so.

[scroguard]

i found some time today, i gave it a test and it still throws the same error.

[tugsi]

@scroguard You got time to try out the latest v3.0.5 and see if the issue has been fixed?

I´ve got also some time to test.
No step forward, NoVNC can be used, but xterm.js not and also the Shell isn´t ok.

[franckleroy91]

Hi,

just tried ; still 501 error on console.

Franck

[tobychui]

Hmmm ok, that is weird, I think I run out of clues why this could be happening.

[KemMuammer]

Is it now possible to do the get it working with the new update?

[tobychui]

It is already on the 3.1.0 branch. For Debian you can copy the link by right clicking the binary of choice and use wget to download it.

[tugsi]

@tobychui Ok, for understanding, I already used the Branch 3.10 in Github yesterday, if you have insert the API already there, then the problem is still alive.. if you fixed it today into 3.10, then I will test it later, back from work

[tugsi]

@tobychui Back from work, make a test and yes ignore my question above, you've fixed inside 3.10, so now I can upload my cert and everything went fine 👍🏻 👍🏻
One little question I got, I´ve done all on a little test-LXC inside Proxmox, to make it happened, I clone it from github in /opt/zoraxy, focused on branch 3.10 and make a build.
Unfortunately I didn´t make a backup from my conf, so first i deleted the opt/zoraxy in order to get a new clone, so I lost my conf (what a fool I am *LOL)
So here is my question, the build is made in zoraxy/src, is everything I made inside zoraxy saved under /conf/ ?

[tobychui]

@tugsi Yes, most of the configs are in the conf folder with some of the legacy settings will left inside the sys.db file. In fact you can just do git pull in the project file and cd ./src/ with go build. That way you do not need to delete everything.

[Groumf44]

I think I got it figured out and patched in here. This was partially caused by chunked encoding but some of the error was actually from the golang net/http package itself.

If anyone is free this weekend, feel free to test out the nightly build and let me know if it is working now 👍🏻

Hello
I just updated and all is working now
Thank you very much for your work dude !
Your product is now perfect for me and my usage <3

[morvane98]

hello

i discover zoraxy and i just saw that there was a bug between zoraxy and proxmox.
i have this bug and i would like to know if it has been fixed?
i installed zoraxy under proxmox in a debian12 LXC container
i am in version 3.1.1

thanks

[tugsi]

Hey @morvane98
please tell us, what happened or better what not happened.
I use zoraxy in a debian12 LXC on Proxmox and everything went fine.
So a little more information please.

[morvane98]

well when I connect to proxmox via my subdomain, I arrive well under proxmox but for example when I want to go to the shell I do not have the command prompt and the interface cannot connect whereas if I directly enter the ip address of my proxmox everything works well

[tugsi]

ok, that was the bug before v3.1, are you sure being on v3.1.1 (shown on the Dashboard down under left).
which Version of Proxmox? I have 8.2.6
do you use the Shell-Link left under the Name or use the Button right above (NoVNC,Spice,xterm.js)?
If you use left, please make a try using the option Spice or anything else.
And last but not least, how do you use your subdomain, through cloudflare or something else or only directly ?

[morvane98]

I am in 3.1.1 and proxmox8.2.6 and I use the left menu "shell".
when I enter proxmox via its ip address it works but not with the subdomain proxmox.xxxx.ovh.
and so I am at ovh I have configured my domain and subdomain at ovh

[morvane98]

[morvane98]

this is the bug

[tugsi]

Ok, let us check your settings for proxmox.xxx.ovh
My Upstream has checked TLS and Skip Websocket

[morvane98]

[tugsi]

Good Morning, so back again.
I´ve seen something special to your failure.
You got an CODE 1006 in the Shell, that´s not the Bug we had at ime ago.
But I´ve take a little search in web, do you have a firewall on your PC like Norton, ESET or something else?
https://forum.proxmox.com/threads/undefined-code-1006.93079/
If yes please take a try to deactivate the firewall.

Otherwise for testing, use NPM instead of zoraxy, so you can see, what happend.
If NPM is ok, then the bug is in zoraxy and @tobychui must take a look.

[tobychui]

Not sure if that is related, but the "Proxmox compatibility mode" only activate when your browser have cookies related to proxmox. That asides, even if the compatibility mode is disabled, you shd be seeing error 501 instead of 1006.

[morvane98]

error 1006 only occurs when I want to access proxmox by the subdomain name proxmox .xxxx.ovh
when I access proxmox locally with 192.168.2.250:8006 no problem everything works well
i don't understand why?

[tugsi]

That's exactly the point, internally it's easier for traffic, but from the outside there may be a firewall, virus scanner or something else in between. The cookies are also set differently.
You can't compare internally and externally 1:1, but of course it's good if it works internally ;-)
That's why I asked whether a firewall or something similar was used, as described in the link I posted (example to ESET Scanner).
Alternatively, you can test NPM (NginxProxyManager), which is quickly set up in Proxmox and redirected to the IP in the router. The entry that has to be entered in NPM for Proxmox is posted above. Because if Code 1006 also comes up, it is a problem on the PC and not from the reverse proxy.

[morvane98]

So on my windows pc I only have windows defender which is the original firewall.
yesterday following your message I deactivated it and it's the same thing.
what can prevent the connection on the pc?

[morvane98]

the same thing on my samsung S24.....

[tugsi]

Sorry, but I don't know what to do now.
Please try NPM to see if it also has the problem.
Is quickly set up as LXC with the script from TTeck.
https://tteck.github.io/Proxmox/#nginx-proxy-manager-lxc

Start by creating a proxy host, checking “Block Common Exploits” and “Websockets Support”.
Under Advanced then:

location / {
proxy_pass https://IP-to-Proxmox:8006;
proxy_set_header Upgrade $http_upgrade;
}

If it works fine, be happy, for the first you have a Reverse Proxy, but if it fails too, then there is more, but @tobychui is could say more.

One more thing it could be, tested only with one browser or also with Firefox, Chrome, Safari... ?

[morvane98]

ok but I don't know nginx at all and I have a bit of trouble with the command line, because I don't always understand what I'm doing
I prefer like zoraxy when there is a gui

[tugsi]

The command line is only to simple create the LXC with NPM in it, you also can create an LXC and install NPM by hand in it.
NPM has also a GUI, only a little bit less informations like zoraxy.
My only intention was, to find a way to know, if your homelab has a failure coming from outside or zoraxy has it.
In the bug mentioned here, we can´t get the shell without getting the Code1006, only running in TimeOutError.
Your behavior seems to be another bug.
Unfortunately I can't help you anymore, I'm just a user with advanced knowledge.

[morvane98]

idem with chrome and firefox

[morvane98]

I will try to see with npm

[rimbat]

Skip verification for self signed certificates:

[NitroHype]

Thank you, this fixed the same issue I was having too

[morvane98]

cool all is done!!! thanks

[jcastro]

Having problems with Proxmox Backup server using Zoraxy. With NPM it works smoothly but with Zoraxy I get this error

INFO: starting new backup job: vzdump 200 --mode snapshot --notification-mode auto --remove 0 --notes-template '{{guestname}}' --storage pbszora --node ms01
INFO: Starting Backup of VM 200 (lxc)
INFO: Backup started at 2025-01-01 17:23:49
INFO: status = running
INFO: CT Name: community
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-200-disk-0_vzdump" created.
INFO: creating Proxmox Backup Server archive 'ct/200/2025-01-01T16:23:49Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1905990_200/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 200 --backup-time 1735748629 --entries-max 1048576 --repository REDACTED:443:bkp --ns usertest
INFO: Starting backup: [usertest]:ct/200/2025-01-01T16:23:49Z    
INFO: Client name: ms01    
INFO: Starting backup protocol: Wed Jan  1 17:23:49 2025    
**INFO: Error: missing Upgrade header**
INFO: Failed at 2025-01-01 17:23:49
INFO: Backup job finished with errors
TASK ERROR: job errors

[tugsi]

Hello,
Unfortunately, I can't understand your case.
I also have a PBS and can reach it at pbs.domain.net, but that is irrelevant for the backup process since the PVE is on the same network and does not address it via the domain.
If I start the backup in my LXC via pve.domain.net, it does it completely normally.
How did you mount the PBS in the PVE?
Is it “local” there?

I still don't fully understand why two local servers on the same network are causing problems.

If you access the PVE locally via IP and then start the backup, does that work?

[jcastro]

hi @tugsi, if I do it "locally" (with a DNS rewrite) but through Zoraxy it fails. If I do it with the IP directly it works fine!

[tugsi]

@jcastro Sorry, but I can't reconstruct your mistake.
As I said, I installed my PVE on a PC, the PBS runs on it as a VM and is backed up on a NAS, everything in my local network.
If I now access the PBS through Zoraxy (whether locally or from outside), it saves the backup without any problems.
Hence my question, is the storage drive also located locally?

My PBS is 3.3.2 and PVE is 8.3.2
Settings Zoraxy Upstream:

[jcastro]

hello! My setup was as follows (I had to change it to NPM to get it working!) PBS 8.3.x (I can't remember the exact version I was using) running on LXC on top of PVE TrueNAS bare metal in another computer, sharing NFS that's mounted on PVE, which PBS uses for the datastore

AFAIR, the settings were also as you are showing in the pic. I might give it a try again to see what happens!

[Smiley-k]

@jcastro Sorry, but I can't reconstruct your mistake. As I said, I installed my PVE on a PC, the PBS runs on it as a VM and is backed up on a NAS, everything in my local network. If I now access the PBS through Zoraxy (whether locally or from outside), it saves the backup without any problems. Hence my question, is the storage drive also located locally?

My PBS is 3.3.2 and PVE is 8.3.2 Settings Zoraxy Upstream:

proxmox works fine with these parameters

[aenima99x]

Hello - Just recently switched back to Zoraxy from Caddy and am seeing similar issues with Proxmox 9.0.10.
When connecting to a console for any of my LXC's or VM's, it sits there with a "Connecting...." message for an extended period of time (20-30 seconds) before either finally connecting or rarely erroring out. The issue is the same with trying either XTerm.js or NoVNC.

[Smiley-k]

all ok, work

[aenima99x]

Are you using Proxmox 9? Because disabling "Skip verification" does not fix the issue.

[aenima99x]

This also seems to be affecting HomeAssistant. It's taking about 20-30 seconds to load the web interface there as well. I suspect this is related to something WebSockets again.

[Smiley-k]

Are you using Proxmox 9? Because disabling "Skip verification" does not fix the issue.

no Proxmox 8 - I don't think there will be any problems with the 9th

I also use the home assistant and everything works correctly.

[aenima99x]

Here is some info showing the proxmox console connection trying to connect for long period, timing out and then finally connecting.

[CrazyWolf13]

@tobychui sadly this is still happening for me general things seem to work, however jsconsole (web-console) does just say undefined:

Funnily enough, when I open devconsole which limits the space the jsconsole visually takes up, it always works. If I close devconsole again, it doesn't work again, is there maybe some size restrictions of the package/headers ?

Sorry if this sounds weird, but was my first guess.

Sadly without proxmox support I can't really use zoraxy as replacement.

[tobychui]

is there maybe some size restrictions of the package/headers ?

No, how much traffic you are pushing through in one request?

Generally speaking, just make sure you understand Virtual Directory are not Locations in nginx then you should be able to set it up or at least provide a proper bug report if needed.

Sadly without proxmox support I can't really use zoraxy as replacement.

Don't be sad, there are still a lot of different proxy out there you can use :)

[Smiley-k]

@CrazyWolf13 This works with proxmox, I have it on many installations

[CrazyWolf13]

@tobychui Could you please elaborate on the Virtual directory thing?

I use just the http proxy to proxy proxmox from a IP:port to my domain with ssl through ACME.

Leaving this unticked made it better, yet still roughly every 8th webshells immediately shows undefined error.

[CrazyWolf13]

Every failure seems like an issue with the websocket connection:

Firefox kann keine Verbindung zu dem Server unter wss://pve.REDACTED.dev/api2/json/nodes/lenovo1/lxc/134/vncwebsocket?port=5901&vncticket=PVEVNC%3AXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX6CSp9Q%3D%3D aufbauen.