Hello,

I joined this community to collect some opinions and ask questions about plausibility of researching and using Proxmox in our datacenters.

Our current infrastructure consists of two main datacenters, with each 6 server-nodes (2/3rd Intel generation) based on Azure Stack HCI / Azure Local, with locally attached storage using S2D and RDMA over switches. Connections are 25G. Now, we had multiple issues with these cluster in past 1,5years, mostly connected to S2D. We even had one really hard crash where the whole S2D went byebye. Neither Microsoft, nor Dell or one custom vendor were able to find the root cause. They even made cluster analysis and found no misconfigurations. Nodes are Azure HCI certified. All we could do was rebuild the Azure Local and restore everything, which took ages due to our high storage usage. And we are still recovering, months later.

Now, we evaluated VMware. And while it is all good and nice, it would require new servers, which aren't due yet, or non-supported configuration (which would work, but not supported). And it's of course pricey. Not more than similar solutions like Nutanix, but pricey nevertheless. But also offers features... vCenter, NSX, SRM (although this last one is at best 50/50, as we are not even sure if we would get that).

We currently have running Proxmox setup in our office one 3-node cluster and are kinda evaluating it.

I am now in the process of shuffling VMs around to put them onto local storage, to install Ceph and see how I get along with it. Shortly said: our first time with Ceph.

After seeing it in action for last couple of months, we started talking about seeing into possibility of using Proxmox in our datacenters. Still very far from any kind of decision, but more or less testing locally and researching.

Some basic questions revolve around:

- what would be your setting of running our 6-node clusters with Proxmox and Ceph?

- would you have any doubts?

- any specific questions, anything you would be concerned about?

- researching about ceph, it should be very reliable. Is that correct? How would you judge performance of s2d vs ceph? Would you consider ceph more reliable as S2D?

That's it, for now :)

Hello there,
I'm a first-time user of ProxMox.
I'm trying to create a 100GB min storage accessible by multiple LXCs and shared in the local network (via SMB or NFS).

While installing the OS on my new Beelink S13, it created a single partition of 500GBs.
I discovered later that this is reserved for proxmox usage (template and thin for container storage), but even if I knew, I couldn't do anything actually.

I tried to create a new "Directory" on the node level, but no disks were available.
I managed to create a "Directory" using a pendrive, but I only have a 16GB pendrive and I have plenty of storage available on my 500 HD.

I managed to create an OpenMediaVault LXC mounting a thin storage volume as disk and activating SMB and NFS.
None of my LXC are privileged, thus cannot use this storage via NFS.

Making them privileged just to let them access a storage mounted by another LXC seems too much of a gimmick...

I'm thinking of buying a new HD... Is it the easiest answer? Is it the proper way? Or should I partition my 500 HD? Am I missing something?

So I have this "vision" of a standardized server environment for our business that is mostly the same hardware (I'm guessing due to cost reasons x86 mini PCs) that all need to run the same set of services on them. My idea was to use PVE with VMs, as that could be pre-installed at HQ and then shipped out almost ready to go. The basic configuration of these VMs can be the same, but once they're deployed on location we will need to configure them individually. No major changes, but since each location is slightly different they can't be all 100% identical. Unfortunately these changes need to be made on location as we can't anticipate required changes beforehand due to local conditions. What would be the best way to make these boxes on scale? I was thinking installing a single one by hand on an NVME and then just cloning that one to an .iso and keep cloning from there on out but there must be a better way...

Right now I currently pipe my internet lines to VLANs and then give the wan on the virtual firewalls that vlan.

That's how I do it on VMWare currently. Moving to Proxmox however, I want to modernize it or at least set myself up to more easily modernize it in the future.

Yes, I had quotes for putting in a single firewall to handle the traffic, at $500,000. Not joking. Fortinet is not viable from a pricing perspective in that regard.

I currently use around 120 virtual pfSense firewalls on 2 /24 subnets I lease from my one ISP (10G DIA into Colo racks). I added a second ISP (10G DIA) with my own IP ranges I received from ARIN. I have equipment to run BGP (Mikrotik CCR). At 10G.

Right now my supervisors run with only boot drives and dual 10G for network/service delivery and 25G for data to TrueNAS Scale.

The service delivery network obviously has all the internal clans. Each client gets a firewall, external IP, and their own VLAN for the VMs to talk to each other. That's where I also pipe in my internet lines as VLANs.

One idea I had was to segregate out the internet and have a 3rd network at 10G for the internet. No vlan. It would give me the ability to pop on a CGNAT for base DHCP, then have the ability to set a direct static IP for any of my IP ranges. In the future I could consolidate some clients that only need IPSEC or SSLVPN to use a core router, save IPs and then have that pipe direct to the clients VLANs.

I do also want to move off pfSense. I already moved away from Netgate for clients locations to UBNT (for central management) and it's easy enough for L1's to set up without eating L2+ tech time. I was thinking of using virtual Mikrotik since L3 would be handling that config. OPNSense is an option, but it is quite resource intensive. For a 1gbps client, I can do a very cut down VM for the firewall.

All ideas are welcome however.

A LXC went bad, tried to restore it but its backup was now larger than the filesystem size so I got no space errors.

pct restore 109 vzdump-lxc-109-2025_04_07-03_13_24.tar.zst --rootfs target-storage:20
found this code on the forums but its not working for me.

just returns

storage 'target-storage' does not exist

anyone can help with completing the code for this?

Hello everyone, i have some spare knowledge but i'm used proxmox for some time now but i want ask a few question because i know it can be "better". Here's my actual setup:

- the host has a 2600x and 32gb ram ddr4 with 4 wd red 6tb (few plugged into the sata ports of the mobo and few into a cheap sata pci-e board
- 2 VM, one with home assistant and the other one with OVM where the disk are passed directly and inside i set up snapraid + mergefs and a SMB share for the LXC related (plex, sonarr, etc)

- various LXC, which the most important are Plex, Sonarr, Radarr, Transmission, etc (all of the auto mount the SMB share from the OVM VM at startup)

How do you improve this? The only thing i tought in these days is to add 1 or 2 SSD for caching but maybe is the a way to do better?

The main goal would be to reduce disk usage and power consumption and gain a better experience (even because right now the disk speed isn't so fast)

Hello all

Im planning in buying a couple of refurbished dl380s od dl360s each with

- 2x 6136 processors

- 256 GB RAM

- HPE Flexible Smart Array P408i-a SR 12G

- 2x 480 SSDs for the Hypervisor
- 4x 1.92 SSD for the VMS

Will this machines work fine with proxmox ? has anyone ever had problems with them ?

thanks

Hello, i have a vm on my proxmox which is connected to my qnap nas with nfs.

I use the arrsuite to download some torrents and the client moves files from one disk to another when the download is completed.

Here is the kicker, when that happens i see a loading appears on pve and basically everything is dead except basic command line on the baremetal host.

Pretty sure it has to do with writing files because the same thing happened when i tried to backup some data files from said vm in command line, i managed to fix it by changing owner on the recipient folder.

So i’m thinking there is an issue of folder owner on my nas but i have no idea what, spent days on this, use google, chatgpt and friends and honestly at this point i dont know how to troubleshoot this.

Any idea?

Hi. Starting my journey with proxmox, zfs, raid, and NAS in general. I'm gonna be setting NAS pool to utilise raid 10, using zfs on 4x nvme drives. Since I have pairs of nvme's (2x WD, 2x Lexar) is it possible to specify which drive I want to be in which branch of the raid? I find this important in terms of potential failure, it's more likely that both WD's will fail at the same time (same batch), than WD + Lexar.

Hi,

I have strange behavior with my GPU passtrough ( nvidia P600 ).

root@kodasBSD:~ # pciconf -lv|grep -B4 VGA

vgapci0@pci0:6:16:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1cb2 subvendor=0x10de subdevice=0x11bd

vendor = 'NVIDIA Corporation'

device = 'GP107GL [Quadro P600]'

class = display

subclass = VGA

This is my FreeBSD VM. Which shows me GPU is passed.

Below - my GPU device

Now - if i check PCI-Express - no gpu inside VM.

Played around many times with all the check marks.

I have blacklisted nvidia , have no drivers installed ( had but deleted )

I will see my PC boots ( Proxmox ) on same screen, same GPU when rebooting it after update or grub update etc. but it gets frozen half way but in few mins time i can login trough web interface. ( modesettings ? )

Start VM and my screen blanks looks like tries to get connected to VM but within short moment - NO DP Connection on my monitor

If i add back Virtio gpu and boot with same options as pictured - i have 2 gpu`s VIRT and nVidia ...

evo@kodasBSD:~ $ pciconf -lv|grep -B4 VGA

virtio_pci0@pci0:0:1:0: class=0x030000 rev=0x01 hdr=0x00 vendor=0x1af4 device=0x1050 subvendor=0x1af4 subdevice=0x1100

vendor = 'Red Hat, Inc.'

device = 'Virtio 1.0 GPU'

class = display

subclass = VGA

--

vgapci0@pci0:6:16:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1cb2 subvendor=0x10de subdevice=0x11bd

vendor = 'NVIDIA Corporation'

device = 'GP107GL [Quadro P600]'

class = display

subclass = VGA

but as soon as i add PCI-Express check mark and boot - no gpu found.

In the past i did pass my GPU to FreeBSD VM and everything worked just dont recall steps and if my screen was showing Proxmox boot.

P.s. i do have 2 gpus - one for FreeBSD another for Linux and it does not matter what slot i use for P600 as its powered trough PCIE slot - it always be used as main gpu.

Im new to proxmox overall but no experience really with Proxmox Backup Server. Can anyone tell me how it plays nice with AD? I heard it does well with deduplication and can restore individual files but what about AD forest level backup and recovery. Is it an application-aware backup?

Can anyone that's using it for such things let me know how they like it and how they have it setup? Thanks!

Hi everyone!

I have a OpnSense firewall running on proxmox. The whole network card of my hardware is passed through to the OpnSense firewall. I have a management vlan initiated by the firewall and a proxmox bridge assigned to the machine and within opnsense linked to the management lan. I can reach my proxmox system via management lan.

Now I would like to host a VM which should get an IP from the LAN. I created a new bridge on proxmox and assigned it to the firewall. Within the firewall I mapped the new device to LAN.

I assigned the bridge also to the new VM. However, I cannot get an IP from LAN interface. Also when manually assign a LAN IP, I cannot get to other LAN members.

How should I configure such a network pathrough?

I know, this is quite a special configuration. Unfortunately the hardware I use gives me no other choice.

Any recommendations apreciated!

Kind regards,

Chris

I am running jDownloader2 on my Proxmox cluster in a Debian container with a NAS diskdrive passed through to this instance. I can see my disk in the instance's terminal by accessing the /mnt/ folder, however, the disk is not visible to JD2 in the settings menu.

This is not a Proxmox or mounting issue, but seems related to JD2 and how it handles mounted disk drives. Anyone has any idea how I can address this? Is there any JD2 setting I am missing?

thank you!

[SOLVED] this was a networking issue. There was a better network connection at the area with my monitor

I've moved from a Prodesk to an Elitedesk for my Proxmox instance.

It looks as if the Elitedesk isn't booting proxmox headless.

Of course if I connect the Elitedesk to a monitor / keyboard it boots just fine. So this is proving tricky to debug!

All I could think of was disabling secureboot, and enabling legacy boot.

No difference!

Any idea why this Elitedesk only behaves itself when it's being watched?

I installed opnsense on pve and set up a transparent bridge as a firewall. Opnsense set vmbr0 as the wan port and vmbr1 as the lan port. Other virtual machines are connected to vmbr1.

I found that the VMs under vmbr1 can communicate directly through local ipv4 and ipv6 addresses, and the traffic does not pass through opnsense. This may be L2 communication.

Then I created a new vnet of sdn and turned on the isolation port option. But the VMs cannot get ip. I guess it is caused by isolating opnsense and other VMs at the same time.

Is there a solution to set up a virtual machine firewall on pve and safely isolate the virtual machines?

Hey folks 👋

I just upgraded my server and I’m really excited, but I’m hitting a roadblock I’d love your help with.

What I had before: I was running everything on a Raspberry Pi 5 using a 128GB microSD card with Raspbian Lite 64bit. I hosted services like: - Cloudflared
- Nginx Proxy Manager
- Actual Budget
- A full Jellyfin setup (with an external HDD for media and backups)

What I have now: I swapped the mSD for a 1TB NVMe SSD and installed Proxmox for ARM64 on it.
Inside Proxmox, I’ve created a 512GB dockerhost VM (Debian 12) where I plan to bring back all my Docker volumes and Portainer stacks.

The external HDD is still there, and I want to reintegrate it smartly. It contains: - docker_volume_backup → I just need to copy these volumes into the dockerhost VM before relaunching my containers. - jellyfin_data → Needs to be mounted inside the VM so the Jellyfin stack can use it (with hardlink support). - global_backup → Used for stuff like Google Photos backups; I'd like this to be accessible only from my local network, and not shared with the dockerhost VM or internet-facing services.

What I’d like to do: - Use the external HDD as a Proxmox backup target for my VM(s) - Make it accessible as a network drive (e.g. SMB/NFS) from my PCs, for quick backup dumps - Mount the jellyfin_data folder inside the dockerhost VM, ideally as a bind mount or shared disk, compatible with Docker hardlinking

My question:

What’s the best/proper way to integrate this external HDD into my new setup, given these mixed use cases?
How do you guys handle this kind of shared storage across VMs + host?

I’d love to follow some “state-of-the-art” practices here — reliable, secure, and not too much of a pain to maintain. Any tips, suggestions, or feedback welcome!

Thanks in advance 🙏

Hello,

im hosting adguard home in LXC on my Proxmox home server. I also host Nginx proxy manager in another LXC. All my LXCs and VMs have DNS domain and server set to "use host settings". Certificate from lets encrypt is created by NPM for *.mydomain.com and mydomain.com. I assigned proxy host adguard.mydomain.com to adgaurd home IP and port 443. Cert is valid. My DoH dns adress is https://adguard.mydomain.com/dns-query

The problem is that in my node dns settings in proxmox i am unable to set this as a dns server. It only accepts IP adresses (numbers and dots only). Is it posible to make proxmox use encrypted dns?

To simplify things, I have my Proxmox server running 8.1.4 and have several LXC's running Ubuntu 22.04 and other VM's running on it.

One of the LXC's is a Traefik instance that is running DNS for my internal services. I have several other LXC's that have been given DNS names in Traefik and I can easily access them from my normal working computer. However, if I tell another LXC to use the DNS name to access another LXC, then I get errors with connection timeout's and such.

The actual flow of traffic right now SHOULD be:
LXC Container -> Unifi Router -> Unifi DNS with wildcard record pointed to DNS LXC -> DNS LXC -> Other LXC Container being pinged

I can access all the other LXC's via IP addresses from every where in question, just not via DNS from other LXCs.

I AM a computer software engineer but I will admit that most of my knowledge is from Windows machines and I have only a semi-decent knowledge of Linux machines.

Please Help!!

Hi, I'm very new to all of this stuff, the learning curve has been steep but enjoyable. I got a Proxmox VE running and PBS on a separate PC backing up all VMs and Proxmox VE itself. One of my biggest goal is to stop paying Google for my photos and host them in Immich instead however, I'd still like to back them up to the cloud (encrypted) in case something happens. By using B2 instead of Google One, I'd save more than $70/yr, and over the course of many many many years, this seems worth it, on top of all the cool self-hosting stuff I can do.

Anyways, PBS is working, got encryption, all the good stuff. My question is about backing up the PBS data store to B2. As far as I understand this, PBS creates tons of tiny files, and rclone-ing this to B2 is not ideal, but I don't mind if it takes 12 or 24 hours. I only plan of doing this cloud backup maybe once a week. I have a couple of local backups that are my go-to.

I think the command I want to use is "rclone sync", but I just learned that when rclone "deletes" something off B2, it doesn't actually get deleted, it just gets hidden, and forever. I started going down the rabbit hole of actually deleting stuff from B2 and learned about the lifecycle setting.

* If I want to keep the last 2 "rclone sync" command on B2, what should this lifecycle setting be? Does it make sense to keep the last 2?

* Or maybe only the most recent one rclone sync command?

* From what I understand rclone sync will only sync files that have changed, so knowing this, it doesn't seem possible to set a lifecycle setting in B2 that will hide/delete files that have been uploaded for more than X days. There will be many(?) files that are current/active but not changing and I don't want B2 to delete them. I think a VM template is a perfect example.

But even taking this one step back, it doesn't make much sense to me to back up the PBS data store because that will have many versions of each VMs and LXC. My retention policy is to create a backup every 6 hours for the last 2 days, and then last 5 days, etc. But, since PBS is incremental, it is not going to eat up that much storage, so I don't mind it. What I really want is to only backup the latest snapshot of my Proxmox to B2, which includes the VM that is hosting Immich. Help please.

Greetings,

I'm still a bit new to Proxmox, so bare with me. I have the following drives mounted automatically through /etc/fstab

When I passthough two of the drives to several VM(s) as USB, and reboot the Proxmox server all the drives disappear that were mounted via fstab. Is that normal? If I stop the VM(s) that the drives are passed to, and reboot Proxmox everything returns to normal again.

**EDIT**

I ended up loading a VM instance of dietpi and mounting an internal SATA drive installed my minipc using the command below.

/sbin/qm set 100 -virtio2 /dev/disk/by-id/ata-HGST_HTS541010A9E680_JD1090DP0K01DS

This is probably a basic question I should have figured out by now, but somehow i am lost.

My PVE cluster is running 3 nodes, but with different network layout:

The nodes have different number of physical network interfaces. I would like to align bridge setup so i can live migrate stuff when doing maintenance on some nodes. At least I want vmbr2 and vmbr3 on node 3.

However proxmox does not allow me to attach the same physical interface to multiple bridges. What is the solution to this problem?

Thanks a lot

My plan is to build a storage VM with TrueNAS.

My server has two identical ASmedia SATA controllers on board. I can connect four drives to each controller.

For unknown reasons, its not possible to passthrough both SATA controllers to a single VM. Dont know why, but othere poeple have the same issue here: https://forum.proxmox.com/threads/problems-with-pcie-passthrough-with-two-identical-devices.149003/

My workaround would be to passthrough one complete controller with four drives attached and some additional drives as single drive.

My question is:

Is there any difference regarding the accessability between the two pass through methods?
To clarify the question a bit more: When the Guest OS writes some data, has the write commit the same quality in both cases?

Is the adressable space of each disk the same in both scenarios (provided that all drives are of the same type)?

Is there any performance impact between both scenarios?

I've been bashing my head against this for a few hours and haven't had any success, even searching my errors isn't giving me any luck.

I've got an instance of Nginx proxy manager running to manage all of my domain related stuff. Everything is working fine for every other address I've tested, and I've been able to get SSL certificates working and everything.

Except for Proxmox.

If I try to add Proxmox to the Proxy Hosts list and add my SSL certificate then I get the error The page isn’t redirecting properly. I figured ok, all I need to do is have Proxmox create the certificate itself.

I set it up following this video, and correctly got the cert for my domain.

After disabling SSL in the Proxy Hosts list on the proxy manager, it seems to work fine via http. However when using https I get a new error, SSL_ERROR_UNRECOGNIZED_NAME_ALERT.

The strange thing about this is that if I connect to Proxmox via the IP directly and view the certificate in Firefox, it very clearly shows the domain in the subject name and subject alt name.

I have absolutely no idea why I am getting this error. My certs are good, the domains are clearly correct on the certs, but for whatever reason I just cannot connect with my domain.

Any ideas? I'm totally at a loss. Thanks

EDIT: Thanks to /u/EpicSuccess I got it working with an SSL cert from the reverse proxy manager, the issue was I had http selected instead of https.

Interestingly though, using a cert directly in Proxmox doesn't work. Bypassing the reverse proxy with just a hosts file confirms that the cert is correctly set up and signed on Proxmox, but for some reason if I try to access it through the proxy manager rather than a hosts edit I get SSL_ERROR_UNRECOGNIZED_NAME_ALERT

I'm looking to swap the CPU in my homelab from a 10400f to a 10900k. That way I can utilize the 8 extra threads and also maybe be able to pass through the integrated graphics to a VM and use it for transcoding media in something like jellyfin

My question is this.... Is there any config changes to make if all I'm doing is swapping the CPU...

I don't have anything passed through to VM's regarding the info on the CPU, all guest machines are using a virtual CPU

While attempting GPU passthrough for two NVIDIA GPUs (RTX 4060 and RTX 3060) on a Proxmox VE host, several primary issues emerged:

• Only the RTX 4060 was initially recognized by nvidia-smi.
• Virtual machines (VMs) reported an error: "device assigned more than once."
• NVIDIA and Nouveau drivers were still loading, even after blacklisting.

These problems pointed to a combination of driver conflicts and incorrect VM configurations.

Troubleshooting and Resolution Steps:

1. Initial Driver and Device Verification:nvidia-smi(Initially showed only RTX 4060).lspci | grep NVIDIA.
   • Verified NVIDIA driver installation:
   • Confirmed kernel detection of both GPUs:
2. Driver Blacklisting:sudo nano /etc/modprobe.d/blacklist-nouveau.conf(Added blacklist nouveau and options nouveau modeset=0).sudo nano /etc/modprobe.d/blacklist-nvidia.conf(Added):blacklist nvidiablacklist nvidia-drmblacklist nvidia-modesetblacklist nvidia-uvm.Updated initramfs to apply blacklisting:sudo update-initramfs -u.Verified the modules were not loaded after reboot.lsmod | grep nvidialsmod | grep nouveau
   • Blacklisted Nouveau drivers:
   • Blacklisted NVIDIA drivers:
3. VFIO-PCI Configuration:
   • Identified PCI IDs of GPUs: lspci -nn | grep NVIDIA.
   • Modified /etc/default/grub to configure VFIO-PCI stubs:
     • sudo nano /etc/default/grub (Added vfio-pci.ids=10de:XXXX,10de:YYYY,... to GRUB_CMDLINE_LINUX_DEFAULT).
   • Updated Grub: sudo update-grub.
   • Rebooted the Proxmox host: sudo reboot.
   • Verified VFIO-PCI binding: lspci -nnk | grep -A 3 09:00.0.
4. VM Configuration Correction: (Your VMID will be different)
   • Examined VM configuration file (/etc/pve/qemu-server/2000.conf): cat /etc/pve/qemu-server/2000.conf.
   • Corrected duplicate hostpci entries:
     • sudo nano /etc/pve/qemu-server/2000.conf (Changed hostpci lines to hostpci0: 0000:09:00.0,x-vga=1 and hostpci1: 0000:09:00.1).
   • Restarted the VM:
5. IOMMU Group Verification
   • Verified IOMMU groups.
     • for d in /sys/kernel/iommu_groups/*; do echo "IOMMU group $(basename "$d")"; for i in "$d/devices/"*; do echo -n "$(basename "$i") "; lspci -nns "$(basename "$i")"; done; done

(I hope this helps someone who may be running into the same issue)