The solution for us was to assign more nic adapters to the virtual machine, so we could stick to 1 IP per nic. This way we don’t have to disable and re-enable multiple IPs per nic, so the port forwarding keeps working even after reboots.
Though this does increase the amount of rules required for things like the firewall.