Now it's 2019 and this is still a problem :-)

I have been struggling with this for a week; I couldn't work out why ICMP from the host and another VM through the pfSense VM would work, but nothing else. I could only SSH into the host if I SSH to the pfSense VM first. In order to have the host be able to connect out I installed Squid and set it up as a transparent proxy, but I shouldn't have had to do this.

Researching, I finally found this thread. I'm replying because I just wanted to say that after I enabled "Disable hardware checksum offload" and pressed save, immediately traffic started flowing to/from the host, and the other VM which had basically been unreachable. No reboot or reconfig or anything else was required.

I now see it's fairly well documented here.. https://docs.netgate.com/pfsense/en/latest/virtualization/virtio-driver-support.html

Perhaps it would be nice if pfSense could automatically disable hardware checksum offload on the virtio driver/NICs :-)

@viragomann But that was after letting him select the .gz in the first place. It was smart enough to know it can't connect to a gzip, but not smart enough to filter out all non-ISO files in the image picker dialog?

I just tried it myself with ESXi 6.7 and it doesn't show any non-ISO files. I couldn't select one even if I wanted to.

Bizarre.

@johnpoz just for learning sake, if this was a cisco router, how would one set the NAT? anyone happen to know the command? is it

enable
configure terminal
ip nat pool name start-ip end-ip {netmask netmask | prefix-length prefix-length }
access-list access-list-number permit source [source-wildcard ]
ip nat inside source list access-list-number pool name
interface type number
ip address ip-address mask
ip nat inside
exit
interface type number
ip address ip-address mask
ip nat outside
end

I've never seen such a thing work with pfSense and a VM host.

pfSense doesn't include the binaries to mount smb/cifs, I don't think nfs is there either (client or server).

Are you trying to copy from the VM host to pfSense, or from pfSense to the VM host? Why not use scp/rsync? The VM host may be able to use ssh as a filesystem, connecting to pfSense. But that wouldn't work the other way.

@tibere86
I'm using Open vSwitch (OVS) instead Linux bridge on PVE.

Show from your PVE:
ip a s
ethtool -I <interface-name-from-previous-command>
and cat /etc/network/interfaces

And why ethX ? Latest PVE using enpX. Or you wrote that just as example? :)

Maybe you must also enable multiqueue inside pfsense VM ?
https://bsdrp.net/documentation/technical_docs/performance

http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/
If one wishes to use multiple queues for an interface in the guest, the driver in the guest operating system must be configured to do so

https://cloudblog.switch.ch/2016/09/06/tuning-virtualized-network-node-multi-queue-virtio-net/
This should be done during interface initialization, for example in a “pre-up” action in /etc/network/interfaces

P.s. Bingo!
(Maybe this step not needed ?)
Add something like in PVE network config:
...
pre-up ethtool -L enpX combined N
...
Then reboot PVE host and check is multiqueue enabled: ethtool -I <PVE-interface-name>

And then https://forum.proxmox.com/threads/kvm-and-multi-queue-nics.27213/ set on PVE side in VM config file (pfsense VM must be stopped!):
...
-netX virtio=XX:XX:XX:XX:XX:XX,bla-bla-bla,queues=N
...

Starting pfsense VM and enable multiqueue within https://www.freebsd.org/cgi/man.cgi?query=vtnet
reboot VM
check is multiqueue worked https://forums.freebsd.org/threads/multiple-network-queues-on-vmx-interface.49080/

P.p.s. https://forum.proxmox.com/threads/virtio-multi-queue-balancing.43744/

Ok, thank you for the clarification. The setup seems correct then.

You probably want to put the pfSense on its untagged interface (ao1) and let vmware do the tagging. In order to pass vlan tags to the VM interface I'm pretty sure you have to put VLAN 4095 on it in vmware.

Moving to Virtualization.

It works, thank You very much!

Hi sorry.

Forget this one, I had found the issue I was having, thanks.

Hi.

Much better
https://xcp-ng.org/ + https://xen-orchestra.com/docs/

Do a fresh Install in your VM, take a Backup for the appliance in Diagnostics > Backup & Restore and Restore this Backup to your VM.
More Information here: https://www.netgate.com/docs/pfsense/backup/configuration-backup-and-restore.html

-Rico

The guide is like that because running with only two NICs total is not recommended.
It states: Host has at least two network interfaces available for WAN and LAN.

If you have used a NIC for the management port then you don't have two available.

However it should work with two NICs total and no dedicated management port as you found.

Steve

@eiger3970 Buy any NIC you want, so long as it's supported by FreeBSD. Stick with Intel if you can.

However not sure if pfSense is clever enough to allocate the 2 ports to LAN and WAN?

pfSense doesn't allocate anything. You define WAN and LAN yourself during setup by telling pfS which NIC is WAN and which is LAN.

refer to this thread
https://social.technet.microsoft.com/Forums/exchange/en-US/ca93a8bc-500a-49e3-be6e-bf3407d8d798/hyperv-is-not-configured-to-enable-processor-resource-controls?forum=win10itprovirt

i used bcdedit /set hypervisorschedulertype classic to fix the latency, intermittent, and bandwidth issues
this is needed on win10 1803 and 1809

also in pfsense under System/Advanced/Networking make sure
Disable hardware checksum offload
Disable hardware TCP segmentation offload
Disable hardware large receive offload
are checkmarked with certain networkcards such as realteks

refer to this thread
https://social.technet.microsoft.com/Forums/exchange/en-US/ca93a8bc-500a-49e3-be6e-bf3407d8d798/hyperv-is-not-configured-to-enable-processor-resource-controls?forum=win10itprovirt

i used bcdedit /set hypervisorschedulertype classic to fix the latency, intermittent, and bandwidth issues
this is needed on win10 1803 and 1809

Did you get your hyper-v pfsense running? If not, feel free to post again. I have a couple of pfsense instances running on a hyper-v server, so it's definitely possible to get it working properly.

i found the link below and a few others on the net but this one explains what i'm trying to do, at least from a vm perspective:

dailysysadmin.com/KB/Article/965/port-mirroring-cisco-switch-virtual-machine-vmware-esxi-host/

made those configurations & mirrored the pfsense LAN switch port to security onion. checking now if i have the VLAN option correct but for now seeing a lot of traffic on the securityonion " ens192 " interface, the one without an ip that, i think, captures on all interfaces. getting there.

i want to get the actual traffic to securityonion for analysis, say versus streaming pfsense syslog to securityonion.

so port mirroring the pfsense LAN port is the way to do so, yes?

@netblues Hi, following up on this, below is a small and crude (sorry...) script for setting up a basic UDP LB with Nginx on-board pfSense. This script assumes that the directory /root/NGINX exists, and you have your custom nginx.conf file in it.

#!/bin/sh if [ -f /usr/local/etc/rc.d/nginx ] then echo "Backup and rename nginx service" cp /usr/local/etc/rc.d/nginx /root/NGINX/nginx-dist mv /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/nginx.sh cp /usr/local/etc/nginx/nginx.conf-dist /root/NGINX/nginx.conf-dist echo 'nginx_enable="YES"' >> /etc/rc.conf.local fi echo "Update nginx config" cp /root/NGINX/nginx.conf /usr/local/etc/nginx/nginx.conf echo "Restart nginx" service nginx.sh restart

...and this is the diff between the default nginx.conf and my custom one, which balances two AWS instances (addresses intentionally changed):

[2.4.4-RELEASE][ec2-user@MY-pfSense.localdomain]/home/ec2-user: diff /usr/local/etc/nginx/nginx.conf-dist /usr/local/etc/nginx/nginx.conf 0a1 > load_module /usr/local/libexec/nginx/ngx_stream_module.so; 15a17 > user root wheel; 122a125,142 > > stream { > > upstream lb_instances { > server 1.1.1.17:1234; > server 1.1.1.147:1234; > server 1.1.1.140:1234; > } > > server { > listen 2.2.2.1:5678 udp; > proxy_pass lb_instances; > proxy_bind $remote_addr:$remote_port transparent; > proxy_responses 0; > } > } >

It seems that the failover feature is an Nginx+ feature, which requires a paid subscription.

Thanks a lot for your help!
Erez