But how would I set a display filter so it only displays the packet
that has "Broadcast" as their destination port?
There is no broadcast port. Ports are transport protocol addresses, and they are individually assigned to processes.
Broadcast addresses are either IPv4 or data-link addresses. The IPv4 limited broadcast address is 255.255.255.255, ...
What you're looking for is called RSPAN - Remote Switch Port ANalyzer. RSPAN copies frames from a monitored port into a (dedicated) VLAN which you connect to the monitoring station.
Set up a VLAN between the monitored switch and the monitoring station.
Configure RSPAN from the monitored port into the RSPAN VLAN.
I found a detailed description for Brocade ...