Crossplay campaign - always disconnects. Horde works... why?!

as described in the topic:
When trying to start a campaign in crossplay (me on the PC, my buddy on the Xbox X) we always get disconnected. The one not hosting gets kicked out with a “timeout” message. Horde works without problems… any idea?

NAT is fine btw on his xbox, I did the port forwarding manually…