In order to get this to work, you need to specify the IP and port each should listen in the domain configuration file (and NOT in the global web-server config).
nginx configuration for a domain (/etc/nginx/sites-enabled/www.mydomain.com) contains:
server_name http://www.mydomain.com mydomain.com;
Apache configuration for another domain contains:
In this case, this was at the top of /etc/apache2/sites-enabled/myotherdomain.com
WIth this setup, each web server has its own IP to listen to and they both serve on port 80.