If it's a cononical issue, learn about and use the cononical tag -
http://www.canonicaltag.com/
The latest theory I've heard is that inbound links can only help, not hurt your site as you are not responsible for those that link to you.
I don't buy that theory but it is the one I hear most quoted. Logically, it just seems that too many inbound links from bad neighborhoods would have to affect you.
That many inbound links from that few of sources might be triggering some kind of flag as an attempt to manipulate the SERPS but I'm not an expert.