At one time, Freemasonry was a religious/political movement. In the United States, it is quite different and the philanthropic activities of Masons far outweigh any harm done by someone who might take the symbolic language literally.
On the other hand, in some countries like Italy, Masons are a matter of concern. The infamous P2 lodge, for example, which was deeply involved in politics and banking fraud. I have been told that the lodge has since been expelled by the governing body for Freemasonry in Italy, but I'm not sure of that.
There are many sites that promote a rather unrealistic and sinister view of Freemasonry. But these are mostly imaginative, and don't really apply to things as they are in the United States now.
Freemasonry is as susceptible to corruption and being turned to evil as all other human institutions. And, you're right, the religious/political nature of Freemasonry was problematic in the past. The Morgan Affair http://www.kinghiramslodge.org/morgan.html in upstate New York was a particularly nasty example, and it led to a very powerful anti-Masonry reaction throughout the country.