Association with criminals doesn't mean you are any more culpable than the guy who makes company with ministers, or at least in the legal sense you are no more culpable.
Apply that same idea to Jesus...
I probably should have been more specific about the accusations that the bible mentions. Jesus was looked down on because he kept company with tax-collectors and "wine-bibbers". When asked about why the Pharisees fasted but they (Jesus and his disciples) didn't fast he explained,
"Can the children of the bridechamber mourn, as long as the bridegroom is with them? but the days will come, when the bridegroom shall be taken from them, and then shall they fast."
Right after this he explained the need for repentance and new birth by analogy saying,
"No man putteth a piece of new cloth unto an old garment, for that which is put in to fill it up taketh from the garment, and the rent is made worse. Neither do men put new wine into old bottles: else the bottles break, and the wine runneth out, and the bottles perish: but they put new wine into new bottles, and both are preserved." (Mat 9:16-17 KJV)
My thought is that the fruit of the earth (the grape) will produce joy. The "sin" analogy of the fermentation process wasn't considered or discussed nor is it universal.
The parallel passage found in Luke 5 shows more detail (for those interested). Here we see a publican named Levi who Jesus associated with and told to follow him. In another passage we see that Jesus also put a point on this teaching when he told the parable of the Pharisee and the Publican stating that the prayers of the humble were heard and the prayer of the proud was not:
(Luk 5:27-35, 39 KJV) - "And after these things he went forth, and saw a publican, named Levi, sitting at the receipt of custom: and he said unto him, Follow me. And he left all, rose up, and followed him. And Levi made him a great feast in his own house: and there was a great company of publicans and of others that sat down with them. But their scribes and Pharisees murmured against his disciples, saying, Why do ye eat and drink with publicans and sinners? And Jesus answering said unto them, They that are whole need not a physician; but they that are sick. I came not to call the righteous, but sinners to repentance. And they said unto him, Why do the disciples of John fast often, and make prayers, and likewise the disciples of the Pharisees; but thine eat and drink? And he said unto them, Can ye make the children of the bridechamber fast, while the bridegroom is with them? But the days will come, when the bridegroom shall be taken away from them, and then shall they fast in those days."
In the gospel of Luke we see that Jesus concluded his teaching about old and new wine by saying ... [vs. 39]:
No man also having drunk old wine straightway desireth new: for he saith, The old is better."
Go read what He (Christ) did! There is all of God's Creation, Adam on! And using ALL OR ANY THING OF THEIR 
See Matt. 4:1-10 
