Is your husband a disciple of Jesus Christ, a Christian? If so, has he been properly taught the nature of marriage from God's perspective, laid out in the Bible? It doesn't sound like he has.
The married couple "forsake all others," they "leave and cleave" to one another, in that they make their relationship to one another their primary, their most precious and intimate, relationship, sacrificing investment in all other relationships in order to nurture and protect their marriage to one another. This is God's design for marriage. And so, any relationship that destructively infringes upon, or diminishes, or distracts from, a couple's marriage relationship must be forsaken.
It seems to me that you know that your husband's "thing" for this younger woman is not right. And no, you're not making too big a thing of his infatuation with her. One of the simplest ways we can tell if what we're doing is wrong is whether or not we hide what we're doing from others. Your husband, it seems to me, has been hiding his interactions with this other gal, his doing so only coming to light when your search exposed that he had. This says a great deal about the nature of his interest in this younger woman - and about him, too, quite frankly.
I think his accusation that you were having an affair is actually a bit of projection going on - especially without any concrete evidence that you're cheating on him. It's pretty common for human beings to accuse others of the things of which they are themselves most guilty. The liar tends to think everybody else is lying just like they do; the porn addict assumes everyone else looks at porn, too; the gossip is certain all her friends backstab her, and so on. Your husband's projecting accusation that you're having an affair doesn't mean necessarily that he's having one, but it sure suggests he's seriously thinking about it.
I say all this having only one side of the story. Perhaps if I had your husband's end of things, I'd have different remarks to make to you.