How about Romans 8:28
When one knows God, it's OK to be sad, it's only human.
But even when sad, I could still feel joyful.
Difficult to explain...
Off course it's ok to be sad. I'm not saying it's not. You know me well enough to know and acknowledge that.
You quote
Romans 8:28
And we know that all things work together for good to those who love God, to those who are the called according to His purpose.
Praise God he works good for those who love God. In the crap of life God promises to work good in it, not make it good but work in it and all the sadness/regret and so on.
But let's take it beyond V:28
Romans 8:29-30
For whom He foreknew, He also predestined to be conformed to the image of His Son, that He might be the firstborn among many brethren. Moreover whom He predestined, these He also called; whom He called, these He also justified; and whom He justified, these He also glorified.
He works good for those who love God, yes to deal with being sad but he takes it further "To be like Jesus"
That being the case we are justified because of the firstborn.
When we get that fully get it we have nothing to be sad about.
Truth is we don't fully get it and that's the problem.
Further what we do not fully get yet if we did it would so make a difference
John 17:23
I in them, and You in Me; that they may be made perfect in one, and that the world may know that You have sent Me, and have loved them as You have loved Me.
GOD LOVES US AS MUCH AS HE LOVES JESUS.
To me when fully get this then we are released from sadness to the joy you mention.
Yes we may be sad at times and God gets it, we are made from dust.
Get that as well then we get God