If we were merely "clay", that is, an inanimate object, issues of justice don't come into play. Since inanimate objects can neither sin nor do what is right, how can in ananimate object be held accountable for things over which it has no control.
Indeed that is the position of Calvinism, namely...