By 'us' he means those who have been called. He is not wishing any of his sheep should perish.It’s God’s will that not any should perish.
The Lord is not slack concerning His promise, as some count slackness, but is long suffering toward us, not willing that any should perish but that all should come to repentance. 2 Peter 3:9
For God so loved the world that He gave His only begotten Son, that whoever believes in Him should not perish but have everlasting life. John 3:16