Even the early church father are against the Roman Catholic beliefs, read this man's words closely.
Johann Joseph Ignaz von Dollinger
Dollinger taught Church history as a Roman Catholic for 47 years in the 19th century and was one of the greatest and most influential historians in the Church of his day. He sums up the Eastern and Western understanding of Matthew 16 in the patristic period:
In the first three centuries, St. Irenaeus is the only writer who connects the superiority of the Roman Church with doctrine; but he places this superiority, rightly understood, only in its antiquity, its double apostolical origin, and in the circumstance of the pure tradition being guarded and maintained there through the constant concourse of the faithful from all countries. Tertullian, Cyprian, Lactantius, know nothing of special Papal prerogative, or of any higher or supreme right of deciding in matter of doctrine. In the writings of the Greek doctors, Eusebius, St. Athanasius, St. Basil the Great, the two Gregories, and St. Epiphanius, there is not one word of any prerogatives of the Roman bishop. The most copious of the Greek Fathers, St. Chrysostom, is wholly silent on the subject, and so are the two Cyrils; equally silent are the Latins, Hilary, Pacian, Zeno, Sulpicius, and St. Ambrose.
St. Augustine has written more on the Church, its unity and authority, than all the other Fathers put together. Yet, from all his numerous works, filling ten folios, only one sentence, in one letter, can be quoted, where he says that the principality of the Apostolic Chair has always been in Rome—which could, of course, be said then with equal truth of Antioch, Jerusalem, and Alexandria. Any reader of his Pastoral Letter to the separated Donatists on the Unity of the Church, must find it inexplicable...that in these seventy–five chapters there is not a single word on the necessity of communion with Rome as the centre of unity. He urges all sorts of arguments to show that the Donatists are bound to return to the Church, but of the Papal Chair, as one of them, he says not a word.
We have a copious literature on the Christian sects and heresies of the first six centuries—Irenaeus, Hippolytus, Epiphanius, Philastrius, St. Augustine, and, later, Leontius and Timotheus—have left us accounts of them to the number of eighty, but not a single one is reproached with rejecting the Pope’s authority in matters of faith.
All this is intelligible enough, if we look at the patristic interpretation of the words of Christ to St. Peter. Of all the Fathers who interpret these passages in the Gospels (Matt. xvi.18, John xxi.17), not a single one applies them to the Roman bishops as Peter’s successors. How many Fathers have busied themselves with these texts, yet not one of them whose commentaries we possess—Origen, Chrysostom, Hilary, Augustine, Cyril, Theodoret, and those whose interpretations are collected in catenas—has dropped the faintest hint that the primacy of Rome is the consequence of the commission and promise to Peter! Not one of them has explained the rock or foundation on which Christ would build His Church of the office given to Peter to be transmitted to his successors, but they understood by it either Christ Himself, or Peter’s confession of faith in Christ; often both together. Or else they thought Peter was the foundation equally with all the other Apostles, the twelve being together the foundation–stones of the Church (Apoc. xxi.14). The Fathers could the less recognize in the power of the keys, and the power of binding and loosing, any special prerogative or lordship of the Roman bishop, inasmuch as—what is obvious to any one at first sight—they did not regard a power first given to Peter, and afterwards conferred in precisely the same words on all the Apostles, as anything peculiar to him, or hereditary in the line of Roman bishops, and they held the symbol of the keys as meaning just the same as the figurative expression of binding and loosing (Janus (Johann Joseph Ignaz von Dollinger), The Pope and the Council (Boston: Roberts, 1869), pp. 70-74).
And to think we haven't even started with scripture yet.