One of the best now arguments for the existence, or alleged existence of infinity in mathematics, outside of set theory, comes from number theory. While infinity is used in the theory of limits and hence the calculus, that symbol is one of convenience, and can be eliminated if necessary, in terms such as “increases without bonds,” what the ancient Greek philosophers would call the “potential infinite,” by contrast to the actual infinite. Strict finitists who would say that there are only a finite number of say natural numbers, 0, 1, 2, 3, … n … such as made by finitists in the following papers:
https://www.math.uni-hamburg.de/home/loewe/HiPhI/Slides/bendegem.pdf
http://www.jstor.org/stable/pdf/2272346.pdf?refreqid=excelsior%3A6e4ad837e563fca884ba1b46e6069c68
file://uofa/users$/users0/a1066120/1468-1449-1-PB.pdf
https://projecteuclid.org/download/pdf_1/euclid.ndjfl/1093634481
https://www.jstor.org/stable/pdf/2273760.pdf?refreqid=excelsior%3A5a69fb898236f1f5f11fb5b134fa9c78
https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1746-8361.1955.tb01332.x
The conventional Platonistic mathematician will argue that given a hypothetical “last” number L, simply consider L + 1, to obtain a larger number. The question-begging assumption is made that L + 1 actually denotes, but we ignore that. The argument assumes that numbers exist in some non-material realm of abstract essence, as has been done from Plato to Gödel. That view is subject to metaphysical difficulties that mathematicians seldom consider, such as how is mathematical knowledge possible if the entities in question have primarily a non-material existence, since causal interaction with the brain is ruled out:
http://www.columbia.edu/~jc4345/benacerraf%20with%20bib.pdf