Agreed. Refactoring is for a purpose, and there are many possible forms you might refactor into.
I disagree with your suggestion that there are an infinite number of forms you could refactor into. For there to be an infinity of forms, it would require an infinite memory space within the computer in which to store some of the more long-winded versions. With a finite computer memory, there will always be a (large) finite number of possibles. But this isn't about your misunderstanding of the extent of infinity.
The old adage "Red-green-refactor" - for write failing test, make it pass, and refactor later - doesn't quite do it for me. I often add a refactoring step up front too. The way I see it, the refactoring you do at the end is the equivalent of reviewing how straightforward the code is, once the tests are passing. Is it self-explanatory? Does it violate the Principle of Least Surprise? Are there any nasty surprises in there? If there are, then you fix them. If there's unnecessary complexity left in from a previously more complex form, then you reduce it. If you've increased complexity by bolting onto an inappropriate pattern, then you change the pattern to better accommodate what you've done. Then you stop before you waste more time.
For me, the opportunity to refactor BEFORE a particular enhancement provides the ability to switch something to a pattern which better accommodates the forthcoming change - provided it can be predicted that the change requires it. The extent of refactoring you do before or after a change will vary.
I was unable to post it to that site for some reason. Still, I've had my say now.