Glossary
Improving the internal structure of code without changing its externally visible behaviour.
The second half of that sentence is the decisive one. As soon as behaviour changes it is no longer refactoring, it is a change that needs testing and sign-off.
It pays off where work is about to happen. Refactoring on spec spends time on places that may never be touched again.