If everything in scrum is all about functional things that a user can see is there really any place for refactoring code unrelated to any new functional requirements?
I don't think that this has as much to do with Scrum as it does with project management philosophy.
Regardless of whether a project uses Scrum or not, many project managers do not like developers spending time on "unnecessary" things like code refactoring or restructuring that doesn't directly advance one of the outstanding functional requirements. It's not "work that yields results" like normal development, it's "work that prevents a delay of results later". Given the typically short time-lines used for Sprints, the benefit is often hard to see and nearly impossible to quantify.
Keeping code maintainable needs to be an item on your burn-down list (if you use a Scrum). It is just as important as new development. While it may not seem like something that is "visible to the user", ignoring it increases your technical debt. Down the road when the technical debt piles up enough that your code's lack of maintainability slows down development, the delays in new feature development will be visible to customers.
It's all a matter of management/philosophy. Instead of looking at refactoring and maintainability enhancements as "extra" work that doesn't impact customers, it should be viewed as a time investment to prevent customer-visible delays (and potentially bugs as well) down the road. Developers can sometimes see these benefits more clearly than managers can; if your manager doesn't understand the disadvantages of neglecting maintainability, you might want to grab several other developers and have a chat with your manager.