/^[a-zA-Z0-9]{1,5}$
For example, I want to match letters or digits less than 5 in length, but the above doesn't work. In particular, it is the quantifier part is wrong.
How to correct it?
This very similar StackOverflow question has a good answer on the same topic.
Your regex isn't working because the default search pattern interpretation mode in Vim requires curly braces beginning a quantifier to be escaped. When the beginning brace isn't escaped using a slash \{, Vim searches for a literal curly brace { instead.
For example, assume we're searching the literal text: abc{1,5}
/[a-z]{1,5} matches c{1,5}/[a-z]\{1,5} matches abcThat being said, your search pattern works when written as /^[a-zA-Z0-9]\{1,5}$ (notice the added leading slash before the quantifier opening brace). It seems like this works fine without escaping the closing brace, however I'm not sure if that will always be the case.
/\v^[a-zA-Z0-9]{1,5}$ will also work. Notice how the leading \v is used to enable "very magic" mode (see :help magic for a full explanation), and that escaping the curly brace is no longer required as a result.
Vim has a few different ways in which the search pattern can be interpreted. See the built-in :help magic page for a full explanation including a table of characters which need to be escaped at each "magic" level.