Sorting alphabetically, such as with sorted() in Python would yield the following:
a
ab
aa
aab
How might one go about achieving the following instead?
a
aa
ab
aab
More specifically, is there a conventional word for this, or is it otherwise a matter of implementing a custom algorithm to do so?
This depends on, among other things, programming language of your choosing. In, say, Java, you can use Collections.sort(List<T> list, Comparator<? super T> c), for which you pass a comparator that can do whategver you want it to do, like return the comparison value based on similarity. In other languages, or when implementing sorting algorithms yourself, you can just swap standard value comparison for your own methods.
That said, I still don't see what you mean by similarity. As far as I can see, you want to order the strings first by length, and then alphabetically wthin a given length. In this case, the Comparator you would have to pass to the method mentioned above would have to first compare the lengths, and if they're equal, the values of two strings.