How to escape history expansion exclamation mark ! inside a double quoted string?

EDIT: the command substitution is not necessary for the surprising behavior, although it is the most common use case. The same question applies to just echo "'!b'"

b=a

# Enable history substitution.
# This option is on by default on interactive shells.
set -H

echo '!b'
# Output: '!b'
# OK. Escaped by single quotes.

echo $(echo '!b')
# Output: '!b'
# OK. Escaped by single quotes.

echo "$(echo '$b')"
# Output: '$b'
# OK. Escaped by single quotes.

echo "$(echo '!b')"
# Output: history expands
# BAD!! WHY??

In the last example, what is the best way to escape the !?
Why was it not escaped even if I used single quotes, while echo "$(echo '$b')" was? What is the difference between ! and $?
Why was does echo $(echo '!b') (no quotes) work? (pointed by @MBlanc).

I would prefer to do this without:

set +H as I would need set -H afterwards to maintain shell state

backslash escapes, because I need one for every ! and it has to be outside the quotes:

echo "$(echo '\!a')"
# '\!a'.
# No good.

echo "$(echo 'a '\!' b '\!' c')"
# a ! b ! c
# Good, but verbose.

echo $(echo '!b') (no quotes), because the command could return spaces.

Version:

bash --version | head -n1
# GNU bash, version 4.2.25(1)-release (i686-pc-linux-gnu)

Solution

In your last example,

echo "$(echo '!b')"

the exclamation point is not single-quoted. Because history expansion occurs so early in the parsing process, the single quotes are just part of the double-quoted string; the parser hasn't recognized the command substitution yet to establish a new context where the single quotes would be quoting operators.

To fix, you'll have to temporarily turn off history expansion:

set +H
echo "$(echo '!b')"
set -H