When there are many connections, we want to reduce the dynamic table size through SETTINGS_HEADER_TABLE_SIZE in settings frame. If the client ignores the settings frame, and does not send a settings frame with an ack flag, will the server use the default value (4096 octets)?If so, the client can send many requests with a 4096 octets dynamic table after receiving a server settings frame. This may cause the server's memory used too much.How to avoid this case?
The client must apply the settings with promptness and send a Settings Ack back, this is not optional. Any other behavior from the client is non-compliant and in those situations the server can close the connection. For the specific case where the client lingers for too long without acknowledging the Setting, the server can close the connection using a GoAway frame with reason SETTINGS_TIMEOUT.
The other thing is that the HPACK dynamic table is "global" for the HTTP/2 connection. So, the only way that an attacker can really abuse the default size is by opening many different connections, not by making many requests on the same connection. In my opinion, you always want to limit the number of connections that can come from a single IP address, otherwise attackers won't even need a botnet to perform a DDOS attack on your server.
Reducing the size of the dynamic table under 4096 bytes could increase headers traffic and maybe be counter-productive for performance.