I wonder what is the difference between a file placed under group_vars/ and group_vars/all/ directories? Aren't they serve for the same purpose of storing inventory group files?
If you place the vars in group_vars/all.yml they will be loaded for the all group (i.e. all machines except the implicit localhost)
If you want to separate your vars for the all group in different files, you can place all the different yaml files in the group_vars/all/ directory. They will all be loaded for the all group in the same way.
You cannot mix the 2 possibilities: if you have a all directory in group_vars, this will take precedence over the all.yml file.
Although the all group was used in the above examples to stick to your question, the same is true for any group in your inventory.