I want to deploy multiple guest executables in one node but I'm not sure how does it works behind the screens? How are the VMs resources divided between each executable? Is it done in an efficient way? Do I need to configure something for getting all the executable well packed in the VM to save memory? How can I know how many executables can be run on the same VM?
In order to understand how Service Fabric allocates services inside nodes you need to understand its hosting model.
Service Fabric hosting model is very powerful when using the Service Fabric programming model (Stateless and Stateful services and Actors), but more limited with Guest Executables and Containers, although it's still efficient.
Basically, Service Fabric will activate a ServicePackage (a process) for each Guest Executable (service). Depending on the configured number of instances for each service, Service Fabric will either run one instance of each service per node (if InstanceCount = -1) or it will distribute the instances across all the nodes (if InstanceCount is a number lower than the number of nodes).
There is no limitation on the number of services you can run on a single node, but each service will consume resources (CPU, RAM, ports, etc) and it can become a problem. In that case, you have several options:
Increase the size of the VM for that node type
Scale out the node type (add more nodes) and specify a lower number of instances per service (so that not every node has all services)
Create more node types and organize the services accordingly using Placement Constraints (for example, you could have High Compute nodes, High Ram nodes, Public Nodes, Backend Nodes, Financial Nodes, Analytics Nodes... it depends on what makes sense for your scenario).
How can I know how many executables can be run on the same VM?
As I mentioned before, this does not depend on the quantity, but in the resources used by these executables and the size of the VM. Depending on your services, you might be able to estimate the resources they need, but you'll definitely need to test and monitor your cluster because no amount of calculations beats reality.
Update: add interesting links
You can help Service Fabric manage your cluster more efficiently by making your services report dynamic metrics and also by providing limits to what resources a single service can take (to avoid a service from consuming all the memory of a node for example):