There is not any research comparing the difference between custom chain in IP tables and IPset.
Can anyone tell me the difference in terms of function and performance between these two please? (especially the efficiency)
If possible to provide any published research paper or information, please?
If you server receives really high traffic, a lot of iptables rules can have significant overhead especially on the CPU. It depends on the types of rules you have and the iptables extensions you use.
You could read these two papers: https://workshop.netfilter.org/2013/wiki/images/a/ab/Jozsef_Kadlecsik_ipset-osd-public.pdf
http://www.netdevconf.org/1.1/proceedings/slides/kadlecsik-ipset-firewalling-iptables.pdf
They provide different scenarios, how iptables works, performance tests and so on.
Not sure if that's what you're looking for but it's a good read nonetheless
UPDATE:
ipset is an extension to iptables that allows you to create firewall rules that match entire "sets" of addresses at once. Unlike normal iptables chains, which are stored and traversed linearly, IP sets are stored in indexed data structures, making lookups very efficient, even when dealing with large sets.
Supposing you want to block 10k ip addresses, with just iptables you'll have to create 10k rules, one for each ip address, while with ipset you can create a single rule for a specific set of those ip addresses. This translates in speed, less rules, less cpu processing power etc