When sending the FTP command "PASV", the FTP server responds with information on which host and port a data connection can be established.
This host information can be problematic if there is a NAT involved. Some clients resolve this by comparing the IP of the proposed data connection host with the one from the control connection: If the one proposed after PASV is a local one while the control connection is not using a local one, only the ports given by the PASV response are used while using the IP of the host connection.
As an example, see the Apache Commons implementation: http://svn.apache.org/viewvc/commons/proper/net/trunk/src/main/java/org/apache/commons/net/ftp/FTPClient.java?revision=1788985&view=markup#l3876
My question is now: Why use the host of the PASV answer at all? Why not just always only use the ports of the PASV answer but use the host of the control connection? As far as I know this is how it's done with IPv6 but I never see it with IPv4. Why?
At the time FTP was designed there were enough IP addresses and NAT was not a thing to be considered. But FTP was designed to support a use case where a client initiated a transfer between two different systems - see figure 2 in RFC 959 or read about File eXchange Protocol for details. And for this use case it was necessary that the IP addresses in the PORT or PASV command could be different from the (single) server resp. client IP.
As far as I know this is how it's done with IPv6 but I never see it with IPv4.
You are probably referring to the EPRT and EPSV commands which only take a port number and no IP address. While these commands were introduced since the original PORT and PASV commands did not support IPv4 these command can be used for both IPv6 and IPv4 and several clients use these for IPv4 too, thereby avoiding the problems with the wrong IP address in PORT/PASV.