I understand that the two facts are expressed using complex terms which again have complex terms as arguments. There are three levels of terms nested inside terms.
vertical(line(point(X,Y), point(X,Z))).
horizontal(line(point(X,Y), point(Z,Y))).
Does this mean the overall complex term has an arity of one as the other complex terms are nested?
Notice that what you call "complex terms" are normally called coumpound terms in Prolog. You can easily check for the arity of Prolog terms yourself in the following way:
?- functor(vertical(line(point(X,Y), point(X,Z))), _, Arity).
Arity = 1.
As you can see, your intuition was correct in this particular case!
Since you have added the SWI-Prolog tag to your question, it may be useful to know that in SWI 7 there is also compound_name_arity/3, which works on compound terms of arity 0. (In other Prologs a compound term of arity zero would be an atom.) For example:
?- functor(f(), _, Arity).
ERROR: functor/3: Domain error: `compound_non_zero_arity' expected, found `f()'
?- compound_name_arity(f(), _, Arity).
Arity = 0.