How does lazy-seq accumulate the result?

Question

Here is the implementation of partition function in clojurescript. The other methods are removed for simplicity.

I have hard time understanding how the lazy-seq accumulates the result. At the end there is a when which if I understand correctly will return nil if the test is false. Where does that nil go in the next iteration of lazy-seq?

(defn partition
  "Returns a lazy sequence of lists of n items each, at offsets step
  apart. If step is not supplied, defaults to n, i.e. the partitions
  do not overlap. If a pad collection is supplied, use its elements as
  necessary to complete last partition up to n items. In case there are
  not enough padding elements, return a partition with less than n items."
  ;; ...
  ([n step coll]
     (lazy-seq
       (when-let [s (seq coll)]
         (let [p (take n s)]
           (when (== n (count p))
             (cons p (partition n step (drop step s))))))))
  ;; ...

Answer 1

The Special Interpretation of nil

cons is a special form (i.e. it is not a function, but a compiler built-in). cons knows that a nil means "no more data is coming".

(cons 7 nil)   => (7)
(cons 7 '())   => (7)
(cons 7  [])   => [7]

So, if either when-let or when fails, nil is returned and we have something like (cons 7 nil). Therefore, the lazy sequence is terminated (the nil is discarded), and at this point it is equivalent to a plain list.

---

Returning nil

Your question surprised me! I hadn't thought it would work, but here is the code:

(defn odd->nil [it]
  (if (odd? it)
    nil
    it))

(defn down-from
  "Count down from N to 1"
  [n]
  (lazy-seq
    (when (pos? n)
      (cons (odd->nil n) (down-from (dec n))))))

(down-from 5) => (nil 4 nil 2 nil)

So we see there is a big difference between nil being the first or 2nd argument to cons. If nil is the first arg, it is added to the beginning of the list as normal. If nil is the second arg, it is (silently) converted into an empty list, and the result is a 1-element list:

(cons nil [99])  => (nil 99)   ; works like normal
(cons  99  nil)  => (99)       ; creates a 1-elem list
(cons nil  nil)  => (nil)      ; for completeness

---

P.S.

Note there is a slight contradiction with seq, since we have:

(seq nil) => nil

---

P.P.S rest vs next

I never use next, since I don't like silent conversions to nil:

(next [1]) => nil
(next [])  => nil
(next nil) => nil

I prefer to use rest, since it will give me an empty list as I expect:

(rest [1]) => ()
(rest [])  => ()
(rest nil) => ()

I can then write a test like so:

  (let [remaining (rest some-seq) ]
    (when-not (empty remaining)      ; says what it means
       ....more processing.... ))

I don't like assumptions about silent conversions:

(when (next some-seq)        ; silently converts [] => nil
  ....more processing.... )  ; & relies on nil <=> false

---

One Last Thing

You may be interested in a small refinement called lazy-cons described here. I think it is a bit simpler than the original lazy-seq.

(defn lazy-countdown [n]
  (when (<= 0 n)
    (lazy-cons n (lazy-countdown (dec n)))))

(deftest t-all
  (is= (lazy-countdown  5) [5 4 3 2 1 0] )
  (is= (lazy-countdown  1) [1 0] )
  (is= (lazy-countdown  0) [0] )
  (is= (lazy-countdown -1) nil ))

It also has a cousin the emulates Python-style generator functions.