Why does For Comprehension generator suppress Option type?

In the code below, version 1 gives the correct result. I make a small variation in V2. The None value had disappeared which is Ok as that is how For Expression works. But what is the reason the yield output in V2 no longer respects the data type returned by myList.lift() which is an Option (as in V1)?

val myList = List(12, 34, "ABC")

Version 1

for { i <- (0 to 3).toList } yield myList.lift(i)
// res1: List[Option[Any]] = List(Some(12), Some(34), Some(ABC), None)

Version 2

for { 
  i <- (0 to 3).toList 
  x <- myList.lift(i)
} yield x
// res2: List[Any] = List(12, 34, ABC)

Solution

Desugaring the first case:

// desugar for comprehension:
(0 to 3).toList.map(
  i => myList.lift(i))

Desugaring the second case:

// desugar for comprehension:
(0 to 3).toList.flatMap(
  i => myList.lift(i).map(
    x => x))

// remove .map(x => x):
(0 to 3).toList.flatMap(
  i => myList.lift(i))

// desugar flatMap:
(0 to 3).toList.map(
  i => myList.lift(i)).flatten

The second case simplifies to the first case with a .flatten at the end, which explains the difference in the results: res2 = res1.flatten.

What's actually going on?

Scala can treat Option as a sequence:

Some(foo) --> Seq(foo)
None      --> Seq()

The .flatten is just flattening the sequence of sequences.

If you're curious about the types:

scala.collection.Seq.flatten requires that the 'inner' type have an implicit conversion to GenTraversableOnce[T]
there's a global implicit conversion from Option[T] to Iterable[T]
Iterable[T] <: GenTraversableOnce[T]

What does <- mean then?

The <- in x <- myList.lift(i) doesn"t just assign a variable to a value, it "gets a value out of" myList.lift(i). When you "get a value out of" an Option[T], you get foo for Some(foo) and nothing for None. "Getting nothing" means the yield doesn"t run at all for a None, so nothing shows up in the result for the "iteration" when i = 3.

If you're curious about this "get a value out of" concept that is defined for Seq, Option, and many other types in Scala, it is defined for any Monad.