restrictions on generator types in scala for comprehension

I'm trying to understand for comprehensions in scala.

When the first generator is a collection other than Option, the second generator can also be a collection other than Option:

case class D(p1: Option[Int], p2: List[Int])

val d = for
    p1 <- List(1)
    p2 <- List(2) 
yield D(Option(p1), List(p2))
assertEquals(d, List(D(Some(1), List(2))))

However, when the first generator is an Option, the second generator cannot be a collection other than Option (but it can be an Option):

val e = for
    p1 <- Option(1)
    // p2 <- List(2)   // cannot compile: Found: List[D], required: Option[Nothing]
yield D(Option(p1), List(2))
assertEquals(e, Some(D(Some(1), List(2))))

Note that the type of the second generator is List[Int], not List[D] as the Intellij Idea 2022.2.1 compiler says. Using scala 3.1.1.

Thank you.

Solution

for-comprehension is a syntactic sugar for a sequence of flatMaps ending with a map.

Your first example is equivalent to this:

val d2 = List(1).flatMap(p1 => List(2).map(p2 => D(Option(p1), List(p2)))) // ok

If your second example would compile with that line uncommented, it would look like this:

val d3 = Option(1).flatMap(p1 => List(2).map(p2 => D(Option(p1), List(2)))) // compile error

As you can see, this is impossible because a flatMap on an Option must result in another Option not a List[D]. You see List[D] because this is the type of the right hand side of the function literal inside the flatMap: List(2).map(p2 => D(Option(p1), List(2))).

The warning is correct in this case. You want an Option there, not a List[D], but it's harder to understand the warning when using for-comprehension.

Generally, the type checker infers the least upper bound, which is Option[Nothing], since Nothing is the subtype of every other type and Option is covariant. You get Option[Nothing] which is a subtype of Option[Int], so it's type sound.

You can play around with this example here on Scastie.