Search code examples
scalaapache-sparkapache-spark-sql

How to generate a DataFrame with random content and N rows?


How can I create a Spark DataFrame in Scala with 100 rows and 3 columns that have random integer values in range (1, 100)?

I know how to create a DataFrame manually, but I cannot automate it:

val df = sc.parallelize(Seq((1,20, 40), (60, 10, 80), (30, 15, 30))).toDF("col1", "col2", "col3") 

Solution

  • Here you go, Seq.fill is your friend:

    def randomInt1to100 = scala.util.Random.nextInt(100)+1
    
    val df = sc.parallelize(
      Seq.fill(100){(randomInt1to100,randomInt1to100,randomInt1to100)}
    ).toDF("col1", "col2", "col3")