apache spark - Convert to dataframe error

Question

I want to make a dataframe with 110 columns, so i create a class with 110 attributes when i try to convert the rdd to dataframe.

case class Myclass(var cin_nb:String,...........,var last:String)
import sqlContext.implicts._
file2.map(_.split("	")).map(a=>Myclass(a(0),a(1),a(2),a(3),.....a(110)).ToDf()

I got this error:

not enough arguments for method apply: (cin_nb: String,...........,last:String)

i'm using scala and spark 1.6. Thank you

Answer 1

You can't do this because there is a hard limit of 22 columns with case classes / StructType schemas. This is due to the Tuple in scala only supporting 22 elements!! To grow a dataframe to more columns you need to expand it using the .withColumn function, or load from file directly into a Dataframe. For example, from parquet, or using the databricks csv parser.

Edit: An example of how to do this with .withColumn

import scala.util.Random

val numCols = 100
val numRows = 5
val delimiter = "	"

def generateRowData = (0 until numCols).map(i => Random.alphanumeric.take(5).mkString).mkString(delimiter)

val df = sc.parallelize((0 until numRows).map(i => generateRowData).toList).toDF("data")

def extractCol(i: Int, sep: String) = udf[String, String](_.split(sep)(i))

val result = (0 until numCols).foldLeft(df){case (acc,i) => acc.withColumn(s"c$i", extractCol(i,delimiter)($"data"))}.drop($"data")

result.printSchema
result.show