Change column value in a dataframe spark scala

Question

This is how my dataframe looks like at the moment

+------------+
|    DATE    |
+------------+
|    19931001|
|    19930404|
|    19930603|
|    19930805|
+------------+

I am trying to reformat this string value to yyyy-mm-dd hh:mm:ss.fff and keep it as a string not a date type or time stamp.

How would I do that using the withColumn method ?

Answer 1

Here is the solution using UDF and withcolumn, I have assumed that you have a string date field in Dataframe

//Create dfList dataframe
  val dfList = spark.sparkContext
    .parallelize(Seq("19931001","19930404", "19930603", "19930805")).toDF("DATE")


  dfList.withColumn("DATE", dateToTimeStamp($"DATE")).show()

  val dateToTimeStamp = udf((date: String) => {
    val stringDate = date.substring(0,4)+"/"+date.substring(4,6)+"/"+date.substring(6,8)
    val format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
    format.format(new SimpleDateFormat("yyy/MM/dd").parse(stringDate))
  })