Search code examples
rapache-sparkrenamesparklyr

Sparklyr - Change columns names in a Spark dataframe

df <- data.frame(old1 = LETTERS, old2 = 1)
df_tbl <- copy_to(sc,df,"df")

df_tbl <- df_tbl %>% dplyr::rename(old1 = new1, old2 = new2)

returns:

> head(df_tbl)
Error: `new1`, `new2` contains unknown variables

Is there an easy way to change the column names using Sparklyr?

Solution

  • First of all you mixed the order:

    df_tbl %>% rename(new1 = old1, new2 = old2)

    but with Sparklyr you have to use select:

    df_tbl %>% select(new1 = old1, new2 = old2)