Use ggplot2 to plot some layers using a paler, but non-transparent, palette

Question

I would like to make a plot in which the trend lines are shown in strong colour and some individual points are shown as paler/less saturated versions of the same colour palette, but not transparent.

This is an idea of the basic plot. Sorry for some poor aesthetic choices, it's just a simple reproducible example.

diamonds %>%
  filter(color %in% LETTERS[4:6]) %>%
  ggplot(aes(x = as.numeric(cut), y = price, colour = color)) +
  geom_smooth(method = "lm", se = FALSE) +
  stat_summary() +
  scale_colour_brewer(palette = "Set2") +
  theme_classic()

Changing alpha nearly does what I want, but leads to ugly effects where points overlap, and also seems to treat lines and points differently. It's the best I can do currently, but I would much prefer a version that changes the colours themselves rather than manipulating transparency. (In my real plot, there are more values along the x axis and it is not practical to use jittering to the extent required to prevent any overlap)

diamonds %>%
  filter(color %in% LETTERS[4:6]) %>%
  ggplot(aes(x = as.numeric(cut), y = price, colour = color)) +
  geom_smooth(method = "lm", se = FALSE) +
  stat_summary(alpha = 0.5) +
  scale_colour_brewer(palette = "Set2") +
  theme_classic()

The package ggnewscale seems like it should help, but I can't get it to work:

diamonds %>%
  filter(color %in% LETTERS[4:6]) %>%
  ggplot(aes(x = as.numeric(cut), y = price, colour = color)) +
  geom_smooth(method = "lm", se = FALSE) +
  scale_colour_brewer(palette = "Set2") +
  ggnewscale::new_scale_colour() +
  stat_summary() +
  scale_colour_brewer(palette = "Pastel2") + 
  theme_classic()

Answer 1

There are a few ways to do this.

One handy tool you can use is colorspace::lighten, which will produce the colors you want.

You can use the after_scale mechanism in ggplot to apply this function directly to the color palette. This requires (ab)using the fill aesthetic:

diamonds %>%
    filter(color %in% LETTERS[4:6]) %>%
    ggplot(aes(x = as.numeric(cut), y = price, color = color, fill = color)) +
    geom_smooth(method = "lm", se = FALSE) +
    stat_summary(aes(color = after_scale(colorspace::lighten(fill, 0.5)))) +
    scale_colour_brewer(palette = "Set2") +
    scale_fill_brewer(palette = "Set2") +
    theme_classic()

As an alternative, using filled dots might give a nicer appearance, again using the fill aesthetic but this time "properly"

diamonds %>%
    filter(color %in% LETTERS[4:6]) %>%
    ggplot(aes(x = as.numeric(cut), y = price, color = color)) +
    geom_smooth(method = "lm", se = FALSE) +
    stat_summary(aes(fill = after_scale(colorspace::lighten(colour, 0.5))), shape = 21) +
    scale_colour_brewer(palette = "Set2") +
    scale_fill_brewer(palette = "Set2") +
  theme_classic()

You can also use ggnewscale as in the following code. This gives you a clearer legend (though it is somewhat redundant having two legends here)

diamonds %>%
  filter(color %in% LETTERS[4:6]) %>%
  ggplot(aes(x = as.numeric(cut), y = price, color = color)) +
  geom_smooth(method = "lm", se = FALSE) +
  scale_colour_brewer("Trend", palette = "Set2") +
  ggnewscale::new_scale_color() +
  stat_summary(aes(color = color)) +
  scale_color_manual("Mean SE", values = colorspace::lighten(
    RColorBrewer::brewer.pal(3, "Set2"), 0.5)) +
  theme_classic()

Just for completeness, you can avoid additional packages by drawing each summary twice - once in white, then once in color with a reduced alpha. This prevents the color mixing you wish to avoid.

diamonds %>%
  filter(color %in% LETTERS[4:6]) %>%
  ggplot(aes(x = as.numeric(cut), y = price, color = color)) +
  geom_smooth(method = "lm", se = FALSE) +
  stat_summary(data = . %>% filter(color == "D"), color = "white") +
  stat_summary(data = . %>% filter(color == "D"), alpha = 0.5) +
  stat_summary(data = . %>% filter(color == "E"), color = "white") +
  stat_summary(data =. %>% filter(color == "E"), alpha = 0.5) +
  stat_summary(data = . %>% filter(color == "F"), color = "white") +
  stat_summary(data = . %>% filter(color == "F"), alpha = 0.5) +
  scale_colour_brewer(palette = "Set2") +
  theme_classic()