Is it possible for the color to 'erase' the background in CSS?

Question

I really doubt what I am asking is possible but it's still worth a try.

I am trying to create a button that normally has background-color: transparent; color: white; and when you hover over it, those properties should swap. The problem is that if you just swap them then all you see is a white button. If you know the background colour of the containing element then you can get the colour from there but If the button is over an image or a canvas then this won't work.

This is how I've been doing it so far

html,
body {
  height: 100%;
}

#container {
  background-color: #38404D;
  height: 100%;
}

.ghost-button {
  background-color: transparent;
  border: 1px solid #ffffff;
  outline: none !important;
  transition: all 0.8s;
  margin: 10px 10px;
  padding: 6px 7px;
  cursor: pointer;
  color: #ffffff;
}

.ghost-button:hover {
  background-color: #ffffff;
  color: #38404D;
}

.ghost-button:active {
  box-shadow: inset 0 0 8px 0px #888888;
}

<div id="container">
  <button class="ghost-button">Hover Here</button>
</div>

UPDATE

It seems that quite a few people were confused by the question. I am asking if there is a way to do the exact same thing I've done above but on top of an image or a canvas (instead of a solid colour). See example below

html,
body {
  height: 100%;
}

#container {
  background-image: url("http://www.freegreatpicture.com/files/147/17878-hd-color-background-wallpaper.jpg");
  height: 100%;
}

.ghost-button {
  background-color: transparent;
  border: 1px solid #ffffff;
  outline: none !important;
  transition: all 0.8s;
  margin: 10px 10px;
  padding: 6px 7px;
  cursor: pointer;
  color: #ffffff;
}

.ghost-button:hover {
  background-color: #ffffff;
  color: #38404D;
}

.ghost-button:active {
  box-shadow: inset 0 0 8px 0px #888888;
}

<div id="container">
  <button class="ghost-button">Hover Here</button>
</div>

Answer 1

Yes, it IS possible in CSS with mix-blend-mode.

Answer's update in April 2021: Currently it have a very solid support (95% globally) although Safari doesn't have hue, saturation, color, and luminosity blend modes. Of course, IE isn't a considerable thing if you expect to use it (like many of other cool CSS features of the last years).

.ghost-button {
  /* Important part */
  mix-blend-mode: screen;
  background: #000;
  color: #fff;

  /* Button cosmetics */
  border: .125em solid #fff;
  font: 2em/1 Cursive;
  letter-spacing: 1px;
  outline: none !important;
  transition: all .8s;
  padding: .5em 1em;
  cursor: pointer;
}

.ghost-button:hover {
  /* Important part */
  background: #fff;
  color: #000;
}

#container {
  background: url('http://www.freegreatpicture.com/files/147/17878-hd-color-background-wallpaper.jpg') center/cover;

  /* Also works with background-color or gradients: */
  /* background: linear-gradient(to right, red, yellow); */

  /* Container positioning */
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100vh;
}

html, body {
  margin: 0;
}

<div id="container">
  <button class="ghost-button">Hover Here</button>
</div>

As you can see, the secret here is using mix-blend-mode: screen along with the black color for the "erased" part, since black is mixed with the background when using this screen mode.