Is it possible for me to take use an image as a green screen type thing like in photo booth where it takes the background out?
Yes, although exactly what you mean by “green screen” will affect the specific answer. Chroma-keying will require a custom CIFilter, as I don't believe Core Image comes with such a filter and I know NSImage by itself doesn't support it.