Search code examples
iosswiftmetalcore-imagecikernel

Core image filter with custom metal kernel doesn't work

I've made a custom CIFilter based on a custom kernel, I can't make it work the output image is filled with black and I can't understand why. Here is the shader:

// MARK: Custom kernels
    float4 eight_bit(sampler image, sampler palette_image, float paletteSize) {
        float4 color = image.sample(image.coord());
        float dist = distance(color, palette_image.sample(float2(0,0)));
        float4 returnColor = palette_image.sample(float2(0,0));
        for (int i = 1; i < floor(paletteSize); ++i) {
            float tempDist = distance(color, palette_image.sample(float2(i,0)));
            if (tempDist < dist) {
                dist = tempDist;
                returnColor = palette_image.sample(float2(i,0));
            }
        }
        return returnColor;
    }

The first sampler is the image that needs to be elaborated the second image is and image that contains the colors of a specific palette that must be used in that image.
The palette image is create from an array of RGBA values, passed to a Data buffer an created by using this CIImage initializer init(bitmapData data: Data, bytesPerRow: Int, size: CGSize, format: CIFormat, colorSpace: CGColorSpace?). The image is 1px in height and number of color wide. The image is obtained correctly and it looks like that:
Image from palette
Trying to inspect the shader I've found:

  • If I return color I get the original image, thus means that the sampler image is passed correctly
  • If I try to return a color from any pixel in palette_image the resulting image from the filter is black

I'm starting to think that the palette_image is somehow not passed correctly. Here how the image is passed through the filter:

 override var outputImage: CIImage? {
        guard let inputImage = inputImage else
        {
            return nil
        }
        let palette = EightBitColorFilter.palettes[Int(0)]
        let paletteImage = EightBitColorFilter.image(from: palette)
        let extent = inputImage.extent
        let pixellateImage = inputImage.applyingFilter("CIPixellate", parameters: [kCIInputScaleKey: inputScale])
//        let sampler = CISampler(image: paletteImage)
        let arguments = [pixellateImage, paletteImage, Float(palette.count)] as [Any]

        let final = kernel.apply(extent: extent, roiCallback: {
                (index, rect) in
                return rect
        }, arguments: arguments)

        return final
    }
Solution

  • Your sampling coordinates are off.

    Samplers use relative coordinates in Core Image, i.e. (0,0) corresponds to the upper left corner, (1,1) the lower right corner of the whole input image.

    So try something like this:

    float4 eight_bit(sampler image, sampler palette_image, float paletteSize) {
    float4 color = image.sample(image.coord());
    // initial offset to land in the middle of the first pixel
    float2 firstPaletteCoord = float2(1.0 / (2.0 * palletSize), 0.5);
    float dist = distance(color, palette_image.sample(firstPaletteCoord));
    float4 returnColor = palette_image.sample(firstPaletteCoord);
    for (int i = 1; i < floor(paletteSize); ++i) {
        // step one pixel further
        float2 paletteCoord = firstPaletteCoord + float2(1.0 / paletteSize, 0.0);
        float4 paletteColor = palette_image.sample(paletteCoord);
        float tempDist = distance(color, paletteColor);
        if (tempDist < dist) {
            dist = tempDist;
            returnColor = paletteColor;
        }
    }
    return returnColor;
}