macos - Swift : Convert byte array into CIImage

Question

I am looking for a way to convert a byte array to CIIimage so that I can feed it into a ML model for classification. I am using a REST HTTP server, where I send a POST request to the server with payload as the image. The image bytes received by the server needs to be processed and converted into a CIImage format for MAC OS so that it can be fed into a ML model which accepts requests of type VNImageRequestHandler(ciImage: <ciimage>).

Could someone give an example to do this in swift ?

VNImageRequestHandler : NSObject

let data = Data(bytes) let imgHandler = VNImageRequestHandler(ciImage: data) The above data variable needs to be typecased to be of type CIImage.

On the HTTP server side, I am receiving the bytes of the image like this: imageData = request.body.bytes

Answer 1

Convert byte array to CGImage using this method. You must make it sure that your bytes are rgba 32 bit pixel raw bytes.

func byteArrayToCGImage(raw: UnsafeMutablePointer<UInt8>, // Your byte array
                        w: Int, // your image's width
                        h: Int // your image's height
    ) -> CGImage! {

    // 4 bytes(rgba channels) for each pixel
    let bytesPerPixel: Int = 4
    // (8 bits per each channel)
    let bitsPerComponent: Int = 8

    let bitsPerPixel = bytesPerPixel * bitsPerComponent;
    // channels in each row (width)
    let bytesPerRow: Int = w * bytesPerPixel;

    let cfData = CFDataCreate(nil, raw, w * h * bytesPerPixel)
    let cgDataProvider = CGDataProvider.init(data: cfData!)!

    let deviceColorSpace = CGColorSpaceCreateDeviceRGB()

    let image: CGImage! = CGImage.init(width: w,
                                       height: h,
                                       bitsPerComponent: bitsPerComponent,
                                       bitsPerPixel: bitsPerPixel,
                                       bytesPerRow: bytesPerRow,
                                       space: deviceColorSpace,
                                       bitmapInfo: [],
                                       provider: cgDataProvider,
                                       decode: nil,
                                       shouldInterpolate: true,
                                       intent: CGColorRenderingIntent.defaultIntent)



    return image;
}

Using this method, you can convert to CIImage like this.

let cgimage = byteArrayToCGImage(raw: <#Pointer to Your byte array#> ,
                                 w: <#your image's width#>,
                                 h: <#your image's height#>)
if cgimage != nil {
    let ciImage = CIImage.init(cgImage: cgimage)
}

According to the comment, your datas might be RGB raw bytes rather than RGBA. In this case, you will have to allocate new buffer, put 255 for each alpha channel manually and pass that buffer to the method.

Updated for convertion of 32 bits RGB to 32 bits RGBA

func convertTo32bitsRGBA(from32bitsRGB pointer: UnsafeMutablePointer<UInt8>!,
                         width: Int,
                         height: Int) -> UnsafeMutablePointer<UInt8> {

    let pixelCount = width * height
    let memorySize = pixelCount * 4
    let newBuffer = malloc(memorySize).bindMemory(to: UInt8.self, capacity: width * height)

    var i = 0;
    while(i < pixelCount) {
        let oldBufferIndex = i * 3;
        let newBufferIndex = i * 4;

        // red channel
        newBuffer.advanced(by: newBufferIndex).pointee = pointer.advanced(by: oldBufferIndex).pointee
        // green channel
        newBuffer.advanced(by: newBufferIndex + 1).pointee = pointer.advanced(by: oldBufferIndex + 1).pointee
        // blue channel
        newBuffer.advanced(by: newBufferIndex + 2).pointee = pointer.advanced(by: oldBufferIndex + 2).pointee
        // alpha channel
        newBuffer.advanced(by: newBufferIndex + 3).pointee = 0xff;

        // &+ is used for little performance gain
        i = i &+ 1;
    }


    return newBuffer;
}

You can call the converter method with your rgb image buffer as follow

let newImageBuffer = convertTo32bitsRGBA(from32bitsRGB: <#Your RGB image buffer#>,
                    width: <#Your image pixel row count or width#>,
                    height: <#Your image pixel column count or height#>)

but remember, like in C, C++ or Objective-C, you are responsible to release the memory allocation returned by this method. These are pointers which memory are not managed by compiler.

You can release with simple function.

newImageBuffer.deallocate();

After deallocation, you must not access the deallocated memory. If you do so, you will get BAD_ACCESS_EXC (Bad access exception thrown by OS for accessing memory you do not own).