Stream remote framebuffer into NSView

Question

I'm trying to implement a simple VNC client in Swift using LibvnccClient.

I have created a subclass of NSView. I create a CGContext and pass the data pointer to the library for use as the framebuffer. libvncclient updates the framebuffer as contents of the screen changes and calls my provided callback.

Here is the relevant code

class VNCClient {

    var localClient: rfbClient
    var view: NSView

    init(view: NSView) {
        guard let client_ptr = rfbGetClient(8, 3, 4) else {
            fatalError("Trouble")
        }

        self.view = view
        self.localClient = client_ptr.pointee

        localClient.MallocFrameBuffer = resize
        localClient.GotFrameBufferUpdate = update
        let fbPointer = UnsafeMutablePointer<UInt8>(OpaquePointer((view as! RFBView).buffer))
        localClient.frameBuffer = fbPointer
        rfbClientSetClientData(&localClient, &viewTag, &self.view)
        var argc: Int32 = 0
        let b = rfbInitClient(&localClient, &argc, nil)
        if b == RFB_TRUE {
            print("Connected!")
        }
        Timer.scheduledTimer(withTimeInterval: 0.001, repeats: true) {_ in
            if WaitForMessage(&self.localClient, 1) > 0 {
                HandleRFBServerMessage(&self.localClient)
            }
        }
    }

func update(client: UnsafeMutablePointer<rfbClient>?, x: Int32, y: Int32, w: Int32, h: Int32) -> Void {
    let cl = client!.pointee
    let view_ptr = rfbClientGetClientData(client, &viewTag)
    let view = view_ptr?.assumingMemoryBound(to: RFBView.self).pointee
    view?.setNeedsDisplay(NSRect(x: Int(x), y: Int(y), width: Int(w), height: Int(h)))
}


class RFBView: NSView {

    let ctx = CGContext(data: nil, width: 800, height: 600, bitsPerComponent: 8, bytesPerRow: 4 * 800, space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue)!

    var buffer: UnsafeMutableRawPointer? {
        return ctx.data
    }
    override func draw(_ dirtyRect: NSRect) {

        let image = ctx.makeImage()

        NSGraphicsContext.current()?.cgContext.draw(image!, in: frame)

   }

It works but the display is not smooth. The server is running on the same machine in a VM so no network issues.

I'm redrawing the whole image for every update which I assume is the cause of the problem. So how can I redraw only the part of the framebuffer that is updated?

Is CoreGraphics fast enough for this or should I use NSOpenGLView?

Answer 1

I assumed that HandleRFBServerMessage would call update function once per iteration if there was an update. But that is not the case.

I replaced update with a stub that printed to console, and noticed it was being called more than thousand times when there was a lot of activity on the screen.

However, with the setNeedsDisplay call inside update, it was only being called a few hundred times. Since it was spending too much time rendering it was missing out on a few frames in between.

I moved the drawing code inside the Timer and it works perfectly now.

localClient.GotFrameBufferUpdate = { _, _, _, _, _ in needs_update = true }
Timer.scheduledTimer(withTimeInterval: 0.016, repeats: true) {_ in
    if needs_update {
        view.display()
        needs_update = false
    }
    if WaitForMessage(&self.localClient, 1) > 0 {
        HandleRFBServerMessage(&self.localClient)
    }
}