JNA tagged union mapping

How to map following "tagged union" structure to JNA? This structure is used by libvterm (link to source code).

/**
 * Tagged union storing either an RGB color or an index into a colour palette
 */
typedef union {
  /**
   * Tag indicating which union member is actually valid. This variable
   * coincides with the `type` member of the `rgb` and the `indexed` struct
   * in memory. */
  uint8_t type;

  struct {
    uint8_t type;
    uint8_t red, green, blue;
  } rgb;

  struct {
    uint8_t type;
    uint8_t idx;
  } indexed;
} VTermColor;

Solution

Although there are many mappings that will work (any 32-bit structure with the right methods used to fetch the values), the canonical way to map this directly is by using JNA's Union class. The Union will have three elements; either a byte, or a structure RGB that you can define (an inner class of the Union is fine), or a structure Indexed that you will define (again, an inner class).

The Union will allocate enough memory on the native side for the largest element (32-bits), and given the structure options you're guaranteed that the first 8 bits of the resulting 32-bit C-side memory will contain the type field; based on that value you'll know what is contained in the remaining 24 bits.

If you look at the source code of JNA's Variant class which maps the tagged union VARIANT, you'll see this implemented on a slightly more complex scale. The _VARIANT class contains these five elements, similar to the 3 elements in your union:

        public VARTYPE vt;
        public short wReserved1;
        public short wReserved2;
        public short wReserved3;
        public __VARIANT __variant;

For a Union, only one value of __variant will be valid. In this case, the type is set here:

   public void setVarType(short vt) {
        this._variant.vt = new VARTYPE(vt);
   }

More generally, you can look at the outer VARIANT class which uses the Union class setType() method to determine whether there's even a valid value: it sets a string corresponding to the active field (in this case "_variant") which is set in the constructor. (You can also set using a class instead of a String.)

In your case, you will want to initialize based on the type value, so you'll start out with the type as the default, read its value, and then switch.

You could define your union like this:

public class VTermColor extends Union {
    public class RGB extends Structure {
        public byte type;
        public byte red;
        public byte green;
        public byte blue;
    }

    public class Indexed extends Structure {
        public byte type;
        public byte idx;
    }

    public byte type;
    public RGB rgb;
    public Indexed indexed;

    public VTermColor() {
        // initialize knowing only the type, read its value
        this.setType("type");
        this.read();

        // switch union based on type, re-read
        if ((this.type & VTERM_COLOR_TYPE_MASK) == VTERM_COLOR_RGB) {
            this.setType("rgb");
        } else {
            this.setType("indexed");
        }
        this.read();
    }

    public VTermColor(Pointer p) {
        super(p);
        // same remaining code as above
    }
}

You probably want to create a few other getter methods to do checking of the type value before returning the appropriate fields.

As mentioned at the start, any 32-bit data structure would work. A somewhat hacky alternative (sacrificing readability and type safety for a lot less code) could simply always use the 4-byte RGB structure as defined above. A getter for type would always work, while the getters for red, green, and blue would work if valid, otherwise you could create a getter for idx that just reads the value of red.