I am attempting to pass a collection of JNA structures to a native method but it's proving very fiddly:
Let's say we have a structure:
class MyStructure extends Structure {
// fields...
}
and a method in a JNA interface:
void pass(MyStructure[] data);
which maps to the native method:
void pass(const MYStructure* data);
Now the complication comes from the fact that the application is building a collection of these structures dynamically, i.e. we are NOT dealing with a static array but something like this:
class Builder {
private final Collection<MyStructure> list = new ArrayList<>();
// Add some data
public void add(MyStructure entry) {
list.add(entry);
}
// Pass the data to the native library
public void pass() {
// TODO
}
}
A naive implementation of the pass() method could be:
MyStructure[] array = list.toArray(MyStucture[]::new);
api.pass(array);
(where lib is the JNA library interface).
Of course this doesn't work because the array is not a contiguous block of memory - fair enough.
One solution is to allocate a JNA array from a structure instance and populate it field-by-field:
MYStructure[] array = (MyStructure[]) new MyStructure().toArray(size);
for(int n = 0; n < array.length; ++n) {
array[n].field = list.get(n).field;
// other fields...
}
This guarantees the array consist of contiguous memory. But we have had to implement a field-by-field copy of the data (which we've already populated in the list) - this is OK for a simple structure, but some of the data I am dealing with has dozens of fields, structures that point to further nested arrays, etc. Basically this approach is just not viable.
Another alternative is to convert the collection of data to a simple JNA pointer, something along these lines:
MyStructure[] array = list.toArray(MyStructure[]::new);
int size = array[0].size();
Memory mem = new Memory(array.length * size);
for(int n = 0; n < array.length; ++n) {
if(array[n] != null) {
array[n].write();
byte[] bytes = array[n].getPointer().getByteArray(0, size);
mem.write(n * size, bytes, 0, bytes.length);
}
}
This solution is generic so we can apply it to other structure as well. But we have to change the method signatures to be Pointer instead of MyStructure[] which makes the code more obtuse, less self-documenting and harder to test. Also we could be using a third-party library where this might not even be an option.
(Note I asked a similar question a while ago here but didn't get a satisfactory answer, thought I'd try again and I'll delete the old one / answer both).
Basically I was expecting/hoping to have something like this:
MyStructure[] array = MyStructure.magicContiguousMemoryBlock(list.toArray());
similar to how the JNA helper class provides StringArray for an array-of-string:
StringArray array = new StringArray(new String[]{...});
But no such 'magic' exists as far as I can tell. Is there another, simpler and more 'JNA' way of doing it? It seems really dumb (and probably incorrect) to have to allocate a byte-by-byte copy of the data that we essentially already have!
Do I have any other options? Any pointers (pun intended) gratefully accepted.
As the author of the previous answer, I realize a lot of the confusion was approaching it one way before realizing a better solution that we discussed primarily in comments to your answer. I will try to answer this additional clarification with an actual demonstration of my suggestion on that answer which I think is the best approach. Simply, if you have a non-contiguous structure and need a contiguous structure, you must either bring the contiguous memory to the structure, or copy the structure to the contiguous memory. I'll outline both approaches below.
Is there another, simpler and more 'JNA' way of doing it? It seems really dumb (and probably incorrect) to have to allocate a byte-by-byte copy of the data that we essentially already have!
I did mention in my answer on the other question that you could use useMemory() in this situation. It is a protected method but if you are already extending a Structure you have access to that method from the subclass (your structure), in much the same way (and for precisely the same purpose) as you would extend the Pointer constructor of a subclass.
You could therefore take an existing structure in your collection and change its native backing memory to be the contiguous memory. Here is a working example:
public class Test {
@FieldOrder({ "a", "b" })
public static class Foo extends Structure {
public int a;
public int b;
// You can either override or create a separate helper method
@Override
public void useMemory(Pointer m) {
super.useMemory(m);
}
}
public static void main(String[] args) {
List<Foo> list = new ArrayList<>();
for (int i = 1; i < 6; i += 2) {
Foo x = new Foo();
x.a = i;
x.b = i + 1;
list.add(x);
}
Foo[] array = (Foo[]) list.get(0).toArray(list.size());
// Index 0 copied on toArray()
System.out.println(array[0].toString());
// but we still need to change backing memory for it to the copy
list.get(0).useMemory(array[0].getPointer());
// iterate to change backing and write the rest
for (int i = 1; i < array.length; i++) {
list.get(i).useMemory(array[i].getPointer());
list.get(i).write();
// Since sending the structure array as an argument will auto-write,
// it's necessary to sync it here.
array[1].read();
}
// At this point you could send the contiguous structure array to native.
// Both list.get(n) and array[n] point to the same memory, for example:
System.out.println(list.get(1).toString());
System.out.println(array[1].toString());
}
Output (note the contiguous allocation). The second two outputs are the same, from either the list or the array.
Test$Foo(allocated@0x7fb687f0d550 (8 bytes) (shared from auto-allocated@0x7fb687f0d550 (24 bytes))) {
int a@0x0=0x0001
int b@0x4=0x0002
}
Test$Foo(allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d550 (8 bytes) (shared from auto-allocated@0x7fb687f0d550 (24 bytes)))))) {
int a@0x0=0x0003
int b@0x4=0x0004
}
Test$Foo(allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d550 (8 bytes) (shared from auto-allocated@0x7fb687f0d550 (24 bytes))))) {
int a@0x0=0x0003
int b@0x4=0x0004
}
If you don't want to put useMemory in every one of your structure definitions you can still put it in an intermediate class that extends Structure and then extend that intermediate class instead of Structure.
If you don't want to override useMemory() in your structure definitions (or a superclass of them), you can still do it "simply" in code with a little bit of inefficiency by copying over the memory.
In order to "get" that memory to write it elsewhere, you have to either read it from the Java-side memory (via reflection, which is what JNA does to convert the structure to the native memory block), or read it from Native-side memory (which requires writing it there, even if all you want to do is read it). Under-the-hood, JNA is writing the native bytes field-by-field, all hidden under a simple write() call in the API.
Your "Rubbish Solution #2" seems close to what's desired in this case. Here are the constraints that we have to deal with, with whatever solution:
Structure, the native memory is not contiguous (unless you pre-allocate contiguous memory yourself, and use that memory in a controlled manner, or override useMemory() as demonstrated above), and the size is variable.Here are the "JNA ways" of dealing with structures and memory:
Structure.getPointer() with a size of (at least) Structure.size().Structure.getByteArray().new Structure(Pointer p) constructor.Structure.toArray() method creates an array of structures backed by a large, contiguous block of native memory.I think your solution #2 is a rather efficient way of doing it, but your question indicates you'd like more type safety, or at least self-documenting code, in which case I'd point out a more "JNA way" of modifying #2 with two steps:
Memory(array.length * size) native allocation with the Structure.toArray() allocation from your solution #1.
length * size block of contiguous native memory and a pointer to it (array[0].getPointer()).mem.write(n * size, ... ) with array[n].getPointer().write(0, ... ).getByteArray() and immediately write() that byte array seem clear enough to me.
write(0, getByteArray(0, size), 0, size), although one might argue if that's more or less clear.So, adapting your method #2, I'd suggest:
// Make your collection an array as you do, but you could just keep it in the list
// using `size()` and `list.get(n)` rather than `length` and `array[n]`.
MyStructure[] array = list.toArray(MyStructure[]::new);
// Allocate a contiguous block of memory of the needed size
// This actually writes the native memory for index 0,
// so you can start the below iteration from 1
MyStructure[] structureArray = (MyStructure[]) array[0].toArray(array.length);
// Iterate the contiguous memory and copy over bytes from the array/list
int size = array[0].size();
for(int n = 1; n < array.length; ++n) {
if(array[n] != null) {
// sync local structure to native (using reflection on fields)
array[n].write();
// read bytes from the non-contiguous native memory
byte[] bytes = array[n].getPointer().getByteArray(0, size);
// write bytes into the contiguous native memory
structureArray[n].getPointer().write(0, bytes, 0, bytes.length);
// sync native to local (using reflection on fields)
structureArray[n].read();
}
}
From a "clean code" standpoint I think this rather effectively accomplishes your goal. The one "ugly" part of the above method is that JNA doesn't provide an easy way to copy fields between Structures without writing them to native memory in the process. Unfortunately that's the "JNA way" of "serializing" and "deserializing" objects, and it's not designed with any "magic" for your use case. Strings include built-in methods to convert to bytes, making such "magic" methods easier.
It is also possible to avoid writing the structure to native memory just to read it back again if you do the field-by-field copy as you implied in your Method #1. However, you could use JNA's field accessors to make it a lot easier to access the reflection under the hood. The field methods are protected so you'd have to extend Structure to do this -- which if you're doing that, the useMemory() approach is probably better! But you could then pull this iteration out of write():
for (StructField sf : fields().values()) {
// do stuff with sf
}
My initial thought would be to iterate over the non-contiguous Structure fields using the above loop, storing a Field.copy() in a HashMap with sf.name as the key. Then, perform that same iteration on the other (contiguous) Structure object's fields, reading from the HashMap and setting their values.