Skip to content

[Experimental] Marshal C Data to []byte

Sebastien Binet requested to merge TacoVox:h5ptappend_marshal_bin into master

Created by: TacoVox

First and foremost: this is an experimental P/R that works. But the main purpose is for me to get your feedback and maybe we can work something out that is nice and could replace at least the suboptimal C.realloc calls.

As @sbinet pointed out in https://github.com/gonum/hdf5/pull/17, it could be nice to marshal data into a []byte. I coulndn't wait, started coding on another branch, and this is the result:

  • Replaced the nasty C.realloc calls with a simple make and copy to increase the []byte size.
  • Got rid of the C.free that was necessary as a result of the memory allocation.
  • Experimented a bit and I think I found an acceptable way to find the "native" endianess of the machine. It seems to work as expected.
  • Replaced most of the C types with Go-native ones.

What I was not able to achieve is to get rid of the usage of unsafe related things. I hope we can find a solution for that. If I am not mistaken, it is obviously a problem to marshal C.float and C.double into a uint32 or uint64 without using unsafe.Pointer and a cast. But while experimenting, I found the same for the unsafe.Pointer address to the C.Strings, and the signed integers. A potential solution would have been to extend the ad.buffer by 8 bytes before the case statement and then just use unsafe.Pointer in those mentioned cases and for the unsigned integers e.g. a nativeEndian.PutUint16 instead. But for consistency, I liked this solution more.

Furthermore, the strings are as they are. I think, limited by the HDF5 library itself, there is no other way for us to move away from that.

With that: I would be very glad about any input. There is no rush in getting this PR in, I just wanted to exchange thoughts with you guys on that.

If and only if we will get this in, I will change the commit message ;)

Merge request reports