hdf5: finalizers are sometimes called before they should

Created by: TacoVox

Hello @sbinet. It's me again 😸

Using this library in production, I noticed some major issues that seem to be caused by the finalizers which are used.

Three out of four times, our application is crashing. I looked at the traces and found an interesting thing:

fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0xb762bb]

runtime stack:
runtime.throw(0xcd5fec, 0x2a)
        /usr/local/go/src/runtime/panic.go:616 +0x81
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:372 +0x28e

goroutine 4 [syscall]:
runtime.cgocall(0x88e8a0, 0xc42006b698, 0x0)
        /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc42006b658 sp=0xc42006b620 pc=0x401f94
decode/vendor/gonum.org/v1/hdf5._Cfunc_H5Tclose(0x30000000000013e, 0x0)
        _cgo_gotypes.go:913 +0x4d fp=0xc42006b698 sp=0xc42006b658 pc=0x71391d
decode/vendor/gonum.org/v1/hdf5.(*Datatype).Close(0xc42022cbd0, 0xc42006b738, 0x0)
        /go/src/decode/vendor/gonum.org/v1/hdf5/h5t.go:153 +0x37 fp=0xc42006b6d0 sp=0xc42006b698 pc=0x717ac7
decode/vendor/gonum.org/v1/hdf5.(*Datatype).finalizer(0xc42022cbd0)
        /go/src/decode/vendor/gonum.org/v1/hdf5/h5t.go:138 +0x2f fp=0xc42006b728 sp=0xc42006b6d0 pc=0x7179ef
runtime.call32(0x0, 0xce3b48, 0xc420432000, 0x1000000010)
        /usr/local/go/src/runtime/asm_amd64.s:573 +0x3b fp=0xc42006b758 sp=0xc42006b728 pc=0x4535fb
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:222 +0x1eb fp=0xc42006b7e0 sp=0xc42006b758 pc=0x415edb
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42006b7e8 sp=0xc42006b7e0 pc=0x455d81
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:156 +0x62

Looks like the (Datatype).finalizer tries to close an already closed Datatype... hmm... This happened right after the beginning of execution where the finalizer should actually not be called.

Looking at other traces, I noticed that the Close() on Datatype is not always the cause of crash, but a segmentation violation in regard to that Datatype is.

Having extensive logging in place that also spits out a log whenever we call Close() on the Datatype ourselves, I noticed that this actually never happened in those cases. So something else must've closed it before.

So I went to our vendor folder, h5t.go, removed the finalizer related code and BOOOM. The error does not occur any longer. Our application calls Close() when it is actually done writing and everything is fine.

As we both know (latest after the article that you linked recently), finalizers are not optimal. Actually, they seem to be causing issues in regard to this library. Therefore, I would suggest to remove them all over the place and would also take care of it in case you agree.

IMHO everybody that uses this library should actually take care of the Close() calls themselves.

For the record, I am running Go 1.10 and libhdf5 1.10.1.