mex error when reading lz4 compressed h5 file... sometimes

I'm testing some code, and to check for bottlenecks I made three copies of my files

1) on a shared network drive

2) on a HDD next to my workstation connectet through a USB dock

3) on a ramdisk on the computer (just to see how fast the code would actually run)

I made 2) as a copy of 1) and 3) as a copy of 2) and the data is 150ish GiB

When I process the data from 1) Everything is fine.

When I process the data from 2) or 3) however, things got weird. for some files I would suddenly get workers to drop dead. It turns out this was due to an error in the MEX file for reading HDF5 files. The files that gave this error turned out not to be the same on 2) and 3), and some files affect on 3) where not affected on 2) and vice versa. By copying the files affected on 3) back to 3) from 2) I somehow succeeded in running my code from 3). But this aproach does not work for 2) for some reason.

When I replace the files matlab can read them, but when I then restart matlab, like it asks me to, it will find some other files in my dataset that it cannot read.

I haven't had this problem before?

What's even weirder is that I can read part of all the files, as I tried:

    fileholder = dir('file/folder/*.h5')
    for ii = numel(fileholder):-1:1
    disp(ii)
    filecontent = h5read(fullfile(fileholder(ii).folder,fileholder(ii).name),'/path/to/data',[1 1 1 1], [1 1 1 1])
    end

I had no problems (most of the time). whereas

    fileholder = dir('file/folder/*.h5')
    for ii = numel(fileholder):-1:1
    disp(ii)
    filecontent = h5read(fullfile(fileholder(ii).folder,fileholder(ii).name),'/path/to/data')
    end

does give an error

I'm getteng the error only on windows from these files (but often it will only be one file in more than 300)

The install is rather new, so I'm wondering, could it be a change in the hdf5plugin for python?

The crash dump I get looks like this:

    --------------------------------------------------------------------------------
                 Access violation detected at 2021-03-02 17:56:03 +0100
    --------------------------------------------------------------------------------
    
    Configuration:
      Crash Decoding           : Disabled - No sandbox or build area path
      Crash Mode               : continue (default)
      Default Encoding         : US-ASCII
      Deployed                 : false
      Graphics Driver          : Unknown hardware 
      Graphics card 1          : NVIDIA ( 0x10de ) NVIDIA GeForce RTX 3090 Version 27.21.14.5671 (2020-9-30)
      Java Version             : Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
      MATLAB Architecture      : win64
      MATLAB Entitlement ID    : 1242273
      MATLAB Root              : C:\Program Files\Matlab R2020b
      MATLAB Version           : 9.9.0.1467703 (R2020b)
      OpenGL                   : hardware
      Operating System         : Microsoft Windows 10 Enterprise LTSC
      Process ID               : 43000
      Processor ID             : x86 Family 143 Model 49 Stepping 0, AuthenticAMD
      Session Key              : 8a396e89-e08e-47dd-a8b3-685e64d67c23
      Window System            : Version 10.0 (Build 17763)
    
    Fault Count: 1
    
    
    Abnormal termination:
    Access violation
    
    Current Thread: 'MCR 0 interpreter thread' id 57536
    
    Register State (from fault):
      RAX = 0000019dcb3819d6  RBX = 0000019dca25cffa
      RCX = 0000000000000000  RDX = 0000019dcb3819d2
      RSP = 00000031a37f78c0  RBP = 0000019dc9d7d050
      RSI = 0000019dcb382928  RDI = 0000000000000000
     
       R8 = 0000000000000000   R9 = 0000000000000000
      R10 = 0000019dcb3819d6  R11 = 0000000000000246
      R12 = 00007ff8e8710000  R13 = 0000019dca266040
      R14 = 0000019dcb38290e  R15 = 0000000000000000
     
      RIP = 00007ff8e8714249  EFL = 00010293
     
       CS = 0033   FS = 0053   GS = 002b
    
    Stack Trace (from fault):
    [  0] 0x00007ff8e8714249 C:\Users\Guldfaxe\AppData\Local\Programs\Python\Python39\Lib\site-packages\hdf5plugin\plugins\libh5lz4.dll+00016969 H5PLget_plugin_type+00008505
    [  1] 0x00007ff8e8711f15 C:\Users\Guldfaxe\AppData\Local\Programs\Python\Python39\Lib\site-packages\hdf5plugin\plugins\libh5lz4.dll+00007957
    [  2] 0x00007ff89dd843a8  C:\Program Files\Matlab R2020b\bin\win64\hdf5.dll+01524648 H5Z_pipeline+00000520
    [  3] 0x00007ff89dc40d7d  C:\Program Files\Matlab R2020b\bin\win64\hdf5.dll+00200061 H5D__chunk_lock+00000845
    [  4] 0x00007ff89dc3c488  C:\Program Files\Matlab R2020b\bin\win64\hdf5.dll+00181384 H5D_btree_debug+00004952
    [  5] 0x00007ff89dc4cabc  C:\Program Files\Matlab R2020b\bin\win64\hdf5.dll+00248508 H5D__read+00001212
    [  6] 0x00007ff89dc4d88a  C:\Program Files\Matlab R2020b\bin\win64\hdf5.dll+00252042 H5Dread+00000522
    [  7] 0x00007ff8912f3236 C:\Program Files\Matlab R2020b\toolbox\matlab\imagesci\private\h5readc.mexw64+00078390 mexFunction+00001334
    [  8] 0x00007ff8b0d9a289 C:\Program Files\Matlab R2020b\bin\win64\libmex.dll+00565897 mexfile::Inspector::needs_upgrade+00015977
    [  9] 0x00007ff8b0d9b67b C:\Program Files\Matlab R2020b\bin\win64\libmex.dll+00571003 MexRetrieveVersion+00003579
    [ 10] 0x00007ff8b0d9b9b4 C:\Program Files\Matlab R2020b\bin\win64\libmex.dll+00571828 MexRetrieveVersion+00004404
    [ 11] 0x00007ff8b0d7c209 C:\Program Files\Matlab R2020b\bin\win64\libmex.dll+00442889 mexUnlock_800+00023993
    [ 12] 0x00007ff8b0c726c1                     bin\win64\pgo\m_dispatcher.dll+00665281 Mfh_file::dispatch_file_common+00000165
    [ 13] 0x00007ff8b0c71d62                     bin\win64\pgo\m_dispatcher.dll+00662882 Mfh_error::lookup+00006498
    [ 14] 0x00007ff8b0c72615                     bin\win64\pgo\m_dispatcher.dll+00665109 Mfh_file::dispatch+00000045
    [ 15] 0x00007ff8ae318208                            bin\win64\pgo\m_lxe.dll+00557576 CGGBuiltinImpl+00181784
    [ 16] 0x00007ff8ae2fb484                            bin\win64\pgo\m_lxe.dll+00439428 CGGBuiltinImpl+00063636
    [ 17] 0x00007ff8ae417861                            bin\win64\pgo\m_lxe.dll+01603681 mwboost::serialization::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >+00649937
    [ 18] 0x00007ff8ae2ff20d                            bin\win64\pgo\m_lxe.dll+00455181 CGGBuiltinImpl+00079389
    [ 19] 0x00007ff8ae2fe4dd                            bin\win64\pgo\m_lxe.dll+00451805 CGGBuiltinImpl+00076013
    [ 20] 0x00007ff8ae2ed187                            bin\win64\pgo\m_lxe.dll+00381319 CGGBuiltinImpl+00005527
    [ 21] 0x00007ff8ae2ecd8f                            bin\win64\pgo\m_lxe.dll+00380303 CGGBuiltinImpl+00004511
    [ 22] 0x00007ff8ae2f8b2c                            bin\win64\pgo\m_lxe.dll+00428844 CGGBuiltinImpl+00053052
    [ 23] 0x00007ff8ae2fd185                            bin\win64\pgo\m_lxe.dll+00446853 CGGBuiltinImpl+00071061
    [ 24] 0x00007ff8ae2fc19c                            bin\win64\pgo\m_lxe.dll+00442780 CGGBuiltinImpl+00066988
    [ 25] 0x00007ff8ae412c1f                            bin\win64\pgo\m_lxe.dll+01584159 mwboost::serialization::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >+00630415
    [ 26] 0x00007ff8ae412b07                            bin\win64\pgo\m_lxe.dll+01583879 mwboost::serialization::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >+00630135
    [ 27] 0x00007ff8ae412a16                            bin\win64\pgo\m_lxe.dll+01583638 mwboost::serialization::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >::singleton<mwboost::serialization::extended_type_info_typeid<MathWorks::lxe::PreLineExecutionEvent> >+00629894
    [ 28] 0x00007ff8ae452d7c                            bin\win64\pgo\m_lxe.dll+01846652 mwboost::archive::detail::oserializer<mwboost::archive::binaryTerm_oarchive,MathWorks::lxe::ClosedAbstractionFunctionDescriptorOwningIrTree>::oserializer<mwboost::archive::binaryTerm_oarchive,MathWorks::lxe::ClosedAbstractionFunctionDescriptorOwningIrTree>+00024116
    [ 29] 0x00007ff8ae452d2a                            bin\win64\pgo\m_lxe.dll+01846570 mwboost::archive::detail::oserializer<mwboost::archive::binaryTerm_oarchive,MathWorks::lxe::ClosedAbstractionFunctionDescriptorOwningIrTree>::oserializer<mwboost::archive::binaryTerm_oarchive,MathWorks::lxe::ClosedAbstractionFunctionDescriptorOwningIrTree>+00024034
    [ 30] 0x00007ff8afde69ba                            bin\win64\pgo\m_lxe.dll+28666298 MathWorks::lxe::profiler::IMexProfilingSupport::~IMexProfilingSupport+00058926
    [ 31] 0x00007ff8cbcd6549 C:\Program Files\Matlab R2020b\bin\win64\libmwbridge.dll+00157001 mnGetPrompt+00016921
    [ 32] 0x00007ff8b0e911ff   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00463359 iqm::Iqm::instance+00001023
    [ 33] 0x00007ff8b0f038b7   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00932023 iqm::UserEvalPlugin::execute+00001351
    [ 34] 0x00007ff8a4fe80be C:\Program Files\Matlab R2020b\bin\win64\nativejmi.dll+00360638 nativejmi::JmiIIP::serializeExplicit+00013438
    [ 35] 0x00007ff8b0eb7a3f   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00621119 iqm::Iqm::setupIqmFcnPtrs+00113615
    [ 36] 0x00007ff8b0ec31b0   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00668080 iqm::Iqm::setupIqmFcnPtrs+00160576
    [ 37] 0x00007ff8b0e86fc3   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00421827 iqm::Iqm::create+00007235
    [ 38] 0x00007ff8cbcc7a3e C:\Program Files\Matlab R2020b\bin\win64\libmwbridge.dll+00096830 ioReadLine+00000478
    [ 39] 0x00007ff8cbcc781c C:\Program Files\Matlab R2020b\bin\win64\libmwbridge.dll+00096284 ioReadLine+00000188
    [ 40] 0x00007ff8cbcd6930 C:\Program Files\Matlab R2020b\bin\win64\libmwbridge.dll+00158000 mnGetCommandLineBuffer+00000304
    [ 41] 0x00007ff8cbcd6db4 C:\Program Files\Matlab R2020b\bin\win64\libmwbridge.dll+00159156 mnParser+00000500
    [ 42] 0x00007ff8b1056302   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00549634 mcr_set_enableReadingFromStdin+00016098
    [ 43] 0x00007ff8b0fee7ad   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00124845 mcrFunctionSignature::set_signature+00103437
    [ 44] 0x00007ff8b100baea   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00244458 mcrFunctionSignature::set_signature+00223050
    [ 45] 0x00007ff8b0ee4504   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00804100 iqm::PackagedTaskPlugin::execute+00000084
    [ 46] 0x00007ff8b10503ce   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00525262 services::lmgr::exception::LicensingStartupException::~LicensingStartupException+00009822
    [ 47] 0x00007ff8b0eb7a3f   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00621119 iqm::Iqm::setupIqmFcnPtrs+00113615
    [ 48] 0x00007ff8b0e87c42   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00425026 iqm::Iqm::create+00010434
    [ 49] 0x00007ff8b0e87421   C:\Program Files\Matlab R2020b\bin\win64\iqm.dll+00422945 iqm::Iqm::create+00008353
    [ 50] 0x00007ff8b10260dc   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00352476 mcrInstantiationError::operator=+00011756
    [ 51] 0x00007ff8b1026a3c   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00354876 mcrInstantiationError::operator=+00014156
    [ 52] 0x00007ff8b10243cc   C:\Program Files\Matlab R2020b\bin\win64\mcr.dll+00345036 mcrInstantiationError::operator=+00004316
    [ 53] 0x00007ff8e959a2a3 C:\Program Files\Matlab R2020b\bin\win64\mwboost_thread-vc141-mt-x64-1_70.dll+00041635 mwboost::thread::swap+00000083
    [ 54] 0x00007ff8f16e268a                   C:\Windows\System32\ucrtbase.dll+00140938 o_exp+00000090
    [ 55] 0x00007ff8f4317974                   C:\Windows\System32\KERNEL32.DLL+00096628 BaseThreadInitThunk+00000020
    [ 56] 0x00007ff8f471a2d1                      C:\Windows\SYSTEM32\ntdll.dll+00369361 RtlUserThreadStart+00000033
    
    
    This error was detected while a MEX-file was running. If the MEX-file
    is not an official MathWorks function, please examine its source code
    for errors. Please consult the External Interfaces Guide for information
    on debugging MEX-files.