Your dll function takes 3 inputs, the first two are scalar numeric indeed. The snippet of documentation you show is not very clear but indeed it looks like you want 0 for the default.
The 3rd input is a pointer to some memory where the function will write something.From your header file snippet it looks like it is indeed a 4-bit unsigned integer (assuming that the dll defines long as a 32-bit integer, which is fairly standard but not guaranteed).
The function only has one output, which is probably an error code or success. I'm unclear why you say you get 3 outputs.
deviceptr = libpointer('uint32');
res = calllib('alp4395', 'AlpDevAlloc', 0, 0, deviceptr);
ALP_ID = deviceptr.Value;
should work according to the information you've provided. However, do check the value of res against the dll documentation as it may indicate an error of some kind.
If you get a matlab error with the above, then we've not properly understood the signature.
Note that when you did:
ALP_ID = libpointer;
res1 = calllib('alp4395','AlpDevAlloc',DeviceNum,InitFlag,ALP_ID);
you passed a NULL pointer to the library. Two things can happen in this case: the library tries to write to the NULL address and crashes, or the dll checks that the pointer is not NULL and returns an error. Since it didn't crash, it must have returned an error code which I assumes is that 1001 value.