MatLab + Windows + UNC + HTCondor (generic scheduler)

Hello all and thanks for reading / giving some input:

We have set up a simple lab with one node acting both as master and execute on Win 7 (no domain). Same unit has MatLab.

There is a NAS device on the network accessed with UNC paths.

We've set up Condor to launch MatLab with user impersonation and the condor_submit description file uses run_as_owner = true.

When invoking Matlab through Condor and watching Task Manager we can see MatLab launching as the submitting user.

Here is the anomaly:

When launching MatLab from the standard desktop by icon, within MatLab the Path entries to the UNC share are there and no warning is posted on startup.

When launching MatLab from a CMD windows with MatLab -r mFile, MatLab launches, executes mFile, and again there are no warnings in respect to Path entries.

When we launch through Condor, we are presented with a warning about all the UNC Path entries:

Warning: Directory access failure: \\Diskstation\Matlab Warning: Directory access failure: \\Diskstation\Matlab\Data

.... [edited for brevity]

This information is being trapped launching MatLab with the -logfile switch within Condor.

No errors are reported within Condor and the instantiated MatLab (under the user's account) simply hangs until we kill it through Task Manager.

There are no access denied logs on the UNC device, and the shares are globally open - read / write / browse.

Any thoughts on how to hunt this problem down is appreciated.

This question has also been submitted to the HTCondor community in hopes of some shared knowledge found.

Many thanks.

Answer 1

[Jason Ross]

I'm suspecting that the account doing the launching is the LocalSystem account. This account (by design) has no access to the network at all. Although you do indicate that user switching should happen, it sounds like it happens too late.

To do a quick and dirty test of this, use the PSTools (<http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx>) to launch a command shell as the LocalSystem account:

 psexec -i -s cmd.exe

In this command shell, map the UNC with the user credentials

 net use \\unc\path /user:USERDOMAIN\username

it will then allow you to authenticate (you can also do this on the command line, but that puts your password in plain text)

Then run your command again and see if it works.

[Mark]

Hi Jason,

Thank you for taking the time to contribute your thoughts.

I don't quite understand what you are trying to confirm with this test. I agree that the LocalSystem account shouldn't have UNC access.

With TaskManager indicating the process is running as the user I don't understand how this is helpful to diagnosis.

FWIW - prior to posting the question, I did try some out of the box testing like mapping the share statically and creating symbolic links. All of these equally failed.

At the end of the day it sounds like a permission issue (as I think you are also believing), but it seems it occurs with a bona-fide and authenticated user account therein a possible MatLab bug.

It doesn't help that this particular ad-hoc lab is not a Active Directory domain, but the local user credentials should work as far as I understand.

In any event again thanks for your thoughts.

[Jason Ross]

The difference is that (I'm assuming, I know little to nothing about the innards of Condor) that the Condor process runs as a service using the LocalSystem account, and somewhere along the way it switches user context via impersonate user in order to run under that UID.

Do you know if your username as been granted the "Run As A Service" right?

The instructions I gave above were to run the mapping as the LocalSystem account. By default it does not have network access but when you run with the pstools you can run as the LocalSystem account, and you can see if having the shares accessible to the LocalSystem account resolves it. It does indeed sound like a permission or acceess issue, and I've encountered them in many stripes -- and I've used the above technique to resolve many more of them than I ever thought possible.

As you mention, this does seem like a permissions or access issue. It's just a matter of "peeling the permissions onion" to determine where something is falling over.

I've also found the Process Explorer to be a very valuable tool in tracking down this kind of issue.

[Mark]

Hi Jason - I wanted to take a quick moment and thank you for your additional response and clarification. I'll have to revert on specific answers nd continue my discovery.

The shame is there is no ability to pass a RunAs w/ credentials from within a Condor job submission (that I am aware of). In an AD domain this issue falls away but when dealing with siloed workstations in a workgroup the challange appears pretty large.

Mark