I figured it out. First, and really the only MATLAB part, create an openxzy.m file and place it in MATLAB's startup path (probably Documents/MATLAB). Then in windows registry you have to replicate what .m, .mat, etcs files do. Here is what I have in my .reg file with explanation below:
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\.xyz]
@="xyzfile"
[HKEY_CLASSES_ROOT\.xyz\OpenWithList]
@=""
[HKEY_CLASSES_ROOT\.xyz\OpenWithList\matlab.exe]
@=""
[HKEY_CLASSES_ROOT\.xyz\OpenWithProgids]
@=""
[HKEY_CLASSES_ROOT\.xyz\OpenWithProgids\xyzfile]
@=""
[HKEY_CLASSES_ROOT\xyzfile]
@="XYZ Data"
[HKEY_CLASSES_ROOT\xyzfile\Shell]
[HKEY_CLASSES_ROOT\xyzfile\Shell\Open]
@="Open"
[HKEY_CLASSES_ROOT\xyzfile\Shell\Open\command]
@="\"C:\\Program Files\\MATLAB\\\\R2010a\\bin\\win64\\matlab.exe\""
[HKEY_CLASSES_ROOT\xyzfile\Shell\Open\ddeexec]
@="open('%1')"
[HKEY_CLASSES_ROOT\xyzfile\Shell\Open\ddeexec\application]
@="shellverbs.MATLAB"
[HKEY_CLASSES_ROOT\xyzfile\Shell\Open\ddeexec\topic]
@="system"
This is more complicated than what is necessary to get it to work. You need to add two keys to the HKEY_CLASSES_ROOT (which requires admin privilages). One is for the exentension (.xyz). The default vaule of .xyz (xyzfile) is where windows will look when trying to open that file. The rest in that key is just making the open with menu look how I want it to.
xyzfile needs a subkey Shell, Shell needs a subkey Open (you can actually have others, just make the default value of Shell the same as the name of the subkey you want as a default command), Open needs a subkey command. The default value of command needs to be the full path of matlab.exe (which can also be followed with commands like -r). That is enough to get it working. The magic happens when you add the the subkey ddeexec with its two subkeys (application and topic with default values shellverbs.MATLAB and system respectively) to the Open key. You set the default value of ddeexec to the function you want called (%1 is Windows' veraible name for the file name) and it will be called in the existing MATLAB session if there is one and open one otherwise.
Since figuring this out I have easily added icons to .xyz files and functions to the right click menu of .mat files.