How to avoid recursion with parenReference

Using Matlab 2021B version

I create a class which inherits both handel & matlab.mixin.indexing.RedefinesParen.

classdef Abc <handle &  matlab.mixin.indexing.RedefinesParen

I create object arrays using this class.

I want to overload the parenthesis for the class so that it can perform special handling in addition to indexing into the object array ( normal operation ).

For example

Abc('special') - performs special handling

Abc(2:3) - down selects only objects 2 to 3 (normal operation without RedefinesParen)

I have written the parenReference function as follows

function varargout = parenReference(obj,indexOp)
    if isscalar(indexOp)
        varargout{1} = obj(indexOp(1).Indices);
    else
%    Other special handling and forward indexing
%                
    end
end

The problem is that no matter how I try to select the subset of objects to return in the third line, it calls parenReference recusively and faults when recursion limit is reached. I have even tried

varargout{1} = builtin('subsref',obj,substruct('()',indexOp(1).Indices))

How do I select a subset of obj in this function without it being called recursively.

Thanks

[Matt J]

Interestingly, this recursion problem doesn't seem to afflict RedefinesDot (in R2021b). In the test classdef below, you can see that I have many expressions of the form obj.something inside my overload of the dotReference() method. They don't trigger any recursions, though.

classdef myclass<matlab.mixin.indexing.RedefinesDot 
%A test class which uses RedefinesDot to make indexing into a struct member more brief.

    properties 
        params  %this is a struct but to access its fields, we want to 
                %do things like obj.a instead of obj.params.a
        c
    end

   methods
        function obj=myclass()

          obj.params.a=1;
          obj.params.b=2;

        end

        function disp_a(obj)
           disp(obj.a);
        end
   end

    methods (Access=protected)
            
        function varargout = dotReference(obj,indexOp)
            if ismember(indexOp.Name,fieldnames(obj.params))
             [varargout{1:nargout}] = obj.params.(indexOp.Name);
            else
              [varargout{1:nargout}]=obj.(indexOp.Name);
            end
        end

        function obj = dotAssign(obj,indexOp,varargin)
            if ismember(indexOp.Name,fieldnames(obj.params))
             [obj.params.(indexOp.Name)] = varargin{:};
            else
              S.type='.'; 
              S.subs=indexOp.Name;
              obj= builtin('subsasgn',obj,S,varargin{1});
            end
        end
        
        function n = dotListLength(obj,~,~)
            n = numel(obj);
        end

    end

end 

>> obj=myclass;

>> obj.a=10; obj.b=20; obj.c=30

obj =

myclass with properties:

params: [1×1 struct]

c: 30

>> obj.params

ans =

struct with fields:

a: 10

b: 20

>> obj.disp_a

10

[James Lebak]

@Matt J -- You don't see recursion with RedefinesDot because the methods of RedefinesDot don't override accessible properties.

We couldn't do something like that for RedefinesParen because it specifically is designed for scalars masquerading as arrays, and in such a case we have no idea how or where array elements are stored by your class.

[Matt J]

the methods of RedefinesDot don't override accessible properties.

@James Lebak I'm not sure what it means for a method to override properties. Am I wrong to assume that dot-indexing statements revert to built-in indexing methods inside dotReference, dotAssign, and dotListLength, but nowhere else?

[James Lebak]

If you set a breakpoint in dotReference you will see it is never called for the public property obj.params, regardless of whether you issue that statement from inside or outside of your class. We don't call dotReference when the property is an actual property that is accessible in the present context.

See the doc for dotReference, which states that it

"handles index referencing operations that begin with dots and index into fields that are inaccessible or nonexistent, such as obj.PrivateProp or obj.NotAProp{1}(2)"

[James Lebak]

You asked "Am I wrong to assume that dot-indexing statements revert to built-in indexing methods inside dotReference, dotAssign, and dotListLength, but nowhere else?"

The idea of 'reverting to built-in indexing methods' is not something that any of the current modular indexing APIs do.

The behavior of dotReference/dotAssign depends on context. When you have a public property named prop, obj.prop does not call dotReference, either inside or outside the class. When you have a private property named privateProp, it's accessible inside the class and so doesn't call dotReference there. But outside the class, it's not accessible, and obj.privateProp calls dotReference.

[Matt J]

I see. So the invocation rules of modular indexing overloads are uniform. You don't have special rules that say built-in indexing will be invoked instead of modular overloads, but only in specific methods.

[James Lebak]

Correct. That uniformity was a design goal for the modular indexing APIs.

[Matt J]

@James Lebak Is there any reason for MathWorks not to offer a syntax for the builtin() command like the following,

tmp=builtin('parenReference',obj,indexOp);

that will call built-in ()-indexing when the user truly wants it? As it stands now, I don't think there is a solution to the OP's recursion issue without something like this.

[James Lebak]

The OP's request can't be accomplished with RedefinesParen. If you inherit from RedefinesParen, your object is always a scalar, and you have to have a property inside the object (or something else) that is the array. The usual paradigm is then to have parenReference/parenAssign index into that other entity, and then there's no recursion problem (because the property is a different class and presumably uses built-in indexing). Internally we call this the 'scalar masquerading as an array' pattern, and it's the only thing RedefinesParen supports. The reason RedefinesParen doesn't have a concept of 'calling builtin parenReference' is that it's really not needed for the specific cases that RedefinesParen handles -- which do not, unfortunately, include the OP's use case.

Subsref and subsasgn support the 'scalar masquerading as an array' pattern, but they also allow you to override indexing into arrays like the OP is trying to do -- for example, to translate character strings into numerical indices. This allows the object to be a built-in array, and also requires some sort of 'map the indices and call built-in paren-indexing' capability as you are suggesting. When we designed RedefinesParen we decided to separate these use cases. If we provided the 'override indexing into arrays' functionality in a modular indexing mixin, we would provide something like the builtin parenReference/parenAssign calls that you're suggesting, and eventually I do hope to provide this functionality. But I think you're correct that the OP's request requires subsref/subsasgn as things currently stand.

[Paul]

"If you inherit from RedefinesParen, [the intent is that] your object is always a scalar,"

Perhaps that idea should be stated very clearly at the top of matlab.mixin.indexing.RedefinesParen - Customize class indexing operations that use parentheses - MATLAB

Answer 1

[Matt J]

Classic subsref doesn't seem to difficult in this case. You can't beat the classics!

function varargout = susbref(obj,S)

    if S(1).type~="()"
       [varargout{1:nargout}] = builtin('subsref',obj,S); return
    end

    T=S(2:end);
    S=S(1);

    if isscalar(S.subs) && isscalar(S.subs{1})
        out = obj(S(1).subs{1});
    else
        %    Other special handling and forward indexing
        %          
       out=...
    end
    
    [varargout{1:nargout}] = builtin('subsref',out,T);
    
    
end

[Mike Falcinelli]

I tried this. The problem is that I am using the RedefinesDot mixin and it is not compatible with subsref. The only reason that I am trying to use the RedefinesParen, is because I the RedefinesDot does not allow compound statements.

for example

V.specialDot       % works fine if V is an Abc object

but

V(1).specialDot     % causes error - compound statements not allowed, must use redefines paren.

[Matt J]

All the more reason to discard RedefinesDot as well... The only reason that I can fathom to use the Redefines is if only one mode of indexing is being overloaded. Otherwise, I can't see the advantage over just using subsref.

[Mike Falcinelli]

Okay - I have discarded Redefines Dot also and am using subsref & subsasgn. In this case, they are pretty simple and not execution time critical.

I had moved to the RedefinesDot because previously ( 2 to 3 years ago) I tried the subsref/subsasgn route for another application. I found it cumbersome and slow because I had to implement much of the built-in functionality. I liked the ease and simplicity of the RedefinesDot, and it added almost no overhead cost in terms of execution time for 'buit-in' capability.

[Thomas Michiels]

@Matt J i am astonished by your advice. redefinesParen is a better version of subsref. You only redefine what you want to redefine and everything else keeps working. In general subsref code is so ugly because you need to handle so many cases where you want default behavior.

@Mike Falcinelli what you are looking for is the fact that you are trying to index on yourself again, causing recursion. (This would be the exact same with subsref EDIT: apparently it is not.) Usually your class contains an array and you want to make the indexing operation operate on that array, which is a property of your class/obj, it isn't obj itself.

[Matt J]

@Thomas Michiels

(This would be the exact same with subsref EDIT: apparently it is not.)

If redefinesParen had the same robustness to recursion as subsref, I would have fully embraced it.

[Thomas Michiels]

@Matt J understandable. it is so strange to me that you cannot access the original indexing operator anymore. i have to agree with you in this case

[Matt J]

@Thomas Michiels Stranger still that you can access the original operator when using RedefinesDot, see my comment above.

Answer 2

[Shanmukha Voggu]

Hi Mike,

The recursion limit is reached because the parenReference is calling itself recursively,

The solution is to create a property that holds the given array in the class Abc

properties (Access=private)
    copyOfArray
end

And in the constructor of class Abc assign the given array to the property above

function obj = Abc(arr)
    obj.copyOfArray = arr;
end

Now you can use indexing on the property to get the desired results, Instead of using below statement in parenReference function

varargout{1} = obj(indexOp(1).Indices);

use the following statements

 obj.CopyOfArray = obj.CopyOfArray.(indexOp(1));
 varargout{1} = obj; %if you want output as object array, or use obj.copyOfArray in order to have array alone as output 

Refer this for more information.

[Mike Falcinelli]

Thanks for the answer. This works fine for a contained array. I am using an array of objects and trying to implement the built in capability.

In my example above, I create array of objects,

Varx 1x5 Abc

If I do not use matlab.mixin.indexing.RedefinesParen, and do the following

Vary = Varx(2:3)

I get

Vary 1x2 Abc <- points to members 2 & 3 of Varx

When I use matlab.mixin.indexing.RedefinesParen

function varargout = parenReference(obj,indexOp)
    
end

obj is a 1x5 Abc. I cannot find a way to access any subset of obj without getting the recursion error from calling parenReference.

How would I code parenReference to implement the built-in function?

Thanks,

Mike

[Matt J]

The recursion limit is reached because the parenReference is calling itself recursively

That is puzzling behavior, I have to say. I would have thought the rules for overloading parenReference would mimic those of subsref, namely that only indexing expressions issued outside the classdef invoke the overload. Not following that rule introduces new pitfalls, and seems to ruin a lot of the simplification over subsref that parenReference is intended to achieve.

[Mike Falcinelli]

Yes, I would expect that also. The dotReference simplication is nice. It handles all the builtin cases and just passes on what it does not understand. I thought parenReference might be similar, but I am stumped.

[James Lebak]

You're correctly picking up on an intentional design decision of the API. One of the most confusing things about subsref/subsasgn is that rule you mention, that indexing inside the class always calls built-in indexing. The good thing about that restriction is exactly that it prevents recursion like the one the OP is experiencing. However, from a usability standpoint, developers inside and outside the company repeatedly told us that they often experienced behavior that they didn't expect because of the rule. In particular, if you overload dot, paren, or brace, there's no good way to invoke your own overloaded indexing behavior inside your class, short of an explicit call to subsref/subsasgn. Those calls hurt the performance of your overloaded indexing and should not be necessary in the new system.

We separated the uses here into two cases. One case is the 'class is a scalar masquerading as an array' and as this answer points out, RedefinesParen is set up to handle this. The other case is the 'class is a built-in array and modifies or logs the indices with which it is called.' Currently, to implement that use case, you still need to use subsref and subsasgn. We have plans to deliver additional modular indexing functionality to address the second use case, but it isn't in the initial delivery.

[Matt J]

@James Lebak

developers inside and outside the company repeatedly told us that they often experienced behavior that they didn't expect because of the rule

It's definitely something people new to Matlab OOP don't initially grasp, but I greatly worry that the recursion pitfall will be highly prevalent without it, in both of the use cases that you've described. It makes me very nervous about embracing the new Redefines.

In particular, if you overload dot, paren, or brace, there's no good way to invoke your own overloaded indexing behavior inside your class, short of an explicit call to subsref/subsasgn.

Seems easy enough to outsource the work to a class-related function or external function, if you really do need a workspace where the overloaded indexing needs to be in force. I personally have never had to do that.

[James Lebak]

@Matt J

I want to offer reassurance that in our experience with using the modular indexing APIs internally, we haven't found recursion to be a big problem. The APIs don't currently do everything that subsref and subsasgn can, but I would hate to see you avoid them just because of the potential to encounter this problem.

It's true that it's easy to write code that encounters the recursion, and we did see developers new to the API write this code. In the case that RedefinesParen is primarily designed for, the scalar that presents itself as an array, we also found it to be a relatively easy problem to avoid, and to be easily uncovered by simple testing.

We are currently moving several MathWorks classes from subsref and subsasgn to modular indexing and in our production test system, we haven't encountered failures due to modular indexing classes unexpectedly calling overloaded indexing recursively.

Answer 3

[Matt J]

The problem is that no matter how I try to select the subset of objects to return in the third line, it calls parenReference recusively and faults when recursion limit is reached.

The attached classdef demonstrates a workaround that permits construction of non-scalar RedefinesParen arrays and extraction of their subarrays with paren-indexing. The key is to rebuild the object array inside parenReference as a cell array of objects, whereupon paren-indexing can be applied freely without recursion:

function varargout = parenReference(Objs, indexOp)

C=cell(Objs); C=C.(indexOp(1));

Objs = reshape([C{:}], size(C));

if numel(indexOp)>1

[varargout{1:nargout}] = Objs.(indexOp(2:end));

else

varargout={Objs};

end

end

This isn't as efficient as one might like, but if

1. you tend to work with only small object arrays,
2. your class constructor doesn't do anything much heavier than assigning properties,

then it should often be acceptable.

DEMO:

objArray = myclass([3;4;5], zeros(2), ones(3)).'

objArray = 
     3         0     0         1     1     1
     4         0     0         1     1     1
     5                         1     1     1
 

size(objArray)

ans = 1×2

1 3

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

objArray([1,3])

ans = 
     3         1     1     1
     4         1     1     1
     5         1     1     1
 

objArray(3) = objArray(1)

objArray = 
     3         0     0         3
     4         0     0         4
     5                         5
 

objArray(1)=[]

objArray = 
     0     0         3
     0     0         4
                     5
 

[Matt J]

Of course, this would all be much easier if they would allow builtin('parenReference',...) or builtin('subsref',...) to work for RedefinesParen, and I hope they will. If RedefinesParen objects really were intended to be scalar all the time, it wouldn't be possible to apply builtin cat() and reshape() operations to them, but as demonstrated in myclass.m, it is.

[James Lebak]

This is not the intended use of RedefinesParen. While this workaround may continue working in future versions of MATLAB, we don't recommend it.

RedefinesParen is meant for classes that take complete ownership of array semantics. The subsref and subsasgn APIs support both extending array indexing and completely customizing it. When we surveyed classes that customize indexing, we found that mixing these two purposes led to confusing code and hard-to-diagnose problems. We also found that complete customization, such as a scalar object presenting itself as an array, was far more common, so we designed RedefinesParen around that case. Consequently, it isn't a good tool when your goal is to extend indexing in a relatively minor way. We plan to update the documentation to clarify this.

Better support for this use case of extending array indexing remains a long-term goal for us. We receive relatively few requests for it, and would welcome real-world examples so we can understand the requirements in more detail. Subsref and subsasgn remain possible ways to extend indexing in MATLAB as needed.

[Catalytic]

RedefinesParen is meant for classes that take complete ownership of array semantics.

In that case, maybe it should have been called 'RedefinesArray'? The parentheses indexing seems like a rather small and incidental piece if the intention was to redefine all array semantics.

[Matt J]

@James Lebak

RedefinesParen is meant for classes that take complete ownership of array semantics.

Yes, from our earlier conversations, I thought that to be the case as well. It was surprising to me therefore, to discover that much of Matlab's built-in array semantics are still accessible to RedefinesParen. As illustrated in the myclass.m attachment, one can create arrays of RedefinesParen objects and access a lof of built-in array functionality (cat, reshape, transpose, etc...) through the builtin() command. Is this effectively a jail-break? Did you mean to seal all that off from us?

If so, I hope MathWorks won't do that now. There doesn't appear to be anything inherently harmful about letting RedefinesParen inherit native array behavior and, of course, it increases the flexibility of its use.

[James Lebak]

We really didn't intend for anyone to use built-in cat on objects that inherit from RedefinesParen. We didn't think that issuing errors was worthwhile, and I doubt that we would go back and introduce them now. However, it is likely that there are places in the code that really do assume any class that inherits from RedefinesParen is a scalar and that will not work properly when it's not.That's why I say I don't recommend it. If it's working for you, that's good, but you might encounter situations where it doesn't work as you expect, and I can't promise we'll fix those.

[James Lebak]

In that case, maybe it should have been called 'RedefinesArray'? The parentheses indexing seems like a rather small and incidental piece if the intention was to redefine all array semantics.

Maybe it's because MATLAB indexing has been my job for a decade, but I think of parentheses indexing as a fundamental component of array behavior in the language. It's funny that you mention "RedefinesArray". We did propose to call the class "RedefinesArray" during class design at one point for precisely this reason.

When we surveyed classes that overloaded indexing with subsref and subsasgn, we found many instances where the array behavior presented by the class was inconsistent, leading to bugs. Our goal with RedefinesParen was to give authors the tools they needed to completely redefine the way arrays work, but also to encourage them to make explicit decisions about array behavior rather than rely on silent defaults.

[Catalytic]

@James Lebak

Consequently, it isn't a good tool when your goal is to extend indexing in a relatively minor way. We plan to update the documentation to clarify this.

Yes, indeed. Most importantly, you will want to remove documentation entries like this one from Code Patterns for subsref and subsasgn Methods --

"For R2021b and later, the recommended process for customizing indexing is to use the modular indexing superclasses instead of overloading subsref and subsasgn. For more information, see Customize Object Indexing."

By your own assertion, RedefinesParen et al. are not meant at all to be a comprehensive replacement for subsref and subsasgn. They are meant rather to support certain very specific use cases that you have identified as common.

[Paul]

"... RedefinesParen et. al."

Are RedefinesDot and/or RedefinesBrace intended to be comprehensive replacements for subsref and subsasgn? Are they intended to support specifice use cases? This thread doesn't really address those two Redefines.

[Catalytic]

@Paul It doesn't look like they could be. RedefinesBrace will trigger the same recursion problem if the indexing is not forwarded to an encapsulated array. Additionally, in his 11 Sep 2023 comment, @James Lebak explains that "accessible properties" are not redefinable through RedefinesDot. IOW, you cannot overload dot indexing for an already existing public property or method name. So RedefinesDot does not override dot-indexing as flexibly as subsref/subsasgn either.

[Matt J]

IOW, you cannot overload dot indexing for an already existing public property or method name. So RedefinesDot does not override dot-indexing as flexibly as subsref/subsasgn either.

Not as flexibly, though it is frequently what you want...

[James Lebak]

The summary is that the union of the three modular indexing mixins -- RedefinesBrace, RedefinesParen, RedefinesDot -- should allow class authors to implement the vast majority of behaviors that we saw implemented with subsref and subsasgn, easier, more correctly, and with better performance. This was the design intent. We used the most complex classes in the MathWorks codebase as test cases for the design and they were able to be implemented using the union of the three mixins, and performed better as well.

In the case of brace, the default behavior for brace-indexing for objects is that an error is issued. If your class doesn't overload brace, this is what you get. If you overload brace and forward brace-indexing to your own object you will have to deal with recursion, and as in paren the most effective way to deal with that would be to have a property with built-in brace indexing that you forward to. You could probably do something like Matt had proposed for paren as well. I'm not aware of any overloaded brace behaviors that we saw implemented with subsref and subsasgn that cannot be done with RedefinesBrace. If your intent is to extend your class to add brace indexing, I would say that is more easily done with RedefineBrace than with subsref/subsasgn, because you need not attempt to reimplement any of the details of paren- or dot- indexing for your class.

RedefinesDot similarly allows implementing the most frequent overloaded behaviors we saw for dot-indexing and allows better performance. It is true that class authors cannot use RedefinesDot to overload dot indexing for an already existing public property or method name. This was a design choice based on several observations about subsref and subsasgn:

• Forcing all dot-indexing to go through subsref and subsasgn disables JIT-compiler optimizations for all properties of classes that overload indexing, resulting in poor performance.
• Property set- and get- methods are overridden by subsref and subsasgn, leading to subtle and confusing behavior.
• Private properties usually end up becoming publicly accessible under subsref/subsasgn unless the class author takes a lot of care (this is probably the most frequent subtle error caused by simply forwarding indexing to built-in subsref/subsasgn).

RedefinesDot was designed to allow custom dot indexing and avoid these problems.

Frequently, you can get the equivalent of 'overloading dot indexing for an existing property' through dependent properties. The remaining case that I'm aware of where people would need something more like the behavior of subsref/subsasgn for an existing public property is when there is a performance optimization enabled by knowing about subsequent indexing. Such optimizations can't be done with dependent properties, but are relatively easy to do with something like RedefinesDot, with the limitation that the name you use can't be the name of a real public property. We don't think this limitation is very important, but we're open to feedback and could provide an enhancement later if customers need it.

It's also worth pointing out one of the most frequently desired overloaded dot behaviors we saw was disallowing dot-access to methods, and that this case is easily implemented using ForbidsPublicDotMethodCall.

[Paul]

@James Lebak

Can you verify that the design intent for RedefinesBrace is to only support a scalar masquerading as a cell array (synonymous to the intent of RedefinesParen)?

[James Lebak]

No, the intent of RedefinesBrace isn't the same as RedefinesParen, and it doesn't make the same assumptions about array behavior. Brace-indexing isn't tied to array nature in MATLAB in the same way as paren-indexing is. RedefinesBrace could certainly support the case of a scalar masquerading as a cell array, but I don't think it's limited to that case.

[Catalytic]

The summary is that the union of the three modular indexing mixins -- RedefinesBrace, RedefinesParen, RedefinesDot -- should allow class authors to implement the vast majority of behaviors that we saw implemented with subsref and subsasgn, easier, more correctly, and with better performance. This was the design intent.

Yes, I see that was the intent back then, but this thread seems to have established that there are a fair range of plausible usages (Prevalence = 10-20% ?) that are not covered, and which call for continued reliance on subsref/subsasgn. You mention that full coverage of subsref/subsasgn functionality is a possibility in the future, but this thread is 5 years old and, in your words, it is still a "long term goal". This hints that modular mixins are either likely to remain specific to the array-encapsulation scenario, or at least that it is premature to document them as a comprehensive replacement for subsref/subsasgn.

[Matt J]

@Catalytic

RedefinesBrace will trigger the same recursion problem if the indexing is not forwarded to an encapsulated array.

I don't think so. The recursion in the OP's case was triggered by attempting to unconcatenate object arrays, which requires access to builtin paren-indexing. But there is no analogue to that with brace-indexing. Brace-concatenation cannot be overloaded in user-defined classes. Concatenating objects with braces always results in an ordinary cell array.

(It might be interesing, though, if overloading brace-concatenation were possible...)

[Paul]

@James Lebak

If the doc page at RedfinesParen is updated as suggested in this comment, then it would also be benficial to update matlab.mixin.indexing.RedefinesBrace - Customize class indexing operations that use braces - MATLAB to indicate that RedefinesBrace does not make the same assumption about intent as RedefinesParen. If I understand the example at matlab.mixin.indexing.RedefinesBrace Class, it shows a scalar object masquerading as a cell array. Maybe it would be benefiical for the example to show an array of objects so as to reinforce the less restrictive intent.

Also, thanks for your insights in this discussion. Always good for the community when knowledgeable Mathworks staff participate.