Overriding methods inherited from a superclass with methods from a mixin

[a***@gmail.com]

Hello there.

I'm trying to override methods inherited from a superclass by methods defined in a mixin class.
Here's an sscce:
https://bpaste.net/show/6c7d8d590658 (never expires)

I've had problems finding the proper way to do that, since at first the base class wasn't to the right and I've assumed the correct order was from left to right. It was previously suggested on IRC that the mixin should be a subclass of "Base"; that worked but I wasn't happy with it because the B class basically serves the purpose of "holding" a list of methods to be inherited and that should override the methods inherited from the "A" classes, so I didn't see why it should derive from "Base".

I eventually found in an article that the problem was the ordering of the superclasses I was deriving from, which should be from right to left, the only article I could find that states that is this one: https://www.ianlewis.org/en/mixins-and-python

Every single example of mixins in Python that I've read -except that one- (and I've seen literally dozens) has the base class to the left, although the other classes aren't overriding any methods (at least in most of them).

That bpaste code is working perfectly for me and makes sense, but I don't really like it, and the people on IRC couldn't convince me the code is fine.

I haven't used Python for some time so I don't feel confident to judge that code, and perhaps there's a better way to achieve that result. However, what really scared me is the obscurity of the mixins usage in Python, and the fact that every example except that single one gets it "wrong", including from notable pythonistas.

Perhaps you guys could help me either convincing me that the bpaste code is OK, or perhaps coming up with a better solution for that problem. What matters to me is code re-usability in this case. I surely could re-implement the overrides in all "Z" classes separately, but that's what I'm trying to avoid. The requirements are:
1. I can't touch the "classes I don't have control over" (as per comment in code).
2. I don't want to pass the superclasses as parameters in the constructor. I see how you could solve the problem that way, but that would only increase the complexity of the code (PEP20).
3. I absolutely need to override methodX, I can't use composition and access the members another way unless I override methodX and access them there. This is to interface properly with other modules.
4. I need to be able to access A#.methodX in the "Z" classes methods.
5. I want to avoid using a factory. It's one of the most over-used patterns in my opinion, and I really don't like that.

Please note that the bpaste code is just an example. The real problem is much bigger and involves multiple methods to override and more classes, so the solution has to scale accordingly.

Thank you in advance.

[dieter]

Post by a***@gmail.com
I'm trying to override methods inherited from a superclass by methods defined in a mixin class.
https://bpaste.net/show/6c7d8d590658 (never expires)
I've had problems finding the proper way to do that, since at first the base class wasn't to the right and I've assumed the correct order was from left to right.

The class' MRO ("Method Resolution Order") determines in which
order attributes are looked up.
Either, you must order your base classes in such a way that the MRO
controlled lookup finds the methods you want to be used or
you must explicitely put a definition for those methods in your
derived class (it may have the form "overridden_method = <BaseClass>.overridden_method").

The rules to determine the MRO are complex. The "inspect" module contains
a function ("get_mro") to show the MRO of a given class. Use it
to get your inheritance order right.

[Peter Otten]

Post by a***@gmail.com
Hello there.
I'm trying to override methods inherited from a superclass by methods
https://bpaste.net/show/6c7d8d590658 (never expires)
#the following represents classes I don't have control over
print "methodX class Base"
class A(Base): #inherits methodX
pass
class A2(Base): #inherits methodX
pass
class A3(Base): #inherits methodX
pass
#the following represents classes I have control over
print "methodX class B"
A.methodX(self)
pass
pass
#what I need is to be able to quickly create these Z classes and override

...

the

Post by a***@gmail.com
#methodX by the functionality in class B, while still being able to call
#class A.methodX
Z().methodX();
Z2().methodX();
Z3().methodX();
I've had problems finding the proper way to do that, since at first the
base class wasn't to the right and I've assumed the correct order was from
left to right. It was previously suggested on IRC that the mixin should be
a subclass of "Base"; that worked but I wasn't happy with it because the B
class basically serves the purpose of "holding" a list of methods to be
inherited and that should override the methods inherited from the "A"
classes, so I didn't see why it should derive from "Base".
I eventually found in an article that the problem was the ordering of the
superclasses I was deriving from, which should be from right to left, the
https://www.ianlewis.org/en/mixins-and-python
Every single example of mixins in Python that I've read -except that one-
(and I've seen literally dozens) has the base class to the left, although
the other classes aren't overriding any methods (at least in most of
them).
That bpaste code is working perfectly for me and makes sense, but I don't
really like it, and the people on IRC couldn't convince me the code is
fine.

...

But the code is fine. Write unit tests to ensure that the correct method is
called.

Usually you use mixins to add methods. In that case the order of base
classes doesn't matter.

Post by a***@gmail.com
I haven't used Python for some time so I don't feel confident to judge
that code, and perhaps there's a better way to achieve that result.
However, what really scared me is the obscurity of the mixins usage in
Python, and the fact that every example except that single one gets it
"wrong", including from notable pythonistas.
Perhaps you guys could help me either convincing me that the bpaste code
is OK, or perhaps coming up with a better solution for that problem. What
matters to me is code re-usability in this case. I surely could
re-implement the overrides in all "Z" classes separately, but that's what
I'm trying to avoid. The requirements are: 1. I can't touch the "classes I
don't have control over" (as per comment in code). 2. I don't want to pass
the superclasses as parameters in the constructor. I see how you could
solve the problem that way, but that would only increase the complexity of
the code (PEP20). 3. I absolutely need to override methodX, I can't use
composition and access the members another way unless I override methodX
and access them there. This is to interface properly with other modules.
4. I need to be able to access A#.methodX in the "Z" classes methods. 5. I
want to avoid using a factory. It's one of the most over-used patterns in
my opinion, and I really don't like that.

...

That looks like you are actively excluding most alternatives ;)

Here's one option that should also work (though I'm not sure if that is what
you mean with point 2 of your list):

def subclass(base):
class Z(base):
def methodX(self):
print "overridden methodX"
return Z

Z = subclass(A)
Z2 = subclass(A2)

Post by a***@gmail.com
Please note that the bpaste code is just an example. The real problem is
much bigger and involves multiple methods to override and more classes, so
the solution has to scale accordingly.

[Ian Kelly]

Post by dieter

Post by a***@gmail.com
I'm trying to override methods inherited from a superclass by methods defined in a mixin class.
https://bpaste.net/show/6c7d8d590658 (never expires)
I've had problems finding the proper way to do that, since at first the base class wasn't to the right and I've assumed the correct order was from left to right.

The class' MRO ("Method Resolution Order") determines in which
order attributes are looked up.
Either, you must order your base classes in such a way that the MRO
controlled lookup finds the methods you want to be used or
you must explicitely put a definition for those methods in your
derived class (it may have the form "overridden_method = <BaseClass>.overridden_method").
The rules to determine the MRO are complex. The "inspect" module contains
a function ("get_mro") to show the MRO of a given class. Use it
to get your inheritance order right.

...

The details are complex, but there are two fairly simple principles
that can be relied upon:

1) Subclasses always appear before their superclasses in the MRO.
2) When a class inherits from multiple base classes, they will always
appear in the MRO in the order they appear in the class statement from
left-to-right.

Note that neither of these rules guarantee that the classes will
appear *sequentially* in the MRO.

So in the case of mixins, listing the mixin class first is absolutely correct:

class Derived(Mixin, Base): pass

This ensures that Mixin will always appear before Base in the MRO (and
thus override Base's methods) for Derived and for any subclass of
Derived.

It is possible to concoct elaborate inheritance hierarchies in which
it is not possible to come up with an MRO that will satisfy both 1)
and 2) above. In such cases, it is also useful to know what Python
will do. Fortunately, the answer to that is also simple: the type
constructor will throw an exception. So this isn't something that
really needs to be worried about.

[Michael Selik]

Post by a***@gmail.com
I haven't used Python for some time so I don't feel confident to judge

Yet, you are clearly judging the code you pasted as not OK.

Perhaps you guys could help me either convincing me that the bpaste code is

Post by a***@gmail.com
OK

It would be helpful for you to explain why you think the code you pasted is
not OK. To me it makes sense. English reads left-to-right, so method
lookups go left-to-right (and children before parents) in the inheritance
list.

[a***@gmail.com]

Thank you for your replies. I don't know if I'm quoting you correctly, I'm quite confused with Google Groups... not sure if it's a "forum", something like a mailing list, or both... or neither.

Post by dieter
The class' MRO ("Method Resolution Order") determines in which
order attributes are looked up.
Either, you must order your base classes in such a way that the MRO
controlled lookup finds the methods you want to be used or
you must explicitely put a definition for those methods in your
derived class (it may have the form "overridden_method = <BaseClass>.overridden_method").
The rules to determine the MRO are complex. The "inspect" module contains
a function ("get_mro") to show the MRO of a given class. Use it
to get your inheritance order right.

...

OK, I'm going to read about the MRO rules. Thanks for the heads up on the inspect module.

----

Post by dieter
But the code is fine. Write unit tests to ensure that the correct method is
called.
Usually you use mixins to add methods. In that case the order of base
classes doesn't matter.

Yeah, I see that in most cases the order doesn't matter, but still I would think that since the correct order is from right to left, that should be the common practice. What if later on you want to expand a mixin and actually override a method? That could cause a situation that's a hell to debug, so people should just get used to keeping their (most) "base" classes to the right when doing multiple inheritance in my opinion.

Post by dieter
That looks like you are actively excluding most alternatives ;)
Here's one option that should also work (though I'm not sure if that is what
print "overridden methodX"
return Z
Z = subclass(A)
Z2 = subclass(A2)

Yes, that violates the second item :). That would increase the complexity of the code because I would need to know which A to use when instantiating the classes, and what I'm doing is exactly to simplify the usage of those classes. Of course some docs and perhaps a well tailored enum (or alike) to be passed as the parameter could help, but a factory could do just as well, however, that isn't going to fulfill my needs at all, and since I'm the one who's going to use the code most of the time, I don't care much about using something somewhat "unconventional".

----

Post by dieter
The details are complex, but there are two fairly simple principles
1) Subclasses always appear before their superclasses in the MRO.
2) When a class inherits from multiple base classes, they will always
appear in the MRO in the order they appear in the class statement from
left-to-right.
Note that neither of these rules guarantee that the classes will
appear *sequentially* in the MRO.
class Derived(Mixin, Base): pass
This ensures that Mixin will always appear before Base in the MRO (and
thus override Base's methods) for Derived and for any subclass of
Derived.
It is possible to concoct elaborate inheritance hierarchies in which
it is not possible to come up with an MRO that will satisfy both 1)
and 2) above. In such cases, it is also useful to know what Python
will do. Fortunately, the answer to that is also simple: the type
constructor will throw an exception. So this isn't something that
really needs to be worried about.

...

That's some very nice information you got there. I also like how Ian Lewis explained that, it may not be very technical and perhaps that "logic" is accidental, but it surely helps you remember the correct order when doing multiple inheritance. (That part about forgetting the brackets and thinking of the code as a lined-up hierarchy, so MyClass => Mixin2 => Mixin1 => BaseClass).
In my opinion Python users should get used to that order. It doesn't matter if *atm* you're not overriding, if that's the correct way to do it, you should get used to it in my opinion because who know how you may need/want to expand your code in the future. Still, pretty much *no one* uses that order. A quick Google search returns (at least in my "bubble") many blog articles from "notable" Python users with that order wrong.

As long as I can rely on that it's OK for me. I couldn't find anything "official" on that subject though (mixins that override).

----

Post by dieter
Yet, you are clearly judging the code you pasted as not OK.

Actually, I didn't say (hard and fast) the code wasn't OK (if memory serves), I said I didn't like it -on a first superficial analysis-. Especially due the lack of recent experience with Python I was afraid I wasn't seeing some obvious caveat, thus I've basically asked you guys to tell me if the code is or isn't OK.

Post by dieter
It would be helpful for you to explain why you think the code you pasted is
not OK. To me it makes sense. English reads left-to-right, so method
lookups go left-to-right (and children before parents) in the inheritance
list.

Basically I wasn't very confident the code was OK because my intuition said the right way to order the classes I was inheriting from was from left to right. I don't see any sense in that analogy you made... sure English and most occidental languages read ltr, but I fail to see how it makes more sense to have the base class on the right and the mixins on the left. Basically, I think of the mixins like plugins of kinds, I'm "adding" functionality to that "base" class I'm inheriting from. Having them to the left sounds like a sandwich recipe that tells you to "slice the ham, put mayonnaise on it, and put it on the bread". I guess that's a lot to do with everyone's individual intuition sense, but I guess your intuition, albeit more correct for this specific case, is very different from the massive majority of people. Don't take my word for it, Google multi inheritance or mixin in Python and let me know if you find someone who thinks it's "natural" to do it that way. Most people just get that wrong without even knowing, and the only one I could find who gets that right simply states that at first that isn't intuitive at all. So you're the first person I've seen who thinks that's the intuitive way to do it.
I'm not complaining or something, I totally don't mind that "rule", and as I said the way Ian put that will surely make it stick to my brain like a scar, but it sure as hell doesn't seem intuitive for me.

Thank you guys for the help. I really appreciated it. I'll go with that code because it makes sense for this specific case now I'm convinced it's OK.

[Gregory Ewing]

Post by a***@gmail.com
I see that in most cases the order doesn't matter, but still I would
think that since the correct order is from right to left, that should be the
common practice.

This order is only "correct" if overriding is what you want.
That's not always going to be the case. The mixin might be
intended to supply default functionality that can be
overridden by the classes it's being mixed into. You
can't say that one order is more correct than the other
in general.

Post by a***@gmail.com
Basically, I
think of the mixins like plugins of kinds, I'm "adding" functionality to that
"base" class I'm inheriting from. Having them to the left sounds like a
sandwich recipe that tells you to "slice the ham, put mayonnaise on it, and
put it on the bread".

To me it's more like adding seasoning to a dish. Normally
you put the main ingredient in first, then add the salt
and pepper. This is probably the way most people are
thinking when they write the mixins after the main base
class.

[Michael Selik]

Post by a***@gmail.com
Thank you for your replies. I don't know if I'm quoting you correctly, I'm
quite confused with Google Groups... not sure if it's a "forum", something
like a mailing list, or both... or neither.

Mailing list. A "forum" in the metaphorical sense, not the sense of phpBB.
You're doing fine with quoting, though it would help to add who said what
when quoting multiple people.

since the correct order is from right to left,


Did you mean left-to-right?
Are you thinking of Java where there's only one parent class allowed and
you specify any extra interfaces you're implementing afterwards? Because
that's not what your code is doing.

Still, pretty much *no one* uses that order [mixin1, mixin2, base]. A quick

Post by a***@gmail.com
Google search returns (at least in my "bubble") many blog articles from
"notable" Python users with that order wrong.

Do you mind providing links? I haven't seen anyone "notable" make this
mistake.

Post by a***@gmail.com

Post by Michael Selik
To me it makes sense. English reads left-to-right, so method

lookups go left-to-right (and children before parents) in the inheritance

Post by Michael Selik
list.

Basically I wasn't very confident the code was OK because my intuition
said the right way to order the classes I was inheriting from was from left
to right.

That is correct! Write them in the order you are prioritizing the
definitions. Overrides before bases, left to right.

Post by a***@gmail.com
I don't see any sense in that analogy you made... sure English and most
occidental languages read ltr, but I fail to see how it makes more sense to
have the base class on the right and the mixins on the left. Basically, I
think of the mixins like plugins of kinds, I'm "adding" functionality to
that "base" class I'm inheriting from.

That's not a good metaphor for what's actually happening. For more detail,
check out Raymond's "super considered super" video (

Post by a***@gmail.com
Don't take my word for it, Google multi inheritance or mixin in Python and
let me know if you find someone who thinks it's "natural" to do it that way.

This might be a bit of selection bias. People who think it makes sense
might not have an incentive to write a blog post about it. People who don't
like it will thus be more vocal.

Ian Lewis seems to be ranked highly by Google (
https://www.ianlewis.org/en/mixins-and-python).
I disagree with his assertion that "most people" read an inheritance
hierarchy top-down with the bases at the left. I think of inheritance as a
tree. Even if it is a diamond at some point, a depth-first-search of a tree
isn't far off from the truth. The root (base) is at the bottom, in my mind,
and the leaves are at the top.

[a***@gmail.com]

Post by Gregory Ewing

Post by a***@gmail.com
I see that in most cases the order doesn't matter, but still I would
think that since the correct order is from right to left, that should be the
common practice.

This order is only "correct" if overriding is what you want.
That's not always going to be the case. The mixin might be
intended to supply default functionality that can be
overridden by the classes it's being mixed into. You
can't say that one order is more correct than the other
in general.

Yes, but (I presume) ordering it rtl doesn't have any caveat, and the code gets 'future-proof' in case you want to override a method in a class that's intended to be used as a mixin. I for instance am making an effort to get used to the rtl order as quickly as possible. But sure, in most cases it's probably never going to make a difference.

Post by Gregory Ewing

Post by a***@gmail.com
Basically, I
think of the mixins like plugins of kinds, I'm "adding" functionality to that
"base" class I'm inheriting from. Having them to the left sounds like a
sandwich recipe that tells you to "slice the ham, put mayonnaise on it, and
put it on the bread".

To me it's more like adding seasoning to a dish. Normally
you put the main ingredient in first, then add the salt
and pepper. This is probably the way most people are
thinking when they write the mixins after the main base
class.

...

Exactly.

----

Post by Gregory Ewing

Post by a***@gmail.com
Thank you for your replies. I don't know if I'm quoting you correctly, I'm
quite confused with Google Groups... not sure if it's a "forum", something
like a mailing list, or both... or neither.

Mailing list. A "forum" in the metaphorical sense, not the sense of phpBB.
You're doing fine with quoting, though it would help to add who said what
when quoting multiple people.

Post by a***@gmail.com
since the correct order is from right to left,

Did you mean left-to-right?
Are you thinking of Java where there's only one parent class allowed and
you specify any extra interfaces you're implementing afterwards? Because
that's not what your code is doing.

...

Nope, I'm certainly not thinking of interfaces of abstract classes, I'm literally thinking about mixins.

Post by Gregory Ewing

Post by a***@gmail.com
Still, pretty much *no one* uses that order [mixin1, mixin2, base]. A quick
Google search returns (at least in my "bubble") many blog articles from
"notable" Python users with that order wrong.

Do you mind providing links? I haven't seen anyone "notable" make this
mistake.

Not at all. Of course the snippets work because they're not overriding, but they all use an 'ltr' order. I feel somewhat bad about posting these links because I don't want to imply the code is wrong or something, since apparently there's no convention on that. So I'm publishing this list with all due respect to the programmers, and this only proves that it's probably more intuitive for most people to put is as [base,mixin]:

class Request(BaseRequest, AcceptMixin, ETagRequestMixin, UserAgentMixin, AuthorizationMixin):
http://stackoverflow.com/a/547714

class RealTestCase(BaseTestCase, MyMixin):
http://nedbatchelder.com/blog/201210/multiple_inheritance_is_hard.html

class TextBook(Book, IndexMixin):
https://ahal.ca/blog/2014/when-would-you-use-python-mixin/

Post by Gregory Ewing

Post by a***@gmail.com

Post by Michael Selik
To me it makes sense. English reads left-to-right, so method

lookups go left-to-right (and children before parents) in the inheritance

Post by Michael Selik
list.

Basically I wasn't very confident the code was OK because my intuition
said the right way to order the classes I was inheriting from was from left
to right.

That is correct! Write them in the order you are prioritizing the
definitions. Overrides before bases, left to right.

I'm sorry but it doesn't seem that obvious to me... It feels like building a house starting from the roof to me. Of course that's entirely subjective.

Post by Gregory Ewing

Post by a***@gmail.com
I don't see any sense in that analogy you made... sure English and most
occidental languages read ltr, but I fail to see how it makes more sense to
have the base class on the right and the mixins on the left. Basically, I
think of the mixins like plugins of kinds, I'm "adding" functionality to
that "base" class I'm inheriting from.

That's not a good metaphor for what's actually happening. For more detail,
check out Raymond's "super considered super" video (
http://youtu.be/EiOglTERPEo

...

I'm going to take a look. Thank you for the link.

Post by Gregory Ewing

Post by a***@gmail.com
Don't take my word for it, Google multi inheritance or mixin in Python and
let me know if you find someone who thinks it's "natural" to do it that way.

This might be a bit of selection bias. People who think it makes sense
might not have an incentive to write a blog post about it. People who don't
like it will thus be more vocal.

Yes, or maybe it's just my Google results that are biased.

Post by Gregory Ewing
Ian Lewis seems to be ranked highly by Google (
https://www.ianlewis.org/en/mixins-and-python).
I disagree with his assertion that "most people" read an inheritance
hierarchy top-down with the bases at the left. I think of inheritance as a
tree. Even if it is a diamond at some point, a depth-first-search of a tree
isn't far off from the truth. The root (base) is at the bottom, in my mind,
and the leaves are at the top.

Or maybe you're just very used to thinking like that :). I do understand your logic, but it's not something that happen "intuitively" for me.

Thank you for your replies.

[Michael Selik]

I ... am making an effort to get used to the rtl order as quickly as
possible.

Funny, you keep saying right-to-left. I think the issue is you think the
parent class is more important. Fight the power! Youth revolt! In Python,
the children are in control. Now you can read it left-to-right, as is
natural.

I feel somewhat bad about posting these links
The StackOverflow answer had an upvoted comment that the BaseRequest should
be written after the Mixins. The two blog posts read like conversation
starters. I don't think the authors intended to write model code.

A famous textbook getting it wrong would be more interesting. But even
Knuth makes mistakes (https://en.wikipedia.org/wiki/Knuth_reward_check).