Discussion:
super().__init__() and bytes
(too old to reply)
Roel Schroeven
2024-12-03 09:41:06 UTC
Permalink
We can use super().__init__() in the __init__() method of a derived
class to initialize its base class. For example:

import string
class MyTemplate(string.Template):
    def __init__(self, template_string):
        super().__init__(template_string)
print(MyTemplate('Hello ${name}').substitute(name="Pedro"))

This works, and prints "Hello Pedro" as expected. Note that I passed
template_string in the super().__init__() call, and that is what used to
initialize the base class. So far nothing special.

When I try the same with bytes as base class though, that doesn't work
(at least in the Python version I'm using, which is CPython 3.11.2
64-bit on Windows 10):

class MyBytes(bytes):
    def __init__(self, data):
        super().__init__(data)
print(MyBytes(b'abcdefghijlkmn'))

This results in an exception:

Traceback (most recent call last):
  File "test_mybytes.py", line 4, in <module>
    print(MyBytes(b'abcdefghijlkmn'))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "test_mybytes.py", line 3, in __init__
    super().__init__(data)
TypeError: object.__init__() takes exactly one argument (the instance to
initialize)

I'm passing two arguments (data and the implicit self), and apparently
that's one too many. Let's try without arguments (i.e. only the implicit
self):

class MyBytes(bytes):
    def __init__(self, data):
        super().__init__()
print(MyBytes(b'abcdefghijlkmn'))

Now it works, and prints b'abcdefghijlkmn'. The same happens with int as
base class, and presumably a number of other classes. That behavior is
beyond my understanding, so I have some questions that might hopefully
lead to a better understanding:

(1) How does that work? How does my argument end up in the code that
initializes the instance state?

(2) How can I customize the argument is passed? For example, what if I
want to do something like (supersimple example) super().__init__(data * 2)?

(3) Why is bytes (and int, ...) different? Is it because it's a builtin?
Or implemented in C? Or something else?
--
"Man had always assumed that he was more intelligent than dolphins because
he had achieved so much — the wheel, New York, wars and so on — whilst all
the dolphins had ever done was muck about in the water having a good time.
But conversely, the dolphins had always believed that they were far more
intelligent than man — for precisely the same reasons."
-- Douglas Adams
Roel Schroeven
2024-12-03 10:01:00 UTC
Permalink
Post by Roel Schroeven
[...]
When I try the same with bytes as base class though, that doesn't work
(at least in the Python version I'm using, which is CPython 3.11.2
        super().__init__(data)
print(MyBytes(b'abcdefghijlkmn'))
  File "test_mybytes.py", line 4, in <module>
    print(MyBytes(b'abcdefghijlkmn'))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "test_mybytes.py", line 3, in __init__
    super().__init__(data)
TypeError: object.__init__() takes exactly one argument (the instance
to initialize)
I'm passing two arguments (data and the implicit self), and apparently
that's one too many. Let's try without arguments (i.e. only the
        super().__init__()
print(MyBytes(b'abcdefghijlkmn'))
Now it works, and prints b'abcdefghijlkmn'. The same happens with int
as base class, and presumably a number of other classes.
As a follow-up, it looks like this behavior is because bytes and int are
immutable. When I try with bytesarray instead of bytes, which works
largely the same but is mutable, things do work as I expect. There's a
hint in the documentation of __new__(): "__new__() is intended mainly to
allow subclasses of immutable types (like int, str, or tuple) to
customize instance creation". But that doesn't tell me why using
super().__init__(<custom arguments>) doesn't work for immutable classes.

The documentation for __init__() says " If a base class has an
__init__() method, the derived class’s __init__() method, if any, must
explicitly call it to ensure proper initialization of the base class
part of the instance; for example: super().__init__([args...])". So does
that mean that bytes and int not have an __init__() method? Is there a
link between being immutable and not having __init__()?
--
"Man had always assumed that he was more intelligent than dolphins because
he had achieved so much — the wheel, New York, wars and so on — whilst all
the dolphins had ever done was muck about in the water having a good time.
But conversely, the dolphins had always believed that they were far more
intelligent than man — for precisely the same reasons."
-- Douglas Adams
Stefan Ram
2024-12-03 10:26:33 UTC
Permalink
Post by Roel Schroeven
As a follow-up, it looks like this behavior is because bytes and int are
immutable.
Alright, so you've stumbled onto some pretty gnarly behavior with
Python's immutable types. Let's unpack this burrito:

1) Here's the skinny on how it goes down with immutable types
like "bytes" and "int":

These bad boys do their thing in "__new__", not
"__init__". It's like trying to change a burrito after
it's been wrapped - ain't going to happen.

When you hit "MyBytes( b'abcdefghijlkmn' )", it's like this:

a) Python calls up "MyBytes.__new__".

b) That passes the buck to "bytes.__new__", which
whips up the new "bytes" object.

c) Then "MyBytes.__init__" gets a shot, but the horse
has already left the barn.

2) Wanna customize those args?

For immutable types, you got to override `__new__`.
It's like swapping out the ingredients before the
burrito's wrapped:

class MyBytes( bytes ):
def __new__( cls, data ):
return super().__new__( cls, data * 2 )

print( MyBytes( b'abc' )) # Spits out: b'abcabc'

3) Why "bytes" (and "int", ...) are different beasts:

You nailed it in your follow-up. It's all about being immutable.
These types are coded in C for speed, but the real kicker
is they can't change after creation.

On your other questions:

- Yeah, immutable types like "bytes" and "int" pretty much
ghost "__init__". All the magic happens in "__new__".

- There's definitely a connection between being immutable
and ditching "__init__". It's like setting your GPS
before you start driving - once you're moving, you can't
change the route.

- That "__init__" doc you quoted? It's legit for mutable
types, but for immutable ones, it's like trying to add
avocado to a sealed burrito - not going to work.

To wrap it up:

- Mutable types: Override "__init__" to set things up.

- Immutable types: "__new__" is where the party's at.

This setup lets Python keep its promises about immutability while
still letting you customize through subclassing. It's like having
your artisanal, gluten-free burrito and eating it too.
Anders Munch
2024-12-03 12:55:59 UTC
Permalink
As a follow-up, it looks like this behavior is because bytes and int are immutable.
Yes.
But that doesn't tell me why using super().__init__(<custom arguments>) doesn't work for immutable classes.
bytes.__init__ does work, but it's just an inherited object.__init__, which does nothing, and takes no parameters.
__init__ cannot change the value of the bytes object; the value is set by bytes.__new__ and cannot change after that.

Best not to define an __init__ method at all, just use __new__.

Something like:

class BytesSubclass(bytes):
def __new__(cls, whatever, arguments, you, like):
bytesvalue = compute(whatever, arguments, you, like)
ob = bytes.__new__(cls, bytesvalue)
ob.some_other_att = compute_something_else(whatever, arguments, you, like)
Roel Schroeven
2024-12-03 14:24:55 UTC
Permalink
Post by Anders Munch
As a follow-up, it looks like this behavior is because bytes and int are immutable.
Yes.
OK.
Post by Anders Munch
But that doesn't tell me why using super().__init__(<custom arguments>) doesn't work for immutable classes.
bytes.__init__ does work, but it's just an inherited object.__init__, which does nothing, and takes no parameters.
__init__ cannot change the value of the bytes object; the value is set by bytes.__new__ and cannot change after that.
I see now why __init__, being a regular method, can't change an object's
value (or attributes in general) if that object is immutable. I'm not
sure why I didn't think of that before.

It's not entirely clear to me though how bytes.__new__ *can* set an
object's value. Isn't __new__ also a regular function? Are these
immutable classes special cases in the language that can't be recreated
in the same way with user-defined classes? Not that that's something I
want to do, and it's also not terribly important to me, but I'm trying
to better understand what's going on.
Post by Anders Munch
Best not to define an __init__ method at all, just use __new__.
bytesvalue = compute(whatever, arguments, you, like)
ob = bytes.__new__(cls, bytesvalue)
ob.some_other_att = compute_something_else(whatever, arguments, you, like)
return ob
Thanks, that works perfectly. That's also more important than
understanding all the nitty-gritty details (I feel a basic understanding
is important, but not necessarily always all the low-level details).
--
"There is no cause so noble that it will not attract fuggheads."
-- Larry Niven
Greg Ewing
2024-12-03 23:14:17 UTC
Permalink
Post by Roel Schroeven
It's not entirely clear to me though how bytes.__new__ *can* set an
object's value. Isn't __new__ also a regular function?
Yes, but the __new__ methods of the builtin immutable objects (int,
str, bytes, etc.) are implemented in C, and so are able to do things
that Python methods cannot.
--
Greg
Roel Schroeven
2024-12-04 11:38:33 UTC
Permalink
Post by Greg Ewing
Post by Roel Schroeven
It's not entirely clear to me though how bytes.__new__ *can* set an
object's value. Isn't __new__ also a regular function?
Yes, but the __new__ methods of the builtin immutable objects (int,
str, bytes, etc.) are implemented in C, and so are able to do things
that Python methods cannot.
Aha, yes, that's what I already suspected, but I wasn't sure. Thanks for
confirming that.

All clear now. Thanks to Anders and Greg for explaining this to me.

"In the old days, writers used to sit in front of a typewriter and stare out of
the window. Nowadays, because of the marvels of convergent technology, the thing
you type on and the window you stare out of are now the same thing.”
-- Douglas Adams

Loading...