Struggling to understand Callable type hinting

[Ian Pilcher]

I am making my first attempt to use type hinting in a new project, and
I'm quickly hitting areas that I'm having trouble understanding. One of
them is how to write type hints for a method decorator.

Here is an example that illustrates my confusion. (Sorry for the
length.)


import collections.abc

class BufferScanner(object):

def __init__(self, buf: str) -> None:
self._buffer = buf
self._index = 0
self._eof = False
self._line = 1
self._column = 1

@property
def eof(self) -> bool:
return self._eof

@staticmethod
def _check_eof(method: collections.abc.Callable -> (
collections.abc.Callable
):
def wrapper(*args, **kwargs):
self = args[0]
if not self._eof:
method(*args, **kwargs)
if self._index >= len(self._buffer):
self._eof = True
return self._eof
return wrapper

@_check_eof
def next_line(self) -> None:
"""Advance the scanner to the beginning of the next line."""
try:
i = self._buffer.index('\n', self._index)
except ValueError:
self._index = len(self._buffer) # decorator will set _eof
return
self._index = i + 1 # move past the newline
self._line += 1
self._column = 1


I cannot figure out how to correctly specify the Callable argument and
return type for _check_eof(). As indicated by the name, method should
be a method (of the BufferScanner class), so its first positional
argument should always be an instance of BufferScanner, but it could
have any combination of positional and/or keyword arguments after that.

I've read the TypeVar and ParamSpec documentation, and me head is
spinning, but neither one really seems to help with this situation.

Any pointers to good resources or the correct way to do this are
appreciated.

Thanks!

[dn]

Post by Ian Pilcher
I am making my first attempt to use type hinting in a new project, and
I'm quickly hitting areas that I'm having trouble understanding.  One of
them is how to write type hints for a method decorator.
Here is an example that illustrates my confusion.  (Sorry for the
length.)
import collections.abc

...

Post by Ian Pilcher
    def _check_eof(method: collections.abc.Callable -> (
        collections.abc.Callable

...

Post by Ian Pilcher
I cannot figure out how to correctly specify the Callable argument and
return type for _check_eof().  As indicated by the name, method should
be a method (of the BufferScanner class), so its first positional
argument should always be an instance of BufferScanner, but it could
have any combination of positional and/or keyword arguments after that.

Is it a typing problem?
The def is not syntactically-correct (parentheses).
What happens once corrected?

Also, which tool is 'complaining', and what does it have to say?

General comment: as far as type-hints go, rather than trying to learn
how to deal with complex situations, it might be better to ease-in
gradually - add the easy stuff now, and come back to deal with the rest
later (otherwise the typing 'tail' is wagging the coding 'dog'!)

[Paul Rubin]

Post by Ian Pilcher
I cannot figure out how to correctly specify the Callable argument and
return type for _check_eof().

It looks like _check_eof() is supposed to be a decorator, which is a
function that accepts a callable and returns another callable with the
same signature. Could something like this work? The idea is to use
type variables to capture the parameter lists. I'm not sure how to
specialize it even further to methods of that class. I haven't tested
any of this. Looking at the 3.12 docs I see that the typing stuff has
changed a noticeably since last time I tried to use it.

T = TypeVar('T')
U = TypeVar('U')

def _check_eof(method: Callable[T: ..., U]) -> Callable[T,U]:
...

Actually it might be best to do this with the new generics features.
I'll look at it some more out of general interest, but I think the other
poster is right to say start out with something approximate. In the
Haskell world there are well known opportunities to go completely crazy
with highly precise types, and with Python (mypy), I found as of 3.8
that there were often no good ways to do exactly what you wanted. The
Mypy type system isn't really sound anyway. It's more of a bug catching
device that works some of the time but not always, so don't expect too
much from it.

[Stefan Ram]

Post by Ian Pilcher
them is how to write type hints for a method decorator.

To get your type hints dialed in for a method decorator,
we're going to need to pull in "ParamSpec", "TypeVar", and
"Callable" from the typing module, plus "Concatenate" from
typing_extensions if you're rolling with Python older than
3.10. Check out how you can soup up your _check_eof method:

from typing import TypeVar, ParamSpec, Callable
from typing_extensions import Concatenate # Use this for Python < 3.10

T = TypeVar('T')
P = ParamSpec('P')

class BufferScanner:
# . . . other methods . . .

@staticmethod
def _check_eof(method: Callable[Concatenate['BufferScanner', P], T]) -> Callable[Concatenate['BufferScanner', P], bool]:
def wrapper(self: 'BufferScanner', *args: P.args, **kwargs: P.kwargs) -> bool:
if not self._eof:
method(self, *args, **kwargs)
if self._index >= len(self._buffer):
self._eof = True
return self._eof
return wrapper

# . . . other methods . . .

Let's break it down like we're dissecting a California roll:

1. "T = TypeVar('T')": This bad boy's a generic type
variable, standing in for whatever the original method
kicks out.

2. "P = ParamSpec('P')": This dude's capturing all the
parameters of the original method, minus the self part.

3. "Callable[Concatenate['BufferScanner', P], T]": This is
the type of the method getting decorated. It's like a
burrito bowl - BufferScanner instance as the base, other
ingredients (P) mixed in, and T as the guac on top.

4. "Callable[Concatenate['BufferScanner', P], bool]": This is
what the decorator's dishing out. It's taking the same
order as the original method (including self) and always
serves up a bool.

In the wrapper function, we're tagging self as "BufferScanner"
and using "P.args" and "P.kwargs" to scoop up the rest of
the arguments. A few more nuggets for you:

1. Stick to "typing" instead of "collections.abc" for the
freshest type hinting flavors.

2. In Python 3, your class can just chill as class
"BufferScanner": without the explicit object inheritance.

3. If you're cruising with Python 3.10 or newer, you can
import "Concatenate" straight from "typing". For the
classic versions, you'll need to grab it from
"typing_extensions".

This approach is like catching the perfect wave at Mavericks for
your method decorator type hinting. It captures the essence of
decorating a BufferScanner method and keeps all the juicy details
about parameters and return types, while making sure the self
parameter doesn't wipe out in the type annotations. Cowabunga!

[Stefan Ram]

Post by Paul Rubin
T = TypeVar('T')

Alright, so type variables in Python are like the Swiss Army
knife of the coding world. They're part of this thing called the
typing module, and they let you whip up functions and classes
that can roll with pretty much any type you throw at them.

Here's the skinny on when you'd need one of these bad boys instead
of just sticking to one type:

Python

from typing import TypeVar, List

T = TypeVar('T')

def first_and_last(items: List[T]) -> List[T]:
return [items[0], items[-1]]

Check it out - this "first_and_last" function is like a food truck
that can serve up any cuisine. It takes a list of whatever and
spits out a list with the first and last items.

Using this T type variable is like having a menu that changes
based on what's fresh that day. It keeps everything kosher between
what goes in and what comes out.

If we'd used a fixed type, like "List[int]", it'd be like a taco
truck that only serves carne asada. By using a type variable,
we're cooking up a function that's as flexible as a yoga
instructor but still gives the lowdown on types to those picky
static checkers and other devs. It's the best of both worlds,
like living in California - you get the beach /and/ the mountains!

"TypeVar" is a factory for creating type variables. It's not a
type itself, but a tool to create types.

When you write "T = TypeVar('T')", you're basically brewing
your own custom type. It's like creating a signature blend
at Philz Coffee - unique and tailored to your needs.

Once you've created this custom type "T", you can use it in
generic constructions like "List[T]". This tells Python,
"Hey, I want a list of whatever type T turns out to be".

The string argument 'T' in TypeVar('T') serves two main purposes:

1. It provides a name for the type variable, which is used
for debugging and error reporting. This name helps
developers and type checkers identify and track the type
variable throughout the code.

2. It's a convention to use the same string as the variable
name to which the TypeVar is assigned. This practice
enhances code readability and maintainability.

While any string could technically be used, it's strongly
recommended to follow the convention of using the same
string as the variable name.

For example:

Python

T = TypeVar('T') # Recommended
AnyStr = TypeVar('AnyStr') # Also follows the convention

Using a different string wouldn't break the functionality, but
it could lead to confusion:

Python

X = TypeVar('Y') # Works, but confusing and not recommended

The name is primarily for human readers and tools. It doesn't
affect the runtime behavior of the program, but it's crucial
for static type checking and code comprehension.

[Stefan Ram]

Post by Paul Rubin
T = TypeVar('T')
P = ParamSpec('P')

We use "ParamSpec('P')" instead of "TypeVar('P')" because "ParamSpec"
serves a specific purpose that "TypeVar" can't fulfill. Here's why:

1. "ParamSpec" is designed to capture the entire parameter
specification of a callable, including both positional
and keyword arguments. This is something a regular
TypeVar can't do.

2. While "TypeVar" represents a single type, "ParamSpec"
represents a collection of parameters. It's a bundle
that includes all the arguments a function might take.

3. ParamSpec allows for more precise typing of
higher-order functions, especially those that manipulate
other functions' signatures. For example, it's crucial
when you want to preserve the exact signature of a
wrapped function.

4. With ParamSpec, you can use the special attributes
".args" and ".kwargs" to refer to the positional and
keyword arguments respectively. This level of
granularity isn't available with TypeVar.

5. ParamSpec enables the use of the Concatenate operator,
which allows you to add parameters to an existing
parameter specification. This is particularly useful for
decorators that add arguments to the functions they wrap.

In essence, ParamSpec is like a Swiss Army knife for function
signatures, while TypeVar is more like a single-purpose tool.

When you're dealing with complex function manipulations, especially
in the realm of decorators or higher-order functions, ParamSpec gives
you the flexibility and precision that TypeVar simply can't match.

[Stefan Ram]

Post by Ian Pilcher
@staticmethod

"P" (ParamSpec) is resolved when the decorator is applied to any
method.

- "P" (ParamSpec) captures all parameters of the decorated
method except for the first "self" parameter.
"Concatenate['BufferScanner', P]" then adds the
"BufferScanner" type as the first parameter, effectively
representing the complete parameter list of the method.

"T" (TypeVar) is inferred based on the return type of the method
being decorated.

So, those type variables are resolved based on the specific
method the decorator is applied to, allowing for generic and
reusable type annotations across different classes and methods

(This "resolution" of a type name corresponds to the
initialization of a traditional name. But a single type name
can have multiple values when the wrapper is being applied
to multiple methods - one value per application.)

[Ian Pilcher]

(Note: I have mail delivery disabled for this list and read it through
GMane, so I am unable to respond with correct threading if I'm not cc'ed
directly.)

Post by dn

Post by Ian Pilcher
I am making my first attempt to use type hinting in a new project, and
I'm quickly hitting areas that I'm having trouble understanding.  One of
them is how to write type hints for a method decorator.
Here is an example that illustrates my confusion.  (Sorry for the
length.)
import collections.abc

...

Post by Ian Pilcher
     def _check_eof(method: collections.abc.Callable -> (
         collections.abc.Callable

...

Post by Ian Pilcher
I cannot figure out how to correctly specify the Callable argument and
return type for _check_eof().  As indicated by the name, method should
be a method (of the BufferScanner class), so its first positional
argument should always be an instance of BufferScanner, but it could
have any combination of positional and/or keyword arguments after that.

Is it a typing problem?
The def is not syntactically-correct (parentheses).

Yes, but only when I re-typed it in my email client to avoid word-
wrapping issues.

Post by dn
What happens once corrected?

Once the missing parenthesis is added, the version in my original post
works just fine, but it's incomplete, because the Callables aren't
parameterized.

Based on what I've read, I would expect to be able to replace:

@staticmethod
def _check_eof(method: collections.abc.Callable) -> (
collections.abc.Callable
):
...

with something like this:

@staticmethod
def _check_eof(
method: collections.abc.Callable[[BufferScanner, ...], None]
) -> (
collections.abc.Callable[[BufferScanner, ...], bool]
):
...

But Mypy gives me a bunch of errors when I do that.

bs.py:19: error: Unexpected "..." [misc]
bs.py:21: error: Unexpected "..." [misc]
bs.py:32: error: Argument 1 to "_check_eof" of "BufferScanner" has
incompatible type "Callable[[BufferScanner], None]"; expected
"Callable[[BufferScanner, Any], None]" [arg-type]
Found 3 errors in 1 file (checked 1 source file)

(And even that wouldn't really be correct, if it worked, because it
doesn't express the fact that the arguments of the two Callables are the
same. I believe that ParamSpecs are supposed to address this, but there
doesn't seem to be any way to use a ParamSpec when one or more of the
parameter types is known.

Post by dn
Also, which tool is 'complaining', and what does it have to say?

Mypy 1.14.1.

Post by dn
General comment: as far as type-hints go, rather than trying to learn
how to deal with complex situations, it might be better to ease-in
gradually - add the easy stuff now, and come back to deal with the rest
later (otherwise the typing 'tail' is wagging the coding 'dog'!)

Makes sense ... except that this decorator is what motivated me to use
type hinting in this project in the first place.

[Ian Pilcher]

(Note: I have mail delivery disabled for this list and read it through
GMane, so I am unable to respond with correct threading if I'm not cc'ed
directly.)

Post by Ian Pilcher
(And even that wouldn't really be correct, if it worked, because it
doesn't express the fact that the arguments of the two Callables are the
same.  I believe that ParamSpecs are supposed to address this, but there
doesn't seem to be any way to use a ParamSpec when one or more of the
parameter types is known.

I think that I figured it out.

__P = typing.ParamSpec('__P')

@staticmethod
def _check_eof(
method: collections.abc.Callable[
typing.Concatenate[BufferScanner, __P], None
]
) -> (
collections.abc.Callable[
typing.Concatenate[BufferScanner, __P], bool
]
):
...

Mypy is happy, and it catches me if I try to use the decorator on some-
thing that isn't a method of BufferScanner (or some other callable that
takes an instance of BufferScanner as its first positional argument).