How to debug TypeError: required field "lineno" missing from expr?

[Mark Lawrence]

Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

The code has a call to the ast fix_missing_locations function. I'm
aware of the Armin Ronacher tweet[3] stating

<quote>
"TypeError: required field "lineno" missing from stmt" — no, what you
actually mean is "tuple is not a statement'.
</quote>

but I don't think this is the problem. Still I've clearly managed to
screw something up somewhere along so line, so any bright ideas as to
how I can proceed?

Further would it be worth raising an enhancement request to get some
better diagnostics from the built-in compile function, or is such a
thing already on the bug tracker? Or is it simply too difficult, or what?

No panic needed, it's past 2am BST and I'm off to bed for my beauty
sleep, so give me a couple of months :)

[1] https://github.com/grantjenks/pypatt_python_pattern_matching
[2] http://alexleone.blogspot.co.uk/2010/01/python-ast-pretty-printer.html
[3] https://twitter.com/mitsuhiko/status/91169383254200320

[Terry Reedy]

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4.

With 2to3 or by hand?

Post by Mark Lawrence
I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.
The code has a call to the ast fix_missing_locations function.

ast.fix_missing_locations does not have a raise statement

Post by Mark Lawrence
I'm aware of the Armin Ronacher tweet[3] stating
<quote>
"TypeError: required field "lineno" missing from stmt" — no, what you
actually mean is "tuple is not a statement'.
</quote>

As near as I can tell, it is impossible for lineno to be missing from a
call to the inner function _fix. Grepping for "TypeError: required
field" in 3.4 .c and .py files (with Idle) does not return any hits.

astpp.py does not raise that particular error either.

A random tweet without context or example is not terribly helpful.

Post by Mark Lawrence
but I don't think this is the problem. Still I've clearly managed to
screw something up somewhere along so line, so any bright ideas as to
how I can proceed?

How many times have you asked people asking such questions to post both
the complete traceback and the code. Where does the traceback say the
exception is from? The What is the code that compiled with 2.7 and the
(modified?) code that does not compile. What is a minimal example that
exhibits the problem?

[Steven D'Aprano]

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

Now now Mark, you know how this works. Where's your COPY and PASTED
traceback? Where's the Short, Self Contained, Correct Example?

http://sscce.org/

Our crystal ball is in no better state than yours :-)

Post by Mark Lawrence
The code has a call to the ast fix_missing_locations function. I'm
aware of the Armin Ronacher tweet[3] stating
<quote>
"TypeError: required field "lineno" missing from stmt" — no, what you
actually mean is "tuple is not a statement'.
</quote>
but I don't think this is the problem.

That's what I dislike about Twitter. There's no context, and no way to tell
what Armin is talking about. I've never seen this error, and cannot imagine
how to get it, or what he means by "tuple is not a statement" for that
matter. A tuple can be a statement:

if condition:
(1, 2) # Construct a tuple, then throw it away.

Post by Mark Lawrence
Still I've clearly managed to
screw something up somewhere along so line, so any bright ideas as to
how I can proceed?
Further would it be worth raising an enhancement request to get some
better diagnostics from the built-in compile function,

If you think it is worth waiting months or years before you can actually use
those enhanced diagnostics, sure. What sort of diagnostics?

Peering into my crystal ball, I see that you're maybe generating some code
dynamically, then running something like:

code = compile(the_text, something, something)

and that's giving you an error. What happens if you do this?

print(the_text)
code = compile(the_text, something, something)


Can you extract the offending line that way? What happens if you copy and
paste the_text into a Python file, and then try to import or run that file?

[Mark Lawrence]

Post by Steven D'Aprano

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

Now now Mark, you know how this works. Where's your COPY and PASTED
traceback? Where's the Short, Self Contained, Correct Example?
http://sscce.org/
Our crystal ball is in no better state than yours :-)

The traceback was effectively that in the subject line. I'd have
provided a SSCCE if I'd had the faintest idea of how to produce one,
which I didn't have a couple of days ago.

Post by Steven D'Aprano

Post by Mark Lawrence
The code has a call to the ast fix_missing_locations function. I'm
aware of the Armin Ronacher tweet[3] stating
<quote>
"TypeError: required field "lineno" missing from stmt" — no, what you
actually mean is "tuple is not a statement'.
</quote>
but I don't think this is the problem.

That's what I dislike about Twitter. There's no context, and no way to tell
what Armin is talking about. I've never seen this error, and cannot imagine
how to get it, or what he means by "tuple is not a statement" for that
(1, 2) # Construct a tuple, then throw it away.

Post by Mark Lawrence
Still I've clearly managed to
screw something up somewhere along so line, so any bright ideas as to
how I can proceed?
Further would it be worth raising an enhancement request to get some
better diagnostics from the built-in compile function,

If you think it is worth waiting months or years before you can actually use
those enhanced diagnostics, sure. What sort of diagnostics?
Peering into my crystal ball, I see that you're maybe generating some code
code = compile(the_text, something, something)
and that's giving you an error. What happens if you do this?
print(the_text)
code = compile(the_text, something, something)
Can you extract the offending line that way? What happens if you copy and
paste the_text into a Python file, and then try to import or run that file?

I managed to find the problem with the aid of
https://github.com/simonpercivall/astunparse which allowed me to compare
the 2.7 output with the 3.4 and pinpoint precisely where I'd gone wrong.

Thanks anyway for your reply and all the others, it certainly got my
brain engaged in a logical manner. I might not be an expert on Python
ASTs but I've certainly enjoyed playing with them. All I've got to do
now is correct the next problem in the pipeline. Maybe I'll bore you
all with that tomorrow :)

[Laura Creighton]

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

Are you trying to use ast.copy_location recursively somewhere?
http://bugs.python.org/issue3530 Maybe you have found something else
that doesn't work recursively?

Laura

[Terry Reedy]

Post by Laura Creighton

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

Are you trying to use ast.copy_location recursively somewhere?
http://bugs.python.org/issue3530 Maybe you have found something else
that doesn't work recursively?

I do not understand the closing of this issue given that a) the failing
part of the code is copied from the doc (3.4 has the same in the
docstring) and b) that there is no recursion for a single node ast,
which the example should be.

The error message comes from
F:\Python\dev\34\Python\Python-ast.c: 3752:
PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from
stmt");

[Terry Reedy]

Post by Terry Reedy

Post by Laura Creighton

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

Are you trying to use ast.copy_location recursively somewhere?
http://bugs.python.org/issue3530 Maybe you have found something else
that doesn't work recursively?

The issue is that copy_location is not recursive and does not fix
children of the node being fixed, whereas the example method, copied
from the NodeTransformer doc, creates 3 new nodes, not just 1, that need
fixing. I reopened the issue as a doc issue to fix the example and
recommend the use of recursive fix_missing_locations() instead of the
non-recursive copy_location.

The only other location-related function in ast is increment_lineno,
which also works on multiple nodes. I would not be surprised is Mark's
issue results from Mark or someone else copying the bad doc example.

Post by Terry Reedy
I do not understand the closing of this issue given that a) the failing
part of the code is copied from the doc (3.4 has the same in the
docstring) and b) that there is no recursion for a single node ast,
which the example should be.

The original ast is actually an Expression with a Name. But the issue
is that the Name is replaced with a Subscript with Name and Str children
and all three need locations added.

Post by Terry Reedy
The error message comes from
PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from
stmt");

[Mark Lawrence]

Post by Terry Reedy

Post by Laura Creighton

Post by Mark Lawrence
Purely as an exercise I've been converting Grant Jenks' pypatt[1] from
2.7 to 3.4. I've managed to sort out most of the required changes by
checking on what I can see with an AST pretty printer[2]. So it's
rather frustrating to have the compile stage throw the error given in
the subject line.

Are you trying to use ast.copy_location recursively somewhere?
http://bugs.python.org/issue3530 Maybe you have found something else
that doesn't work recursively?

The issue is that copy_location is not recursive and does not fix
children of the node being fixed, whereas the example method, copied
from the NodeTransformer doc, creates 3 new nodes, not just 1, that need
fixing. I reopened the issue as a doc issue to fix the example and
recommend the use of recursive fix_missing_locations() instead of the
non-recursive copy_location.
The only other location-related function in ast is increment_lineno,
which also works on multiple nodes. I would not be surprised is Mark's
issue results from Mark or someone else copying the bad doc example.

FTR there is no call to copy_location in Grant's existing code. Whether
or not one is actually needed in 3.4 when compared to 2.7 is another
question.