How to manage accented characters in mail header?

[Chris Green]

I have a Python script that filters my incoming E-Mail. It has been
working OK (with various updates and improvements) for many years.

I now have a minor new problem when handling E-Mail with a From: that
has accented characters in it:-

From: Sébastien Crignon <***@amvs.fr>


I use Python mailbox to parse the message:-

import mailbox
...
...
msg = mailbox.MaildirMessage(sys.stdin.buffer.read())

Then various mailbox methods to get headers etc.
I use the following to get the From: address:-

str(msg.get('from', "unknown").lower()

The result has the part with the accented character wrapped as follows:-

From: =?utf-8?B?U8OpYmFzdGllbiBDcmlnbm9u?= <***@amvs.fr>


I know I have hit this issue before but I can't rememeber the fix. The
problem I have now is that searching the above doesn't work as
expected. Basically I just need to get rid of the ?utf-8? wrapped bit
altogether as I'm only interested in the 'real' address. How can I
easily remove the UTF8 section in a way that will work whether or not
it's there?

[Stefan Ram]

In Python, when you roll with decode_header from the email.header
module, it spits out a list of parts, where each part is like
a tuple of (decoded string, charset). To smash these decoded
sections into one string, you’ll want to loop through the list,
decode each piece (if it needs it), and then throw them together.
Here’s a straightforward example of how to pull this off:

from email.header import decode_header

# Example header
header_example = \
'From: =?utf-8?B?U8OpYmFzdGllbiBDcmlnbm9u?= <***@amvs.fr>'

# Decode the header
decoded_parts = decode_header(header_example)

# Kick off an empty list for the decoded strings
decoded_strings = []

for part, charset in decoded_parts:
if isinstance(part, bytes):
# Decode the bytes to a string using the charset
decoded_string = part.decode(charset or 'utf-8')
else:
# If it’s already a string, just roll with it
decoded_string = part
decoded_strings.append(decoded_string)

# Join the parts into a single string
final_string = ''.join(decoded_strings)

print(final_string)# From: Sébastien Crignon <***@amvs.fr>

Breakdown

decode_header(header_example): This line takes your email header
and breaks it down into a list of tuples.

Looping through decoded_parts: You check if each part is in
bytes. If it is, you decode it using whatever charset it’s
got (defaulting to 'utf-8' if it’s a little vague).

Appending Decoded Strings: You toss each decoded part into a list.

Joining Strings: Finally, you use ''.join(decoded_strings) to glue
all the decoded strings into a single, coherent piece.

Just a Heads Up

Keep an eye out for cases where the charset might be None. In those
moments, it’s smart to fall back to 'utf-8' or something safe.

[Chris Green]

Post by Stefan Ram
In Python, when you roll with decode_header from the email.header
module, it spits out a list of parts, where each part is like
a tuple of (decoded string, charset). To smash these decoded
sections into one string, you’ll want to loop through the list,
decode each piece (if it needs it), and then throw them together.
from email.header import decode_header
# Example header
header_example = \
# Decode the header
decoded_parts = decode_header(header_example)
# Kick off an empty list for the decoded strings
decoded_strings = []
# Decode the bytes to a string using the charset
decoded_string = part.decode(charset or 'utf-8')
# If it’s already a string, just roll with it
decoded_string = part
decoded_strings.append(decoded_string)
# Join the parts into a single string
final_string = ''.join(decoded_strings)
Breakdown
decode_header(header_example): This line takes your email header
and breaks it down into a list of tuples.
Looping through decoded_parts: You check if each part is in
bytes. If it is, you decode it using whatever charset it’s
got (defaulting to 'utf-8' if it’s a little vague).
Appending Decoded Strings: You toss each decoded part into a list.
Joining Strings: Finally, you use ''.join(decoded_strings) to glue
all the decoded strings into a single, coherent piece.
Just a Heads Up
Keep an eye out for cases where the charset might be None. In those
moments, it’s smart to fall back to 'utf-8' or something safe.

Thanks, I think! :-)

Is there a simple[r] way to extract just the 'real' address between
the <>, that's all I actually need. I think it has the be the last
chunk of the From: doesn't it?

[Stefan Ram]

Post by Chris Green
Is there a simple[r] way to extract just the 'real' address between
the <>, that's all I actually need. I think it has the be the last
chunk of the From: doesn't it?

Besides the deal with the pointy brackets, there's also this
other setup with round ones, like in

***@amvs.fr (Sébastien Crignon)

. The standard library has:

email.utils.parseaddr(address)

Parse address – which should be the value of some
address-containing field such as To or Cc - into its
constituent realname and email address parts. Returns a tuple
of that information, unless the parse fails, in which case a
2-tuple of ('', '') is returned.

.

[Peter J. Holzer]

Post by Chris Green
Is there a simple[r] way to extract just the 'real' address between
the <>, that's all I actually need. I think it has the be the last
chunk of the From: doesn't it?

No,
From: <***@amvs.fr> (Sébastien Crignon)
would also be permissible (properly encoded, of course), and even
From: < sebastien (Sébastien) . crignon (Crignon) @ amvs . fr >
(although I think the latter is deprecated).

And also, there can be more than one address in a From header.

To properly extract email addresses from a header, use
email.utils.getaddresses(). You don't have to decode the header first.
The MIME-encoding is supposed to not interfere with parsing headers for
machine-readable information like addresses or message ids.

hp

[Peter Pearson]

Post by Chris Green
I have a Python script that filters my incoming E-Mail. It has been
working OK (with various updates and improvements) for many years.
I now have a minor new problem when handling E-Mail with a From: that
has accented characters in it:-
I use Python mailbox to parse the message:-
import mailbox
...
...
msg = mailbox.MaildirMessage(sys.stdin.buffer.read())
Then various mailbox methods to get headers etc.
I use the following to get the From: address:-
str(msg.get('from', "unknown").lower()
The result has the part with the accented character wrapped as follows:-
I know I have hit this issue before but I can't rememeber the fix. The
problem I have now is that searching the above doesn't work as
expected. Basically I just need to get rid of the ?utf-8? wrapped bit
altogether as I'm only interested in the 'real' address. How can I
easily remove the UTF8 section in a way that will work whether or not
it's there?

This seemed to work for me:

import email.header
text, encoding = email.header.decode_header(some_string)[0]