Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeEncodeError: 'latin-1' codec can't encode character '\uf0e0' in position 0: ordinal not in range(256) #1557

Open
teto opened this issue Dec 13, 2020 · 6 comments

Comments

@teto
Copy link
Contributor

teto commented Dec 13, 2020

Before you submit a bug report, please make sure that the issue still exists on the master branch!

Describe the bug
On start I get


Traceback (most recent call last):
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1346, in run
    reraise(*exc_info)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1360, in wrapper
    rval = f(*args,**kargs)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1319, in _twisted_idle_callback
    callback()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 574, in entering_idle
    self.draw_screen()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 589, in draw_screen
    self.screen.draw_screen(self.screen_size, canvas)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 868, in draw_screen
    self.write(l)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 286, in write
    self._term_output_file.write(data)
UnicodeEncodeError: 'latin-1' codec can't encode character '\uf0e0' in position 0: ordinal not in range(256)

Software Versions

  • Python version: 3.8
  • Notmuch version: [if relevant]
  • Alot version: 0s94m17yph1gq9f2svipb3bbwbw1s4j3zf2xkg5h91006v8286r6

To Reproduce
Steps to reproduce the behaviour:

  1. for me just type "alot"

Error Log
Please include all error information from the log file.
relevant part from the log

Traceback (most recent call last):
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/logging/__init__.py", line 1084, in emit
    stream.write(msg + self.terminator)
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 30-34: ordinal not in range(256)
Call stack:
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1341, in run
    self.reactor.run()
  File "/nix/store/012k72gbxdbd76j0cckpz3sn51bsc4mx-python3.8-Twisted-20.3.0/lib/python3.8/site-packages/twisted/internet/asyncioreactor.py", line 267, in run
    self._asyncioEventloop.run_forever()
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/nix/store/i9lzacdfrwlj1ayw551c016s0fq71p7j-python3-3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 277, in apply_commandline
    await apply_this_command(c)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 723, in apply_command
    cmd.apply(self)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/commands/globals.py", line 125, in apply
    ui.buffer_open(buffers.SearchBuffer(ui, self.query,
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 417, in buffer_open
    self.buffer_focus(buf)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 477, in buffer_focus
    self.update()
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 668, in update
    self.mainloop.draw_screen()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 588, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/container.py", line 1089, in render
    body = self.body.render((maxcol, maxrow-ftrim-htrim),
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/buffers/buffer.py", line 19, in render
    return self.body.render(size, focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 439, in render
    middle, top, bottom = self.calculate_visible(
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 322, in calculate_visible
    self._set_focus_complete( (maxcol, maxrow), focus )
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 686, in _set_focus_complete
    return self._set_focus_first_selectable(
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 656, in _set_focus_first_selectable
    middle, top, bottom = self.calculate_visible(
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/listbox.py", line 385, in calculate_visible
    next, pos = self._body.get_next( pos )
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/walker.py", line 46, in get_next
    return self._get_at_pos(start_from + self.direction)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/walker.py", line 72, in _get_at_pos
    widget = self._get_next_item()
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/walker.py", line 85, in _get_next_item
    next_widget = self.containerclass(next_obj, **self.kwargs)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 26, in __init__
    self.rebuild()
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 60, in rebuild
    width, part = build_text_part(partname, self.thread,
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 145, in build_text_part
    content = prepare_string(name, thread, maxw)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 212, in prepare_string
    s = content(thread)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/widgets/search.py", line 177, in prepare_authors_string
    return thread.get_authors_string() or '(None)'
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 199, in get_authors_string
    for aname, aaddress in self.get_authors():
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 162, in get_authors
    msgs = sorted(self.get_messages().keys(),
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 248, in get_messages
    self._toplevel_messages.append(accumulate(self._messages,
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 243, in accumulate
    acc[M].append(accumulate(acc, m))
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 243, in accumulate
    acc[M].append(accumulate(acc, m))
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 243, in accumulate
    acc[M].append(accumulate(acc, m))
  [Previous line repeated 2 more times]
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/thread.py", line 240, in accumulate
    M = Message(self._dbman, msg, thread=self)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/message.py", line 59, in __init__
    sender = decode_header(msg.header('From'))
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/db/utils.py", line 539, in decode_header
    logging.debug("unquoted header: |%s|", header)
Message: 'unquoted header: |%s|'
Arguments: ('\u0410\u0440\u0442\u0451\u043c \u041c\u0443\u0445\u0430\u043c\u0435\u0434-\u041a\u0430\u0440\u0438\u043c\u043e\u0432 \u041c\u041f\u0411-802 <[email protected]>',)
Traceback (most recent call last):
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1346, in run
    reraise(*exc_info)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1360, in wrapper
    rval = f(*args,**kargs)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1319, in _twisted_idle_callback
    callback()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 574, in entering_idle
    self.draw_screen()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 589, in draw_screen
    self.screen.draw_screen(self.screen_size, canvas)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 868, in draw_screen
    self.write(l)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 286, in write
    self._term_output_file.write(data)
UnicodeEncodeError: 'latin-1' codec can't encode character '\uf0e0' in position 0: ordinal not in range(256)
@lucc
Copy link
Collaborator

lucc commented Dec 13, 2020

This looks like it is related to some message in your initial buffer. Can you try to find the message and confirm by starting alot with alot search id:my-special-message. Assuming you did not change bug_on_exit or initial_command you might use notmuch search --output messages tag:inbox AND NOT tag:killed | xargs -t -I{} -n1 alot search {} ';' exit in your shell to figure out which message is at fault.

@teto
Copy link
Contributor Author

teto commented Dec 13, 2020

your command gives me

notmuch search --output messages tag:inbox AND NOT tag:killed | xargs -t -I{} -n1 alot search {} ';' exit                                                                                 ~
alot search id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com ';' exit 
Traceback (most recent call last):
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/bin/.alot-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/__main__.py", line 137, in main
    UI(dbman, cmdstring)
  File "/nix/store/wq2i5ajl6vx3az3zkvvamfjpwlxh3gvv-alot-dev/lib/python3.8/site-packages/alot/ui.py", line 146, in __init__
    self.mainloop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1346, in run
    reraise(*exc_info)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/main_loop.py", line 1360, in wrapper
    rval = f(*args,**kargs)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 417, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 449, in get_available_raw_input
    codes = self._get_gpm_codes() + self._get_keyboard_codes()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 523, in _get_keyboard_codes
    code = self._getch_nodelay()
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 661, in _getch_nodelay
    return self._getch(0)
  File "/nix/store/1h2sxiczmjnq420f3s4x8zp9zaz1zljf-python3.8-urwid-2.1.2/lib/python3.8/site-packages/urwid/raw_display.py", line 571, in _getch
    return ord(os.read(fd, 1))
TypeError: ord() expected a character, but string of length 0 found

And I've uploaded the content of notmuch show id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com > msg.txt here: msg.txt

@lucc
Copy link
Collaborator

lucc commented Dec 13, 2020

That is the file as rendered by notmuch(1). Please upload the raw file from your mail folder. You can find it with notmuch search --output files id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com.

@teto
Copy link
Contributor Author

teto commented Dec 13, 2020

sry, here you go:
msg.txt
note that the message appears 3 times in the output

notmuch search --output files id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com                                                                           
/home/teto/maildir/gmail/[Gmail]/Tous les messages/cur/1604851446.4383_1515.jedha,U=22591:2,S
/home/teto/maildir/gmail/Inbox/cur/1604851319.4383_514.jedha,U=106227:2,S
/home/teto/maildir/gmail/Inbox/cur/1605397110.26667_3953.jedha,U=173701:2,S

@lucc
Copy link
Collaborator

lucc commented Dec 14, 2020

That message looks harmless (it claims to be ascii and actually is ascii). Can you confirm that alot search id:99aa6841f5fd378b65518716f23457f6b52aea0c.1604845519.git.geliangtang@gmail.com produces the original error?

I just reread your #1557 (comment) and tested it. That is actually another error and not your original one. That was my bad I am sorry, alot seems to not like it when stdin is not a terminal. So we have to search the offending message like this:

for q in $(notmuch search --output messages tag:inbox AND NOT tag:killed); do
  echo $q
  alot search $q \; exit
done

If you find a message please also confirm that it reproduces your original error.

@lucc
Copy link
Collaborator

lucc commented Dec 14, 2020

I also reread the first exception and it seems to fail when encoding to latin1 (instead of decoding from latin1). So what is your $TERM and $LANG just in case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants