Allow hooks to output arbitrary bytes: UnicodeDecodeError: 'utf8' codec can't decode byte · Issue #245 · pre-commit/pre-commit · GitHub
Skip to content

Allow hooks to output arbitrary bytes: UnicodeDecodeError: 'utf8' codec can't decode byte #245

Description

@Lucas-C

I initially mentioned this issue in #207, but only found an easy-to-reproduce test this week.
I confirm this bug is still present in pre-commit v0.5.3 :

cat <<EOF >> .pre-commit-config.yaml
- repo: local
  hooks:
  - id: utf8_error_test
    name: UTF8 cmd output Error test
    language: script
    entry: echo_problematic_char.sh
    files: ''
EOF

cat <<EOF >echo_problematic_char.sh
#!/bin/bash
echo 'Problematic character: �'
exit 1
EOF
chmod u+x echo_problematic_char.sh

git add .pre-commit-config.yaml
pre-commit run utf8_error_test

The command output:

UTF8 cmd output Error test.........................................................................................................................................An unexpected error has occurred: UnicodeDecodeError: 'utf8' codec can't decode byte 0xbb in position 23: invalid start byte
Check the log at ~/.pre-commit/pre-commit.log

And the error log:

An unexpected error has occurred: UnicodeDecodeError: 'utf8' codec can't decode byte 0xbb in position 23: invalid start byte
Traceback (most recent call last):
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/error_handler.py", line 34, in error_handler
    yield
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/main.py", line 144, in main
    return run(runner, args)
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/commands/run.py", line 185, in run
    return _run_hooks(repo_hooks, args, write, environ)
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/commands/run.py", line 117, in _run_hooks
    retval |= _run_single_hook(hook, repo, args, write, skips)
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/commands/run.py", line 88, in _run_single_hook
    retcode, stdout, stderr = repo.run_hook(hook, filenames)
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/repository.py", line 123, in run_hook
    self.cmd_runner, hook, file_args,
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/languages/script.py", line 19, in run_hook
    retcode=None,
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/prefixed_command_runner.py", line 41, in run
    return cmd_output(*replaced_cmd, __popen=self.__popen, **kwargs)
  File "/home/lucas/.local/lib/python2.7/site-packages/pre_commit/util.py", line 153, in cmd_output
    stdout = stdout.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbb in position 23: invalid start byte

The problematic UTF8 character is '»' (U+00BB), and it originally came from a maven command stderr output.

I won't have time to try to solve this issue right now, but I'll try to get a look at it soon.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions