With Python 3.9.16, clush performs full buffering on stdout/stderr instead of line buffering.
Environment
# cat /etc/redhat-release
Rocky Linux release 9.2 (Blue Onyx)
# which python
/usr/bin/python
# ls -l /usr/bin/python
lrwxrwxrwx. 1 root root 9 Jun 21 23:53 /usr/bin/python -> ./python3
# python --version
Python 3.9.16
Problem
Lines are not displayed live anymore, console or not. Example with a pipe to pv:
# clush -w @oss 'for i in {1..3}; do date; sleep 1; done' | pv -nlt
1.0811 0
2.0722 0
3.0633 0
4.0543 0
elm-rcf-io1-s2: Sat Sep 16 01:25:57 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:25:57 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:25:58 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:25:58 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:25:59 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:25:59 PM PDT 2023
4.2953 2
Same for stderr:
# clush -w @oss 'for i in {1..3}; do date >&2; sleep 1; done' |& pv -nlt
1.0812 0
2.0722 0
3.0632 0
elm-rcf-io1-s2: Sat Sep 16 01:28:21 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:28:21 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:28:22 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:28:22 PM PDT 2023
elm-rcf-io1-s2: Sat Sep 16 01:28:23 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:28:23 PM PDT 2023
3.4652 2
We should have the following instead:
# clush -w @oss 'for i in {1..3}; do date; sleep 1; done' | pv -nlt
elm-rcf-io1-s2: Sat Sep 16 01:41:30 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:41:30 PM PDT 2023
1.0797 2
elm-rcf-io1-s2: Sat Sep 16 01:41:31 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:41:31 PM PDT 2023
2.0814 4
elm-rcf-io1-s2: Sat Sep 16 01:41:32 PM PDT 2023
elm-rcf-io1-s1: Sat Sep 16 01:41:32 PM PDT 2023
3.0838 6
3.4693 8
This is an issue with the Python 3 port because we use sys.stdout.buffer (_io.BufferedWriter) instead of sys.stdout (_io.TextIOWrapper).
With Python 3.9.16:
- if we use
_io.BufferedWriter (eg. sys.stdout.buffer), we always get full buffering
- if stdout is a console,
_io.TextIOWrapper will perform line buffering by default (we want that)
- if stdout is NOT a console,
_io.TextIOWrapper will perform full buffering, but we want line buffering! Thankfully there is a way to enable line buffering with sys.stdout.reconfigure(line_buffering=True)
If line_buffering is True, flush() is implied when a call to write contains a newline character or a carriage return.
Resolution
We could add explicit flush() calls – it would be easy thanks to CLI/Display.py – but I don't love it, because they will potentially be called a lot of time. It's better to let Python handle that. So let's try to avoid using sys.stdout.buffer completely + io.TextIOWrapper.reconfigure() if available in Python 3 and see how it goes. I will submit a patch shortly.
With Python 3.9.16,
clushperforms full buffering on stdout/stderr instead of line buffering.Environment
Problem
Lines are not displayed live anymore, console or not. Example with a pipe to
pv:Same for stderr:
We should have the following instead:
This is an issue with the Python 3 port because we use
sys.stdout.buffer(_io.BufferedWriter) instead ofsys.stdout(_io.TextIOWrapper).With Python 3.9.16:
_io.BufferedWriter(eg.sys.stdout.buffer), we always get full buffering_io.TextIOWrapperwill perform line buffering by default (we want that)_io.TextIOWrapperwill perform full buffering, but we want line buffering! Thankfully there is a way to enable line buffering withsys.stdout.reconfigure(line_buffering=True)Resolution
We could add explicit
flush()calls – it would be easy thanks toCLI/Display.py– but I don't love it, because they will potentially be called a lot of time. It's better to let Python handle that. So let's try to avoid usingsys.stdout.buffercompletely + io.TextIOWrapper.reconfigure() if available in Python 3 and see how it goes. I will submit a patch shortly.