You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Switch from ronn to AsciiDoc to generate manual pages #4897
The Git LFS project has used the ronn(1) utility to generate its roff and HTML format manual pages since at least 2014, which is a good run for any type of software tooling.
However, the ronn project itself (https://github.com/rtomayko/ronn) does not appear to have had any updates in that time, and with the recent migration of GitHub Actions workflows to Windows Server 2022 runners, the need to first install a Ruby build environment in order to run ronn has proven something of a challenge (see #4883 and ruby/setup-ruby#293 for details).
Rendering Issues
There are also a number of glitches in ronn's conversions of Markdown to roff and HTML; see #4851 as an example, but also the HTML output for the git-lfs-install(1) man page, which uses <dt class="flush"> for just two out of a number of bullet points and therefore renders them oddly:
A further complication with the use of ronn is that the ronn-format(7)specification calls for the use of angle brackets to delimit user-supplied arguments and variables. This is one of four different inline delimiters supported by ronn, the others being backticks (`), double-stars (**), and underscores (_), all of which we use more-or-less consistently in our existing manual source pages.
The use of angle brackets as delimiters, though, means that when our source manual pages are viewed in the GitHub UI, some of their contents are elided, leading to confusion. A recent issue pointed out the problem (see #4857 (comment)); at my suggestion the user opened a GitHub Community discussion on the topic.
To demonstrate the problem we can look at the Git LFS manual page for the git-lfs-migrate(1) command. The underlying
source Markdown contains this line:
`git lfs track` [options] [<pattern>...]
which describes how the command should be used.
After conversion by ronn(1) into the roff format, and then rendered from roff into HTML by Debian's online manual page viewer (https://manpages.debian.org/), this looks like:
However, when viewed on GitHub without the ?plain=1 query argument, it looks like:
The elision of the <pattern> term makes the page confusing and incorrect.
Moreover, not all angle bracket-delimited words are treated identically by ronn. Various HTML4 elements are specified and then exempted from conversion. We even rely on this in a few places where we intermingle custom angle bracket-delimited terms with HTML elements like <br>. And of course the list of HTML4 elements in ronn has not kept pace with the current HTML specifications.
Desired Attributes
We would like any potential replacement for ronn in our tooling to have at least the following characteristics:
Ability to generate high-quality roff and HTML versions of our manual pages.
Supports linking between manual pages and to external URLs in the HTML versions.
Easily installable on all GitHub Actions runners (Windows, macOS, and Linux).
Limited need to reformat our existing manual source pages.
Source format can be displayed without serious degradation in the GitHub UI.
On that last desired characteristic, when responding to user issues it is often convenient to be able to link to a recent version of a Git LFS manual page. Unlike the Git project, we don't host rendered HTML versions of our manual pages on, say, git-lfs.github.com, so it is highly convenient to be able to refer users to the source versions of the pages; see, for instance, #3805 (comment). We could try to remember to always append the ?plain=1 query argument to such links, but this adds friction when replying to users, and moreover, users themselves refer to these pages on the GitHub UI, both for reference and in their questions, such as in #4887 for instance.
Possible and Recommended Solutions
One potential option is to adopt the AsciiDoc source format and either AsciiDoc or Asciidoctor. This would align our source format with that of the Git project, for one thing, and if we named our source files with the .adoc extension, the GitHub UI should render them. However, we would then need to convert all our source files to a new format, and further tooling appears to be necessary to convert the DocBook output of these utilities into roff-format manual pages.
Another solution, and the one recommended in this proposal, is to replace ronn with Pandoc, which also offers direct conversion from Markdown to both HTML and roff formats.
The pandoc(1) utility should be available for installation via apt(8) on Debian, Homebrew on macOS, and Chocolatey on Windows. This would eliminate need to install a Ruby build environment with a ruby/setup-ruby@v1 GitHub Actions workflow step, which in turn would allow the Git LFS test suite to pass on Windows Server 2022.
The Markdown source format would mean our manual source pages would continue to render on GitHub's UI, and inline and reference links are fully supported for the HTML output.
We would need to revise all of our manual source pages, though; the existing ronn-format(7) ones will not convert cleanly when using pandoc(1). This effort, while significant, should not be as demanding as converting to an entirely different format like AsciiDoc.
One particular required change will be to replace the existing setext-style title headings with Pandoc's pandoc_title_block Markdown extension format, because Pandoc expects this in order to generate the roff manual pages' headers and footers, as described in the documentation:
The man page writer extracts a title, man page section number, and other header and footer information from the title line. The title is assumed to be the first word on the title line, which may optionally end with a (single-digit) section number in parentheses. (There should be no space between the title and the parentheses.) Anything after this is assumed to be additional footer and header text. A single pipe character (|) should be used to separate the footer text from the header text.
We will also have to replace any angle bracket-delimited words (other than HTML elements) with some other delimited form, as Pandoc does not support ronn's use of that technique for demarcating user-supplied arguments and variables in manual pages.
And we will need to write custom HTML templates for Pandoc to use, as its default HTML templates use a proportionally-spaced font and does not match the layout of a manual page displayed by man(1). We would perhaps also want to override the default template/styles.html template so that <em> elements were rendered as underlined instead of italicized, to align with their rendering by man(1), although this is not of major importance.
Additional Recommendations
If this proposal is adapted, we will need to edit all the manual source pages, and so we would have an opportunity to make some of our formatting more consistent between various pages and sections.
As noted above, ronn supports four types of inline formatting delimiters, and yet traditional roff-format manual pages are only displayed using two types of text highlighting, underlined text and/or emphasized (bold) text. And of course man(1) displays typically such pages in a monospaced font, so the use of Markdown backticks to delimit code sections seems somewhat superfluous.
We therefore propose that we eliminate the use of backticks and angle brackets as delimiters in our Markdown source, and restrict our inline formatting to just double stars (**) for emphasized text and single stars (*) for underlined text. The latter will not render in the GitHub UI, of course, but will leave the delimited text intact, unlike the use of angle brackets. Both should be correctly rendered in the output roff and HTML pages by Pandoc.
In places where we want to enclose a term in angle brackets to indicate it is a user-supplied argument or variable, we would use <...>. Often this would be combined with double stars to refer to a user-supplied value within a body of text. For instance, if we wanted to replicate the formatting of this portion of the Git project's git-checkout(1) manual page:
we would write something like:
\--orphan <new_branch>
Create a new *orphan* branch, named **<new_branch>**, started from
**<start_point>** and switch to it. The first commit made on this new
which would appear in the GitHub UI as:
--orphan <new_branch>
Create a new orphan branch, named <new_branch>, started from <start_point> and switch to it. The first commit made on this new
So, having read this, I have a couple of thoughts:
First, in general, this seems like a reasonable approach. Pandoc is well-known and used elsewhere.
However, I do want to point out that Asciidoctor provides built-in manual page generation (with -b manpage) as well as built-in HTML5 and XHTML5 output, so that's not a good reason to exclude it. AsciiDoc is my preferred language here, but I know it's not everyone's. (I have also contributed to the project multiple times, so I could feel comfortable fixing any bugs.)
I do think that it's fine to retain backticks, especially considering that we generate HTML pages and in some cases, we'll want to preserve monospace there.
We may need to adjust our built-in help generator to handle escape sequences such as < and >, since I don't think it does that.
We'll need to consider how Pandoc will work when building packages for e.g. CentOS 7.
Since Pandoc supports multiple variants of Markdown, we'll need to consider what variant we'd like to use (which should take into consideration what's available in older versions of our supported distros).
So with those thoughts, I would be fine with the given proposal, and I would also be fine with switching to AsciiDoc if folks prefer that.
On Wed, Mar 2, 2022, 8:18 AM brian m. carlson ***@***.***> wrote:
So, having read this, I have a couple of thoughts:
- First, in general, this seems like a reasonable approach. Pandoc is
well-known and used elsewhere.
- However, I do want to point out that Asciidoctor provides built-in
manual page generation (with -b manpage) as well as built-in HTML5 and
XHTML5 output, so that's not a good reason to exclude it. AsciiDoc is my
preferred language here, but I know it's not everyone's. (I have also
contributed to the project multiple times, so I could feel comfortable
fixing any bugs.)
- I do think that it's fine to retain backticks, especially
considering that we generate HTML pages and in some cases, we'll want to
preserve monospace there.
- We may need to adjust our built-in help generator to handle escape
sequences such as < and >, since I don't think it does that.
- We'll need to consider how Pandoc will work when building packages
for e.g. CentOS 7.
- Since Pandoc supports multiple variants of Markdown, we'll need to
consider what variant we'd like to use (which should take into
consideration what's available in older versions of our supported distros).
So with those thoughts, I would be fine with the given proposal, and I
would also be fine with switching to AsciiDoc if folks prefer that.
—
Reply to this email directly, view it on GitHub
<#4897 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AXQ3N7FV77UVPK2MRHCXFQDU552CVANCNFSM5PWZFJUA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
Thanks for the speedy feedback, @bk2204! A few further thoughts and notes, and a possible change of recommendation as a result. I had spotted that Asciidoctor can generate roff man pages, but it seemed like:
The Git project preferred to generate DocBook and then use xmlto(1) to turn it into man pages (but I may be mistaken about that!) I'm guessing that's because AsciiDoc doesn't support the -b manpage option, or for historical reasons.
We'd have to make a bigger investment in reformatting our source manual pages, and either reworking docs/man/mangen.go to process an entirely different input format or replacing it altogether.
Installing Asciidoctor in GitHub Actions might require the ruby/setup-ruby@v1 workflow step that's causing us grief on Windows Server 2022 runners.
So I forgot to mention those points last night, and just skipped ahead to suggesting Pandoc, which was precipitous of me. I was mostly concerned with trying to find a way around the Ruby build problems on Windows Server 2022 (see also ruby/setup-ruby#293) that are seemingly inevitable if we stick with ronn.
That said, I too prefer the AsciiDoc format as I think it would give us greater control and precision in our text formatting; Markdown has a number of limitations in this regard (when considered for this purpose). That's why I wanted to look at it as a possible option.
If we're dreaming big, I would love to publish versioned copies of our manual pages as HTML to git-lfs.github.com, akin to how git-scm.org has pages like https://git-scm.com/docs/git-checkout with a dropdown that lets you choose the specific Git version you have. Currently our site just has a "Docs" link that points to this repo's docs/ folder. And I'd also like to include HTML versions of our manual pages in our Debian and RPM packages.
I just made a test run of our GitHub Actions CI job which skips the make man step (on Windows, anyway, since it comes first there) and drops all the actions/setup-ruby@v1 steps, and adds an apt-get asciidoctor, brew install asciidoctor, or gem install asciidoctor command for Linux, macOS, and Windows respectively. The first two succeed, as expected, and so does the latter, which is in contrast to what gem install ronn does (see actions/runner-images#5100), so that eliminates one of my concerns with using Asciidoctor.
Another possibility I didn't consider last night is that we might use the asciidoc(1) and its a2x(1)utility, which appears to also require xstlproc(1) and maybe some other bits and pieces. Getting those working on Windows might not be as easy as installing the Asciidoctor gem, though.
One other thing which occurs to me is that we might be able to replace the custom mangen.go processor with some kind of Asciidoctor extensions or other tools to generate plain text output, which we could then embed into the git-lfs Go binary. The a2x(1) utility appears to convert HTML into text using w3m(1) or lynx(1). That feels more complex than necessary to me, though. Perhaps there's some other tool or extension we could use to convert the AsciiDoc input, either directly or indirectly, to plain text, so we don't need to write a new mangen.go.
All of which is to say that if you're willing to take on a bigger project than I'd initially envisioned (since I was just trying to work around the problems with ronn in our CI jobs), a conversion to AsciiDoc as the source format would be great.
Yeah, I definitely think Asciidoctor is the way to go here. It has the nice benefit that it's self-contained, so as long as we have a reasonably up-to-date Ruby, things should work very nicely.
I know we already have a bunch of stuff for 3.2.0, but maybe we could consider this for 3.3.0, considering that it might make our release process a little simpler, and due to the need to no longer build RPMs for ronn's dependencies, potentially slightly faster as well. That should give us several months to work on it at our leisure and come up with a solution we'd be happy with.
changed the title [-]Switch from ronn to Pandoc to generate manual pages[/-][+]Switch from ronn to AsciiDoc to generate manual pages[/+]on Apr 21, 2022
Activity
bk2204 commentedon Mar 2, 2022
So, having read this, I have a couple of thoughts:
-b manpage) as well as built-in HTML5 and XHTML5 output, so that's not a good reason to exclude it. AsciiDoc is my preferred language here, but I know it's not everyone's. (I have also contributed to the project multiple times, so I could feel comfortable fixing any bugs.)<and>, since I don't think it does that.So with those thoughts, I would be fine with the given proposal, and I would also be fine with switching to AsciiDoc if folks prefer that.
Cassandra1983 commentedon Mar 2, 2022
chrisd8088 commentedon Mar 2, 2022
Thanks for the speedy feedback, @bk2204! A few further thoughts and notes, and a possible change of recommendation as a result. I had spotted that Asciidoctor can generate roff man pages, but it seemed like:
xmlto(1)to turn it into man pages (but I may be mistaken about that!) I'm guessing that's because AsciiDoc doesn't support the-b manpageoption, or for historical reasons.docs/man/mangen.goto process an entirely different input format or replacing it altogether.ruby/setup-ruby@v1workflow step that's causing us grief on Windows Server 2022 runners.So I forgot to mention those points last night, and just skipped ahead to suggesting Pandoc, which was precipitous of me. I was mostly concerned with trying to find a way around the Ruby build problems on Windows Server 2022 (see also ruby/setup-ruby#293) that are seemingly inevitable if we stick with ronn.
That said, I too prefer the AsciiDoc format as I think it would give us greater control and precision in our text formatting; Markdown has a number of limitations in this regard (when considered for this purpose). That's why I wanted to look at it as a possible option.
If we're dreaming big, I would love to publish versioned copies of our manual pages as HTML to git-lfs.github.com, akin to how git-scm.org has pages like https://git-scm.com/docs/git-checkout with a dropdown that lets you choose the specific Git version you have. Currently our site just has a "Docs" link that points to this repo's
docs/folder. And I'd also like to include HTML versions of our manual pages in our Debian and RPM packages.I just made a test run of our GitHub Actions CI job which skips the
make manstep (on Windows, anyway, since it comes first there) and drops all theactions/setup-ruby@v1steps, and adds anapt-get asciidoctor,brew install asciidoctor, orgem install asciidoctorcommand for Linux, macOS, and Windows respectively. The first two succeed, as expected, and so does the latter, which is in contrast to whatgem install ronndoes (see actions/runner-images#5100), so that eliminates one of my concerns with using Asciidoctor.Another possibility I didn't consider last night is that we might use the
asciidoc(1)and itsa2x(1)utility, which appears to also requirexstlproc(1)and maybe some other bits and pieces. Getting those working on Windows might not be as easy as installing the Asciidoctor gem, though.One other thing which occurs to me is that we might be able to replace the custom
mangen.goprocessor with some kind of Asciidoctor extensions or other tools to generate plain text output, which we could then embed into thegit-lfsGo binary. Thea2x(1)utility appears to convert HTML into text usingw3m(1)orlynx(1). That feels more complex than necessary to me, though. Perhaps there's some other tool or extension we could use to convert the AsciiDoc input, either directly or indirectly, to plain text, so we don't need to write a newmangen.go.All of which is to say that if you're willing to take on a bigger project than I'd initially envisioned (since I was just trying to work around the problems with ronn in our CI jobs), a conversion to AsciiDoc as the source format would be great.
bk2204 commentedon Apr 19, 2022
Yeah, I definitely think Asciidoctor is the way to go here. It has the nice benefit that it's self-contained, so as long as we have a reasonably up-to-date Ruby, things should work very nicely.
I know we already have a bunch of stuff for 3.2.0, but maybe we could consider this for 3.3.0, considering that it might make our release process a little simpler, and due to the need to no longer build RPMs for ronn's dependencies, potentially slightly faster as well. That should give us several months to work on it at our leisure and come up with a solution we'd be happy with.
chrisd8088 commentedon Apr 20, 2022
Sounds good. I know
$DAYJOBis going to be busy for a while, but a provisional target of v3.3.0 would be great.[-]Switch from ronn to Pandoc to generate manual pages[/-][+]Switch from ronn to AsciiDoc to generate manual pages[/+]