Docstring perf by aaltat · Pull Request #5686 · robotframework/robotframework · GitHub
Skip to content

Docstring perf#5686

Closed
aaltat wants to merge 2 commits into
robotframework:masterfrom
aaltat:docstring_perf
Closed

Docstring perf#5686
aaltat wants to merge 2 commits into
robotframework:masterfrom
aaltat:docstring_perf

Conversation

@aaltat

@aaltat aaltat commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

There is simple performance script and I made some measurements.

  1. form docstring
  Showing: median / min / max / mean / stdev  (10 runs per measurement)
  direct / prose_only
    time:  median=0.033s  min=0.033s  max=0.034s  mean=0.033s  stdev=0.001s
    mem:   median=0.00MB  min=0.00MB  max=0.01MB  mean=0.00MB  stdev=0.00MB
  libdoc / prose_only
    time:  median=3.004s  min=2.962s  max=3.024s  mean=3.001s  stdev=0.020s
    mem:   median=21.77MB  min=21.42MB  max=33.74MB  mean=22.92MB  stdev=3.81MB
  rf_import / prose_only
    time:  median=0.662s  min=0.606s  max=0.723s  mean=0.669s  stdev=0.033s
    mem:   median=14.27MB  min=13.33MB  max=14.92MB  mean=14.05MB  stdev=0.66MB
  direct / args_only
    time:  median=0.102s  min=0.102s  max=0.105s  mean=0.103s  stdev=0.001s
    mem:   median=0.01MB  min=0.01MB  max=0.01MB  mean=0.01MB  stdev=0.00MB
  libdoc / args_only
    time:  median=3.405s  min=3.350s  max=3.460s  mean=3.403s  stdev=0.038s
    mem:   median=24.73MB  min=24.73MB  max=39.61MB  mean=26.90MB  stdev=4.59MB
  rf_import / args_only
    time:  median=0.705s  min=0.617s  max=0.738s  mean=0.682s  stdev=0.051s
    mem:   median=13.54MB  min=13.53MB  max=16.77MB  mean=14.15MB  stdev=1.29MB
  direct / full_structured
    time:  median=0.158s  min=0.157s  max=0.161s  mean=0.159s  stdev=0.001s
    mem:   median=0.01MB  min=0.01MB  max=0.02MB  mean=0.01MB  stdev=0.00MB
  libdoc / full_structured
    time:  median=3.848s  min=3.679s  max=3.912s  mean=3.842s  stdev=0.064s
    mem:   median=25.47MB  min=25.47MB  max=40.64MB  mean=27.76MB  stdev=4.81MB
  rf_import / full_structured
    time:  median=0.647s  min=0.634s  max=0.789s  mean=0.699s  stdev=0.075s
    mem:   median=13.94MB  min=13.94MB  max=18.55MB  mean=14.41MB  stdev=1.46MB
  1. from master
  Showing: median / min / max / mean / stdev  (10 runs per measurement)
 direct / prose_only
   time:  median=0.004s  min=0.004s  max=0.005s  mean=0.004s  stdev=0.000s
   mem:   median=0.00MB  min=0.00MB  max=0.00MB  mean=0.00MB  stdev=0.00MB
 libdoc / prose_only
   time:  median=2.835s  min=2.746s  max=2.860s  mean=2.825s  stdev=0.037s
   mem:   median=21.00MB  min=20.65MB  max=32.93MB  mean=22.15MB  stdev=3.79MB
 rf_import / prose_only
   time:  median=0.637s  min=0.625s  max=0.678s  mean=0.638s  stdev=0.015s
   mem:   median=13.50MB  min=12.56MB  max=14.16MB  mean=13.29MB  stdev=0.66MB
 direct / args_only
   time:  median=0.004s  min=0.004s  max=0.004s  mean=0.004s  stdev=0.000s
   mem:   median=0.00MB  min=0.00MB  max=0.00MB  mean=0.00MB  stdev=0.00MB
 libdoc / args_only
   time:  median=3.131s  min=3.085s  max=3.204s  mean=3.131s  stdev=0.036s
   mem:   median=20.69MB  min=20.69MB  max=35.57MB  mean=22.86MB  stdev=4.59MB
 rf_import / args_only
   time:  median=0.688s  min=0.591s  max=0.704s  mean=0.653s  stdev=0.052s
   mem:   median=12.77MB  min=12.77MB  max=16.00MB  mean=13.38MB  stdev=1.29MB
 direct / full_structured
   time:  median=0.004s  min=0.004s  max=0.004s  mean=0.004s  stdev=0.000s
   mem:   median=0.00MB  min=0.00MB  max=0.00MB  mean=0.00MB  stdev=0.00MB
 libdoc / full_structured
   time:  median=3.514s  min=3.490s  max=3.897s  mean=3.557s  stdev=0.122s
   mem:   median=21.10MB  min=21.10MB  max=36.28MB  mean=23.39MB  stdev=4.81MB
 rf_import / full_structured
   time:  median=0.691s  min=0.604s  max=0.770s  mean=0.688s  stdev=0.075s
   mem:   median=13.18MB  min=13.18MB  max=17.79MB  mean=13.64MB  stdev=1.46MB

No idea is this good way to test perf or would be there be a better way. But now there is starting point to discuss.

Copilot AI review requested due to automatic review settings June 18, 2026 22:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new docstring parser (robot.running.docstring.parse_docstring) with extensive unit tests, plus a standalone benchmark script intended to measure parser performance in different Robot Framework-related scenarios.

Changes:

  • Added ParsedDocString dataclass and parse_docstring() implementation for parsing Google-style Args:/Returns: sections.
  • Added comprehensive unit tests covering plain prose, structured sections, malformed inputs, indentation edge cases, and RF variable syntax.
  • Added a performance benchmark script under utest/perf to time/allocate/profile different benchmark phases.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 6 comments.

File Description
src/robot/running/docstring.py New docstring parsing API returning structured doc/args/returns.
utest/running/test_docstring.py New unit test suite validating parsing behavior across many edge cases.
utest/perf/bench_docstring.py New benchmark script to measure timing/memory/profile across “direct/libdoc/rf_import” phases.
utest/perf/__init__.py Adds perf package marker (empty).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +39 to +45
def parse_docstring(doc: str) -> ParsedDocString:
"""Always returns a ParsedDocString — never raises."""
try:
return _parse(doc)
except (TypeError, AttributeError):
return ParsedDocString(doc="", args={}, returns="")

Comment on lines +9 to +12
libdoc — calls ``LibraryDocumentation(path)`` on a generated library.
This is the *only* RF execution phase that actually calls
``parse_docstring`` (via KeywordImplementation.update_docs).

Comment on lines +13 to +16
rf_import — imports the library via RF's own ``TestLibrary.from_name``
(the same path used during test execution). Confirms that
``parse_docstring`` is NOT called during library import — that
only happens in the libdoc phase.
Comment on lines +334 to +336
# 2. libdoc (the only RF path that calls parse_docstring)
if baseline and _KW_IMPL_HAS_PARSE:
_kw_impl.parse_docstring = _noop_parse
Comment on lines +345 to +347
# 3. RF library import (no parse_docstring — only in libdoc)
s = _measure_many(lambda p=lib_path: _run_rf_import(p), runs)
_print_entry("rf_import", style, s, runs)
Comment on lines +350 to +353
@aaltat aaltat closed this Jul 5, 2026
@aaltat aaltat deleted the docstring_perf branch July 5, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants