iframe-proxy

febb0e · 2025-09-10T10:38:39Z

Custom Metadata Feature

Overview

Introduces custom metadata feature for test cases and user keywords. The implementation allows users to define metadata for both test cases and keywords using the [CustomName] syntax.

Implementation Details

Custom Metadata Syntax:
- Any setting in square brackets (e.g., [Owner], [Requirement], [My Custom Metadata]) is now treated as custom metadata.
- Custom metadata is supported in both test cases and user keywords.
- Metadata values can be single or multi-valued, and support variable resolution.
Parsing and Output:
- Custom metadata is parsed and stored in the test/keyword model.
- Custom metadata is included in the output XML and HTML reports by default.
- Output and parsing logic was adapted to treat all unknown square bracket settings as custom metadata.
Backwards Compatibility:
- Test cases that previously expected errors for undefined square bracket settings (e.g., [UndefinedSetting]) will now treat these as valid custom metadata.

CLI Functionality: `--custommetadata` Flag

Purpose:
- The --custommetadata flag allows users to filter which custom metadata keys are considered "valid" for display and parsing in output files.
Behavior:
- No flag set: All custom metadata keys are valid and included in output and parsing.
- Flag set with one or more keys: Only the specified custom metadata keys are considered valid and included in output and parsing. Others are ignored. Test cases and user keywords do not fail.
- Important: The flag does not influence the execution of tests or keywords. It only affects which custom metadata is displayed in output or parsed through execution.

Test Coverage

Acceptance Tests:
- Acceptance tests were created to cover major scenarios:
  - Basic custom metadata parsing
  - Edge cases (empty values, special characters, Unicode, etc.)
  - Performance (many metadata entries)
  - Variable resolution and built-in expressions
  - CLI filtering with --custommetadata
  - Output and report validation
Unit Tests:
- Unit tests ensure parsing, model storage, and output generation for custom metadata.

Notable Changes

Error Handling:
- Square bracket settings that do not match known settings are now always treated as custom metadata. This means tests expecting errors for undefined settings will no longer fail; instead those settings are parsed as metadata.

Summary

This feature allows users to create tests and keywords with metadata they need. Custom metadata values support the same formatting as settings metadata (*bold*, _italic_, `code`), variable resolution, multi-line content, etc. The --custommetadata flag provides control over output, without affecting test execution.

Usage Examples

Basic Custom Metadata

*** Test Cases ***
User Login Test
    [Documentation]    Test user authentication functionality
    [Owner]             Robot
    [Requirement]      REQ-001
    [Priority]         High
    [Component]        Authentication
    Log    Testing user login

Variable Resolution

*** Variables ***
${BUILD_NUMBER}       2024.1.0
${TEST_ENVIRONMENT}   ${{'Development' if os.getenv('CI') is None else 'CI/CD'}}
${TEAM_EMAIL}         qa-team@company.com

*** Test Cases ***
API Integration Test
    [Documentation]    Test API integration with dynamic metadata
    [Build Version]    ${BUILD_NUMBER}
    [Environment]      ${TEST_ENVIRONMENT}
    [Contact]          ${TEAM_EMAIL}
    [Created Date]     ${{'%Y-%m-%d' | strftime}}
    Log    Test API integration

Text Formatting

*** Test Cases ***
Formatted Metadata Test
    [Documentation]    Test with text formatting
    [Description]      *Critical* test for _user registration_ flow
    [Steps]       1. Navigate to registration page
    ...                2. Fill form with _valid test data_
    ...                3. Submit and verify *success message*
    [Bug Report]       BUG-123: See ``validate_user()`` function
    [Test Data]        File: ``test_users.json`` with *valid* entries
    [Requirements]     REQ-001    REQ-002    REQ-003
    Log    Test with formatted metadata

Inline Python Evaluation

*** Test Cases ***
Dynamic Metadata Test
    [Documentation]    Test with dynamically calculated metadata
    [System Info]      ${{platform.system()}} ${{platform.release()}}
    [Python Version]   ${{sys.version.split()[0]}}
    [Timestamp]        ${{datetime.now().isoformat()}}
    [File Count]       ${{len(os.listdir('/tmp'))}} files in temp
    Log    Test with dynamic metadata

Keyword Metadata

*** Keywords ***
Database Connection
    [Documentation]    Establish database connection with metadata
    [Owner]            DB Team
    [Timeout]          ${{'30s' if os.getenv('FAST_MODE') else '60s'}}
    [Dependencies]     [http://www.example.com|Example Dependency]
    [Performance]      *Optimized* for _high-load_ scenarios
    [Notes]            Connection enabled with ``max_connections=50``
    ...
    ...                Auto-retry on connection failure
    Log    Connecting to database
    No Operation

CLI Filtering Examples

# Include all custom metadata (default)
robot tests.robot

# Only include 'Owner' and 'Priority' metadata in output
robot --custommetadata Owner --custommetadata Priority tests.robot

# Only include 'Bug Report' and 'Test Data' metadata in output
robot --custommetadata "Bug Report" --custommetadata "Test Data" tests.robot

- Add support for custom metadata tags in test cases and keywords - Include test file with examples of custom metadata usage - Enhance parsing and model handling for custom metadata - Update XML schema and output generation for metadata support

…ancements

…us/robotframework into feature/custom-metadata-enhancements

- Introduced `CustomMetadata` setting in RobotSettings. - Updated parsing and lexing logic to validate custom metadata. - Enhanced command-line options to specify allowed custom metadata names. - Modified relevant classes and methods to accommodate custom metadata.

…ting documentation

…ve initialization in transformers and model classes; update tests to accommodate changes in metadata structure.

…ing names and updating documentation for command line options

…rsing logic

… in settings

…mprove validation checks, and ensure consistent initialization across settings and builders.

- Implement tests for custom metadata in the TestSuiteBuilder, covering basic filtering, keywords, special values, case sensitivity, and integration with FileSettings. - Create tests for custom metadata modifiers, including access and modification during pre-run and execution phases, complex filtering, and error handling. - Develop tests for UserKeyword custom metadata, ensuring handling of various data types, normalization, and integration. - Validate custom metadata behavior with embedded arguments, setup/teardown, and keyword binding.

Bouska · 2025-09-26T09:40:06Z

Noordsestern · 2025-10-10T14:25:12Z

Hello @Bouska ,

thank you for the valuable feedback!

Typos have caused some discussion when developing this feature. Originally, the most popular opinion has been to let

robot  mytest.robot

fail, when it has custom metadata. You would have to provide a list of all custom metadata at launch so that typos may be prevented:

robot --custommetadata Owner --custommetadata Priority tests.robot   # would fail, if i had a typ like [Oner]

That approach has been altered for the opinion that metadata should be only metadata and not influence the default execution. Also, the use cases we know want to provide many custom metadata pairs. Adding them on CLI would not be feasible.

I like your highlight on typo management, compatibility and least surprise.

Do you have an idea how to satisfy all expectations?

fhennig42 · 2025-10-29T10:22:22Z

I really would like to see this merged as it would help us to store custom Zephyr Scale related metadata directly in the tests.

Bouska · 2025-10-29T10:34:40Z

Do you have an idea how to satisfy all expectations?

I would go for the following:

My test case
    [Metadata]    My key1    My value1
    [Metadata]    My key2    My value2

So it is a mirror on how Metadata works in Settings. If you have a typo on the key, it is a you problem but it won't impact RobotFramework; if you make a typo on Metadata RobotFramework will rightfully fail. There is only one defined new setting Metadata, there is no problem with backward or forward compatibility.

pekkaklarck · 2026-01-15T10:44:53Z

Now that RF 7.4 is out hand the holiday season is behind us, I finally have time to review this PR. This is a pretty big change so I expect there to be quite a bit of work before we can get this merged, but we also have good time for that and hopefully can get this in for RF 7.5.

Before I take a deeper look at this, I'd like to make sure @febb0e and/or others who have worked with this are available to answer questions and react to review comments. Related to that, is the PR in general ready for review or is there something you are still working with? At least the unfortunate conflicts would need to be resolved at some point. I don't know why, but GitHub doesn't show me what conflicts there are.

…us/robotframework into feature/custom-metadata-enhancements

Copilot

Pull request overview

Adds first-class custom metadata support for Robot Framework test cases and user keywords using [CustomName] settings, with optional filtering via the new --custommetadata CLI option, and surfaces the metadata in result models and log/report outputs.

Changes:

Extend lexer/parser/builder pipelines to recognize unknown [Setting] tokens in test/keyword contexts as CUSTOM_METADATA and store them on models.
Add CLI plumbing (--custommetadata) and filtering so only selected metadata keys propagate to built models and outputs.
Emit and consume custom metadata in outputs (XML <meta>, JSON, JS model, HTML log/report) and add extensive unit/acceptance tests plus schema updates.

Reviewed changes

Copilot reviewed 51 out of 51 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
utest/running/test_userkeyword_custom_metadata.py	Unit tests for `UserKeyword.custom_metadata` behavior and serialization.
utest/running/test_custom_metadata_modifiers.py	Verifies modifiers/listeners can read/write custom metadata on execution model.
utest/running/test_custom_metadata_builder.py	Tests builder-layer parsing and CLI filtering behavior for tests/keywords.
utest/running/test_builder_custom_metadata.py	Additional builder tests for edge cases (multiline, unicode, variables).
utest/result/test_keyword_custom_metadata.py	Tests result-model `Keyword` custom metadata behavior and serialization.
utest/reporting/test_jsmodelbuilders.py	Updates JS model expectations for added metadata fields and index shifts.
utest/parsing/test_custom_metadata_lexer.py	Lexer-level tests for `CUSTOM_METADATA` tokenization and context validation.
utest/model/test_testcase_custom_metadata.py	Model-level tests for `TestCase.custom_metadata` and copy/deepcopy semantics.
utest/model/test_keyword.py	Adds tests for keyword custom metadata and bind/copy behavior.
utest/model/test_custom_metadata_model.py	Cross-model tests ensuring consistent metadata behavior across model types.
utest/cli/test_custom_metadata_cli.py	Verifies `--custommetadata` is documented and settings propagation works.
src/robot/running/userkeywordrunner.py	Copies resolved keyword custom metadata into result keywords during execution.
src/robot/running/suiterunner.py	Copies resolved test custom metadata into result tests during execution.
src/robot/running/resourcemodel.py	Adds `custom_metadata` + `has_custom_metadata` for running `UserKeyword` and serializes it.
src/robot/running/model.py	Adds running `TestCase.custom_metadata` and `has_custom_metadata`.
src/robot/running/builder/transformers.py	Captures `CustomMetadata` statements into running models; applies CLI filtering.
src/robot/running/builder/settings.py	Adds `FileSettings.custom_metadata` allowlist and `should_include_custom_metadata()`.
src/robot/running/builder/parsers.py	Threads custom metadata allowlist through parsing/building pipeline.
src/robot/running/builder/builders.py	Adds `custom_metadata` parameter to suite builder construction.
src/robot/run.py	Documents `--custommetadata` and passes it into suite building.
src/robot/result/xmlelementhandlers.py	Allows `<meta>` under test/keyword and routes it to `custom_metadata` when applicable.
src/robot/result/model.py	Adds result keyword custom metadata support and includes it in `to_dict()`.
src/robot/reporting/jsmodelbuilders.py	Adds custom metadata to JS model for tests/keywords used by HTML log/report.
src/robot/parsing/parser/parser.py	Threads `allowed_custom_metadata` through `get_model/get_resource_model/get_init_model`.
src/robot/parsing/model/statements.py	Introduces `CustomMetadata` statement type derived from `DocumentationOrMetadata`.
src/robot/parsing/lexer/tokens.py	Adds `Token.CUSTOM_METADATA`.
src/robot/parsing/lexer/settings.py	Recognizes `[CustomName]` as `CUSTOM_METADATA` in test/keyword settings; errors in invalid contexts.
src/robot/parsing/lexer/lexer.py	Adds `allowed_custom_metadata` plumbing into lexer contexts.
src/robot/parsing/lexer/context.py	Passes allowlist into `FileSettings`-backed lexing contexts.
src/robot/output/xmllogger.py	Writes test/keyword custom metadata as `<meta name="...">value</meta>` elements.
src/robot/output/jsonlogger.py	Adds test `custom_metadata` emission to JSON output.
src/robot/model/testcase.py	Adds `custom_metadata` and `has_custom_metadata` to the base model `TestCase`.
src/robot/htmldata/rebot/testdata.js	Parses metadata from JS model arrays; adjusts indexes for new metadata field.
src/robot/htmldata/rebot/model.js	Stores metadata on Test/Keyword client-side models.
src/robot/htmldata/rebot/log.html	Renders custom metadata rows for tests/keywords in the log UI.
src/robot/conf/settings.py	Adds `CustomMetadata` to settings mapping and accessor.
src/robot/api/parsing.py	Exposes `CustomMetadata` in public parsing API exports.
doc/schema/running_suite.json	Adds `custom_metadata` to running-suite JSON schema.
doc/schema/running_json_schema.py	Adds `custom_metadata` fields to running schema Python models.
doc/schema/result_suite.json	Adds `custom_metadata` to result-suite JSON schema.
doc/schema/result_json_schema.py	Adds `custom_metadata` field to result schema Python models.
doc/schema/result.xsd	Allows `<meta>` under `<test>` and `<kw>` elements.
doc/schema/result.json	Adds `custom_metadata` to result JSON schema variant.
atest/testdata/parsing/custom_metadata_with_variables.robot	Acceptance data for variables/expressions inside custom metadata.
atest/testdata/parsing/custom_metadata_missing_vars.robot	Acceptance data for missing variable handling in metadata.
atest/testdata/parsing/custom_metadata_env_vars.robot	Acceptance data for environment variable handling in metadata.
atest/testdata/parsing/custom_metadata.robot	Acceptance data for broad metadata edge cases and keyword metadata.
atest/robot/parsing/custom_metadata.robot	Acceptance tests validating parsing and variable resolution behaviors.
atest/robot/output/custom_metadata_output.robot	Acceptance tests validating XML/HTML output contains custom metadata.
atest/robot/cli/runner/custom_metadata.robot	Acceptance tests for `--custommetadata` filtering behavior in `robot`.
atest/robot/cli/rebot/custom_metadata.robot	Acceptance tests for preservation/filtering in `rebot` workflows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-25T21:20:15Z

+        "lineno",
+        "_setup",
+        "_teardown",
+        "_custom_metadata",
+    )


_custom_metadata is added to __slots__, but the actual storage used by the @setter-decorated custom_metadata attribute is _setter__custom_metadata (added automatically by SetterAwareType). The extra _custom_metadata slot (and the later self._custom_metadata = None) appears unused and can be removed to avoid confusion and unnecessary per-instance storage.

Copilot · 2026-03-25T21:20:15Z

+        # Validate custom metadata name pattern - allow letters, numbers, spaces, hyphens, underscores, dots, slashes
+        import re
+
+        if not re.match(r"^[a-zA-Z0-9\s\-_./]+$", inner_content):
+            return False


import re and re.match(...) are executed every time _is_custom_metadata_setting() runs. Because this code is on the hot path when lexing large files, it would be better to move the regex compilation to module scope (e.g. a precompiled pattern) and reuse it here.

Copilot · 2026-03-25T21:20:15Z

+        # Validate custom metadata name pattern - allow letters, numbers, spaces, hyphens, underscores, dots, slashes
+        import re
+
+        if not re.match(r"^[a-zA-Z0-9\s\-_./]+$", inner_content):
+            return False


This function imports re and runs re.match(...) on every call. Since lexing runs this for many settings lines, consider using a module-level precompiled regex/pattern to avoid repeated imports/compilation work.

Copilot · 2026-03-25T21:20:15Z

+    __slots__ = (
+        "status",
+        "message",
+        "_start_time",
+        "_end_time",
+        "_elapsed_time",
+        "_custom_metadata",
+    )


robot.result.model.TestCase inherits robot.model.testcase.TestCase, which now defines a _custom_metadata slot. Re-declaring _custom_metadata in this subclass’ __slots__ will cause a TypeError at class creation due to duplicate slot names. Remove _custom_metadata from this __slots__ (and the corresponding self._custom_metadata assignment) and rely on the base custom_metadata setter storage (_setter__custom_metadata).

Copilot · 2026-03-25T21:20:16Z

        self.template = template
        self.error = error
+        # Initialize the custom metadata setter attribute to prevent AttributeError during copy
+        self._setter__custom_metadata = None


Setting self._setter__custom_metadata = None here overrides the base model.TestCase initialization and causes the custom_metadata getter to return None (not an empty Metadata). That can break code that expects test.custom_metadata to be dict-like even when empty (e.g. dict(test.custom_metadata)). Consider removing this assignment (and relying on the base initialization) or initializing it to Metadata() instead of None.

Copilot · 2026-03-25T21:20:16Z

+        # Handle line continuations by replacing newlines with spaces
+        if node.value:
+            metadata_value = " ".join(node.value.split())
+        else:
+            metadata_value = ""


metadata_value = " ".join(node.value.split()) collapses all whitespace and removes newlines, which loses formatting and makes multi-line metadata impossible to preserve. If the intent is only to handle continuations, consider keeping node.value as-is (it already contains the correctly joined lines) or doing a more targeted newline handling instead of split().

Copilot · 2026-03-25T21:20:16Z

+        # Handle line continuations by replacing newlines with spaces
+        if node.value:
+            metadata_value = " ".join(node.value.split())
+        else:
+            metadata_value = ""


This whitespace handling (" ".join(node.value.split())) collapses all whitespace and strips newlines from custom metadata values, which discards intentional formatting. Using node.value directly (or only replacing the continuation newlines) would preserve the value more faithfully.

bund5chuh · 2026-06-11T06:58:57Z

febb0e and others added 20 commits July 4, 2025 16:05

Merge branch 'robotframework:master' into feature/custom-metadata-enh…

43b9c9d

…ancements

Add support for python evaluations and multiline metadata values

1d224a9

Merge branch 'feature/custom-metadata-enhancements' of github.com:imb…

dcaa2d5

…us/robotframework into feature/custom-metadata-enhancements

added document formatting to custom metadata

2f23328

Enhance custom metadata handling by adding validation checks and upda…

e7af030

…ting documentation

Refactor custom metadata handling to prevent AttributeError and impro…

32916a4

…ve initialization in transformers and model classes; update tests to accommodate changes in metadata structure.

Improve custom metadata handling by adding checks for translated sett…

c62d29a

…ing names and updating documentation for command line options

Add support for custom metadata in JSON schemas and update related pa…

2660e50

…rsing logic

Add custom metadata attribute to TestCase and update validation regex…

e6f9605

… in settings

Refactor custom metadata handling to allow None as a default value, i…

8c9ee67

…mprove validation checks, and ensure consistent initialization across settings and builders.

Add custom metadata support to JSON schemas and update related classes

4d5e958

Added tests for custom metadata parsing, filtering, and output handling

9d1d1ab

Refactor custom metadata test data files.

b115eee

Merge branch 'master' into feature/custom-metadata-enhancements

96d4aaf

chore: linting and formatting

69f27e3

Merge branch 'master' into feature/custom-metadata-enhancements

8239066

Refactor metadata test cases to enhance formatting and clarity

7c763fd

febb0e mentioned this pull request Sep 16, 2025

[ENHANCEMENT] Add custom metadata flag to robot.toml robotcodedev/robotcode#510

Open

bhirsz mentioned this pull request Jan 15, 2026

Support custom test metadata MarketSquare/robotframework-robocop#1628

Open

febb0e added 2 commits February 16, 2026 16:08

Merge branch 'master' into feature/custom-metadata-enhancements

732fd81

Merge branch 'feature/custom-metadata-enhancements' of github.com:imb…

8c50a16

…us/robotframework into feature/custom-metadata-enhancements

Snooz82 requested a review from Copilot March 25, 2026 21:12

Copilot started reviewing on behalf of Snooz82 March 25, 2026 21:13 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

This was referenced Jun 15, 2026

Custom Metadata Feature #5685

Open

Custom Test Settings a.k.a. Test Metadata #5281

Open

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

febb0e commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Custom Metadata Feature

Overview

Implementation Details

CLI Functionality: --custommetadata Flag

Test Coverage

Notable Changes

Summary

Usage Examples

Basic Custom Metadata

Variable Resolution

Text Formatting

Inline Python Evaluation

Keyword Metadata

CLI Filtering Examples

Uh oh!

Bouska commented Sep 26, 2025

Uh oh!

Noordsestern commented Oct 10, 2025

Uh oh!

fhennig42 commented Oct 29, 2025

Uh oh!

Bouska commented Oct 29, 2025

Uh oh!

pekkaklarck commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

bund5chuh commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

febb0e commented Sep 10, 2025 •

edited

Loading

CLI Functionality: `--custommetadata` Flag