Add event fields as attributes in DDG log format#6549
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1039a6dffe
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
% string fields as strings
When a field is logged with %, tracing records the display wrapper through record_debug as well, so this branch re-parses display-rendered strings like payload=%json_text whenever the text starts with a valid JSON object or array. That makes DDG attributes change type based on their contents, unlike the JSON formatter and unlike record_str, which can break facets/queries for any % string field that sometimes contains JSON-looking text.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This also parses arbitrary Debug output that merely looks like a JSON container. Existing log fields such as index_ids = ?PrettySample::new(&index_ids, 5) in quickwit-index-management/src/index.rs:331 render as a valid JSON array while the sample is not truncated, but render as a non-JSON string once and N more is appended, so the same DDG attribute flips between array and string depending only on list length; that makes Datadog facets/queries inconsistent for those logs.
Useful? React with 👍 / 👎.
1 participant
This allows us to do some aggregations on our logs (instead of using metrics)
Example Log
This is an example log from coalescing.
This allows aggregations, e.g. on which fields we coalesced over time. Other nice infos would be e.g. published split size over time, etc.