Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.46%. Comparing base (d44cd39) to head (e12087a).
Report is 366 commits behind head on main.

@@            Coverage Diff            @@
##               main    #1458   +/-   ##
=========================================
  Coverage     98.46%   98.46%           
- Complexity     3846     3869   +23     
=========================================
  Files           345      345           
  Lines          9553     9603   +50     
  Branches        613      616    +3     
=========================================
+ Hits           9406     9456   +50     
  Misses          142      142           
  Partials          5        5           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

I have some high level questions because this PR seems have more influence than it looks:

1. There is no issue related to this main feature. I think we should create something and put it to our 2.7 or roadmap

I will add all the required issues.

2. The new .ql-data-sources index is shifting stateless SQL plugin to stateful? If the index is red for some reason, want to confirm if SQL plugin still work as before? At lease for query on default OpenSearch data source.

It should work for queries on default Opensearch Connector even in case of red .ql-datasources index. I will go through the code again and list if there are any scenarios where it breaks.

3. Data source related code and dependencies keep growing, should we move all of them to a new datasource module. Because as I understand, core only depends on resolved DataSource or Table actually.

I was thinking on similar lines whether this code is really required in core or some other place. will discuss with you and try to move this.

** Captured important comments from other PR**

High level comments,

1. The CRUD operation on .ql-datasource could be rewrite as plan, then we can reuse existing query execution logic.

Any advantage in doing so and also I need to stash current context of thread to behave like an admin and make changes to system index. (.ql-datasources)

2. Failure cases we should consider. for example, (1) if cluster write blocked. then plugin can not create/write. (2) if the data in .ql-datasource lost.

Need to look into other plugins when cluster writes are blocked.[Mostly apis stop working until we fix it.]
we need to enable snapshots for .ql-datasources for better resillience.

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1458-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 12bc2b4b51828d2fbcdcbf7d316dd06551f5f26e
# Push it to GitHub
git push --set-upstream origin backport/backport-1458-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1458-to-2.x.