Create datasource API by vamsimanohar · Pull Request #1427 · opensearch-project/sql · GitHub
Skip to content

Create datasource API#1427

Closed
vamsimanohar wants to merge 1 commit into
opensearch-project:2.xfrom
vamsimanohar:create-ds
Closed

Create datasource API#1427
vamsimanohar wants to merge 1 commit into
opensearch-project:2.xfrom
vamsimanohar:create-ds

Conversation

@vamsimanohar

@vamsimanohar vamsimanohar commented Mar 11, 2023

Copy link
Copy Markdown
Member

Description

This PR covers Create datasource REST API.

Create datasource [REST API]

Screen Shot 2023-03-06 at 6 09 58 PM

Get datasource along with storage engine and other artifacts [Internal use case only]

Screen Shot 2023-03-06 at 6 13 39 PM

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov-commenter

codecov-commenter commented Mar 11, 2023

Copy link
Copy Markdown

@vamsimanohar vamsimanohar force-pushed the create-ds branch 2 times, most recently from 7e1ff62 to f893785 Compare March 14, 2023 18:25
@vamsimanohar vamsimanohar marked this pull request as ready for review March 14, 2023 18:46
@vamsimanohar vamsimanohar requested a review from a team as a code owner March 14, 2023 18:46
@vamsimanohar vamsimanohar force-pushed the create-ds branch 4 times, most recently from 9ba491c to 6d18aeb Compare March 16, 2023 07:58
ps48
ps48 previously approved these changes Mar 16, 2023
@dai-chen

Copy link
Copy Markdown
Collaborator

Any special reason this has to merge to 2.x first?

@dai-chen dai-chen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some high level questions because this PR seems have more influence than it looks:

  1. There is no issue related to this main feature. I think we should create something and put it to our 2.7 or roadmap
  2. The new .ql-data-sources index is shifting stateless SQL plugin to stateful? If the index is red for some reason, want to confirm if SQL plugin still work as before? At lease for query on default OpenSearch data source.
  3. Data source related code and dependencies keep growing, should we move all of them to a new datasource module. Because as I understand, core only depends on resolved DataSource or Table actually.

Comment thread .gitignore
Comment thread core/src/main/java/org/opensearch/sql/datasource/DataSourceUserRoleHelper.java Outdated
Comment thread core/src/main/java/org/opensearch/sql/datasource/model/DataSourceType.java Outdated
@vamsimanohar

Copy link
Copy Markdown
Member Author

Any special reason this has to merge to 2.x first?

I need this for pen testing on 2.7 version. I will resolve conflicts with main branch and will soon raise a PR. Following main -> 2.x -> 1.x is becoming little difficult as we mostly test and verify against the next minor version in current running major.

@vamsimanohar

vamsimanohar commented Mar 20, 2023

Copy link
Copy Markdown
Member Author

I have some high level questions because this PR seems have more influence than it looks:

  1. There is no issue related to this main feature. I think we should create something and put it to our 2.7 or roadmap

I will add all the required issues.

  1. The new .ql-data-sources index is shifting stateless SQL plugin to stateful? If the index is red for some reason, want to confirm if SQL plugin still work as before? At lease for query on default OpenSearch data source.

It should work for queries on default Opensearch Connector even in case of red .ql-datasources index. I will go through the code again and list if there are any scenarios where it breaks.

  1. Data source related code and dependencies keep growing, should we move all of them to a new datasource module. Because as I understand, core only depends on resolved DataSource or Table actually.

I was thinking on similar lines whether this code is really required in core or some other place. will discuss with you and try to move this.

@vamsimanohar vamsimanohar linked an issue Mar 20, 2023 that may be closed by this pull request
@vamsimanohar vamsimanohar force-pushed the create-ds branch 2 times, most recently from a08550e to 2747293 Compare March 20, 2023 23:01
Signed-off-by: vamsi-amazon <reddyvam@amazon.com>
@penghuo

penghuo commented Mar 21, 2023

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Create Datasource API.

5 participants