[Improvement]LocalStorage init use multi thread #71 by xianjingfeng · Pull Request #72 · apache/uniffle · GitHub
Skip to content

[Improvement]LocalStorage init use multi thread #71#72

Merged
roryqi merged 4 commits into
apache:masterfrom
xianjingfeng:issue_71
Jul 29, 2022
Merged

[Improvement]LocalStorage init use multi thread #71#72
roryqi merged 4 commits into
apache:masterfrom
xianjingfeng:issue_71

Conversation

@xianjingfeng

@xianjingfeng xianjingfeng commented Jul 26, 2022

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

use multi thread to clean local storage

Why are the changes needed?

If shuffle server exit abnormally, there will be many files need to be clear when shuffle server start again and this operation will cost a lot of time

Does this PR introduce any user-facing change?

No

How was this patch tested?

No need

Comment thread server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java Outdated
@colinmjj

Copy link
Copy Markdown

@roryqi

roryqi commented Jul 26, 2022

Copy link
Copy Markdown
Contributor

Could you modify the description according to our pull request template? We will use the description as commit message, the commit message will provide more information to help people understand the code.

@codecov-commenter

codecov-commenter commented Jul 26, 2022

Copy link
Copy Markdown

Codecov Report

Merging #72 (a0f8c5b) into master (aa18be0) will decrease coverage by 0.01%.
The diff coverage is 79.31%.

@@             Coverage Diff              @@
##             master      #72      +/-   ##
============================================
- Coverage     56.39%   56.38%   -0.02%     
- Complexity     1173     1179       +6     
============================================
  Files           149      149              
  Lines          7953     8013      +60     
  Branches        761      767       +6     
============================================
+ Hits           4485     4518      +33     
- Misses         3226     3250      +24     
- Partials        242      245       +3     
Impacted Files Coverage Δ
...he/uniffle/server/storage/LocalStorageManager.java 56.00% <79.31%> (+5.36%) ⬆️
...e/spark/shuffle/reader/RssShuffleDataIterator.java 89.70% <0.00%> (-3.95%) ⬇️
.../apache/uniffle/coordinator/CoordinatorServer.java 68.67% <0.00%> (-2.22%) ⬇️
...org/apache/uniffle/server/ShuffleFlushManager.java 76.70% <0.00%> (-1.71%) ⬇️
...e/uniffle/server/storage/SingleStorageManager.java 65.57% <0.00%> (-1.64%) ⬇️
.../java/org/apache/uniffle/server/ShuffleServer.java 63.41% <0.00%> (-1.30%) ⬇️
...he/uniffle/client/impl/ShuffleWriteClientImpl.java 25.95% <0.00%> (-0.10%) ⬇️
...he/uniffle/coordinator/CoordinatorGrpcService.java 2.36% <0.00%> (-0.06%) ⬇️
...java/org/apache/uniffle/common/rpc/GrpcServer.java 0.00% <0.00%> (ø)
.../apache/uniffle/common/exception/RssException.java 0.00% <0.00%> (ø)
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

@roryqi roryqi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for your contribution @xianjingfeng .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants