Code of Conduct
Search before asking
Describe the bug
This bug is caused by #706 . After this, the buffers in WriterBuffer will be visited by multi threads.
Stacktrace:
23/04/10 07:38:03 ERROR Executor: Exception in task 3025.0 in stage 0.0 (TID 676)
java.lang.NullPointerException
at org.apache.spark.shuffle.writer.WriterBuffer.getData(WriterBuffer.java:77)
at org.apache.spark.shuffle.writer.WriteBufferManager.createShuffleBlock(WriteBufferManager.java:224)
at org.apache.spark.shuffle.writer.WriteBufferManager.clear(WriteBufferManager.java:213)
at org.apache.spark.shuffle.writer.WriteBufferManager.addRecord(WriteBufferManager.java:198)
at org.apache.spark.shuffle.writer.RssShuffleWriter.doWrite(RssShuffleWriter.java:213)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:167)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Affects Version(s)
master
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
No response
Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Describe the bug
This bug is caused by #706 . After this, the
buffersinWriterBufferwill be visited by multi threads.Stacktrace:
Affects Version(s)
master
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
No response
Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?