CellType reification is overly slow and memory intensive · Issue #343 · locationtech/rasterframes · GitHub
Skip to content

CellType reification is overly slow and memory intensive #343

Closed

Description

@metasim

True source of issue is in GeoTrellis, but until there's a published release we need a workaround.

Activity

  1. self-assigned this
    on Sep 12, 2019
  2. added
    bugWhen it really isn't a "feature".
    on Sep 12, 2019
  3. metasim commented on Sep 12, 2019

    @metasim
    MemberAuthor

    Backtrace from profile:

    Stack Trace	TLABs	Total TLAB Size(bytes)	Pressure(%)
    java.util.Arrays.copyOfRange(char[], int, int)	24,402	192,152,047,392	92.372
       java.lang.String.<init>(char[], int, int)	24,374	191,878,530,344	92.24
          java.lang.String.substring(int, int)	23,174	190,519,159,040	91.587
             scala.collection.immutable.StringLike$class.split(StringLike, char)	23,113	189,923,312,952	91.3
                scala.collection.immutable.StringOps.split(char)	23,113	189,923,312,952	91.3
                   geotrellis.raster.CellTypeEncoding$class.name(CellTypeEncoding)	23,058	189,299,881,744	91.001
                      geotrellis.raster.CellTypeEncoding$uint16raw$.name()	4,243	34,774,393,120	16.717
                         geotrellis.raster.FixedNoDataEncoding$class.unapplySeq(FixedNoDataEncoding, String)	4,242	34,768,360,496	16.714
                            geotrellis.raster.CellTypeEncoding$uint16raw$.unapplySeq(String)	4,242	34,768,360,496	16.714
                               geotrellis.raster.CellType$.fromName(String)	4,242	34,768,360,496	16.714
                                  org.locationtech.rasterframes.encoders.StandardSerializers$$anon$6.from(Object, CatalystSerializer$CatalystIO)	4,242	34,768,360,496	16.714
                                     org.locationtech.rasterframes.encoders.StandardSerializers$$anon$6.from(Object, CatalystSerializer$CatalystIO)	4,242	34,768,360,496	16.714
                                        org.locationtech.rasterframes.encoders.CatalystSerializer$class.fromInternalRow(CatalystSerializer, InternalRow)	4,242	34,768,360,496	16.714
                                           org.locationtech.rasterframes.encoders.StandardSerializers$$anon$6.fromInternalRow(InternalRow)	4,242	34,768,360,496	16.714
                                              org.locationtech.rasterframes.encoders.CatalystSerializer$WithFromInternalRow$.to$extension(InternalRow, CatalystSerializer)	4,242	34,768,360,496	16.714
                                                 org.locationtech.rasterframes.encoders.CatalystSerializer$CatalystIO$$anon$2.get(InternalRow, int, CatalystSerializer)	4,242	34,768,360,496	16.714
                                                    org.locationtech.rasterframes.encoders.CatalystSerializer$CatalystIO$$anon$2.get(Object, int, CatalystSerializer)	4,242	34,768,360,496	16.714
                                                       org.locationtech.rasterframes.model.TileDataContext$$anon$1.from(Object, CatalystSerializer$CatalystIO)	4,242	34,768,360,496	16.714
                                                          org.locationtech.rasterframes.model.TileDataContext$$anon$1.from(Object, CatalystSerializer$CatalystIO)	4,242	34,768,360,496	16.714
                                                             org.locationtech.rasterframes.encoders.CatalystSerializer$class.fromInternalRow(CatalystSerializer, InternalRow)	4,242	34,768,360,496	16.714
                                                                org.locationtech.rasterframes.model.TileDataContext$$anon$1.fromInternalRow(InternalRow)	4,242	34,768,360,496	16.714
                                                                   org.locationtech.rasterframes.encoders.CatalystSerializer$WithFromInternalRow$.to$extension(InternalRow, CatalystSerializer)	4,242	34,768,360,496	16.714
                                                                      org.locationtech.rasterframes.encoders.CatalystSerializer$CatalystIO$$anon$2.get(InternalRow, int, CatalystSerializer)	4,242	34,768,360,496	16.714
                                                                         org.locationtech.rasterframes.encoders.CatalystSerializer$CatalystIO$$anon$2.get(Object, int, CatalystSerializer)	4,242	34,768,360,496	16.714
                                                                            org.locationtech.rasterframes.tiles.InternalRowTile.cellContext()	4,242	34,768,360,496	16.714
                                                                               org.locationtech.rasterframes.tiles.InternalRowTile.cols()	4,228	34,658,454,496	16.661
                                                                               org.locationtech.rasterframes.tiles.InternalRowTile.rows()	13	103,990,960	0.05
                                                                               org.locationtech.rasterframes.tiles.InternalRowTile.realizedTile$lzycompute()	1	5,915,040	0.003
  4. metasim commented on Sep 12, 2019

    @metasim
    MemberAuthor

    Before

    Screen Shot 2019-09-12 at 1 29 16 PM

    After

    Screen Shot 2019-09-12 at 1 42 22 PM

  5. added a commit that references this issue on Sep 13, 2019
    1c1db3c
  6. metasim commented on Sep 13, 2019

    @metasim
    MemberAuthor

    @pomadchin FYI: Planning on creating a PR for GT.

  7. vpipkt commented on Sep 13, 2019

    @vpipkt
    Member
  8. added a commit that references this issue on Sep 13, 2019
    e71a0c4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugWhen it really isn't a "feature".performanceWe can't wait any faster

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions