JavaPackage'object is not callable · Issue #581 · locationtech/rasterframes · GitHub
Skip to content

JavaPackage'object is not callable #581

Open
Open
JavaPackage'object is not callable#581
Labels
questionFurther information is requested

Description

@radi2015
radi2015
opened on Apr 2, 2022

in Getting Started
python3 -m pip install pyrasterframes
import pyrasterframes
from pyrasterframes.utils import create_rf_spark_session
spark = create_rf_spark_session()
Error setting up SparkSession; cannot find the pyrasterframes assembly jar
'JavaPackage' object is not callable

I haven't installed the pirasterframes for a day

Activity

  1. pomadchin commented on Apr 2, 2022

    @pomadchin
    pomadchin
    on Apr 2, 2022
    Member

    Hey @radi2015, I checked the same steps in a fresh conda environment:

    name: rf
channels:
 - conda-forge
dependencies:
 - python=3.9.10
 - pyrasterframes

    And the output that I see:

    $ python3         
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:28:27) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrasterframes
>>> from pyrasterframes.utils import create_rf_spark_session
>>> spark = create_rf_spark_session()
2022-04-02 00:07:56,589 WARN util.Utils: Your hostname, macbook.local resolves to a loopback address: 127.0.0.1; using 10.0.0.224 instead (on interface en0)
2022-04-02 00:07:56,590 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/.../miniconda/envs/geoparquet/lib/python3.9/site-packages/pyspark/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-04-02 00:07:57,056 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
>>>

    I will make an assumption that or you don't have a properly installed Java or you have some other specifics of the infra you're using (i.e. Windows) and not sharing in the issue description, please try to follow the Getting Started guide one more time; to write a more comprehensive response I need to know more, since as I said, the setup in a fresh conda env just works for me; in the worst case you may try it in a prepared Docker env.

    We're open to PRs, and if you'd like to improve docs, you're more than welcome.

    I'd also like to remind you that is an opensource project, and people on this project often volunteering their time to maintain it. If you have any questions / issues feel free to reach us here or in the gitter channel for the interactive chat; but all questions / comments related to other projects please redirect to Stackoverflow / other projects repositories instead.

  2. pomadchin
    added
    questionFurther information is requested
    on Apr 2, 2022

  3. radi2015 commented on Apr 2, 2022

    @radi2015
    radi2015
    on Apr 2, 2022
    Author

    Sorry, there is something wrong with my words. This is a great project. I really want to learn to use it. Maybe there is something wrong with my operation.

    In the Python interpreter above you, I saw how to start, and I succeeded
    I'll tell you what I'm wrong

    pyspark
Python 3.7.11 (default, Jul 27 2021, 07:03:16) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
22/04/02 13:58:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
import pyrasterframesWelcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.3
      /_/

Using Python version 3.7.11 (default, Jul 27 2021 07:03:16)
Spark context Web UI available at http://10.30.108.108:4040
Spark context available as 'sc' (master = local[*], app id = local-1648879113934).
SparkSession available as 'spark'.
>>> import pyrasterframes
>>> from pyrasterframes.utils import create_rf_spark_session
>>> spark = create_rf_spark_session()
Error setting up SparkSession; cannot find the pyrasterframes assembly jar
 'JavaPackage' object is not callable
>>>

    I shouldn't start pyspark first, I should start Python first

    thank you very much

  4. pomadchin commented on Apr 2, 2022

    @pomadchin
    pomadchin
    on Apr 2, 2022
    Member

    @radi2015 great that it works now! how have you tried it before and what is the difference?

    In this case Spark session creation makes your Python interpreter process a driver;

    if you want to try the same in the spark-shell, I think you’d need to configure the class path a bit, so it knows where to find Python libs as well as Java jars.

  5. radi2015 commented on Apr 2, 2022

    @radi2015
    radi2015
    on Apr 2, 2022
    Author

    At the beginning, I started the pyspark command first, and then the sparksession was created. Then I executed the create_rf_spark_session() function and reported an error. When I saw the error, I thought it was the jar package of rasterframe that was not put into the Lib directory of spark. If I put the jar package into the Lib directory, it also report an error. Now, it seems that the reason is that if pyspark cmd create a sparksession first,then executing the create_rf_spark_session() function create spark session again will be error。

  6. pomadchin commented on Apr 2, 2022

    @pomadchin
    pomadchin
    on Apr 2, 2022
    Member

    @radi2015 oh it makes sense, I think you'd need to try smth like spark.withRasterFrames() while being in the spark-shell to make it work; indeed it is impossible to create several sessions within the same jvm

  7. mathiaskvist commented on Apr 6, 2022

    @mathiaskvist
    mathiaskvist
    on Apr 6, 2022

    I have the same issue when trying to install on Databricks. When installing pyrasterframes==0.10.1 on the cluster it is successful, but when I go to the notebook I get the error:

    >>> import pyrasterframes
>>> spark = spark.withRasterFrames()
TypeError: 'JavaPackage' object is not callable

    Also, it is unclear to me based on the documentation, whether it is necessary to install these java packages in order to use pyrasterframes:

    org.locationtech.rasterframes:rasterframes_2.12:0.10.1
org.locationtech.rasterframes:pyrasterframes_2.12:0.10.1
org.locationtech.rasterframes:rasterframes-datasource_2.12:0.10.1

    The installation of the last two packages fail with the error:

    Library resolution failed. Cause: java.lang.RuntimeException: unresolved dependency: com.github.everit-org.json-schema:org.everit.json.schema:1.12.2: not found

  8. pomadchin commented on Apr 29, 2022

    @pomadchin
    pomadchin
    on Apr 29, 2022
    Member

    hey @mathiaskvist that's due to stac4s in deps and due to the json schema validation library we use, add the following https://jitpack.io URI as an extra resolver.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions