{{ message }}
Repository navigation
GDAL not available #605
Description
When I try gdal_version,it seem not available.It's my code:
from osgeo import gdal
from pyrasterframes.utils import create_rf_spark_session,gdal_version
spark = create_rf_spark_session()
print("gdal:" + gdal.__version__)
print("---------------")
print("resterframes:" + gdal_version())
and the result is:
gdal:3.4.1
---------------
resterframes:not available
Activity
pomadchin commentedon Apr 3, 2023
Hey @yang162132!
GDAL Support is exposed via JNI bindings. Most likely
LD_LIBRARY_PATHis not set, this env variable should be set both on a driver node and on all executors pointing to a proper GDAL location.It can be a Spark Session property config (
spark.yarn.appMasterEnv.LD_LIBRARY_PATH,spark.executorEnv.LD_LIBRARY_PATH; see example and docs) as well as the ENV variable (Dokerfile setting example).yang162132 commentedon Apr 4, 2023
But it's seem not useful
yang162132 commentedon Apr 4, 2023
and my gdal is in LD_LIBRARY_PATH
pomadchin commentedon Apr 5, 2023
@yang162132 oh, I think that's due to GDAL versions mismatch; since GDAL 3.x it is sensitive to gdal minor versions upgrades;
I suggest just to symlink the new .so to match the old one, should be smth like:
ln -s /usr/local/lib/libgdal.28.dylib /usr/local/lib/libgdal.27.dylib(/usr/local/lib/libgdal.27.dylibis the expected filename)yang162132 commentedon Apr 6, 2023
Oh, It's uesful!
shendannan commentedon Apr 12, 2023
I have same question, with windows operating system.
My code is same as yang162132, and the result is:
since I use windows OS, how can I config Spark Session property?
pomadchin commentedon Apr 12, 2023
@shendannan that's the bindings repo we're using: https://github.com/geotrellis/gdal-warp-bindings
Its GDAL
3.0.4compatible only (mingw, windows), but you may try to symlink it as well.yang162132 commentedon Apr 19, 2023
@shendannan I suggest you install a linux virtual machine for linux , it is really inconvenient to use GDAL on windows