{{ message }}
Repository navigation
Compatibility with Spark 3.2.1 #605
Description
A change in Spark 3.2.1 broke binary compatibility with 3.2.0 by adding a parameter with default value to class Invoke. This breaks the call in Frameless RecordEncoder:
[info] java.lang.NoSuchMethodError: 'void org.apache.spark.sql.catalyst.expressions.objects.Invoke.<init>(org.apache.spark.sql.catalyst.expressions.Expression, java.lang.String, org.apache.spark.sql.types.DataType, scala.collection.immutable.Seq, scala.collection.immutable.Seq, boolean, boolean)'
[info] at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:154)
I believe that cutting a new release could break compatibility with 3.2.0. I filed https://issues.apache.org/jira/browse/SPARK-38077.
Activity
pomadchin commentedon Jan 31, 2022
@thesamet thx for the report, yes we can definitely rollback the dep. back if that’s the case.
thesamet commentedon Jan 31, 2022
I think it'd be impossible to have a single frameless build that works with both Spark 3.2.0 and Spark 3.2.1 without resorting to reflection or some other means to figure out which instance of
Invokeis available at runtime... Moving in either direction will break it for some people :/pomadchin commentedon Jan 31, 2022
One thing that I can suggest is to wait for a little, mb this regression will be fixed in the future Spark releases, at this point 0.11.1 and snapshots are pretty much the same, with the Spark version difference only :/
chris-twiner commentedon Apr 7, 2022
Is it not possible to just put "-3.2.1" into the artefact name for a 'special' 3.2.1 build? Source code wise it's fine*, we'd just need to reflect the special status of the dataset for 3.2.1 users e.g. frameless-dataset-3.2.1_2.12 etc.
This would force users to specify which version to use only downside is dependant libraries would have to either use provided scope (and force their users to choose) or follow suite with embedding the version in the name.
If people are ok with the concept I'm happy to offer a pr.
* I've tested this by simply rebuilding and running against a large test suite with injectors, loads of type derivation and some fun array nesting etc. I also have to take this approach to provide Databricks runtime specific Expressions / HigherOrderFunctions etc.
pomadchin commentedon Apr 7, 2022
@chris-twiner the build at this point is overly complicated: we publish
3.2 / 3.1 / 3.0, and the plan is to keep at most 3 latest (minor) versions released;I'd like not to publish
3.2.1specific artifacts, does it mean to make a release for all minor versions, how many than? As a workaround, we could publish the latest package with the3.2.1Spark version for folks who are interested.Anyway, the right solution to this issue is to make apache/spark#35378 to the next Spark release if possible.
chris-twiner commentedon Apr 7, 2022
I'd definitely be up for a clean 3.2.1 official version, I've already got lots of workarounds in play, using official frameless again would be one less
pomadchin commentedon Apr 7, 2022
@chris-twiner cool, if it helps you, I can cut
0.12.0a release rightaway (also snapshots should work for you at the moment), if you need it; I think we had a fair amount of changes and + you asked for a release. I hope no one has anything against it; cc @imarios @cchantep do you have anything else you'd like to get into 0.12.0?cchantep commentedon Apr 7, 2022
Fine for me
pomadchin commentedon Apr 9, 2022
@chris-twiner https://github.com/typelevel/frameless/releases/tag/v0.12.0 🚀
chris-twiner commentedon Apr 11, 2022
Many thanks!!, tested on oss and Databricks 10.4 dbr - works wonderfully and lets me scrap my frameless classpath workarounds for these versions completely!
pomadchin commentedon May 15, 2022
TLDR; This is internal API, won't fix (c)
See https://issues.apache.org/jira/browse/SPARK-38077 and apache/spark#35378 to follow the discussion chain.