What changes were proposed in this pull request?

Jira: SPARK-59516

Return zero for the infinity norm of a positive-dimensional sparse vector with no stored entries. Apply the same fix to pyspark.ml.linalg and pyspark.mllib.linalg, with tests for both APIs.

Why are the changes needed?

With NumPy before 2.3, SparseVector.norm passes its empty values array to numpy.linalg.norm, which raises on the maximum reduction. Dense vectors and sparse vectors storing an explicit zero return 0.0 for the same logical vector.

from pyspark.ml.linalg import SparseVector
SparseVector(3, [], []).norm(float("inf"))
# ValueError: zero-size array to reduction operation maximum which has no identity

NumPy 2.3 changed empty-array infinity norms to return zero (release notes). This fix provides consistent behavior for positive-dimensional sparse zero vectors on older supported NumPy releases too.

Does this PR introduce any user-facing change?

Yes. All-zero sparse vectors return 0.0 for infinity norm. Zero-dimensional vectors and invalid norm orders retain their existing behavior.

How was this patch tested?

The new regression tests failed before the fix and passed afterward on Python 3.10.11 with NumPy 2.2.6. After correcting a NumPy-version assumption in the zero-dimensional test, all 6 focused tests pass on Python 3.10.11 / NumPy 2.2.6 and Python 3.11 / NumPy 2.3.0, including parameterized subtests. Zero-dimensional sparse vectors are checked against existing dense-vector behavior: ValueError on older NumPy, zero on NumPy 2.3. Ruff lint and formatting pass for the updated tests.

Focused test commands:

PYTHONPATH=python TERM=dumb python -m unittest pyspark.ml.tests.test_linalg.SparseVectorNormTests pyspark.mllib.tests.test_linalg.SparseVectorNormTests -q
PYTHON_EXECUTABLE=python ./dev/lint-python --compile --custom-pyspark-error --ruff

Run the commands with the Python virtual environment activated; the local run shared an environment from a sibling checkout. The tests call public Python APIs directly and require no SparkSession. Python compilation, custom-error checks, Ruff lint, and Ruff formatting passed. JVM-backed suites were not run because these fixes are confined to Python-side logic.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6).