Add more location information to Test messages · Issue #674 · build-server-protocol/build-server-protocol · GitHub
Skip to content

Add more location information to Test messages #674

Open
Open
Add more location information to Test messages#674

Description

@Arthurm1
Arthurm1
opened on May 31, 2024

Describe the use-case for this feature

Currently BuildTargetTest leads to feedback messages of TaskStartParams containing a TestStart and TaskFinishParams containing a TestFinish.

Both TestStart and TestFinish contain Location which points to a position in a file.

For JUnit tests (JVM languages) there seems to be no feedback from the library about file location of the test. Instead there is suite, class, method. But there is no way of communicating this info to the BSP client.

There is also no option in TestFinish to passback a stacktrace which is helpful to diagnose test failures.

What do you propose

I'd like to propose a JVMTestStartType that can be added to TestStart#data and JVMTestFinishType to be added to TestFinish#data. TestStart currently doesn't have a dataKind or data member and would need to be added.

Both JVMTestStartType and JVMTestFinishType would contain suite, class, method and JVMTestFinishType would also contain stacktrace.

I see Bazel already has the JUnitStyleTestCaseData bespoke type.

I've called the types JVMTestXXXType but I'm not sure whether this as applicable to all test frameworks or all JVM test frameworks or a subset of JVM test frameworks.

Here's some JVM examples...
Junit seems to pass back method, package, class, file (with optional position).
TestNG seems to pass back class, method, method parameters.
Gradle passes back suite, class, method for JUnit and TestNG
SBT test Interface appears to pass back the FQN of suite/method

The idea of parameterized tests is a bit hazy. The file location, suite, class, method should be the same regardless of the parameters used to run the test but I guess it's possible there are overloaded methods of the same name with different parameter types in the file which would mean the BSP client would not know which method was run from the above new data structures. I feel it's unlikely but 🤷.

I've checked to make sure there isn't already a way to support this in the current protocol

  • I've checked the existing protocol and there isn't a way to do this currently

Maintainer approval (This is for the maintainers)

  • Jetbrains
  • Metals
  • Scala Center

Activity

  1. jdneo commented on Jun 2, 2024

    @jdneo
    jdneo
    on Jun 2, 2024
    Contributor

    Vote for this. class/method/suite name can be optional fields.

    The idea of parameterized tests is a bit hazy. The file location, suite, class, method should be the same regardless of the parameters used to run the test

    At least for JUnit 5 running with Gradle, each execution of the parameterized tests will be appended by a sequence number. For example, running:

    @ParameterizedTest(name = "{0}")
@CsvSource({
   "0",
   "1",
   "2"
})
void isParameterized(int input) {
  assertTrue(true);
}

    will get

    isParameterized(int)[1]
isParameterized(int)[2]
isParameterized(int)[3]

    as the method name.

  2. Arthurm1 commented on Mar 4, 2025

    @Arthurm1
    Arthurm1
    on Mar 4, 2025
    Author

    Just to document what we ended up with in the Gradle BSP, this custom solution seemed to cope with all the cases we came up with...

    class TestStartEx extends TestStart {    // extends from ch.epfl.scala.bsp4j.TestStart
  TestName testName    // custom test definition
}

class TestFinishEx extends TestFinish {  // extends from ch.epfl.scala.bsp4j.TestFinish
  TestName testName    // custom test definition
  String   stackTrace  // stacktrace for errors isn't in the current spec
}

class TestName {
  String displayName
  String suiteName
  String className
  String methodName
  TestName parent
}

    The recursive nature of TestName copes with JUnit's @Nested that allows inner classes to defined tests. It also copes with DynamicContainer that allows nesting of tests as deep as you like.

    Another option was putting TestName and stackTrace in the data: TestFinishData field of TestFinish but TestStart lacks any data: ??? field.

    The is purely for handling buildTarget/test. Ideally test discovery buildTarget/scalaTestClasses and buildTarget/jvmTestEnvironment would return tests at a level other than Class, e.g. Method, to allow the BSP client to discover and run a single method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions