feat(core): add task tracking to Testability by heathkit · Pull Request #16863 · angular/angular · GitHub
Skip to content

feat(core): add task tracking to Testability - #16863

Closed
heathkit wants to merge 1 commit into
angular:masterfrom
heathkit:task-tracking
Closed

heathkit wants to merge 1 commit into
angular:masterfrom
heathkit:task-tracking

Conversation

@heathkit

Copy link
Copy Markdown
Contributor

Allow passing an optional timeout to Testability's whenStable(). If
specified, if Angular is not stable before the timeout is hit, the
done callback will be invoked with a list of pending macrotasks.

Also, allows an optional update callback, which will be invoked whenever
the set of pending macrotasks changes. If this callback returns true,
the timeout will be cancelled and the done callback will not be invoked.

Implements #15917

Please check if the PR fulfills these requirements

What kind of change does this PR introduce? (check one with "x")

[ ] Bugfix
[X] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior? (You can also link to an open issue here)

Testability.whenStable() will wait an indefinite amount of time for Angular to become stable. Protractor relies on a script timeout to know when waiting has timed out (typically longer than 10 seconds or so).

What is the new behavior?

Users can now pass a timeout to whenStable(). If there are still macrotasks pending when the timeout occurs, the callback will receive a list of pending tasks (provided by the TaskTracking zone spec).

whenStable() also now accepts an optional callback to invoke whenever the state of pending macrotasks changes. If this callback returns true, whenStable() will cancel the pending done callback, allowing users more control over which macrotasks they wait for.

Does this PR introduce a breaking change? (check one with "x")

[ ] Yes
[X] No

@heathkit heathkit changed the title feat(testability): Improvements to the Testability API. feat(core): Improvements to the Testability API. May 18, 2017
@juliemr
juliemr requested review from juliemr and vikerman May 18, 2017 18:34
@juliemr juliemr added area: testing Issues related to Angular testing features, such as TestBed feature Label used to distinguish feature request from other issues labels May 18, 2017
@heathkit
heathkit force-pushed the task-tracking branch 2 times, most recently from 4e0a2c9 to be84dc9 Compare May 18, 2017 20:11
@heathkit heathkit changed the title feat(core): Improvements to the Testability API. feat(core): add task tracking to Testability May 18, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for cleaning up the old AsyncTestCompleter business.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, let's make these twoasync tests to increase confidence

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expect(tasks.length).toEqual(1)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe instead of execute2 make a new spy and call it something like updateCallback?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be a number|Timer then?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is Timer is only defined in the NodeJS typings, which is only brought in for the platform-server tsconfig. I can't figure out a type that builds for both browser and server platforms, since in the browser setTimeout returns a number.

Of course, with Zone.js, setTimeout returns the same thing on both platforms - a Zone task. So the typings are wrong, anyway ¯_(ツ)_/¯

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is casting here necessary? Shouldn't TS know it's an array of WaitCallbacks?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I accidentally dropped the not null assertion !, so I needed a cast. I put it back, but I do find the! a little easy to miss.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and no further updates will be issued.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@heathkit
heathkit force-pushed the task-tracking branch 2 times, most recently from a5333f8 to 90629ba Compare May 22, 2017 08:43
@juliemr

juliemr commented May 22, 2017

Copy link
Copy Markdown
Member

LGTM, cc @vikerman for additional approval.

@heathkit
heathkit force-pushed the task-tracking branch 3 times, most recently from fee9b00 to cf3fc55 Compare May 22, 2017 23:27
@heathkit

heathkit commented May 22, 2017

Copy link
Copy Markdown
Contributor Author

PTAL - I had to make some slight changes to get CI to pass - updated the public API. Also changed it so you only get an error when the task tracking zone isn't available if you pass a timeout or update callback to whenStable()

@vikerman

Copy link
Copy Markdown
Contributor

LGTM - can we add an end to end test with Protractor for this?

@heathkit

heathkit commented Jun 2, 2017

Copy link
Copy Markdown
Contributor Author

Added an integration test for whenStable() timeout.

@juliemr

juliemr commented Jun 2, 2017

Copy link
Copy Markdown
Member

I believe Vikram said that Igor should take a look at this - ping @IgorMinar

@juliemr
juliemr requested a review from IgorMinar June 2, 2017 19:09
Comment thread tools/public_api_guard/core/core.d.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UpdateCallback does not seem to be exported?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UpdateCallback and DoneCallback should show up in the API guard. The fact that they don't implies that they are not visible to the developer (ie they are not exported at top level).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@IgorMinar IgorMinar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we expect developers to receive this update?

I suspect that once they update to karma that depends on this API they will get the missing zone spec error, not understanding what that means or what to do about it.

If they read up on the change in some changelog they'll figure out that they need to add the new zone spec.

If they do add these zone spec, would they know how to remove them in production?

And if they are not removed from production builds, do we understand the performance impact of having task tracking enabled all the time?

It seems to me that the PR as is will create a lot of pain for developers not suspecting or understanding this change. Am I missing something?

Comment thread tools/cjs-jasmine/index-tools.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't this mean that task tracking must be enabled in all apps for testability to work?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the timeout and update callback are optional. If those aren't passed, testability should continue to work as before.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it what? '' :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, fixed :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"for application to be stable.."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I were a regular developer, I'd have no idea what to do about this error. :-(

@heathkit heathkit Jun 16, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to make it more clear. Also, this API isn't really intended for end users as much as it is for tool developers, like Protractor. On the Protractor side, we'll try to turn this into a more actionable, understandable error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean that the old way of using this api still works? what happens if you have task tracking enabled and you use older protractor that doesn't have the task tracking support yet? Will it just work?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the old way of using the API should still work without errors. If task tracking is enabled but Protractor isn't passing an update callback, it will just work with the current behavior.

Comment thread tools/public_api_guard/core/core.d.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to export these types? Can't we just inline them?

It's weird that someone can do import {DoneCallback} from '@angular/core'; - not only that the name is too ambiguous, I'm not sure if we want to be on the hook to support such api if someone decides to create their implementation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I kind of like having the types for the callbacks available so I know what parameters they need to accept, but it's probably not a good thing to have this in the public API. The Protractor side of this is written in ES5, anyway, so it's not like I'd actually be able to use these types yet.

Comment thread tools/public_api_guard/core/core.d.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above. we should expose as little as possible. Do we need to provide all of this type info?

@heathkit heathkit Jun 16, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, I'll just inline this or make it Function.

Comment thread tools/public_api_guard/core/core.d.ts Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inline (as above)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

@heathkit

Copy link
Copy Markdown
Contributor Author

How do we expect developers to receive this update?

I suspect that once they update to karma that depends on this API they will get the missing zone spec error, not understanding what that means or what to do about it.

That should not happen - at least it's not the intention. The error for the missing zone spec is only thrown when someone tries to call whenStable() with an update callback. Protractor's current behavior should not be affected.

The current behavior for Protractor is to pass a done callback and wait forever. The timeout is actually handled by WebDriver's script execution timeout. Moving the timeout and task tracking logic to Testability will allow us to greatly simplify Protractor's browser.waitForAngular

If they read up on the change in some changelog they'll figure out that they need to add the new zone spec.

If they do add these zone spec, would they know how to remove them in production?

That's a good point, which Misko brought up when we talked this idea over with him. The goal is to fix that with #17390, which adds a way to explicitly set which zone specs get loaded by setting flags in session state. The idea is that if NG_DEFER_BOOTSTRAP is in window.name (which is how Protractor signals to AngularJS that it's under test and needs to load mock modules and such), then the task tracking zone is enabled. So users that want to use Protractor's task tracking would need to always include the zone spec (similar to how they include long-stack-trace), but it will only be forked during a Protractor test.

It seems to me that the PR as is will create a lot of pain for developers not suspecting or understanding this change. Am I missing something?

The goal of this PR is to preserve current behavior, but allow us to add task tracking in future Protractor releases. This will provide better debugging information when Protractor times out waiting for stability, and will also enable us to build a more high-level API for waiting on macrotasks. The change should be transparent. If Protractor attempts to use task tracking but the zone isn't present, it can log a warning (or error or whatever we decide is appropriate).

Once #17390 is in we'll be able to continue this work on the Protractor side. We're also planning on doing the same changes to whenStable for AngularJS.

@IgorMinar IgorMinar added action: merge The PR is ready for merge by the caretaker merge: caretaker note Alert the caretaker performing the merge to check the PR for an out of normal action needed or note and removed action: review The PR is still awaiting reviews from at least one requested reviewer labels Mar 14, 2018
@ngbot

ngbot Bot commented Mar 14, 2018

Copy link
Copy Markdown

I see that you just added the PR action: merge label, but the following checks are still failing:
    failure missing required label: "PR target: *"
    failure status "ci/circleci: build" is failing
    failure status "continuous-integration/travis-ci/pr" is failing
    pending status "google3" is pending
    pending 1 pending code review

If you want your PR to be merged, it has to pass all the CI checks.

If you can't get the PR to a green state due to flakes or broken master, please try rebasing to master and/or restarting the CI job. If that fails and you believe that the issue is not due to your change, please contact the caretaker and ask for help.

@IgorMinar IgorMinar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discussed this with @heathkit and we resolved all the remaining issues. lgtm.

@IgorMinar

Copy link
Copy Markdown
Contributor

caretaker note: can you please run a g3 presubmit on this one before merging? thanks!

@kara

kara commented Mar 14, 2018

Copy link
Copy Markdown
Contributor

presubmit

@IgorMinar

Copy link
Copy Markdown
Contributor

great. the presubmit passed. @kara can you please merge this? thanks

@IgorMinar IgorMinar added the target: major This PR is targeted for the next major release label Mar 14, 2018
@IgorMinar

Copy link
Copy Markdown
Contributor

@heathkit can you please check why the circleci tests are failing? they are the same unit tests that execute on travis, but they run under bazel which also need the new zone spec configuration. you should be able to reproduce this failure locally.

@IgorMinar IgorMinar added action: cleanup The PR is in need of cleanup, either due to needing a rebase or in response to comments from reviews and removed action: merge The PR is ready for merge by the caretaker merge: caretaker note Alert the caretaker performing the merge to check the PR for an out of normal action needed or note labels Mar 14, 2018
Allow passing an optional timeout to Testability's whenStable(). If
specified, if Angular is not stable before the timeout is hit, the
done callback will be invoked with a list of pending macrotasks.

Also, allows an optional update callback, which will be invoked whenever
the set of pending macrotasks changes. If this callback returns true,
the timeout will be cancelled and the done callback will not be invoked.

If the optional parameters are not passed, whenStable() will work
as it did before, whether or not the task tracking zone spec is
available.

This change also migrates the Testability unit tests off the deprecated
AsyncTestCompleter.
@heathkit

Copy link
Copy Markdown
Contributor Author

@IgorMinar Thanks, that was it. Updated BUILD.bazel and init_node_spec.ts, circle is passing now.

@IgorMinar

Copy link
Copy Markdown
Contributor

awesome! thanks @heathkit

@IgorMinar IgorMinar added action: merge The PR is ready for merge by the caretaker merge: caretaker note Alert the caretaker performing the merge to check the PR for an out of normal action needed or note and removed action: cleanup The PR is in need of cleanup, either due to needing a rebase or in response to comments from reviews labels Mar 14, 2018
@IgorMinar

Copy link
Copy Markdown
Contributor

@kara the previous presubmit is still valid since the fixes that Michael made do not affect g3.

@angular-automatic-lock-bot

Copy link
Copy Markdown

@angular-automatic-lock-bot angular-automatic-lock-bot Bot locked and limited conversation to collaborators Sep 13, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

action: merge The PR is ready for merge by the caretaker area: testing Issues related to Angular testing features, such as TestBed cla: yes feature Label used to distinguish feature request from other issues hotlist: google merge: caretaker note Alert the caretaker performing the merge to check the PR for an out of normal action needed or note target: major This PR is targeted for the next major release

Projects

None yet

Development

Successfully merging this pull request may close these issues.