Repository navigation
Don't require rebuilds for downstream modules if module interface is unchanged #3724
Description
Currently, any change to a module will force all downstream modules to be recompiled. For example:
-- src/Upstream.purs
module Upstream where
info = 0
-- src/Downstream.purs
module Downstream where
import Upstream (info)
result = { info }Suppose we compile this, and then update Upstream as follows:
--- a/src/Upstream.purs
+++ b/src/Upstream.purs
@@ -1,2 +1,2 @@
module Upstream where
-info = 0
+info = 1Then, the next time we compile, both Upstream and Downstream will be recompiled: Upstream because the source file changed, and Downstream because an upstream module changed. However, the changes to Upstream do not necessitate any changes to Downstream. I think skipping recompiling downstream modules has the potential to speed builds up quite dramatically in certain cases.
This will need a bit of careful thought and some changes to codegen and externs file formats, as I wrote in #3145 (comment):
Currently we implicitly depend on the current behaviour that if a module is rebuilt then all of its downstream modules must be rebuilt too in a few places, in the sense that information which is more than one hop away in the module dependency graph can leak through into the output artifacts. If we want to try to make things more granular we will need to address this first. The first example which comes to mind is re-exports, as re-exports are currently always generated as coming from the module they were defined in (as opposed to the module they were locally imported from). If module M1 exports foo, module M2 re-exports foo, and M3 imports foo from M2, then in the generated code for M3 we will have an import of M1. This means that if foo later moves to a different module (but continues to be re-exported from M2), and if M2 is determined not to need rebuilding, we could end up in a situation where the generated code for M3 still refers to M1.foo.
At the moment, when a module gets rebuilt, it has access to the externs files for every module which it depends on (directly or transitively). If/when we have addressed the above, and to ensure we've addressed this properly once and for all, I think we should try to have our build system only provide externs for the direct dependencies of any given module when that module is being built; that way, we make it easier not to end up accidentally leaking information through the module graph transitively, which then makes it easier to provide fast and safe incremental builds.
Activity
[-]Don't require rebuilds for downstream modules if interface is unchanged[/-][+]Don't require rebuilds for downstream modules if module interface is unchanged[/+]timjs commentedon Aug 28, 2019
This is wat Elm is going to do in 0.19.1. Would be awesome. Apparently the speed bumps are big!
hdgarrood commentedon Oct 14, 2019
So since this is generating a bit of interest in Slack I thought it would be good to clarify what changes we'd need to implement first:
BuildPlanso that when each module is type-checked and codegen'd, it only gets access to the externs files for the modules which it imports, as opposed to every upstream module. This will help performance (as we avoid constructing a far largerEnvironmentthan what we actually need) but mainly it would be there to verify that we aren't accidentally pulling information through from further up in the dependency graph, as that would invalidate the main assumption which makes this optimization sound.Each of these should happen in its own PR, and in this order. This is probably not an exhaustive list; in particular I suspect the last step (only providing externs for direct dependencies) will uncover similar issues with information leaking further through the module dependency graph than we want it to, which we'll also need to address.
kritzcreek commentedon Apr 28, 2020
When thinking about this I realized that our type-directed-search is going to become a lot less useful if we don't consider it when making this change.
Right now it finds so many things because the
Environmentfor the currently typechecked module is built from every transitive upstream module. Which means lots of definitions are in scope for type search to find. With only the direct dependencies in scope it would only find functions in the imported modules. No blocker just something to think about.hdgarrood commentedon May 7, 2020
That's a really good point. I don't think it should be necessary to restrict the
Environmentas I described, it would be more of an extra safeguard against things going wrong. Perhaps we could continue to supply the entireEnvironment, but then issue warnings in dev builds if typechecking pulls information out of modules which aren't directly imported, and silence those warnings during type search?ajnsit commentedon May 8, 2020
This is a great optimisation and much needed! I'm a bit concerned about this though -
Will this impact jump to definitions in purs ide?
kritzcreek commentedon May 8, 2020
It will probably need some modifications in the IDE code, but all the relevant information is there so I'm sure it's possible to reconstruct the current behaviour. I think the reexport resolving is tested reasonably well so we should notice if it changes.
drathier commentedon Oct 21, 2022
I'm working on this in #4339