The ghc -M / -c approach is not sufficient for correct builds in all cases

[nh2]

See: http://ghc.haskell.org/trac/ghc/ticket/481

While there is a way to get around the external dependencies problem, the cases regarding TemplateHaskell described in that bug can probably not be implemented using ghc -M and ghc -c.

(Also, http://ghc.haskell.org/trac/ghc/ticket/7277 is not even handled properly by ghc --make itself.)

This is probably important for the GSoC project, because it means that any system based on ghc -M alone cannot produce always-correct builds.

The only workaround that I see for now is to call ghc --make after our parallel build, and live with the singlethreadedness that comes with that.

[23Skidoo]

Thanks, will look into this.

[23Skidoo]

I wonder if we can also use our persistent ghc server for deps analysis and extend it to support outputting information that is unavailable via -M.

[nh2]

I guess it will be quite difficult, as GHC discovers these dependencies on the fly, and according to what they say in Trac, even down to the per-function level.

[dcoutts]

I don't understand why http://ghc.haskell.org/trac/ghc/ticket/481 is a problem for ghc -M and ghc -c. The ghc -c uses the recompilation check too to decide if it has to compile or if it can be skipped. More details?

[nh2]

@dcoutts Because in that case TH forces you to do a recompilation that you would not consider necessary only at the -M dependencies and the time stamps of the files. Parmake does that. If it didn't, it would call ghc -c on every module, every time.

(If I'm wrong with this: I would love that, I really want this to work.)

[nh2]

Another point:

ghc/ghc@b994313

The commit mentions No support for ghc -M.

It also looks like there is no way no support addDependentFile in any way: If dependencies can be chosen (e.g. conditionally) at compile time of a module, the dependencies for that module cannot be known before its compilation. They cannot be made part of the ghc -M output, because for this, actual compilation must already have run through and evaluated all TH code.

It looks like only a subset of Haskell can be compiled with ghc -M these days.

[23Skidoo]

@nh2

ghc/ghc@b994313

This is the DEPENDS pragma, I believe. Another related issue is http://ghc.haskell.org/trac/ghc/ticket/3588

[nh2]

This is the DEPENDS pragma, I believe

addDependentFile and the DEPENDS pragma are different:

Being a pragma, DEPENDS carries static knowledge, could be evaluated by ghc -M and would not make a problem; addDependentFile however is Template Haskell and its output can depend on any IO knowledge, e.g. the current date or the weather; you can only get its value by actually compiling the module.

[23Skidoo]

Aha, thanks. Yeah, that's a problem. Perhaps I can make the persistent server output this info during compilation.

[nh2]

Also relevant: http://ghc.haskell.org/trac/ghc/ticket/8144 and nh2@8f2833a.

[nh2]

Shall we close this issue?

We can probably simply keep the final GHC pass because your ghc-server will make it a no-op / very fast.

[23Skidoo]

@nh2 I'm still a bit unsatisfied with that approach, so I'd like to keep this open.

[ezyang]

Why not write some programs using the GHC API to help you get out the information you need? You can define a private interface between ghc-parmake and the GHC API program which is shared across versions of GHC, so you can have different versions of the GHC API program for different versions of GHC. GHC 8.0 is even coming with frontend plugins (https://ghc.haskell.org/trac/ghc/ticket/11194) which should make it even easier to write programs which accept GHC-style arguments.

Once you figure out exactly what you need, and what a good format is, you can submit the frontends to GHC for upstreaming.

[23Skidoo]

Thanks, I'll try to find some time to experiment with frontend plugins.

The master branch actually contains a GHC API-based reimplementation of ghc-parmake (which is not up to par with the 0.1 branch yet).

[ezyang]

Yeah, the GHC API can be really helpful. For example, in https://github.com/ezyang/ghc-shake I can run the preprocessor and parseHeader, get the dependencies, and then feed it directly into the build system. For this approach to scale, I need to separate the GHC API bits from the build system bits, but it works very nicely.

[23Skidoo]

For example, in https://github.com/ezyang/ghc-shake I can run the preprocessor and parseHeader, get the dependencies, and then feed it directly into the build system.

Looks pretty cool!