IFD Improvement Discussion --- STATUS: Migrating to https://wiki.lix.systems/books/lix-contributors/page/improving-ifd ## What is IFD? IFD is a little tricky to describe, but the gist of it is: IFD is a NixLang construct where Nix expression depends on some other expression that needs evaluation and building. It's described in good detail in CppNix 2.24 manual: https://nix.dev/manual/nix/2.24/language/import-from-derivation#import-from-derivation ## What's the problem with IFD? The very practical problem with IFD is performance. IFD blocks evaluation (which is single-threaded and sequential) until the nested expression is built. The problem is best illustrated with a diagram from CppNix 2.24 manual: https://nix.dev/manual/nix/2.24/language/import-from-derivation#illustration There is also a conceptual problem that IFD heavily breaks the eval/build boundary. It's not great, but opinions on how bad this is differ. Here's a summary of opinions. **raito:** this is an acceptable tradeoff, because dynamic builds are very valuable; there's not much of a difference between `builtins.runWASM` and IFD **niko:** `builtins.runWASM` doesn't break the eval/build boundary, so it's conceptually "cleaner", but I dislike the idea of running arbitrary code just to get to eval stage **Qyriad:** without IFD we have very clean "eval steps -> build steps"; feels bad losing it, but living without IFD is worse **niko:** dyndrv feels cleaner, because it decouples eval from build **niko:** IFD is platform-specific due to eval, dyndrv is not **K900:** dyndrv is also platform-specific **K900:** the fundamental difference between IFD and dyndrv is that IFD produces NixLang code and dyndrv produces .drv files **KFears:** is that true? Some of the examples on dyndrv with rec-nix use NixLang code to generate .drv files, so pretty much exactly IFD without nested eval **KFears:** the boundary is broken because we nest eval+build inside of another eval; if we move that nested eval+build outside of the outer eval (e.g. make it a derivation dependency instead), the entire problem goes away ## Solutions ### Async-y Eval **K900:** we can keep IFD thunked **raito:** Snix EvalIO can suspend execution and go evaluate further stuff while waiting on IO **K900:** there's an evil hack that will allow Hydra to use IFD today: completely ignore IFD thunks. That allows Hydra to get all attribute paths without doing IFD, and then we just do the eval again on the builder **raito:** feels like a weird eval mode, but whatever :P **KFears:** in relation to IFD, we already have "default" and "disabled" modes; adding a "hydra" mode is probably fine and not much work? **K900:** we'd like to have something that is easy to implement on CppNix side **raito:** agreed **KFears:** Snix async thunks should be pretty easy to implement in CppNix, because parallel-eval [already uses them](https://determinate.systems/posts/parallel-nix-eval/#:~:text=There%20are%20three%20new%20value%20types) **KFears:** well actually, parallel-eval is a [Dix](https://chaos.social/@fpletz/114114877909534943) feature; it will probably never land in CppNix, but independently pushing refactors that will enable async thunks should hopefully not encounter any resistance from CppNix team ### Bytecode **K900:** we can serialize thunks to bytecode **K900:** that allows us to evaluate up to IFD, do a build, resume eval **KFears:** that would be great in so many ways, but it will take years to do **Qyriad:** yeah, it will take a lot of work **niko:** it would be great to have high level bytecode-ish spec as part of Nix, but it's a ton of work ### RPC Improvements **KFears:** instead of implicitly doing nested eval+build, we should communicate to the builder to do eval+build **K900:** so the outer eval process just sits there and waits for the build to be done? **KFears:** exactly **K900:** it's the same thing as what we currently have, the only difference is what queues the builds (in this suggestion it's daemon, in current IFD it's the build) **KFears:** yes, but this solves the conceptual problem, because now there's no nested eval+build - we depend on a different build that is on the same horizontal level **KFears:** if we get RPC between the build and the builder, we can also later do things much more similar to dyndrv+rec-nix: write a package that does RPC calls to eval and build more packages **KFears:** wild idea for IFD: what if when we encounter IFD in a derivation, we patch the derivation to use IFD as a built input? **KFears:** example impl: eval drv -> encounter IFD -> abort eval, return the IFD itself and a special error code -> daemon sees special error code -> queues IFD eval -> patches initial drv to depend on IFD output -> evals patched drv **KFears:** this example impl is problematic with nested IFD or multiple IFD, but the core idea here is to treat IFD as a special, fundamentally different operation from regular eval **KFears:** this makes me think of C23's #embed: instead of forcing IFD into the otherwise clean eval/build process, let's treat it as a fundamentally different operation ### Expand Build Process **Qyriad:** I wonder if the general purpose solution is worth it **Qyriad:** we can formally expand "eval build plan -> execute build plan" pipeline into "eval fetch plan -> execute fetch plan -> evaluate build plan -> execute build plan", that would solve 90% of IFD usecases **Qyriad:** I'm not convinced we should actually do that, but it's worth considering **K900:** I think we should roll a proper, Snix-like solution **KFears:** I also think Snix-like solution is good, though I also support RPC **K900:** IFD is a good general solution, because it's "just generating code", which allos you to express basically anything **KFears:** (thinking out loud) yay metaprogramming!