-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
implement flattenTreeSystem #13
Conversation
It turns out that we also need to filter by the supported system to get a useful output out of flattenTree. In order to keep back-compat, a new implementation has been created that also looks at the meta.hydraPlatforms or meta.platforms if the former is undefined, and only keep the derivations that have the provided system in it.
Should we filter for broken packages as well? |
What is the use-case? Maybe the best is to use |
I've found a strange case where the behavior of this seems to differ between a flake and flake-compat. I was trying to update the lock file, and put the update in a separate branch to troubleshoot ci failures before pushing to the main branch. Hercules ci doesn't understand flakes yet so ci is managed by flake compat. After the build for kakoune failed in ci, I went to build it locally to see if I could get some more details. That's when I realized that the flake doesn't even have kakoune as an output package (packages are filtered using flattenTreeSystem). I then pulled the flake into a repl to confirm, and also pulled it via flake compat to compare, and for whatever reason kakoune exists in the packages output when pulled using flake-compat, but not in the regular flake. Not 100% sure if |
Do you mind trying #23? Hitting subtle issues like that probably means that it's not worth it. |
That did resolve it. Can't say I'm less confused though 😆 |
Same :-D It would require to dig into the problem but I am satisfied with the workaround. |
setup configurations API NixOS file, configurations/<name>.nix, will be easily installable via `nixos-rebuild --flake ".#<name>"`. succinct `description` and README.md utils: create utility functions to ease repetition `reqImport` in particular, is useful for easily importing an entire directory of nix files into an attribute set. use my fork of `nixpkgs` until flakes stabilize utils: import from configurations home-manager: init flake: nix flake is broken, use pkgs.nixFlakes rename configurations to hosts pkgs: import all packages into core overlay flake: move overlays to flake outputs flake: move modules to outputs flake: provide overlay and overlays niximg: init iso configuration Easily build the iso using `nix build $ISO` thanks to direnv. flake: add packages to outputs shell.nix: init minimal shell.nix: basic command to build nixos `nixos-rebuild` has unstable flake support. By adding a simple rebuild command to the shell, we don't need to reply on it to switch configurations anymore. At least until it stablizes. shell.nix: add git-crypt dependency Initialize template branch Remove all idiosyncratic stuff and only leave profiles, modules and pkg definitions, which are general and applicable for any configuration. secrets: create on entering a nix-shell users: create on entering nix-shell flake.nix: name flake inputs The flake inputs can now be arbitrarily referenced from hosts/default.nix as they are all passed into it. Any input not declared at the top of hosts/default.nix can still be referenced as args.<input>. export profiles via flake outputs Export those listed in `profiles/default.nix` to be included in the flake output `nixosModules` for easy use in other flakes. graphical#qutebrowser: init The current qutebrowser configuration is sufficiently complex that it benefits from being factored out into a separate subprofile. An additional benefit is that this subprofile is now exported via the `nixosModules` flake output. profiles: export additional subprofiles change formatter to nixfmt modules: convert to list to match nixpkg sematics profiles: convert to list same as modules utils: now lives in lib.utils globally Revert "utils: now lives in lib.utils globally" This reverts commit 5a83edbc7cd7f3f690ebf86f2247a354e05b7afb. Causes infinite recursion, revert for now while exploring possible solutions. overlays: init overlays directory Overlays now live globally in the overlays directory. They will be pulled into the flake and the rest of your configuration automatically from there. flake: changed by mistake; revert virt: init profile shell.nix: run nix build as root Variables mist be preserved so use `sudo -E`. Fixes #4. pkgs#pure: init at 0e87bd8 virt: add to exported profiles security#mitigations: init module Resolves #6 by breaking out the disabling of mitigations into it's own module. Now users must explicitly accept the risk of disabling Spectre and Meltdown mitigations with `security.mitigations.acceptRisk` in addition to actually disabling them with `security.mitigations.disable`. flake: initial NUR support Revert "Merge branch 'nur' into template" This reverts commit 9ec7ea50ef3ba50e97128b1102b3fb381df7cbf0, reversing changes made to 8c5c10fdc44eeef8f24aa2dbe408beb2debd0d0b. flake.nix: consistent `default.nix` semantics DOC.md: split off from README.md Keep the README.md lean and use DOC.md for more detailed explanations. While revising user documentation, a default `nixos` user profile was created, which serves as a goood base example. flake.nix: define `pkgs` in flake Set the global package set via `hosts/default.nix` passed in from its definition in `flake.nix`. remove unnecessary recursive attribute sets flake: update to upstream Fix: nix run -> nix shell graphical#xmonad: init profile flake: update lockfile security#mitigations: avoid http call laptop: init profile flake: remove epoch It was renamed to edition: NixOS/rfcs@cc5d0a2 leading to the following error: error: flake 'git+file:///home/jtojnar/Projects/nixflk?ref=template&rev=846b1eb023fca51603d460532516c2f6bce1da91' has an unsupported attribute 'epoch', at /nix/store/yndrc91vlh5vm0k4nngx303q1cjm77z9-source/flake.nix:4:3 but that itself has been deprecated: warning: flake 'git+file:///home/jtojnar/Projects/nixflk' has deprecated attribute 'edition' flake: Refactor nixosModules collection We will want to reuse the prep method for overlays so let's move it higher and simplify it a bit. flake: Fix overlays for `nix flake check` It complains that: value is a list while a set was expected, at /nix/store/0qdcc831rj27wz07lrri6bnfmhvm4wrm-source/flake.nix:26:7 But nixpkgs still expects a list, so we need to pass it just the values. laptop: add acpi package flake: add option to install packages from nixos-unstable laptop: add bluetooth support games: update retroarch to latest version games: add wii-u-gc-adapter package and service laptop: add usbutils xmonad: use volnoti and eliminate dzvol games: add gamemode service ssh: init profile pkgs: add metadata and standardize retroarch: 1.8.8 -> 1.8.9 graphical: set volume settings via window manager sound.mediaKeys doesn't work with pulseaudio enabled, but some programs e.g. Firefox require it. flake: base unstable on master modules#steam: create xsession for Steam cachix: add personal cachix information No profile imports it by default, but if the user wishes to benefit from having some of the packages expressed in this flake pulled down from the cache instead of built locally, they can easily import ./cachix.nix from any profile or host. shell.nix: remove `rebuild` command `nixos-rebuild` now works perfectly with flakes so this is no longer required. A new command `build-iso` is still available so that building a NixOS image is still possible. direnv: use `nix print-dev-env` In order to increase determinism in the build environment, we are having direnv load the shell environment from the flake's `devShell` output, instead of the traditional `nix-shell` approach. Of course, one can still enter a `nix-shell` manually on a system without flake support installed. shell.nix: pass command line options to build-iso direnv: add flake profile so shell isn't GC'd shell.nix: don't override NIX_CONF_DIR flake#nixosModules: add binary cache as module shell: use starship for bash and zsh flake: move pathsToImportedAttrs to utils fix various linting errors throughout the repo * replace `rec` attrs with let bindings * remove unused args format: apply nixpkgs-fmt lint: remove `//`, replace with `recursiveUpdate` flake: rename flake inputs and add overrides Now you can add packages to the list in pkgs/override.nix and they will be pulled in from nixpkgs master instead of the default NixOS flake when installed. shell: proper shell escape flake: add template output users#git: init profile users#alacritty: init profile users#direnv: init profile envrc: `use flake` if available overrides: fix overrides Revert "shell.nix: remove `rebuild` command" This reverts commit 445226005631153481a41c8d0cea026ce4cce8ea. README: simply and update packages: export overlays as packages nix-zsh-completions: enable `nix-command` support graphical: reorganize dependant profiles gamemode: remove pkg from packages games: fix dependency profiles: add symlink to user/profiles profiles: reorganize .gitattributes: match the entire secrets/** subtree A single star would only match a single level. update to 20.09 Also fixes #24 by using recommended home-manager retroarch: 1.8.9 -> 1.9.0 update: small fixes after version bump shell: alias rebuild to nixos rebuild Fixes #27 pkgs#miningcore: init at 50 envrc: flakes should be considered pure profiles#virt: add vagrant core: add manix and nix-index virt: overlay no longer useful virt: enable oci containers via podman users#git: avoid warning on `git pull` direnv: flake based envs are pure by default We should reflect this in the IN_NIX_SHELL envar, since it doesn't get set properly otherwise. shell: don't overwrite nix by default For users who already have nix-command setup, we don't want to override the `nix` in their path. shell: for zsh users provide useful competion Default nix completions fail with the new nix-command interface, at least for users of zsh and direnv, this can be avoided on first startup. nix-zsh-completions: update overlay shell: silent nix-command check qutebrowser: 1.3.1 -> 1.4.1 flake: clean up by moving implementation to utils Fixes #28 by adding an `externOverlays` list to easily import overlays from external flakes. virt: use docker alias so completion works direnv: use nix-direnv instead ad hoc hack shell: rename rebuild to flk shell: add a nixpkgs-compat file We are now able to pull in the same revision of nixpkgs as listed in the lock file. Add NUR support virt: enable container configs flake: initial multiarch support flake: add `externModules` list Fixes #44. Easily add external modules from other flakes by dropping them in the list. Fix any-nix-shell breaking `nix-run` Simply apply the PR from haslersn/any-nix-shell#6 until it is merged upstream. chore: add editorconfig flake: add apps output Packages that contain an "$out/bin" dir are exported as `apps` as well. shell: pull in more dependencies * Pull recent enough versions on nixos-(rebuild|install) to ensure compatibility with flakes. * Add grip for viewing changes to README.md locally factor out a generic fetch from nixpkgs-compat Create fetch function which takes a valid flake input, and calls fetchTarball with the revision and hash information from the flake.lock. Useful for pulling the exact revision of flake inputs from not flake expressions, such as those in shell.nix. shell: switch to using mkDevShell shell: fix flk command shell: add command descriptions any-nix-shell: update overlay editorconfig: fixup hooks: declare nixpkgs-fmt pre-commit hook ref: remove overlay xref to pkgs since packages are exposed through output.packages by their own, a second export through overlays.pkgs would be smudging the public api of this repository. don't be fancy flake: add pkgs back to overlays Packages from `pkgs/default.nix` are meant to be exposed to all nixos configurations. However, #61 removed a redundancy in the output api which was relied up to do so. overlays: fix darkwin builds flake: simply `packages` with flattenTreeSystem Based on numtide/flake-utils#13. Little hint Update flake.nix Shell: pass flags to iso build like --show-trace Fix realod better fix for #75 Evaluate exported pkgs against repo baseline nixos The baseline pkgs for own consumption is nixos, only pkgs/override.nix provides a mechanism to "backport" packages from unstable. Therefore, we should expose our packages to others in the same context as we use them ourselves. shell: add `flk up` command Automatically generate a configuration for your current system using nixos-generate-config, placed in the `up` directory. A file is added in hosts/up-$(hostname).nix importing said configuration. shell: ignore `flk up` generated configs shell: namespace `flk up` Namespace and stage generated files so that the user can build it right away. flake: remove apps to reduce complexity As pointed out here numtide/flake-utils#17, apps are somewhat redundant and, for the sake of simplicity, should be removed for now. pkgs: simplify override develop#haskell: init profile kakoune: add haskell lsp support flake/host: add nixos-hardware - add nixos-hardware flake. - configuration in hosts/default.nix to be able to include hardware specifics per host. - add corresponding documentation in readme default: add default.nix using flake-compat miningcore: remove from outputs packages: apply overlays to output packages Apply overlays to output packages so we can easily use a modified package. default: x86_64-linux packages Add nrdxp cachix to substituter flake list courtesy of: NixOS/nix@343239f default: build all platforms in ci Revert "Add nrdxp cachix to substituter flake list" compat: fixes * fix nixos-option * use flake-compat for all backward compat concerns * rebuild devshell after any nix file changes * reexport system pkgs as legacyPackages * provide lib flake output * provide flake lib to nixos configs via `lib.flk` * clean up flake.nix flake: apply overrides first Fixes other overlays not applying when pulling package from master. profiles#ci-agent: init flake: format fix shell: use DEVSHELL_ROOT shell: use devshell-native pre-commit hooks users: home-manager only configurations for non-root users or non-nixos systems flk script: add home-manager only configuration support shell: fix pre-commit on deleted files home: move hmActivationPackages to own output Fixes `nix flake check` failure. host: add deault implementation for system.build.isoImage target (per host) iso: move iso target to ./lib iso-cd: style cleanup (#2) shell#flk: consistent quoting shell: add editorconfig hook profiles: add concept of suites * Import attrs of profiles automatically with `defaultImport`. * Refactor profiles to ensure all are functions returning a module. * Add a suites.nix with collections of profiles. * Add suites as `specialArgs` to modules. * Add suite import to NixOS host. suites: add users to suites profiles: add concept of suites * Import attrs of profiles automatically with `defaultImport`. * Refactor profiles to ensure all are functions returning a module. * Add a suites.nix with collections of profiles. * Add suites as `specialArgs` to modules. * Add suite import to NixOS host. suites: add users to suites ci: move into build attribute create core branch without any profiles This core will be used as the new base for the reuse branch, which is essentiall the same as the existing template branch, except that it is rebased from this core, simplifying future rebasing. ci: fix ci after core creation shell: fix formatted files not commited various refactors * Move extern lists to their own folder * Move unstable package and module imports to their own folder * Create a genPkgs function to avoid using legacyPackages for the whole package set * Move hmActivationPackages to legacyPackages shell: workaround NixOS/nix#4529 lib: rename defaultImports flk: add get command flake: use same nixos versions for inputs shell: fix pkgs for nix-shell secrets: fix subtle error with .gitkeep Add secrets .gitattributes to secrets dir and avoid encrypting it. Fixes an issue with .gitkeep causing build failures on unlocked repos. shell: cleanup devshell * Add update command to flk * Add cleaner usage to flk command * Factor out scripts to be more readable shell: remove cachix from i686-linux Build of cachix is failing on i686-linux right now. Remove it so shell is still usable. doc: new mdbook documentation flake: nixpkgs points to master by default No need to explicitly specify it. flake: manage flake-compate as input flake: rename flake-utils input deploy-rs: init support Managing servers remotely is a big usecase, and deploy-rs is a flake first deployment tool. By default, all nixosConfigurations are also setup as deploy-rs nodes. You'll still need to do some manual ssh setup, but other than that, the system is ready to deploy. deploy: move logic to lib shell: use devshell.toml nix: patch nix directly for experimental features The bash wrapper was causing issues with deploy-rs and the experimental-features option in the config was causing hercules-ci to spit out a bunch of junk in the logs. Patching nix with the default values required for flakes fixes both of these issues. project rename: nixflk -> devos profiles: simplify profiles to suites * Leave importing to nixpkgs module implentation. Provide a path instead; resolves #136. * Allow profiles which are not lambdas but simple attribute sets, relaxing the constraints a bit. * Update profile README.md * defaultImports -> mkProfileAttrs: allow importing subprofiles even if parent directory does not contain a default.nix. pkgs: use subflake to manage package sources * Resolves #118 * Leverage flakes to manage package sources & hashes * Update documentation with an example. * Add `mkVersion` function to autogenerate a version string. * Add srcs package via overlay containing all sources defined in _pkgs/flake.nix_ * Extend `flk update` with the ability to only update the given input pkgs#flake: fix flake-compat support edolstra/flake-compat#18 flake.lock: update lib#genPkgs: reorder overlays Resolves #146 by allowing external overlays to ve overriden by local overlays. lib: can depend on pkgs (a la nixpkgs#pkgs/pkgs-lib) treewide cleanups and refactoring for initial tests (#157) - [x] refactor lib into separate files, similar to NixOS/nixpkgs/lib. - [x] refactor ci to automatically generate derivations from flake outputs - [x] remove cluttered indirection statements throughout the codebase - [x] refactor hosts to allow for upcoming integration tests - [x] improve ambiguity in the existing docs - [x] add [BORS](https://bors.tech) support - [x] add initial integration test - [x] write tests documentation - [x] test lib - [x] improve version string generation, and do so automatically for pkgs/flake.nix sources Clean up the codebase as best we can in preparation for #152 and add tests. From now on, all PRs will be merged with BORS. subflakes: make first class citizens Subflakes should provide their wares as outputs, so wire up the pkgs flake to reflect that. Due to the unstable nature of flakes, updating the root flake doesn't currently update the subflake lock file. Therefore, add additional logic to flk update script in order to do this behind the scenes. Nix is now pulled in from the "nix" registry flake in order for users to take advantage of improvements to the UI since its last update in nixpkgs. lib: add rgxToString function lib/devos: bake devos repo into live cd (#168) fix #167 This worked for me to bootstrap another machine. lib: pass `inputs` to `./lib` library functions now can access inputs (and their sources!) directly, for example in order to copy their closure into the iso registry so that evaluating the flake does not need to fetch inputs again hosts: set nixpkgs.pkgs based on nixpkgs.system flake/tests: improve multi arch support for flake checks flake: nixos -> nixos-unstable 20.09 is getting stale as we move toward a new release so track unstable for now. nix: remove patch for default unstable features nix: don't use nix flake I originally wanted to use the nix flake to allow users to take advantage of the latest changes. Just so happened that nixpkgs was recently updated with a new version around the same time, and this just adds complexity for no real gain. flake/home: release -> master users: add and export user modules and import them in hosts export homeConfigurations for all user@host pairs and add better home-manager only defaults lib/filterPackages: filter by flake requirements returns only non-broken derivations that support the system. flake: use filterPackages instead of flattenTreeSystem lib/mkPkgs: fix function arguments take extern and overrides as arguments as to not import external files get inputs as a function dependency from callLib shell: get extern and overrides to pass to mkpkgs flake: update to use mkSuites and mkHosts lib: only readDir if path exists with safeReadDir lib: init mkFlake to create a devos flake general lib function - not devos specific flake: use mkFlake this is where we create devos's template structure treewide: format with nixpkgs-fmt init pkgs-lib, move tests and shell there fix some small bugs in mkFlake/evalArgs add old evalArgs as evalOldArgs, so flake works ref: extract lib into subflake workarround for unkown problem fix: various left-overs doc: move all docs under `doc` cachix: move out of root and into profile lib: move compat to lib mv `nix/ci.nix` directly to `default.nix` overrides: move to extern suites: move to `profles/suites.nix` update devos template to use new api add pkgs overlay, pkgs.overlay is just for srcs add cachix to base suite to follow the new api auto-flatten coercedListOf and type fixes move overrides.nix to overlays also update overrides docs it is a pure devos overlay now, it makes sense to go in overlays auto import hosts in flake.nix this allows host-specific settings to be overriden with mkMerge don't use lockfile to get flake-compat lock file format isn't very reliable with naming inputs fix flake by removing flake-compat follows use __dontExport property for lib and overrides pass builderArgs to config and extract builds have mkHomeConfiguration create its own custom build within the function create a externalModule for customBuilds so its easy to add more improve home-manager and deploy-rs integration move both to be setup in template only set home-manager options if they exist switch to devlib repo and remove in-tree lib add in-tree local lib, ourlib flake.nix: explicitly append overlays from path Co-authored-by: David Arnold <[email protected]> flake: format and update to devlib api changes don't import cachix in suites its already imported in the core profile flake.nix: use `imports` for auto-import of hosts This makes the auto-importing of hosts obvious and explicitly indicates how the options would end up getting merged. tree: rename devlib -> digga flake: format ourlib for consistency with digga flake: update to latest digga api allows overlays to also be used with `imports` and renames importHosts suites: re-add `rec` to simplify suite definitions Add agenix integration to secrets and flake default.nix: fix path to compat flake: un-comment defaultTemplate This was accidentally commented. fixes #288 flake: update to new digga api flake: add agenix to devShell gitignore: add flk command results flake: switch to digga master To use the now released v0.2.0 init: common flake compat fixes for other nixpkgs fixes #315 ref: pass our lib as lib.our to the module system At least we don't find ourselves with strange module signatures and use something that looks like a dedicated namespace within he module system: `config.lib` bump digga to latest develop feature: Add pkgs fetch manager nvfetcher Co-authored-by: David Arnold <[email protected]> fix hasekllPackages overlay ref: move devshell module here to `devos` again work-around for NixOS/nix#4641 fix: hm pin to corresponding nixos version imp: reduce flake inputs footprint and increase conciseness imp: add portable user definition According to nix-community/home-manager#2161 there might be a better formal distinction between host-spaced and portable arch-spaced hm users in the future. rm: community the commmunity branch was fallen too far behind and there where no contributions to it, anyways. We will start linking usage examples out in the wild, instead. ref: make hm common module explicit fix: move legacy nix path definitions back to devos imp: add digga's nixConfig and patchedNix add flake's nixConfig values that go well with this repo fmt update: digga/fup refactoring ref: adapt to using plain digga module for boostrap ISO ref: point `digga` to `main` (default) branch imp: use bud nixos-/devhsllModules and hide rebiding both modules depend on `self`/`inputs` available as special args for both, devshell & nixos modules, so bump to an improved `digga`, as well. flake: remove ci-agent Ensure pinning latest channel to nixpkgs nixos-unstable branch docs: soft update (fix the most obvious drift) fix/hm: direnv flakes support bump: nvfetcher and cleanup fix: dedicated bootstrap config fix: input ref / bump digga the indirect reference started failing with ``` warning: Git tree '/home/blaggacao/ghq/github.com/divnix/devos' is dirty error: NAR hash mismatch in input 'github:NixOS/nixpkgs/release-21.05' (/nix/store/0r0xrqj5l88071cama7hlha5ajya06c8-source), expected 'sha256-MhVtkVt1MFfaDY3ObJu54NBcsaPk19vOBZ8ouhjO4qs=', got 'sha256-jQgXeJ9NQQS0Eobb/qQOvS+RRULkqRikAeXkkFKOPDA=' ```
It turns out that we also need to filter by the supported system to get
a useful output out of flattenTree.
In order to keep back-compat, a new implementation has been created that
also looks at the meta.hydraPlatforms or meta.platforms if the former is
undefined, and only keep the derivations that have the provided system
in it.
Fixes #4