| Age | Commit message (Collapse) | Author |
|---|
|
This is the core architecture transformation from Phase 3 of the
performance plan. Each Haskell module is now built as a separate
Nix derivation, enabling true incremental builds where only changed
modules and their dependents are rebuilt.
Implementation: - buildHsModuleGraph: Analyzes transitive module
dependencies and builds DAG - TH detection: Falls back to monolithic
build if Template Haskell detected - SCC cycle detection: Falls
back if import cycles found - Per-module Nix builder: Each module ->
separate derivation with .hi and .o - Module dependencies: Copy .hi
files to build dir, use -i flags for imports - Final link: Use ghc
--make with entry point source + -i paths to .hi files - Entry point
fix: Explicitly analyze entry point module separately from deps
Architecture: - Module compilation: ghc -c with -i paths to dependency
.hi files - Source filtering: Each module derivation includes only
its source file - Dependency DAG: Expressed as recursive Nix attrset
with lib.fix - Link phase: ghc --make with entry source file + all .hi
search paths - Fallback: Monolithic ghc --make when hsGraph is null
(TH/cycles)
Performance characteristics: - Change one module -> rebuild only
that + dependents + relink - Nix handles DAG scheduling and caching
automatically - Parallel module compilation (Nix orchestrates) -
Content-addressed caching across machines
Testing: - Added test_buildHsModuleGraph unit test - Verified with
Omni/Bild/Example.hs (4 modules) - Tested incremental rebuild triggers
correct subset
This completes Phase 2 and Phase 3 core optimizations from the plan.
|
|
Implements Steps 1-4 of per-module Nix architecture: - Add
HsModuleGraph and HsModuleNode types to represent module DAG -
Implement buildHsModuleGraph that:
- Parses imports for each module in the transitive closure -
Detects Template Haskell usage (language pragma or $( patterns) -
Detects import cycles using SCC analysis - Returns Nothing (fallback
to monolithic) if TH or cycles found
- Wire hsGraph into Target type for all builders - Populate hsGraph
for Haskell targets during analysis
Module graph is now emitted in --plan output showing per-module
dependencies. This enables the next step: per-module Nix derivations
in Builder.nix for true incremental builds.
Part of Phase 3 from the performance plan.
|
|
Implements optimization #3 from Phase 2 of the performance plan.
Changes: - Cache stored in _/var/ghc-pkg-cache-<hash>.json - Hash
based on GHC version + GHC_PACKAGE_PATH for automatic invalidation -
Loads cache at startup, saves on successful completion - Uses atomic
write (tmp + rename) to prevent corruption - Gracefully handles
missing/corrupt cache files - Accumulates cache entries across builds -
Works with parallel builds (in-memory IORef + disk persistence)
Performance impact: - Eliminates redundant ghc-pkg invocations across
runs - Near-zero ghc-pkg overhead once cache is populated - No impact
on single-run performance (still uses in-memory IORef)
|
|
- Add Analyzing state to BuildState enum - Refactor from sequential
foldM analyze to concurrent analyzeAll - Initialize all lines with
[+] during analysis phase - Update to [...] (Pending) after each
analysis completes - Uses mapConcurrentlyBounded with concurrency of
8 for analysis - Remove Log.info from analyzeOne (now handled by line
state) - Analysis now runs in parallel, improving efficiency - Flow:
[+] analyzing → [...] pending → [~] building → [✓]/[x] complete
|
|
- Allocate one line per namespace (not per concurrent job) - Add
Pending state shown as [...] when build hasn't started - Initialize all
namespace lines at start showing [...] - Update to [~] when building,
[✓]/[x] when complete - Each namespace keeps its line throughout
the build - At end, all namespaces show their final status - --jobs
controls concurrency, not line count
|
|
- Remove [+] display from reserveLine (was causing mangled output
from concurrent writes) - Add [+] display in analyze function where
it's single-threaded - Simplify cleanup to just move cursor down
without clearing lines - This eliminates race conditions and terminal
clearing issues
|
|
- Remove all label prefixes from build logs (bild:, nix:, etc) -
Show [+] Namespace when first reserving a line (analyzing) - Show [~]
Namespace: output when building with subprocess output - Show [✓]
Namespace (green) on success - Show [x] Namespace (red) on failure
|
|
- Remove Log.info from analyze function - Remove Log.info from build
target compilers (Guile, NixBuild, Rustc, Sbcl) - Remove Log.good
from proc and nixBuild success handlers - Remove 'info: bild:' prefix
from streaming subprocess output - Output now shows only [✓]/[x]/[~]
status lines with subprocess stdout
|
|
- Add ansi-terminal to ghcPackageSetBild dependencies - Add
Omni/Log/Concurrent.hs to source fileset - Fix unused mLineNum warning
in buildTarget - Alphabetize deps list (hostname moved up)
Parallel builds now fully functional with proper Nix dependencies.
|
|
- Add global IORef for currentLineManager and namespaceLines
mapping - Update logs function to use LogC.updateCurrentLine -
Add updateCurrentLine and releaseCurrentLine helpers - Fallback to
normal printing when no LineManager active - Simplify buildTarget to
use global helpers instead of threading
Tasks: t-1a1EaJy
|
|
- Reserve line for each concurrent build - Release line on completion
with success/failed state - Fix hlint eta reduce warning
Task: t-1a1E3j1
|
|
- Add mapConcurrentlyBounded using QSemN for bounded parallelism -
Refactor build function to extract buildTarget worker - Build up to
--jobs targets concurrently - Preserves all existing functionality -
Output will be interleaved (will fix with LineManager next)
Related tasks: - t-1a0OVBs: mapConcurrentlyBounded helper - t-1a16ame:
refactor build function - t-1a1DdSB: replace forM with concurrent map
|
|
Replaced the old slow depth-first search with a breadth-first search
for detecting imports. This should be way faster when building a
single namespace because it doesn't have to visit the same file
multiple times.
The ghc-pkg caching means we only have to run ghc-pkg once per bild
invocation.
|
|
detectPythonImports now recursively analyzes imported modules to find
transitive dependencies, matching the behavior of detectHaskellImports.
Previously it only detected direct imports, which caused build failures
when Python modules had nested dependencies.
- Changed signature from [Text] -> IO (Set FilePath)
to Analysis -> [Text] -> IO (Set FilePath)
- Added filepaths, findDeps, and onlyPython helper functions -
Recursively calls analyze() on imported modules to find transitive
deps - Updated tests to pass empty Analysis map
|
|
This does the full transition: web server, mail server, xmpp. I expect some
disruption, but hopefully this is everything and it just switches over without
any problem.
|
|
This replaces the manually-curated nixTargets list in Bild.hs. Now any nix file
that has the executable bit set will be built.
I added run.sh shebangs to each of the buildable nix targets as well. When
executing these, they will succeed at building, but they have no 'out' metadata,
and so when run.sh tries to exec them, it will fail. This is fine for now.
How would one go about execing a linux tree anyway? If all of the nix targets
output something standard like a qemu image or a container, then I could have a
standard wrapper that calls the image and starts the system. That might be the
ideal way to have a runnable nix target. But this would require rethinking my
infrastructure and how to deploy things, so I can't quite do that yet.
|
|
With run.sh, we can build and run the file in one go. This means we can also use
it as an interpreter in a shebang line and properly use the Unix executable bit.
This is pretty cool and gives a few advantages: running any executable file is
just `exec file.hs` or even `./file.hs`, finding all executables is `fd -t x`,
you don't need to specify or know an `out` name to run something, execution of a
program is standardized.
There is a hack to get this to work. In C and Common Lisp, `#!` is illegal
syntax, so I had to use shell syntax to invoke run.sh, call it on the current
file, and then exit the shell script. Meanwhile, run.sh takes the file and evals
the whole thing, building and running it. As long as either `//` or `;` is a
comment character in the target language, then this works. Maybe a better thing
to do would be to pre-process the file and remove the `#!` before passing it to
the C compiler, like [ryanmjacobs/c][1] and [tcc][2]? However this won't work in
Lisp because then I can't just load the file directly into the repl, so maybe
the comment hack needs to stay.
[1]: https://github.com/ryanmjacobs/c/tree/master
[2]: https://repo.or.cz/tinycc.git/blob/HEAD:/tccrun.c
|
|
I put the storybook into a new Biz.nix deploy target. The idea here is that any
Biz/* targets should be hosted by this one VM for simplicity. Over time I can
grow this as need be, but this should work to host a few services.
|
|
I need a way to reliably get a NixOS VM provisioned in the cloud, and the
easiest way to do this is to create a qcow2 image, upload it to Digital Ocean,
and use that to start a droplet. This is very much a manual process, but that's
fine, I shouldn't need to do it very often (for now).
|
|
This was all dead weight, just delete it and move on.
|
|
I had forgotten to add this feature, apparently, so bild --test just didn't do
the test part.
|
|
I was getting confused about what is a product and what is internal
infrastructure; I think it is good to keep those things separate. So I moved a
bunch of stuff to an Omni namespace, actually most stuff went there. Only things
that are explicitly external products are still in the Biz namespace.
|
