Introduction

Image from Jurassic Park © Universal

What is this?

Nedryland is a collection of utilities and a build system for microservice architecture solutions. Nedryland is built upon Nix. Nix is a functional package manager for Unix systems that promises reproducible and deterministic builds and build outputs. There is also nixpkgs, a collection of over 80 000 packages for the Nix package manager that can be used to get dependencies for builds etc.

There is also NixOS which builds on the ideas of Nix to create a whole Linux distribution where all configuration of the OS is also described in Nix and becomes 100% deterministic and without side effects.

What's up with the name?

Dennis Nedry is one of the main antagonists in the original Jurassic Park movie. His computer system for orchestrating the park is called Nedryland JP. A screenshot of this magnificent system can be seen above.

Concepts

Nedryland introduces some new concepts for building microservices. This chapter describes these components in more detail.

Project

A project in Nedryland is a folder with a collection of components. They can be components of only one type or of different types. A project is created by importing Nedryland in a file and then exposing the matrix in flake.nix and/ordefault.nix and shells in shell.nix so that it works with the standard Nix tools.

To depend on Nedryland in a project, any standard Nix fetcher can be used. More info on how to set up a new project can be found in Declaring The Project. Niv provides an alternative method of pinning and managing a nix dependency.

Component

A component is the most fundamental building block in Nedryland and the architechtural building block in a microservice based system.

Nedryland contains plenty of helper functions to create different kinds of components. Components can furthermore be dependent on eachother and Nedryland also contains many helpers for deployment of components.

Base

In addition to the package collection, components also get access to something called base. base is a collection of Nedryland utilities for component declaration so it is essentially a big set of functions that relates to declaring components. The set of functions available in base can also be extended by other projects as described here.

Documentation

Components can also have associated documentation declared in the docs attribute. See here.

Matrix

A central concept in Nedryland is the matrix. The matrix is how build targets are accessed and the reason it is called a matrix is since the target layout can be thought of as a two-axis matrix. On one axis, we have all components and on the other axis, we have all targets. In addition to this, there are also some one-dimensional targets that always work on all components (deployment is usually one example).

An example matrix might look like:

⬇️ component \ target ➡️	package	docs	target1	target2
nedry
hammond
malcolm

This means that to build the package target of hammond we would access hammond.package:

⬇️ component \ target ➡️	package	docs	target1	target2
nedry
hammond	🧨
malcolm

To build all targets for hammond, we would access hammond:

⬇️ component \ target ➡️	package	docs	target1	target2
nedry
hammond	🧨	🧨	🧨	🧨
malcolm

and to build the target package for all components we would simply access targets.package:

⬇️ component \ target ➡️	package	docs	target1	target2
nedry	🧨
hammond	🧨
malcolm	🧨

Deployment

Deployment in Nedryland is handled by using a set of convenience functions when declaring your components. These are then picked up in the project setup and deployment is declared there.

Deployment configuration is very flexible and you can combine deployments of components in any way you might like.

Declaring the Project

Machine Setup

You will need to install Nix on your computer. Visit nixos.org/nix/ to do so.

The next thing you will need to create is a project. A project consits of three files: project.nix, default.nix and shell.nix. So, let's go ahead and set up a project.nix.

Project Setup

There are two main ways of setting up a project, flakes and everything else. Here we will cover flakes and fetchgit but any other method of fetching files will of course also work.

1 Using flakes

We need two files, flake.nix and project.nix, flakes are used to handle dependencies and set up entry points and the project file declares the components, shells and other attributes in the project.

# flake.nix
{
  description = "Project description.";

  inputs = {
    nedryland.url = github:goodbyekansas/nedryland;
    pkgs.url = github:NixOS/nixpkgs/nixos-22.11;
    flake-utils.url = github:numtide/flake-utils;
  };

  outputs = { nedryland, pkgs, flake-utils, ... }:
  flake-utils.lib.eachDefaultSystem (system:
    let
      pkgs' = pkgs.legacyPackages.${system};

      project = (import ./project.nix {
        pkgs = pkgs';
        nedryland = nedryland.lib.${system} { pkgs = pkgs'; };
      });
    in
    {
      packages = project.matrix // {
        # This make `$nix build` (without arguments)
        # result in a linkfarm of all components.
        default = project.all;
      };
      devShells = project.shells;
    });
}

# project.nix
{ nedryland, pkgs }:
(nedryland { inherit pkgs; }).mkProject {
  name = "project-name";

  components = { callFile }: {
    component1 = callFile ./path/to/component.nix { };
    # more components here
  };
}

After adding these files to the git index, nix flakes will recognize them. Then to build component1 just run nix build .#component1. To get a development shell run nix develop .#component1. To build an inspectable file tree of the project run just nix build.

2 With fetchGit

We will use three files, one for creating the project, one for building and one for opening development shells.

First, you will need to import Nedryland. In this example we use fetchGit, and since Nedryland needs a set of packages we also fetch nixpkgs and send it to Nedryland. The we use mkProject to create a project with a component.

# project.nix
let
  pkgs = import (
    builtins.fetchGit {
      name = "nixpkgs";
      url = ;
    } { });

  nedryland = import (
    builtins.fetchGit {
      name = "nedryland";
      url = "git@github.com:goodbyekansas/nedryland.git";
      ref = "refs/tags/8.2.1";
    } { inherit pkgs; });
in
nedryland.mkProject {
  name = "project-name";

  components = { callFile }: {
    component1 = callFile ./path/to/component.nix { };
    # more components here
  };
}

To be able to build more ergonomically, we create a default.nix and expose the project's matrix.

# default.nix
(import ./project.nix).matrix

Now nix build -f default.nix component1 (or nix-build -A component1 for the old output) will build component1.

To add development shells we add the shell.nix file:

# shell.nix
(import ./project.nix).shells

Then nix-shell -A component1 will launch a shell for working on component1.

mkProject

mkProject is the main function to create a Nedryland project, the function accepts a name for the project and a default config file to use for project configuration, and some other inputs explained below. In the project set returned from the above call to mkProject, there will be more utility functions that can be called to set up the project further.

Creating the Matrix

A project consists of components, and components have targets, which are you build artifacts. To create components use the components argument in mkProject. This argument (and all other arguments to mkProject) can be a function and the function can request arguments from base (including base extension). An essential argument to request is callFile which will call a component with a set of packages and components. This function accepts either a nix file and a set of overrides. It works similar to callPackage in nixpkgs. There is also callFunction which does the same but without importing a path and instead directly calling a function.

mkProject {
  components = { callFile }: {
    example-component = callFile ./example-component/example-component.nix {};
  };

  # ...
}

deployment is described here and is simply a set with your different deployment targets, no magic there.

extraShells is any standard nix shells (as described here) that you wish to have exposed by the matrix.

lib is a conventional key for any nix functions that you wish to expose to projects using your project. These will simply be exposed on your project instance as .lib.

More details on the matrix can be found under concepts.

Configuration

A project might need configuration for example for server addresses, deploy credentails or really anything. Nedryland contains helpers for this that can be accessed by setting the configFile argument to mkProject. This file is a TOML file that may or may not exist. If it exists, Nedryland will parse it and make it available to components.

Environment variable overrides

Environment variables can also be used to override parts of, or the whole configuration. To override all the content of the config, use the environment var PROJECT_NAME_config, where PROJECT_NAME is the name of your project.

To override specific settings, use the environment var PROJECT_NAME_key_setting_name where PROJECT_NAME is once again the name of your project, key is the key used below in base.parseConfig and setting_name is the name of the setting used in structure in the call to base.parseConfig.

Using Config in Components

When declaring a component, you can declare a dependency on some configuration from the config file like this

config = base.parseConfig {
  key = "my-component";
  structure = {
    database = {
      url = "";
      user = "";
      password = "";
    };
  };
};

The assignments set the default values in case the requested keys are not present in the configuration. In this example, the corresponding TOML could look like

[my-component]
database = { url = "tcp://something", user = "user", password = "pass" }

Then, in your component declaration, you can use config any way you want.

Extensions

Nedryland can be extended by passing in extensions when creating the project. This is done by passing a set in a list as the baseExtenstions argument to mkProject.

This might look something like

# project.nix
nedryland.mkProject {
  name = "my-project";
  configFile = ./my-project.toml;
  baseExtensions = [
    ./nedryland-extensions/stuff.nix
  ];
}

and inside ./nedryland-extensions/stuff.nix you declare the extension by providing a set that will be merged with base from Nedryland.

# nedryland-extensions/stuff.nix
{ base, pkgs }: # all extensions are functions called with base and pkgs
{
  mkMyAwesomeComponentType1 = args: {
    # ...
  };

  mkMyAwesomeComponentType2 = args: {
    # ...
  };

  utilityFunction = arg1: arg2:
    true;

  # ...
}

Base will now contain these functions and can be called with for example base.utilityFunction. Note that base is also an input to the extension function, this is the base before the extension is applied. And since extensions are applied in the order they are declared in baseExtesions in mkProject, extensions can use functions defined in extensions earlier in the list.

Using extensions from another repo

You can also configure Nedryland to use base extensions from other repositories by importing that repository. This will give you access to the matrix of that project and it can also be sent in as projectDependencies when declaring your project in Nedryland. Doing so will give you access to all base extensions from that project.

Example

# ...

otherProject = import path/to/other/project/ { };

project = nedryland.mkProject {
  name = "my-project";
  configFile = ./my-project.toml;
  baseExtensions = [
    (import ./nedryland-extensions/my-extension.nix)
  ];
  projectDependencies = [ otherProject ];
};

This will give you access to all baseExtensions declared in otherProject.

Defining Components

To define components, there is a set of functions to use and the most flexible (but also lowest level) is base.mkComponent. There is also a set of convenience functions to create different types of components easier. Furthermore, there is also programming language specific versions of the aforementioned functions.

So, let's declare a simple component. A component usually lives in a sub-folder in the repository so create a folder example-component and create a file example-component.nix in it. In this file, put the following content:

{ base }:

base.mkComponent {
  package = base.mkDerivation {
    name = "example-component";
    src = ./.;

    buildPhase = ''
      touch $out/hello-world
    '';
  };
}

This declares a component with a package target and uses Nedryland's mkDerivation which is a wrapper around the standard stdenv.mkDerivation function but with an added filter to exclude git-ignored files from the source before building. .

Exposing your Component

In order for your component to be used, it needs to be added to the build matrix. This is done by adding it to the call to mkProject (usually inside project.nix). A component is exposed by a call to callFile followed by optional arguments. callFile is part of base and can therefore be accepted as an argument when declaring components on a project.

So, something like this:

nedryland.mkProject {
  name = "my-project";

  components = { callFile } : {
    exampleComponent = callFile ./example-component/example-component.nix {};
  };

  # ...
}

The component will be exposed under the nix attribute exampleComponent so to build it you can use nix-build -A exampleComponent.<target> or nix build .#exampleComponent.<target> where target is any target on the component (see matrix)

Component Dependencies

Nedryland supports components being dependent on other components. This is done by declaring your dependency as an input. This is true for both for packages available in pkgs and your defined components. Add your dependency to the argument list to your file and Nedryland will automatically send it to the function call if available in either components or pkgs.

{ pkgs, base, myDependency }:

base.mkComponent {
  # ...
}

Component Types

Nedryland contains a standard set of component types which are described below. Further component types can be added in projects by extending base like explained here.

Services

A service is a component that has no user interface and instead exposes some sort of a remote API. Helpers exists to create gRPC services.

To define a service, your Nix file might look something like:

{ pkgs, base }:

base.mkService {
  name = "example-service";
  src = ./.;

  buildPhase = ''
    touch $out/hello-world
  '';
  };
}

Clients

A client is a component that presents some sort of user interface. It can be either a GUI, a command line interface or something else that a user interacts with (VR experience, etc.).

To define a client, your Nix file might look something like:

{ pkgs, base }:

base.mkClient {
  name = "example-client";
  src = ./.;

  buildPhase = ''
    touch $out/hello-world
  '';
}

Library

This component type is not exposed directly in base but rather by the different language helpers (see below) and should be used to share common functionality in a library for the language in question.

Documentation

Documentation can be added to a component by the argument docs, this argument has to be a set with derivations or metadata. In base.documentation there are helpers to create documentation derivations. When building the docs target of the component, a combination of all the derivation attributes will be created. Any non-derivation will be serialized to json and stored in metadata.json in share/doc/<component-name>.

base.mkComponent {
  name = "documented-component";
  ...
  docs = {
    user = base.documentation.mkMdbook {
      src = ./docs/user;
    }
  }
}

The Development Environment

Nedryland uses nix-shell (default.nix) or nix develop (flakes) to provide a development environment for defined components. For this to work, shell.nix in the root of the repository needs to expose the output of project.mkShells.

Working on a component

To start a shell for working on a component called hammond, issue the command

$ nix-shell -A hammond

or if using flakes

$ nix develop .#hammond

This will download and expose all dependencies necessary to work on hammond, change the directory to where hammond is defined and create a new shell session.

Get a Shell with all Components

The special target all (nix-shell -A all/nix develop .#all) will drop you in a shell session with access to all build outputs of the whole project. This can be useful to test a set of components together.

Nedryland linters

Nedryland provides a set of linters/formatters, they are accessed in the checks attribute, and as separate apps. The currently provided checks are:

nixfmt: To format nix code. This tool has been customized to compare current code with correctly formatted code by default and take --fix to correct the code.
nix-lint: To lint nix expressions for mistakes and code quality.
shellcheck: To check shell scripts.
check: To run all checks.

To forward linters from Nedryland to a project to something like this:

# flake.nix
{
  inputs = {
    nedryland.url = github:goodbyekansas/nedryland;
    flake-utils.url = github:numtide/flake-utils;
  };

  outputs =
    { nedryland
    , flake-utils
    , ...
    }:
    flake-utils.lib.eachDefaultSystem (system:
    {
      apps = {
        checks = nedryland.apps.${system}.checks;
      };
    })
}

And then run it with nix run .#checks.

Selecting Files

Sometimes a project contains files which should not be linted, for example 3rd party files or generated scripts. To customize which files will be checked set $NEDRYLAND_CHECK_FILES to point to a file containing a list (newline separated) of all files to check.

Extending the Set of Linters

check will run all scripts in its bin folder, which means that to extend the lint toolset in a project, simply symlinkJoin it with a derivation containing more linters. When writing a custom linter use $NEDRYLAND_CHECK_FILES to select files from and fall back to some other way of discovering files. Use $NEDRYLAND_CHECK_COLOR to check if colors should be in the output. The script should be able to run without any arguments.

Setting Up Deployment

Nedryland provides a set of convenience functionality for creating deployments. Note that deployment is not actually done inside the derivation, instead it generates a deploy script that can be run outside of a nix context. This is because things like credentials and network access make deployment inherently impure.

An example might look like

base.mkComponent {
  ...
  deployment = {
    terraform = base.deployment.mkTerraformDeployment {
      terraformPackage = package;
      inherit preDeployPhase postDeployPhase deployShellInputs;
    };
  };
}

Building a deployment script from this declaration can be done in two ways.

The attribute <componentName>.deployment.terraform.
The attribute <componentName>.deploy which is a combined target containing all deployments (which in this example is only one).

A deploy can now be performed by running the script in bin/deploy in the resulting derivation (usually accessed by the result symlink). Most deployments also comes with a deployment shell that is typically used to interact with the deployed environment. To enter this shell run bin/shell in the resulting derivation.

Setting up project deployment targets

mkProject accepts extra targets and deploy is conventionally used for project deployment targets and can be accessed through deploy.<attribute>.

All values in this set should be deployment derivations, or combinations of them which can be created with the convenience function mkCombinedDeployment available in base.

nedryland.mkProject {
  ...
  deploy = { mkCombinedDeployment }: {
    group = {
      comp1 = component1.deploy;
      comp2 = component2.deploy;
    }
    combined = mkCombinedDeployment "combined" {
      comp3 = component3.deploy;
      comp4 = component4.deploy;
    }
  }
}

deploy.group.comp1 will create a deploy script for component1.
deploy.group will create two separate deploy script for component1 and component2.
deploy.combined will create one deploy script for component3 and component4.
deploy.combined.comp3 will not work.

Creating new deployment types

To aid in creating the deploy script and shell Nedryland provides the function base.deployment.mkDeployment. This function takes one mandatory argument.

deployPhase: This is the actual deploy script. This can either be the inline deploy script or call an external utility. The environment is similar to the shell one, but it is recreated outside of nix

Furthermore it accepts these optional arguments.

preDeployPhase: Script code to execute before the deploy phase and when opening the shell. A typical usecase is to aquire credentials and set up connection details. postDeployPhase: Script code to execute after the deploy phase. Used to cleaning up resources aquired during preDeployPhase. inputs: A list of derivations that will be added to path for the deployment script. These are typically dependencies you only need to do the deployment of the artifact that aren't necessarily dependencies of the artifact itself. shellInputs: A list of derivations that will be added to path for the shell. deployShell: Is true by default, set it to false if you do not want to have a deploy shell generated in your output.

Deployment will also pick up preDeploy, postDeploy and preDeployShell hooks.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.