ibizaman's Blog

Connect Home Assistant to Nextcloud Calendar

2026-09-01T00:00:00Z

Home-Assistant can connect to “subscription calendars” and you can then use those events to trigger automations.

I wanted to connect HA to my Nextcloud server so I could use my personal calendar this way too. But it wasn’t that obvious how to do it. So here are the step by step instructions.

Locate calendar in Nextcloud

First, pick one calendar to share and click on its “Edit and share calendar”:

Then click on the three dots next to the “Share link” row and click on “Copy subscription link”:

Add calendar to Home Assistant

On the Home Assistant side, add the Remote Calendar integration:

Give a name to the calendar and copy the subscription link. Then click done.

Conclusion

And that’s it! Now you can see your Nextcloud calendar from Home Assistant.

Testing LDAP and SSO Integrations in NixOS with Playwright

2026-08-22T00:00:00Z

Tags: nix, tests, playwright, ldap, sso, selfhostblocks

There is one thing I hate the most in the DevOps world. It’s having the ground shift under my feet.

I take a while to implement some integration but it’s done and I’m happy. Then 6 months later, something changes, some default is different, I don’t notice it and I deploy. This breaks my carefully crafted integration and I only realize that 2 weeks later or if a user notices it. And now I need to spelunk and figure what changed caused this and fix it, all that in a hurry.

This blog article is about how I pushed NixOS VM tests to the max to fix this once and for all.

Automated Updates

In my project SelfHostBlocks, I add a layer on top of services like Nextcloud, Home Assistant, Jellyfin, Immich, etc. that sets up a lot of features declaratively. One particularly rare feature I’ve seen in other projects is integrating those services with an LDAP provider (LLDAP) and an SSO provider (Authelia) completely self-hosted and declaratively.

I also wanted to provide automated updates. A cron job would open a PR after running nix flake update nixpkgs and if the tests succeed, it would be merged automatically. But… it was hard to get the integrations right, it’s quite the balancing act. Automating the updates like this would always lead to issues cropping up in unexpected ways. Even just manually updating was letting some slip through!

After updating nixpkgs, there are 4 possible scenarios:

Everything works fine. Yeah 🎉
The evaluation is failing. This usually means a NixOS module option changed. The fix is to update how I use it.
The build is failing. This means something in the package built or its dependencies broke. The fix is usually to pick another nixpkgs commit. I attempted to fix this previously, see my other blog post.
The integration fails subtly. The fix is… to test the integration. So let’s see what we can do.

NixOS VM Tests

NixOS VM tests are a wonderful piece of technology. You can spin up one or more VMs that talk together, write some NixOS config to start whatever service you want in those, just like you would deploy on your server, and then write some python code to exercise some properties you want to test.

First, let’s see a minimal NixOS VM test.

{ pkgs, ... }:
pkgs.testers.runNixOSTest {
  name = "jellyfin";

  nodes.machine = {
    services.jellyfin = {
      enable = true;
    };
  };

  testScript = ''
    machine.start()
    machine.wait_for_unit("jellyfin.service")
    machine.wait_for_open_port(9096)
    machine.wait_until_succed("curl --fail http://localhost:9096/login")
  '';
}

This test is named jellyfin, it creates one VM called machine and launches the jellyfin service with services.jellyfin.enable=true. The testScript is Python code, here it starts the machine, waits for the jellyfin systemd service to start and the port 9096 to be open and finally.

For more info, you can check the services.jellyfin NixOS module options and the NixOS VM test manual which is a complete reference on what you can do and what Python function you can use. Or just look at all the existing tests in the nixpkgs repository.

This looks quite simple in the end but it’s extremely powerful because it does three things:

It builds jellyfin (or pulls from the cache) which makes sure the package can be built.
It evaluates the jellyfin NixOS module, making sure the options are still used correctly.
It runs the jellyfin service and runs some smoke tests making sure it starts successfully.

Isn’t this magically solving all the problems we had with automated updates? Well yes it is!

Where this minimal test is too minimal is it does not test the LDAP or SSO integration. So let’s add that.

Playwright Tests

One way of testing the LDAP or SSO integration would be to make some curl requests here and there and poke around in the background. I tried this at first but stopped quickly because discovering those curl requests was hard and anyway, I wasn’t really testing the integrations this way. What I needed was to do some black box testing by opening up a browser, navigating to the service URL, filling out the login form and then see I was correctly logged in the web UI.

That’s where Playwright comes in. It’s a program that can automate actions in a web UI. Setting it up in the NixOS VM test was not obvious, but had to be done only once. In the following snippet, we create a NixOS module that a NixOS VM test can import.

# clientLoginModule.nix
{ config, pkgs, ... }:
let
  cfg = config.test.login;
in
{
  options.test.login = {
    browser = mkOption {
      type = enum [
        "firefox"
        "chromium"
        "webkit"
      ];
      default = "firefox";
    };
  };

  config = {
    environment.variables = {
      PLAYWRIGHT_BROWSERS_PATH = pkgs.playwright-driver.browsers;
    };

    environment.systemPackages = [
      (pkgs.writers.writePython3Bin "login_playwright"
        {
          libraries = [ pkgs.python3Packages.playwright ];
          flakeIgnore = [
            "F401"
            "E501"
          ];
        }
        (
          let
            testCfg = pkgs.writeText "users.json" (builtins.toJSON cfg);
          in
          ''
            import json
            from playwright.sync_api import sync_playwright


            def test(page):
                # Actual test goes here


            browsers = {
                "chromium": {'args': ["--headless", "--disable-gpu"], 'channel': 'chromium'},
                "firefox": {'args': ["--reporter", "html"]},
                "webkit": {},
            }

            with open("${testCfg}") as f:
                testCfg = json.load(f)

            browser_name = testCfg['browser']
            browser_args = browsers.get(browser_name)

            with sync_playwright() as p:
                browser = getattr(p, browser_name).launch(**browser_args)

                context = browser.new_context(ignore_https_errors=True)
                context.set_default_navigation_timeout(2 * 60 * 1000)
                context.tracing.start(screenshots=True, snapshots=True, sources=True)
                try:
                    page = context.new_page()

                    test(page)
                finally:
                    context.tracing.stop(path=f"trace/{i}.zip")

                browser.close()
          ''
        )
      )
    ];
  };
};

This big snippet only sets up the scaffolding. The actual test would go in # Test goes here. A few things to note:

We create a test.login.browser option to choose which browser to test with.
We set PLAYWRIGHT_BROWSERS_PATH = pkgs.playwright-driver.browsers; so the playwright executable can find NixOS provided browsers.
We ignore some python linting issues. I explicitly allowed for long lines because dealing with those for a test is annoying.
We create a login_playwright Python executable that launches the Playwright test. It will be called from the test’s testScript.
We pass the whole test.login config to the Python script by the intermediary of a json file. This makes it easy to pass any kind of values and to add options later on.
We enable taking a full trace and write that trace. It will then be available in the build output.

Using the module looks like so:

{ pkgs, ... }:
let
  clientLoginModule = import clientLoginModule.nix;
in
pkgs.testers.runNixOSTest {
  name = "jellyfin";

  nodes.server = {
    services.jellyfin = {
      enable = true;
    };
  };

  node.client = {
    imports = [
      clientLoginModule
    ];
  };

  testScript = ''
    machine.start()
    machine.wait_for_unit("jellyinf.service")
    machine.wait_for_open_port(9096)
    machine.wait_until_succed("curl --fail http://localhost:9096/login")

    with subtest("Login from client"):
        code, logs = client.execute("login_playwright")
        print(logs)
        try:
            client.succeed("""
              mkdir -p /tmp/shared/
              cp -r trace /tmp/shared/
            """)
            client.copy_from_machine("trace")
        except:
            print("No trace found on client")
        if code != 0:
            raise Exception("login_playwright did not succeed")
  '';
}

Here we create two nodes, two VMs, one called server with the jellyfin service and the second called client from which we will exercise the browser tests. The idea is to make the test as accurate as possible by making sure a remote node can access the service. Also, we take care of copying the traces from the client VM to the build output.

Declarative LDAP

Before being able to write the test, we must figure out how to create the LDAP and SSO config declaratively.

For the LDAP provider, I chose LLDAP for its nice web UI. It did provide a bootstrap script but this one was not wired into the NixOS module system. After adding support for that in my project (link to manual) and upstreaming a few fixes (link to PRs if you’re curious) we can now enable those services in our test and create some users declaratively.

# ldapModule.nix
{
  networking.hosts = {
    "127.0.0.1" = [ "${config.shb.lldap.subdomain}.${config.shb.lldap.domain}" ];
  };

  shb.lldap = {
    enable = true;
    domain = "example.com";
    subdomain = "ldap";
    ldapPort = 3890;
    webUIListenPort = 17170;
    dcdomain = "dc=example,dc=com";
    ldapUserPassword.result = config.shb.hardcodedsecret.ldapAdminPassword.result;
    jwtSecret.result = config.shb.hardcodedsecret.jwtSecret.result;
  };

  shb.hardcodedsecret.ldapAdminPassword = {
    request = config.shb.lldap.ldapUserPassword.request;
    settings.content = "ldapAdminPassword";
  };
  shb.hardcodedsecret.jwtSecret = {
    request = config.shb.lldap.jwtSecret.request;
    settings.content = "jwtSecret";
  };

  shb.lldap.ensureUsers = {
    alice = {
      email = "alice@example.com";
      displayName = "Alice Alice";
      groups = [ "user_group" ];
      password.result = config.shb.hardcodedsecret.alice.result;
    };
    bob = {
      email = "bob@example.com";
      displayName = "Bob Bob";
      groups = [ "admin_group" ];
      password.result = config.shb.hardcodedsecret.bob.result;
    };
    charlie = {
      email = "charlie@example.com";
      displayName = "Charlie Charlie";
      groups = [ "other_group" ];
      password.result = config.shb.hardcodedsecret.charlie.result;
    };
  };

  shb.hardcodedsecret.alice = {
    request = config.shb.lldap.ensureUsers.alice.password.request;
    settings.content = "AlicePassword";
  };
  shb.hardcodedsecret.bob = {
    request = config.shb.lldap.ensureUsers.bob.password.request;
    settings.content = "BobPassword";
  };
  shb.hardcodedsecret.charlie = {
    request = config.shb.lldap.ensureUsers.charlie.password.request;
    settings.content = "CharliePassword";
  };

  shb.lldap.ensureGroups = {
    user_group = { };
    admin_group = { };
    other_group = { };
  };
}

We do use hard-coded secrets here in the tests as that is good enough for testing. The shb.hardcoded module makes sure the file containing the secret has the correct Unix user, group and mode thanks to the request option. So although it hard-codes the secret, it at least makes sure permissions are correct.

The service will be accessible at ldap.example.com and we make it so requests to that fqdn goes to 127.0.0.1. We also create 3 users:

Alice has password AlicePassword and belongs to the LDAP user_group group which grants normal access to the service we will test.
Bob has password BobPassword and belongs to the LDAP admin_group group which grants admin privileges to the service we will test.
Charlie has password CharliePassword and belongs to some other group that do not grant access to the service we will test.

Those 3 users allow us to test most of what the upstream services allow with their LDAP and SSO integration. We also verify with Charlie that a user not part of the authorized groups cannot login into the service.

Declarative SSO

I chose Authelia as the SSO provider, which hopefully is declarative by default.

One thing to note with Authelia is it refuses to work without HTTPS. So we need to set that up using self-signed certificates.

{
  shb.certs = {
    cas.selfsigned.myca = {
      name = "My CA";
    };
    certs.selfsigned = {
      "example.com" = {
        ca = config.shb.certs.cas.selfsigned.myca;
        domain = "*.example.com";
        group = "nginx";
      };
    };
  };

  networking.hosts = {
    "127.0.0.1" = [ "${config.shb.authelia.subdomain}.${config.shb.authelia.domain}" ];
  };

  shb.authelia = {
    enable = true;
    domain = "example.com";
    subdomain = "auth";
    ssl = config.shb.certs.certs.selfsigned."example.com";

    ldapHostname = "127.0.0.1";
    ldapPort = config.shb.lldap.ldapPort;
    dcdomain = config.shb.lldap.dcdomain;

    secrets = {
      jwtSecret.result = config.shb.hardcodedsecret.autheliaJwtSecret.result;
      ldapAdminPassword.result = config.shb.hardcodedsecret.ldapAdminPassword.result;
      sessionSecret.result = config.shb.hardcodedsecret.sessionSecret.result;
      storageEncryptionKey.result = config.shb.hardcodedsecret.storageEncryptionKey.result;
      identityProvidersOIDCHMACSecret.result =
        config.shb.hardcodedsecret.identityProvidersOIDCHMACSecret.result;
      identityProvidersOIDCIssuerPrivateKey.result =
        config.shb.hardcodedsecret.identityProvidersOIDCIssuerPrivateKey.result;
    };
  };
}

I left out all the shb.hardodedsecret assignments from the above SSO snippet this time because they just clutter the view.

The service will be accessible at auth.example.com and we make it so requests to that fqdn goes to 127.0.0.1. We also setup self-signed certificates for that domain using another one of my NixOS module that hides that complexity.

Declarative Module Integration

It’s worth taking a little detour to understand what’s needed for a NixOS module to integrate with the LDAP and SSO integration.

To integrate with LDAP, Jellyfin requires the LDAP plugin to be installed and to create the appropriate config file. None of this is supported declaratively by upstream Jellyfin nor the nixpkgs Jellyfin module. The LDAP config looks like so and I’ll leave it to you to check this link in my repository to see how the plugin is installed.

# jellyfin.nix
pkgs.writeText "LDAP-Auth.xml" ''
  
  
    ${cfg.ldap.host}
    ${builtins.toString cfg.ldap.port}
    false
    false
    false
    uid=admin,ou=people,${cfg.ldap.dcdomain}
    %SECRET_LDAP_PASSWORD%
    ou=people,${cfg.ldap.dcdomain}
    (memberof=cn=${cfg.ldap.userGroup},ou=groups,${cfg.ldap.dcdomain})
    ou=people,${cfg.ldap.dcdomain}
    (memberof=cn=${cfg.ldap.adminGroup},ou=groups,${cfg.ldap.dcdomain})
    false
    uid, cn, mail, displayName
    
    
    
    true
    false
    uid
    userPassword
    true
    
    
  
'';

Not shown is how the secret %SECRET_LDAP_PASSWORD% is replaced in this file safely. For this too I needed to write some code in my project but it’s getting upstreamed into nixpkgs in a PR.

By the way, thanks to the declarative LDAP, you can add your user to the Jellyfin LDAP group with:

{ config, ... }:
{
  shb.lldap.ensureGroups = [ config.shb.jellyfin.ldap.userGroup ];

  shb.lldap.ensureUsers = {
    me.groups = [ config.shb.jellyfin.ldap.userGroup ];
  };
}

For SSO, we need to install another plugin and create another file:

# jellyfin.xni
pkgs.writeText "SSO-Auth.xml" ''
  
  
    
    
      
        
          ${cfg.sso.provider}
        
        
          
            https
            ${cfg.sso.endpoint}
            ${cfg.sso.clientID}
            %SECRET_SSO_SECRET%
            true
            true
            true
            
            
              ${cfg.ldap.adminGroup}
            
            
              ${cfg.ldap.userGroup}
            
            false
            
            groups
            
              groups
            
            
            true
          
        
      
    
  
'';

And also create some Authelia config:

# jellyfin.nix
{
  shb.authelia.oidcClients = lib.optionals cfg.sso.enable [
    {
      client_id = cfg.sso.clientID;
      client_name = "Jellyfin";
      client_secret.source = cfg.sso.sharedSecretForAuthelia.result.path;
      public = false;
      authorization_policy = cfg.sso.authorization_policy;
      redirect_uris = [
        "https://${cfg.subdomain}.${cfg.domain}/sso/OID/r/${cfg.sso.provider}"
        "https://${cfg.subdomain}.${cfg.domain}/sso/OID/redirect/${cfg.sso.provider}"
      ];
      require_pkce = true;
      pkce_challenge_method = "S256";
      userinfo_signed_response_alg = "none";
      token_endpoint_auth_method = "client_secret_post";
      scopes = [
        "openid"
        "profile"
        "email"
        "groups"
      ];
    }
  ];
}

I’m leaving out some minute details on how the secret is shared and how the options are defined because there’s already a lot of information.

I wanted to show you all this because it’s quite overwhelming. It’s overwhelming for me too. And it’s horrible to test each time there’s an update. I hope this helps you understand why I thought having tests for these was useful.

Testing LDAP and SSO Integrations

We have all the pieces now to write the tests.

Let’s create 3 modules settings up Jellyfin. One for the basic case - no LDAP nor SSO integration - one for the LDAP integration with the ldapModule.nix file and for the SSO integration with the file ssoModule.nix. This way we can combine them easily and create our 3 test scenarios.

{ pkgs, ... }:
let
  clientLoginModule = import clientLoginModule.nix;
  ldapModule = import ldapModule.nix;
  ssoModule = import ssoModule.nix;

  basicTestModule = {
    networking.firewall.allowedTCPPorts = [
      80
      443
    ];

    shb.jellyfin = {
      enable = true;
      domain = "example.com";
      subdomain = "jellyfin";
      ssl = config.shb.certs.certs.selfsigned."example.com";
      port = 9096;
      admin = {
        username = "admin";
        password.result = config.shb.hardcodedsecret.jellyfinAdminPassword.result;
      };
    };

    shb.hardcodedsecret.jellyfinAdminPassword = {
      request = config.shb.jellyfin.admin.password.request;
      settings.content = "adminPassword";
    };
  };

  ldapTestModule = {
    imports = [
      basicTestModule
      ldapModule
    ];

    shb.jellyfin = {
      ldap = {
        enable = true;
        host = "127.0.0.1";
        port = config.shb.lldap.ldapPort;
        dcdomain = config.shb.lldap.dcdomain;
        userGroup = "user_group";
        adminGroup = "admin_group";
        adminPassword.result = config.shb.hardcodedsecret.jellyfinLdapAdminPassword.result;
      };
    };

    shb.hardcodedsecret.jellyfinLdapAdminPassword = {
      request = config.shb.jellyfin.ldap.adminPassword.request;
      settings.content = config.shb.hardcodedsecret.ldapAdminPassword.settings.content;
    };
  };

  ssoTestModule = {
    imports = [
      basicTestModule
      ssoModule
    ];

    shb.jellyfin = {
      ldap = {
        userGroup = "user_group";
        adminGroup = "admin_group";
      };
      sso = {
        enable = true;
        endpoint = "https://${config.shb.authelia.subdomain}.${config.shb.authelia.domain}";
        sharedSecret.result = config.shb.hardcodedsecret.jellyfinSSOPassword.result;
        sharedSecretForAuthelia.result = config.shb.hardcodedsecret.jellyfinSSOPasswordAuthelia.result;
      };
    };

    shb.hardcodedsecret.jellyfinSSOPassword = {
      request = config.shb.jellyfin.sso.sharedSecret.request;
      settings.content = "ssoPassword";
    };

    shb.hardcodedsecret.jellyfinSSOPasswordAuthelia = {
      request = config.shb.jellyfin.sso.sharedSecretForAuthelia.request;
      settings.content = config.shb.hardcodedsecret.jellyfinSSOPassword.settings.content;
    };
  };

  clientLoginTestModule = {
    imports = [
      clientLoginModule
    ];

    networking.hosts = {
      "192.168.1.2" = [
        "jellyfin.example.com"
        "auth.example.com"
      ];
    };
  };
in
{
  basic = pkgs.testers.runNixOSTest {
    name = "jellyfin_basic";
    nodes.server.imports = [ basicTestModule ];
    node.client.imports = [ clientLoginModule ];
    testScript = ...
  };

  ldap = pkgs.testers.runNixOSTest {
    name = "jellyfin_ldap";
    nodes.server.imports = [ ldapTestModule ];
    node.client.imports = [ clientLoginModule ];
    testScript = ...
  };

  sso = pkgs.testers.runNixOSTest {
    name = "jellyfin_sso";
    nodes.server.imports = [ ssoTestModule ];
    node.client.imports = [ clientLoginModule ];
    testScript = ...
  };
}

Here we setup Jellyfin to be accessible from jellyfin.example.com. We open up ports 80 and 443 so the client node can talk to the server node and we setup the client node so calls to Jellyfin and Authelia goes to the server node.

With all this behind us, we can finally write the testScript. Do you remember the test(page) Python function we created in the login_playwright executable? We will fill it out here.

page.goto("https://jellyfin.example.com")
page.get_by_label(re.compile('[Uu]ser')).fill("alice")
page.get_by_label(re.compile('[Pp]assword')).fill("AlicePassword")
page.get_by_role("button", name=re.compile('Sign In')).click()
expect(page.get_by_text(re.compile('[Ii]nvalid'))).not_to_be_visible()

Wouldn’t it be nice if it were this simple? Don’t get me wrong, this works, but it does not take into account all use cases nor all quirks of Jellyfin.

First, we need to decline these tests to handle the basic - no LDAP nor SSO integration - the LDAP integration and SSO integration cases. We then need to test the 3 users and make sure Alice can login, Bob can login with admin privileges and Charlie cannot connect. We also need to test the same 3 users but each with wrong passwords to make sure none can login. That makes for quite a few combinations in total.

The basic and LDAP integrations work the same way, they both use the service’s own login form:

We fill out the username and password.
We click the Sign In button.
We assert the user is logged in.

In code, a successful login looks like so:

def loginBasic(username, password):
  page.goto("https://jellyfin.example.com")
  page.get_by_label(re.compile('[Uu]ser')).fill(username)
  page.get_by_label(re.compile('[Pp]assword')).fill(password)
  page.get_by_role("button", name=re.compile('Sign In')).click()
  expect(page.get_by_text(re.compile('[Ii]nvalid'))).not_to_be_visible()
  expect(page.get_by_label(re.compile('^[Uu]ser'))).not_to_be_visible()

The expect combined with not_to_be_visible() is to make sure the popup saying the user could not log in has not shown up and that we’re not seeing the login form anymore.

And a bad login:

def loginBasicFail(username, password):
  page.goto("https://jellyfin.example.com")
  page.get_by_label(re.compile('[Uu]ser')).fill(username)
  page.get_by_label(re.compile('[Pp]assword')).fill(password)
  page.get_by_role("button", name=re.compile('Sign In')).click()
  expect(page.get_by_text(re.compile('[Ii]nvalid'))).to_be_visible()
  expect(page.get_by_label(re.compile('^[Uu]ser'))).to_be_visible()

Note that Jellyfin cannot create users declaratively so I’m only testing login with the admin user in that one. Having a declarative admin user is something I added myself too and upstreamed into Jellyfin.

The SSO integration has a different flow.

We first need to click on the SSO button
which redirects to the login flow.
We then we have to grant access
and then we can finally login.

Which in code looks like so:

def loginSSO(username, password):
  page.goto("https://jellyfin.example.com")
  with context.expect_page() as p:
    page.locator('text=Sign in with Authelia').click()
  page = p
  page.get_by_label('Username').fill(username)
  page.get_by_label('Password').fill(password)
  page.get_by_role("button", name=re.compile('Sign In')).click()
  page.get_by_text(re.compile('[Aa]ccept')).click()
  expect(page.get_by_text(re.compile('[Ii]nvalid'))).not_to_be_visible()
  expect(page.get_by_label(re.compile('^[Uu]ser'))).not_to_be_visible()

Although Jellyfin has an SSO plugin, it has no official way to show a “log in with SSO” button. To still show a button on the login page, we abuse the system designed to add your own branding to Jellyfin by adding the button there. Unfortunately that always opens the login page in a new tab, which we need to catch above with the context.expect_page() call.

Here, the fail login does not have the exact same flow. To avoid waiting for redirection, we navigate back to the Jellyfin base URL and make sure the login form shows up:

def loginSSOFail(username, password):
  page.goto("https://jellyfin.example.com")
  with context.expect_page() as p:
    page.locator('text=Sign in with Authelia').click()
  page = p
  page.get_by_label('Username').fill(username)
  page.get_by_label('Password').fill(password)
  page.get_by_role("button", name=re.compile('Sign In')).click()
  page.goto("https://jellyfin.example.com")
  expect(page.get_by_text(re.compile('[Ii]nvalid'))).to_be_visible()
  expect(page.get_by_label(re.compile('^[Uu]ser'))).to_be_visible()

Taking all this together, we can now complete the various testScripts:

# ...
{
  basic = pkgs.testers.runNixOSTest {
    name = "jellyfin_basic";
    nodes.server.imports = [ basicTestModule ];
    nodes.client.imports = [ clientLoginModule ];
    testScript = ''
      start_all()
      loginBasic("admin", config.shb.hardcodedsecret.jellyfinAdminPassword.settings.content)
      loginBasicFail("admin", "BadPassword")
    '';
  };

  ldap = pkgs.testers.runNixOSTest {
    name = "jellyfin_ldap";
    nodes.server.imports = [ ldapTestModule ];
    nodes.client.imports = [ clientLoginModule ];
    testScript = ''
      start_all()
      loginBasic("admin", config.shb.hardcodedsecret.jellyfinAdminPassword.settings.content)
      loginBasicFail("admin", "BadPassword")
      loginBasic("alice", config.shb.hardcodedsecret.alice.settings.content)
      loginBasicFail("alice", "BadPassword")
      loginBasic("bob", config.shb.hardcodedsecret.alice.settings.content)
      loginBasicFail("bob", "BadPassword")
      loginBasicFail("charlie", config.shb.hardcodedsecret.alice.settings.content)
      loginBasicFail("charlie", "BadPassword")
    '';
  };

  sso = pkgs.testers.runNixOSTest {
    name = "jellyfin_sso";
    nodes.server.imports = [ ssoTestModule ];
    nodes.client.imports = [ clientLoginModule ];
    testScript = ''
      start_all()
      loginSSO("alice", config.shb.hardcodedsecret.alice.settings.content)
      loginSSOFail("alice", "BadPassword")
      loginSSO("bob", config.shb.hardcodedsecret.alice.settings.content)
      loginSSOFail("bob", "BadPassword")
      loginSSOFail("charlie", config.shb.hardcodedsecret.alice.settings.content)
      loginSSOFail("charlie", "BadPassword")
    '';
  };
};

start_all() starts all VM nodes. In the SSO integration case, we don’t login with the admin user since it is not provisioned by the SSO integration. I aslo left out how to import the login* Python function for brevity.

The actual test framework is actually organised a bit differently to provide more reusable pieces and accommodate to the various particularities of the different services. You can take a look at the real jellyfin test file and the real common testing framework by clicking on the links.

Conclusion?

I know, the conclusion is not at the end of the blog post, but we did achieve our original goal now!

We can finally make sure no update will break our previous integrations with LDAP and SSO providers without us noticing.

But let’s continue a bit, we’re not totally done yet.

Debugging

Thanks to saving the traces, it’s easy to figure out what’s wrong. After running a test, you can see the traces with:

$ nix build .#checks.x86_64-linux.vm_jellyfin_https -L --keep-failed

$ ls -1 result/trace/
0.zip
1.zip
2.zip
3.zip

$ nix run github:ibizaman/selfhostblocks#playwright show-trace result/trace/0.zip

The --keep-failed is necessary to use if the test failed, because it will then print where the temp files are locate. In that case, you can access them with:

$ sudo cp -r /nix/var/nix/builds/nix-256919-1305278864/build trace && sudo chown -R $USER: trace

$ nix run github:ibizaman/selfhostblocks#playwright show-trace trace/shared-xchg/trace/0.zip

You then get this nice detailed view of everything that happened from the browser’s perspective.

It’s usually best to start with this view and then look at the test logs, to at least narrow down where to look.

Bonus Test

But that’s not all folks. Something I noticed in the Nextcloud service is that one can correctly setup the LDAP and SSO integration but still make them inconsistent with each other. What I mean is if you set up both integrations and login once using the LDAP flow and once through the SSO flow, you might end up with two different users!

Now, how do you test something like this? Well, NixOS provides us with something quite unique, specializations. Think of those as parallel configurations for our NixOS host that we can switch between at runtime, without needing to deploy again.

So let’s create this final test:

# ...
{
  users = pkgs.testers.runNixOSTest {
    name = "jellyfin_users";

    nodes.server = {
      imports = [
        basicTestModule
      ];

      specialisation.ldap.configuration = {
        imports = [ ldapTestModule ];
      };

      specialisation.sso.configuration =
        { config, ... }:
        {
          imports = [ ssoTestModule ];

          # https://github.com/ibizaman/selfhostblocks/issues/843
          shb.jellyfin.admin.username = lib.mkForce "jellyfin2";
       };
    };

    nodes.client = {
      imports = [
        clientLoginModule
      ];
    };

    testScript =
      let
        specializationsServer = "${nodes.server.system.build.toplevel}/specialisation";
      in
      ''
        def switch_to_specialization(name):
            with subtest(f"switch specialization to {name}"):
                server.succeed(f'${specializationsServer}/{name}/bin/switch-to-configuration test')
                server.wait_for_unit("multi-user.target")

        start_all()
        server.wait_for_unit("multi-user.target")

        switch_to_specialization("ldap")

        # This sleep is needed because Jellyfin reports it is ready before it truly is ready.
        # See ticket https://github.com/ibizaman/selfhostblocks/issues/842
        print("sleeping 60 seconds...")
        import time
        time.sleep(60)

        with subtest("find alice"):
            users = json.loads(server.succeed("sqlite3 /var/lib/jellyfin/data/jellyfin.db -json 'SELECT * FROM Users;'"))
            aliceUsers = [u for u in users if u["Username"] == "alice"]
            if len(aliceUsers) != 1:
                raise Exception(f"Unexpected number of users for alice, got {len(aliceUsers)}\n{json.dumps(users, indent=4)}")
            alice = aliceUsers[0]
            print(f"Users: \n{json.dumps(users, indent=4)}")
            if alice["AuthenticationProviderId"] != "Jellyfin.Plugin.LDAP_Auth.LdapAuthenticationProviderPlugin":
                raise Exception("Unexpected authentication provider id, got {alice['AuthenticationProviderId']}")
            if alice["PasswordResetProviderId"] != "Jellyfin.Plugin.LDAP_Auth.LdapAuthenticationProviderPlugin":
                raise Exception("Unexpected password resiet provider id, got {alice['PasswordResetProviderId']}")

        switch_to_specialization("sso")
        time.sleep(60)

        with subtest("find alice"):
            users = json.loads(server.succeed("sqlite3 /var/lib/jellyfin/data/jellyfin.db -json 'SELECT * FROM Users;'"))
            aliceUsers = [u for u in users if u["Username"] == "alice"]
            if len(aliceUsers) != 1:
                raise Exception(f"Unexpected number of users for alice, got {len(aliceUsers)}\n{json.dumps(users, indent=4)}")
            alice = aliceUsers[0]
            print(f"Users: \n{json.dumps(users, indent=4)}")
            if alice["AuthenticationProviderId"] != "Jellyfin.Plugin.LDAP_Auth.LdapAuthenticationProviderPlugin":
                raise Exception("Unexpected authentication provider id, got {alice['AuthenticationProviderId']}")
            if alice["PasswordResetProviderId"] != "Jellyfin.Plugin.LDAP_Auth.LdapAuthenticationProviderPlugin":
                raise Exception("Unexpected password resiet provider id, got {alice['PasswordResetProviderId']}")
      '';
  };
}

The meat of the test is the if clause len(aliceUsers) != 1. This makes sure only one alice is ever found in the database. There are some extra checks to make sure Alice was created with the LDAP provider but those are maybe superfluous, we’ll see in the long run.

In the test, we create 2 specializations. One for LDAP integration and one for SSO. We switch from one to the other mid test. So on top of making sure only one use is created, we make sure changing the configuration works fine too.

You’ll also notice comments linking to issues 842 and 843. Those are quirks of Jellyfin I needed to circumvent. One day I’ll get back and fix them.

Conclusion

We’re really at the end here. If you followed along, well, thanks! It was a long and frustrated journey to get this result but I don’t regret it. I like those tests and plan to cover more services.

Feel free to check the repo or join the Matrix chat to talk about it.

ESPSomfy on Atom Lite with NixOS

2026-07-04T00:00:00Z

Tags: nix, esp, iot

I have an Atom Lite and wanted to use it to install ESPSomfy-rts on it with a E07-M1101D antenna.

The goal is to control Somfy covers with Home Assistant.

If you have the same setup, you might enjoy some opinionated instructions. So these are the steps I followed.

Installation

These are the steps are followed to flash my Atom Lite.

1. Plug the Atom Lite in the computer

With a USB cable!

2. Find the Atom Lite

If you have multiple ttyUSB devices, repeat the command after disconnecting the Atom Lite and see which one disappeared.

$ ls -l /dev/ttyUSB*
crw-rw---- 1 root dialout 188, 0 Jun  4 12:02 /dev/ttyUSB0

3. Enter shell with needed tools

Yay for nix!

nix-shell -p unzip curl esptool

4. Check Atom Lite is flashable

For good measure but I suppose you could skip this step.

$ sudo esptool --port /dev/ttyUSB0 flash-id
esptool v5.2.0
Connected to ESP32 on /dev/ttyUSB0:
Chip type:          ESP32-PICO-D4 (revision v1.1)
Features:           Wi-Fi, BT, Dual Core + LP Core, 240MHz, Embedded Flash, Vref calibration in eFuse, Coding Scheme None ta
Crystal frequency:  40MHz
MAC:                64:b7:08:b7:98:08

Stub flasher running.

Flash Memory Information:
=========================
Manufacturer: c8
Device: 4016
Detected flash size: 4MB
Flash voltage set by a strapping pin: 3.3V

Hard resetting via RTS pin...

5. Erase the Atom Lite

I’m not sure this is needed but it won’t hurt.

$ sudo esptool --port /dev/ttyUSB0 erase-flash

6. Download the firmware

From the release page, download the *.onboard.esp32.bin.zip archive or copy paste the curl command and update the release version:

$ curl -LO https://github.com/rstrouse/ESPSomfy-RTS/releases/download/v2.4.7/SomfyController.onboard.esp32.bin.zip
$ unzip SomfyController.onboard.esp32.bin.zip
Archive:  SomfyController.onboard.esp32.bin.zip
  inflating: SomfyController.onboard.esp32.bin

7. Flash the Atom Lite

This step took about a minute for me.

$ sudo esptool --port /dev/ttyUSB0 \
    write-flash 0x0 SomfyController.onboard.esp32.bin
esptool v5.2.0
Connected to ESP32 on /dev/ttyUSB0:
Chip type:          ESP32-PICO-D4 (revision v1.1)
Features:           Wi-Fi, BT, Dual Core + LP Core, 240MHz, Embedded Flash, Vref calibration in eFuse, Coding Scheme None
Crystal frequency:  40MHz
MAC:                64:b7:08:b7:98:08

Stub flasher running.

Configuring flash size...
Flash will be erased from 0x00000000 to 0x003effff...
Wrote 4128768 bytes (929394 compressed) at 0x00000000 in 91.0 seconds (363.1 kbit/s).
Hash of data verified.

Hard resetting via RTS pin...

8. Verify Installation

Now, the Atom Lite will create its own WiFi access point with name ESPSomfyRTS.

Hardware

1. Connect to Atom Lite’s WiFi AP

Connect to the ESPSomfyRTS network.

2. Load ESPSomfyRTS web interface

Go to http://192.168.4.1. The IP address is hardcoded. The webpage should show “RADIO NOT INITIALIZED”.

3. Configure device

Click on the gear icon. Go through the all the tabs first, we’ll configure the radio afterwards.

Don’t forget to click “save” before switching tab otherwise your changes will be lost.

You know what? Click twice on “save”, for good measure.

System > Options

System > Security

I store the username and password in Bitwarden (using my Vaultwarden self-hosted server).

Network > Adapter

You don’t need to setup WiFi, you could always connect to the device through its ESPSomfyRTF WiFi AP but then you’ll lose on the Home Assistant integration. So I recommend doing it.

Somfy

Skip the Somfy tab for now, we need to setup the radio first.

4. Wiring

Radio > Transceiver

We will now configure the pinout with the following wiring. I chose this wiring specifically to make the physical wiring as simple as possible, considering the antenna would be located underneath the Atom Lite, with a breadboard in between.

The pinout is shown here:

Combining both diagrams, we can see a quite neat wiring:

The correspondance is:

Antenna	Atom Lite	ESPSomfyRTS
(1) Ground	Ground	-
(2) 3V3	3V3	-
(3) GD0	GPIO 33	TX
(4) CSN	GPIO 23	CSN
(5) SCK	GPIO 25	SCLK
(6) MOSI	GPIO 19	MOSI
(7) MISO	GPIO 21	MISO
(8) GDO2	GPIO 22	RX

Now, we can set the pinout correspondance in the web UI:

5. Test

I’ll do first a test using a solderless breadboard. I’m using the male and female socket headers I’ll use later when soldering.

The breadboard setup looks like so. I had a hard time wrapping my head around where to put the wires because I needed to mirror the diagram from above.

Now, let’s enable the radio in the Radio > Transceiver tab in the web interface by toggling the “Enable Radio” checkbox. Then, click save.

To see if everything works fine, go to the Radio > Logs tab and press on the official Somfy remote, you should see some log lines like so:

You could also configure already the rooms and covers but I’ll leave that for after the soldering.

6. Soldering

Since everything went well, it’s time to solder.

Comparing the breadboard used for soldering to the diagram, you’ll see there’s an extra connection that we need to get rid of between the GD0 pin from the antenna and the 5V pin from the Atom Lite. I marked it in red.

Note the diagram is mirrored at the pink line.

I used a drill with the smallest wood drill bit I had to cut the connection.

And a dremel to cut the female header sockets for the antenna.

Now that we have all the pieces set, let’s solder!

We will need two wires to connect the antenna and the Atom Lite. After decades of stripping wires with a pair of scissors, I invested in a wire stripper and what a quality of life improvement that is!

I did a horrendous job at the soldering so I will only allow you to look it from afar.

I did cut the extra board with a dremel.

The finished product

Next Steps

Now we can plug back the Atom Lite in a USB charger and configure the covers.

I’ll leave you to the official wiki because now you have all that’s needed to follow it.

Bisect Experiment at the PR Level

2026-05-12T00:00:00Z

Tags: nix, forge action, auto update

First Automation Attempt
Naive Commit Selection
Smarter Commit Selection
Manual Run
Links
Possible Improvement
Upstreaming

I maintain the SelfHostBlocks project. It relies on nixos/nixpkgs/unstable as its sole dependency and it attempts to follow the latest commit as close as possible.

If you’re wondering why, it’s because I like the shiny new stuff.

First Automation Attempt

To automate this update process, I initially created a crude workflow which worked like follows:

Workflow runs at midnight. It runs nix flake update nixpkgs and if this results in a change in the flake.lock file, it commits that and creates a PR with the automerge label. If a PR already exists, it instead force pushes to that PR.
When the PR is created with the automerge label, the auto-merge feature of Github gets activated.
Custom NixOS VM tests run on this PR and if they all succeed, the flake.lock update PR gets merged automatically.

I wrote more extensively about this in another blog post.

Naive Commit Selection

Although this is automated and did produce PRs that get auto-merged, more often than not the commit selection is too naive. Constantly trying to update to the tip of unstable can lead to test failure.

The most frequent reasons for failure are:

A package I rely on cannot be built.
A module option got updated leading to some breaking change.
The behavior of a module changed and broke an assumption I had.

The workflow being naive does not realize the PR it had created has a failure and it will happily update the flake.lock file to the latest unstable commit. That might fix the failure when a package cannot be built but will never fix the other two.

So when that happens, this update PR will be failing until I take a look at it which is annoying.

Smarter Commit Selection

I changed the workflow to check if the PR already existed and if so to check its test status.

If the tests are failing, it will then clone the nixpkgs repo and choose the commit in the middle between the last good commit (the tip of my project’s main branch) and the latest failing commit (the PR’s nixpkgs commit).

It then uses this new middle commit to run a flake update similarly as before and pushes that change to the existing PR.

nix flake update nixpkgs \
  --override-input \
    nixpkgs \
    github:nixos/nixpkgs/$commit

If the PR still fails, on next run it will choose again a commit in the middle between the last good commit and the last failing one.

If the PR succeeds, then it gets auto-merged and the cycle repeats. The workflow will see no PR is opened so it will update the flake.lock file using the latest commit on unstable.

If the tests in the PR continuously fail, there might be no more commit to bisect on as the good commit will become the parent of bad commit. In that case, the workflow will go back to the latest commit on unstable and will try anew.

Manual Run

I’m running the workflow every 3 hours but it is possible to run the workflow manually to speed up the process. This will be useful when I’m catering it and trying to fix the issues that appear.

When running the workflow manually, I also added an option to bisect a commit in the future. That is to choose a commit between the currently failing one in the PR and the tip of nixpkgs unstable.

Possible Improvement

Always choosing a commit in the past is not necessarily the best strategy. It could be worth parsing the log output and figuring out what failed. Indeed, when a package fails to build, choosing a commit in the future might work too as a broken package is usually fixed in the next couple of days at most.

Choosing a commit in the future is possible thanks to the manual run mode but this could be made automatic.

Better selection of commits could be helped by using one of the tools in the wiki. Hydrasect is particularly interesting to be able to pick a commit with cached derivations.

Upstreaming

I’m not even sure this is a good idea and will be useful in practice. Time will tell.

If others still might like a tool like this, it could be extracted as its own repo but it should be made generic first. It should be able to let the user choose which flake input to update. It is hardcoded to “nixpkgs” now.

I’m also not sure how to best handle updating multiple inputs as the number of combination of commits to try would explode quickly.

Use Bitwarden as auth source in Emacs

2026-05-01T00:00:00Z

Tags: nix, emacs

1. Setup Emacs
2. Setup nix config
3. Configure secret
Misc

From Magit, it is possible to create pull requests. Of course, we need to setup a secret token to be able to create a pull request on the forge.

One way is to store the secret in a file read by emacs. By default, auth-sources variable is:

("~/.authinfo"
 "~/.authinfo.gpg"
 "~/.netrc")

The issue is these files store the secrets in clear text. Instead, let’s configure emacs to read the secret from Bitwarden.

1. Setup Emacs

We add usage of the emacs-bitwarden package in our init.el file:

(use-package bitwarden
  :ensure t
  :config
  (bitwarden-auth-source-enable))

The package is only available on github so we’ll need to package it ourselves in the next section.

2. Setup nix config

I use the emacs-overlay NixOS module to install emacs. Assuming the following base nix config:

let
  emacsWithPackages = pkgs.emacsWithPackagesFromUsePackage {
    config = emacs-conf + "/init.el";
    defaultInitFile = true;
    package = pkgs.emacs-unstable-pgtk;
  };
in
  config = {
    nixpkgs.overlays = [
      emacs-overlay.overlays.emacs
    ];

    environment.systemPackages = [
      emacsWithPackages
    ];
  };
};

We can then compile the package ourselves:

 let
   emacsWithPackages = pkgs.emacsWithPackagesFromUsePackage {
     config = emacs-conf + "/init.el";
     defaultInitFile = true;
     package = pkgs.emacs-unstable-pgtk;
 
+    override = final: prev: {
+      bitwarden = final.melpaBuild {
+        pname = "bitwarden";
+        version = "0.1.0";
+        src = pkgs.fetchFromGitHub {
+          owner = "seanfarley";
+          repo = "emacs-bitwarden";
+          rev = "50c0078d356e0ac0bcaf26b40113700ba4123ec3";
+          hash = "sha256-5zAkoCdBDI7sNLtxOy4t91A4IGV84lD3Cz5nnsQ0P4Q=";
+        };
+      };
+    };
+  };
 in
   config = {
     nixpkgs.overlays = [
       emacs-overlay.overlays.emacs
     ];

     environment.systemPackages = [
       emacsWithPackages
+      pkgs.bitwarden-cli
     ];
   };
 };

3. Configure secret

Now we have an emacs with the emacs-bitwarden package loaded. We can verify this by looking at auth-sources:

(bitwarden
 "~/.authinfo"
 "~/.authinfo.gpg"
 "~/.netrc")

Now, in Magit, we can open a repository and try to create a pull request:

M-x forge-create-pullreq

After writing the description, I tried to create the pull request with C-c C-c but got this error message:

ghub--token: Required Github token
("ibizaman^forge" for either "api.github.com"
or "api.github.com") does not exist.

This tells me exactly what secret I need to add in Bitwarden.

First, let’s create a PAT in Github in https://github.com/settings/personal-access-tokens with fine-grained permissions and give the following permissions:

Administration - write
Pull Requests - write

Now, we store this value in Bitwarden in a secret with:

Username = ibizaman^forge
Password = PAT from Github
Website = api.github.com
Name = Name can be anything of your choosing.

Before being able to see the password, auth-sources will need some cleanup:

M-x auth-source-forge-all-cached
M-x bitwarden-login
M-x bitwarden-unlock

See next section for more explanation on those options.

Misc

A few more random tips.

Clear auth-sources cache

Emacs aggressively caches secrets, even failure to find secrets. Reset the cache with M-x auth-source-forget-all-cached.

Self-hosted Bitwarden

To use a self-hosted Bitwarden or Vaultwarden, run on the command line:

bw config server https://<fqdn>

Log into vault

To login into bitwarden, run on the command line:

bw login

or in emacs:

M-x bitwarden-login

Unlock vault

Afterwards, you will need to unlock the vault with one of the following commands. You will for example need to do this on reboot too.

bw unlock

or in emacs:

M-x bitwarden-unlock

Sync vault

After updating a secret, to synchronize the vault, run either:

bw sync

or in emacs:

M-x bitwarden-sync

Qemu tips for cdrom, ide and nvme hard drives

2025-06-01T00:00:00Z

Tags: qemu, nix

Nix
CDROM (sr0)
Iso file in Nix
EFI
SATA (sdX) drive
NVMe (nvmeX) drive

Qemu’s documentation is a bit scattered around the web.

I managed to follow the nvme documentation but just couldn’t find anything in qemu.org about SATA drives apart from some small snippets in other sections.

Some other examples could be found on StackOverflow.

This blog post regroups snippets geared towards concrete examples to start a Qemu VM with EFI, SATA, NVMe and/or cdrom.

Nix

You can test with:

nix run nixpkgs#qemu --

For EFI though, using the following script will help you to get easily the needed firmware:

let
  nixos-qemu = pkgs.callPackage "${pkgs.path}/nixos/lib/qemu-common.nix" {};
  qemu = nixos-qemu.qemuBinary pkgs.qemu;
in
  pkgs.writeShellScriptBin "script" ''

  ${qemu} ... 
  '';

We purposely use the facilities provided by qemu-common.nix because all the wiring is done for us already for EFI and some options matching our architecture are set.

CDROM (sr0)

Assuming you have a ISO file available at ./iso. See next section if you want to create an ISO file with nix.

--drive media=cdrom,format=raw,readonly=on,file=./iso

It will show up as sr0.

Iso file in Nix

With nixos-generators:

let
  iso = nixos-generators.nixosGenerate {
    inherit system;
    format = "install-iso";

    modules = [
    ];
  };
in

EFI

To boot using EFI, add:

--drive if=pflash,format=raw,unit=0,readonly=on,file=${pkgs.OVMF.firmware}

Here I’m using the Nix snippet to get the OVMF firmware.

SATA (sdX) drive

With the following variable:

diskSata1=./diskSata1.qcow2

Create a drive:

${qemu} create -f qcow2 $diskSata1 20G

Then use it as a SATA drive:

--drive format=qcow2,file=$diskSata1,if=none,id=diskSata1 \
--device ide-hd,drive=diskSata1

Pay attention that the values for id= and drive= can be arbitrary but must match.

It will show up as a sdX drive:

$ lsblk
sda

To add another drive as sdb, use the same method but create a new file and replace occurrences of diskSata1 in id= and drive= with another value.

To specify the device name yourself, add serial= to the --device options.

NVMe (nvmeX) drive

The method for NVMe drives is exactly the same as for SATA drives but replace --device ide-hd,... with --device nvme,....

Epilogue

I’m using this in my project Skarabox whose goal is to be the fastest way to install NixOS on host with all bells and whistles included.

Specifically, I’m using a qemu VM to test the installation on a VM.

Access a Host Through SSH on NixOS

2025-03-23T00:00:00Z

Tags: nix, server

Scaffolding
Generate a SSH keypair
Configuration for the Server Host
Configuration for the Client Host
Cleanup
Conclusion

Let’s see how to grant ourselves ssh access form a client host to a server host using NixOS.

This post assumes you can already deploy to the server host, by means of another ssh key. So this is not about bootstrapping a server. Here, we’ll see how to grant another user access to the host, for example a backup user with reduced access.

Scaffolding

For this post, I’ll be using:

flakes,
deploy-rs to deploy on the server host,
nixos-rebuild to deploy on the client host,
sops-nix to store the ssh private key encrypted
and home-manager to configure the client host.

This requires a bit of trial and error to get all these pieces to fit together, so here is a flake that defines the scaffolding for both the client and the server host:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";

    sops-nix.url = "github:Mic92/sops-nix";
  };

  outputs = { self, nixpkgs, home-manager, sops-nix }: let
    system = "x86_64-linux";

    specialArgs = {
      serverUser = "vorta";
      clientUser = "me";
      serverHost = "server";
      clientHost = "client";
    };
  in {
    nixosModules.server = {
      imports = [
        sops-nix.nixosModules.default
        ./server.nix
      ];
    };

    nixosConfigurations.server = nixpkgs.lib.nixosSystem {
      inherit system specialArgs;
      modules = [ self.nixosModules.client ];
    };

    nixosModules.client = {
      imports = [
        home-manager.nixosModules.default
        sops-nix.nixosModules.default
        ./client.nix
      ];
    };

    nixosConfigurations.client = nixpkgs.lib.nixosSystem {
      inherit system specialArgs;
      modules = [ self.nixosModules.client ];
    };
  };
}

We use specialArgs here to pass arguments to both the client and the server configuration instead of copying values around.

Deploying to the server host is done with:

nix run nixpkgs#deploy-rs .#server

# Or, if deploy-rs is installed on your system:
deploy .#server

Deploying to the client host is done with:

sudo nixos-rebuild --flake . switch

Generate a SSH keypair

This step is still manual:

nix shell nixpkgs#openssh --command \
  ssh-keygen -t ed25519 -f mykey

Pick no passphrase if you intend this user to get automated access.

This will create two files, mykey with the private key and mykey.pub with the public key. We will use both in the later sections.

Configuration for the Server Host

In this example, I create a new vorta user to allow access for backups by the VortaBackup software into the folder /srv/backup/vorta/me.

This goes in ./server.nix:

{ config, serverUser, clientUser, ... }:
{
  users.users.${serverUser} = {
    isSystemUser = true;
    # Needed to be able to log in.
    useDefaultShell = true;

    home = "/srv/backup/${serverUser}/${clientUser}";
    homeMode = "770";
    createHome = true;
    group = "backup";

    isSystemUser = true;
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAAA..."
    ];
  };
  users.groups.backup = {};
}

The users.users.${serverUser}.openssh.authorizedKeys.keys field contains the public key from the mykey.pub file.

If you intend to keep the file laying around, you could instead do:

users.users.${serverUser}.openssh.authorizedKeys.keys = [
  (builtins.readFile ./mykey.pub)
];

Both versions give the same result and it’s really a matter of taste here.

Configuration for the Client Host

Copy the private key from mykey under some yaml field in your sops file. I’ll assume it’s under ssh/server/client-me-vorta. The convention is ssh/${serverHost}/${clientHost}-${clientUser}-${serverHost} and is an organized to let me have multiple access per server-client pair.

This goes in ./client.nix:

{ config, serverUser, clientUser, serverHost, clientHost, ... }:
{
  sops.secrets."ssh/${serverHost}/backup" = {
    owner = user;
    path = "/home/${clientUser}/.ssh/${serverHost}-${serverUser}";
    key = "ssh/${serverHost}/${clientHost}-${clientUser}-${serverHost}";
  };

  home-manager.users.${clientUser} = {
    programs.ssh = {
      enable = true;
      matchBlocks = {
        "${serverHost}-${serverUser}" = {
          user = serverUser;
          hostname = serverHost;
          identityFile = config.sops.secrets.
            "ssh/${serverHost}/backup".path;
        };
      };
    };
  };
}

I use the key field of sops-nix to give a nickname to the secret, making it a bit easier to recall later in the home-manager config. This is optional.

Cleanup

The mykey files laying around this way is not good. Let’s delete them securely with the shred tool:

nix shell coreutils --command shred mykey
nix shell coreutils --command shred mykey.pub
rm mykey
rm mykey.pub

Omit the mykey.pub file if you used the readFile version of the code.

Conclusion

After deploying, we now can access the server with the new user with:

ssh ${serverHost}-${serverUser}

With the example values I chose above, it’s:

ssh server-vorta

Enable Hibernation on NixOS

2025-03-10T00:00:00Z

Tags: nix

Hibernation was broken for me since a long time. I postponed trying to fix it for such a long time. Then, I stumbled onto this post which had the fix!

To celebrate the finding, I’m sharing a NixOS module to enable hibernation.

As a prerequisite, you need to create a swap partition, as noted in the NixOS wiki.

As for the module, here it is:

{ config, pkgs, lib, ... }:

let
  cfg = config.base;
in
{
  options.base = {
    hibernation = lib.mkOption {
      description = ''
        Options to configuration hibernation.
      '';
      type = lib.types.submodule {
        options = {
          enable = lib.mkEnableOption "hibernation";

          device = lib.mkOption {
            type = lib.types.str;
            description = ''
              Device used to store hibernation
              information. Use lsblk to find it.
            '';
            example = "/dev/disk/by-label/swap";
          };

          hibernateAfterSleepDelay = lib.mkOption {
            type = lib.types.str;
            description = ''
              Hibernate after sleeping for this long.
            '';
            default = "2h";
          };
        };
      };
    };
  };

  config = {
    powerManagement.enable = true;

    # Specifies where the hibernation info will be stored.
    boot.kernelParams = [
      "resume=${cfg.hibernation.device}"
    ];

    # Allow hibernation
    security.protectKernelImage = !cfg.hibernation.enable;

    # Enable hibernation in menus
    environment.etc = lib.mkIf cfg.hibernation.enable {
      "/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla".text = ''
        [Re-enable hibernate by default in upower]
        Identity=unix-user:*
        Action=org.freedesktop.upower.hibernate
        ResultActive=yes

        [Re-enable hibernate by default in logind]
        Identity=unix-user:*
        Action=org.freedesktop.login1.hibernate;org.freedesktop.login1.handle-hibernate-key;org.freedesktop.login1;org.freedesktop.login1.hibernate-multiple-sessions;org.freedesktop.login1.hibernate-ignore-inhibit
        ResultActive=yes
        '';
    };

    # Set suspend-then-hibernate as defaults
    services.logind = lib.mkIf cfg.hibernation.enable {
      lidSwitch = "suspend-then-hibernate";
      extraConfig = ''
        HandlePowerKey=suspend-then-hibernate
        IdleAction=suspend-then-hibernate
        IdleActionSec=2m
      '';
    };

    # 
    systemd.sleep.extraConfig = lib.mkIf cfg.hibernation.enable
      "HibernateDelaySec=${cfg.hibernation.hibernateAfterSleepDelay}";
  };
}

Happy hibernation!

Parental Control with Jellyfin and Radarr/Sonarr

2025-03-01T00:00:00Z

Tags: jellyfin

The goal is to create a user in Jellyfin that can only see media files that have a given tag, say “kids”, and that tag was set inside Radarr, Sonarr or another *arr application.

*arr Configuration

Within Settings > Metadata > Kodi, check “Enable” and “Movie Metadata”.
Then, when adding or editing a movie, add a tag to mark media that children can watch, like “enfants”.

This will create a .nfo file that includes the tag:

<movie>
  ...
  <tag>enfantstag>
  ...
movie>

If you add this tag to an existing movie that is already synced with Jellyfin, click the “Refresh & Scan” button.

Jellyfin Configuration

In Jellyfin, under Administration > Dashboard > Users, create a user and in “Parental Control” tab, only allow items with the tag chosen above:

The tag seems to be case-insensitive.

That’s it!

Pre-Seeded Known Host to Access New Server

2024-11-28T00:00:00Z

Tags: nix

Why Skipping this Step is a Good Idea
Why Does Host Key Verification Exist
Pre-Seed the Host Key

This post goes over how to create a new server and be able to ssh into it without needing to go through the ssh host verification manual step.

This step is annoying, error prone and tedious enough that I’m sure most of us answer “yes” to the question as fast as when reading some Terms of Services.

To skip to the solution, click here.

Why Skipping this Step is a Good Idea

As said in the preamble, this step is tedious and error prone. One needs to extract the newly generated public key from the new server, then generate the fingerprint of the key and compare it visually with the one printed by SSH.

Usually, extracting the public key requires accessing the server in some less secure way, by for example using the username/password protocol to SSH into the server. This is not ideal.

I would even argue this step is inherently insecure especially in a cloud environment because how can you be really sure this public key/fingerprint is from the server you just booted unless it’s physically in front of you?

Why Does Host Key Verification Exist

In NixOS, when OpenSSH’s systemd unit starts for the first time, it generates some SSH keys using ssh-keygen by default at /etc/ssh/ssh_host_rsa_key and /etc/ssh/ssh_host_ed25519_key.

This step is also done in most other distributions.

The location, number and flags used to generate the keys can be changed with the services.openssh.hostKeys option.

Now, let’s assume one can SSH into the server by having shared their own private key with the server. On the first attempt, SSH will prompt them with the following question:

The authenticity of host '[XXX.XXX.XXX.XXX]:22
([XXX.XXX.XXX.XXX]:22)' can't be established.
ED25519 key fingerprint is
SHA256:YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY.
Are you sure you want to continue connecting
(yes/no/[fingerprint])?

This fingerprint corresponds to the public key of one of the SSH keys generated by OpenSSH. One can see a key fingerprint with:

$ ssh-keygen -lvf <public key file>

If they say yes, the following will get printed:

Warning: Permanently added '[XXX.XXX.XXX.XXX]:22'
(ED25519) to the list of known hosts.

And indeed, a new entry is added to their ~/.ssh/known_hosts file:

$ cat ~/.ssh/known_hosts | grep XXX.XXX.XXX.XXX
[192.168.1.30]:22345 ssh-ed25519 ZZZZZ...

The whole reason is to protect them if, in the future, the host changes without them knowing about it. There are a myriad of reasons for this, some innocuous and some malevolent. Anyway, the next time they’ll try to connect, they will see this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW.
Please contact your system administrator.
Add correct host key in /home/them/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/them/.ssh/known_hosts:1
Host key for [XXX.XXX.XXX.XXX]:22 has changed and you have requested strict checking.
Host key verification failed.

Essentially, it’s saying the fingerprint of the host’s public key changed and warns that it could be potentially a bad sign.

If they happen to know this is okay, because for example they indeed generated a new host key for some reason, getting rid of the fingerprint saved the first time and restarting the host key verification procedure is done by deleting the entry in the ~/.ssh/known_hosts file with:

ssh-keygen -R "[XXX.XXX.XXX.XXX]:22"
# Host [XXX.XXX.XXX.XXX]:22 found: line 1
/home/them/.ssh/known_hosts updated.
Original contents retained as
/home/them/.ssh/known_hosts.old

Pre-Seed the Host Key

We can generate the SSH key pair with:

$ ssh-keygen -N "" -f hostkey

Then, assuming we use SOPS with sops-nix to handle secrets, we add the content of the hostkey and hostkey.pub files to our SOPS secrets file:

hostkey:
  privateKey: XXXX...
  publicKey: YYYY...

Finally, the two NixOS snippets for the remaining work are for the server we just let SOPS populate the correct location by using the path option:

sops.secrets."hostkey/privateKey" = {
  path = "/etc/ssh/ssh_host_rsa_key";
};

And for our laptop we add it to our system known hosts:

sops.secrets."hostkey/publicKey" = {};

programs.ssh.knownHosts."myserver".publicKeyFile =
  sops.secrets."hostkey/publicKey".path;

Display Pictures from Nextcloud to a Kodi Photo Frame

2024-09-03T00:00:00Z

Tags: nix, nextcloud, systemd, iot, server

So I wanted to show family pictures on the TV. My current setup is:

a server using NixOS to host Nextcloud,
an OrangePi 3 with Libreelec and Kodi,
and mobile phones with the iOS Nextcloud app.

What I wanted is to be able to select, from Nextcloud, which pictures to show on the TV and those would appear there automatically. This post goes over how I did it.

Btw, I’m hosting Jellyfin and using the Jellyfin For Kodi plugin on the OrangePi but this does not matter for the following setup.

Wanted User Experience

To show pictures:

Pictures are uploaded from the iOS app to Nextcloud.
User goes over pictures and shares good ones with the Photoframe Nextcloud user.
Wait for screensaver on OrangePi box to start and see new pictures!

To remove pictures:

Unshare pictures with Photoframe Nextcloud user.

Setup

I didn’t want the OrangePi to connect to the Nextcloud server using WebDav. I tried that first and got into scenarios where I was sharing so many pictures that the screensaver could not load them correctly. There are multiple reasons for this and one can search to optimize them. But this got me thinking, why couldn’t I instead copy all the shared pictures to the OrangePi directly? This makes the solution very robust to any networking mishap. And that’s what I did.

On Nextcloud

I created a Photoframe user in Nextcloud. I logged in with that user once and configured it to automatically accept incoming shares.
With my user, I shared some pictures with that new user to test that I could and that the Photoframe user would see them.

On the Server

The idea here is to rsync the shared pictures to the OrangePi box. This implies that:

rsync has access to the shared pictures only.
rsync can ssh into the OrangePi box.
rsync is installed on the server and the OrangePi box.

To make rsync access the shared pictures only, I mounted the Photoframe Nextcloud folder through WebDav in a directory on the server. This is done using this Self Host Blocks module and with the following config on my server:

shb.davfs.mounts = [
  # Mount a WebDav folder in the /srv/photoframe.
  {
    remoteUrl = "https://$MYDOMAIN/remote.php/dav/files/photoframe";
    mountPoint = "/srv/photoframe";
    username = "photoframe";
    passwordFile = config.sops.secrets."photoframe".path;
    uid = 992;
    gid = 991;
  }
];
# Password for Photoframe user.
# For now, it must be in format:
#
#     
#
# In this example, it is:
#
#   /srv/photoframe photoframe XHsbaf...
#
sops.secrets."webdav/nextcloud" = {
  sopsFile = ./secrets.yaml;
  mode = "0600";
  path = "/etc/davfs2/secrets";
};
users.groups.photoframe = {
  # Must correspond to the gid above.
  gid = 991;
};
users.users.photoframe = {
  isSystemUser = true;
  # Must correspond to the uid above.
  uid = 992;
  group = "photoframe";
  home = "/var/lib/photoframe";
  createHome = true;
  packages = [
    pkgs.rsync
  ];
};

I could verify this worked by making sure the secret looked good with cat /etc/davfs2/secrets and also by seeing that the /srv/photoframe directory was created and not empty. In case of error, check systemctl status srv-photoframe.mount.

To be able to ssh into Kodi, I needed to create an ssh key pair. So I ran the following and got two files ssh-orangepi and ssh-orangepi.pub:

ssh-keygen -t ed25519 -N "" -f ssh-orangepi

I wrote the private key in my Sops config. I copied over the public key into the OrangePi’s /root/.ssh/authorizedKeys file. I sshed once from the server to the OrangePi to accept the host key fingerprint. Finally, I could put a cron job that would run rsync on a schedule:

systemd.services.sync-to-orangepi = {
  description = "Sync Pictures to OrangePi";
  after = [ "network.target" "srv-photoframe.mount" ];
  bindsTo = [ "srv-photoframe.mount" ];
  path = [ pkgs.openssh ];
  serviceConfig = {
    User = "photoframe";
    Group = "photoframe";
    Type = "oneshot";
    ExecStart = ''
      ${pkgs.rsync}/bin/rsync \
        --rsh 'ssh -i /etc/davfs2/ssh-salon' \
        # Add more things to exclude if needed.
        --exclude='lost+found' \
        --delete \
        --delete-excluded \
        -a \
        /srv/photoframe/ \
        root@orangepi.$MYDOMAIN:/storage/pictures
      '';
  };
};
# The private ssh key.
sops.secrets."webdav/ssh-orangepi" = {
  sopsFile = ./secrets.yaml;
  mode = "0600";
  owner = "photoframe";
  path = "/etc/davfs2/ssh-salon";
};
systemd.timers.sync-to-orangepi = {
  wantedBy = [ "timers.target" ];
  timerConfig.OnBootSec = "10m";
  timerConfig.OnUnitActiveSec = "2h";
  timerConfig.RandomizedDelaySec = "10m";
};

It is important to put --delete and --delete-excluded in the rsync config. This way, unsharing pictures will effectively delete them from the OrangePi.

The BindsTo option is important too because it deactivates the service if the WebDav mount point gets stopped. Otherwise, in that case rsync will happily synchronize an empty directory - the unmounted directory - and delete all pictures on the OrangePi.

I checked for any issues with systemctl status sync-to-orangepi.service, systemctl status sync-to-orangepi.timer and journalctl -u sync-to-orangepi.service -f. Btw, before this steps worked, I needed to install rsync on the OrangePi. See next step.

The files get stored in /storage/pictures on the OrangePi, which is on the internal eMMC flash memory.

On the OrangePi

I first needed to install rsync. For that, I installed the Network Tools addon in Kodi.

I then navigated to the Settings menu, chose Interface and went to the Screensaver menu. There, I said to display pictures from the /storage/pictures folder and that was it!

I then tested the screensaver and saw the pictures I shared earlier to test.

Possible Improvements

All the manual steps here were tedious and error-prone. Coming from the NixOS world, I want this all to be declarative. This means creating users in Nextcloud declaratively and settings user options. On the OrangePi side, that will mean probably switching from LibreElec to NixOS, but I’m not sure if that’s necessary. Anyway, I’ll be working on this.

A Story about Garbage Collection on NixOS

2024-08-07T00:00:00Z

Tags: nix

Context
Starting Small
Going Deeper
Analyzing Store Size
Analyzing Store Size Some More
Let’s step back

Context

So you want to cleanup the Nix store. I wanted too. This led me down a medium sized rabbit whole. Read this post to follow it too! You might learn a couple things.

I realized my disk was close to full. 74Gb available from a 1Tb SAD. Time to clean up the Nix store!

Starting Small

The ubiquitous command to delete old NixOS profiles is nix-collect-garbage. So I used just that:

$ df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/disk/by-uuid/...  909G  789G   74G  92% /

$ nix-collect-garbage --delete-older-than 2d --dry-run
removing old generations of profile /home/ibizaman/.local/state/nix/profiles/home-manager
would remove profile version 77
would remove profile version 76
would remove profile version 75
would remove profile version 74
would remove profile version 73
would remove profile version 72
would remove profile version 71
would remove profile version 70
would remove profile version 69
removing old generations of profile /nix/var/nix/profiles/per-user/ibizaman/profile
removing old generations of profile /nix/var/nix/profiles/per-user/ibizaman/profile


$ nix-collect-garbage --delete-older-than 2d
note: currently hard linking saves 22381.91 MiB
16721 store paths deleted, 22869.11 MiB freed

$ df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/disk/by-uuid/...  909G  765G   97G  89% /

Not bad, but my disk is still quite full.

Going Deeper

Next step is to understand what Nix GC root exists, their size and determine which one I’m okay to delete and garbage collect.

From the wiki, one can print the GC roots with:

$ nix-store --gc --print-roots | egrep -v "^(/nix/var|/run/\w+-system|\{memory|/proc)"

This command ignores some paths, notably ones that are used by the current system.

This gave me quite a few GC roots. Instead of listing them here, I’ll break them down into categories

First, the project result directories. For example:

/home/ibizaman/Projects/selfhostblocks/result-modules
  -> /nix/store/v8va9mzlwnyplgzdjv0n4hy82m6dr9di-nix-flake-tests-success

If you don’t need to retain those and you’re okay to rebuild them later, just delete the link.

$ rm /home/ibizaman/Projects/selfhostblocks/result-modules

I had also some home-manager GC roots like:

/home/ibizaman/.local/state/home-manager/gcroots/current-home
  -> /nix/store/srp7f0jfjr8bq4vf030cw1jmgh2yqz34-home-manager-generation

Understanding why they’re needed is done with nix-store -q --root:

$ nix-store -q --roots /nix/store/srp7f0jfjr8bq4vf030cw1jmgh2yqz34-home-manager-generation
/nix/var/nix/profiles/system-145-link -> /nix/store/9fj82c87fphb6j6cxd9i0ns6wiag9gyh-nixos-system-laspin-24.05pre-git
/nix/var/nix/profiles/system-144-link -> /nix/store/yczj1ln41syk95krrw6n26m6c1wy8q3r-nixos-system-laspin-24.05pre-git
/nix/var/nix/profiles/system-147-link -> /nix/store/gyldgm273sgjb214f99h0ks3dbga2qd0-nixos-system-laspin-24.05pre-git
/run/current-system -> /nix/store/gyldgm273sgjb214f99h0ks3dbga2qd0-nixos-system-laspin-24.05pre-git
/nix/var/nix/profiles/system-146-link -> /nix/store/zj4p4fkyjam69ndgydkl8rrl3m09qyrw-nixos-system-laspin-24.05pre-git
/run/booted-system -> /nix/store/zj4p4fkyjam69ndgydkl8rrl3m09qyrw-nixos-system-laspin-24.05pre-git
/home/ibizaman/.local/state/home-manager/gcroots/current-home -> /nix/store/srp7f0jfjr8bq4vf030cw1jmgh2yqz34-home-manager-generation
/home/ibizaman/.local/state/nix/profiles/home-manager-78-link -> /nix/store/srp7f0jfjr8bq4vf030cw1jmgh2yqz34-home-manager-generation

This one is clearly used by a lot of stuff, I probably won’t be able to clean it up. It makes sense, it’s the current-home one after all. Another one is easier to delete:

/home/ibizaman/.local/state/nix/profiles/home-manager-77-link
  -> /nix/store/aawzc42n09xg5vm9n3phzagl95b6zfhg-home-manager-generation

$ nix-store -q --roots /nix/store/aawzc42n09xg5vm9n3phzagl95b6zfhg-home-manager-generation
/home/ibizaman/.local/state/nix/profiles/home-manager-67-link
  -> /nix/store/aawzc42n09xg5vm9n3phzagl95b6zfhg-home-manager-generation

We also have the direnv related ones:

/home/ibizaman/Projects/oxo/.direnv/flake-profile-1-link
  -> /nix/store/8b6kd8lcq5vckq0dkpab9hr558088q45-nix-shell-env

$ nix-store -q --roots /nix/store/8b6kd8lcq5vckq0dkpab9hr558088q45-nix-shell-env
/home/ibizaman/Projects/oxo/.direnv/flake-profile-1-link
  -> /nix/store/8b6kd8lcq5vckq0dkpab9hr558088q45-nix-shell-env

Same story as the project ones, you can delete them if they’re not needed anymore.

And then, I had a bunch of:

{temp:2597292} -> /nix/store/3d62w5jhcjgdjay4wg351d1jl7l2j2yh-gmp-6.2.1.tar.bz2.drv

This means a currently running process is holding on to this path. If nothing else holds on to the path, stopping the process will allow it to get garbage collected. Let’s check it:

$ nix-store -q --roots /nix/store/3d62w5jhcjgdjay4wg351d1jl7l2j2yh-gmp-6.2.1.tar.bz2.drv
/home/ibizaman/Projects/blog/result -> /nix/store/ikvnbi9w7ikbwv7501v9hj4ghnc01y9i-site
/home/ibizaman/Projects/blog/.direnv/flake-profile-6-link -> /nix/store/sn3wz8n8md5y31w2505557ylb2ygillk-site-env-env

No luck here. The path is also held by two links.

Analyzing Store Size

I used a pretty crude method (output formatted by myself):

$ sudo ncdu /nix/store

[...]
1.6 GiB  /lp2w7k4x4fsaidm18xgqjp2b70i6w6ci-ghc-9.6.4
1.6 GiB  /gjng6wrl7dsk8d98yjfiz9qb3g64h9f2-ghc-9.6.4
1.6 GiB  /2qqlva2zbkdhbyrz4qyacgq57s8kfy1l-ghc-9.4.8
1.6 GiB  /k033yvpca1r7fi0hwgkh5wnky5iixvlk-ghc-9.4.8
1.6 GiB  /psds2pz1qhlr4z8qcahqii6kq1xsawb8-ghc-9.4.8
1.6 GiB  /dpc4240h48a4kmgj684w1wkjmmi8ccxq-ghc-9.2.8
[...]

I probably don’t need all those GHC version. Let’s see.

$ nix-store -q --roots /nix/store/dpc4240h48a4kmgj684w1wkjmmi8ccxq-ghc-9.2.8
/home/ibizaman/Projects/blog/result -> /nix/store/ikvnbi9w7ikbwv7501v9hj4ghnc01y9i-site
/home/ibizaman/Projects/blog/.direnv/flake-profile-6-link -> /nix/store/sn3wz8n8md5y31w2505557ylb2ygillk-site-env-env

$ nix-store -q --roots /nix/store/psds2pz1qhlr4z8qcahqii6kq1xsawb8-ghc-9.4.8
/tmp/tmpbldxg_9z/ahsdpqwigcr399x602z9hlyrbmp3pa4q-vm-test-run-postgresql-peerAuth.drv -> /nix/store/ahsdpqwigcr399x602z9hlyrbmp3pa4q-vm-test-run-postgresql-peerAuth.drv
/tmp/tmpbldxg_9z/i00k3m9h6f0ihijmklfk1lhg4dr3cjw7-vm-test-run-postgresql-peerWithoutUser.drv -> /nix/store/i00k3m9h6f0ihijmklfk1lhg4dr3cjw7-vm-test-run-postgresql-peerWithoutUser.drv
/tmp/tmpbldxg_9z/8jc6kjg3qryhzrkggnw3f5kp8xsgzis2-vm-test-run-postgresql-tcpIpWithoutPasswordAuth.drv -> /nix/store/8jc6kjg3qryhzrkggnw3f5kp8xsgzis2-vm-test-run-postgresql-tcpIpWithou>
/tmp/tmpbldxg_9z/mgy2s5368srbps5bdan8774xdf2q1p3f-vm-test-run-monitoring-basic.drv -> /nix/store/mgy2s5368srbps5bdan8774xdf2q1p3f-vm-test-run-monitoring-basic.drv
/home/ibizaman/Projects/blog/.direnv/flake-profile-6-link -> /nix/store/sn3wz8n8md5y31w2505557ylb2ygillk-site-env-env
/tmp/tmpbldxg_9z/la2qw92yhz2y2kd6qv2wn9jjvcy286z0-vm-test-run-postgresql-tcpIPPasswordAuth.drv -> /nix/store/la2qw92yhz2y2kd6qv2wn9jjvcy286z0-vm-test-run-postgresql-tcpIPPasswordAuth.>
/tmp/tmpbldxg_9z/4blarcmd0hrggpvz8qbmkxmgnndmy9ss-vm-test-run-ldap-auth.drv -> /nix/store/4blarcmd0hrggpvz8qbmkxmgnndmy9ss-vm-test-run-ldap-auth.drv

I want to keep the GC roots of my blog here, so I won’t mess with those.

$ nix-store -q --roots /nix/store/k033yvpca1r7fi0hwgkh5wnky5iixvlk-ghc-9.4.8
/home/ibizaman/Projects/selfhostblocks/result-vm_authelia_basic -> /nix/store/xrihrmahqg32xwd7f7gjpcj02kpdjp67-vm-test-run-authelia-basic

This one can go away.

$ rm /home/ibizaman/Projects/selfhostblocks/result-vm_authelia_basic

$ nix-store --gc --print-dead
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/44wr5bc9szn0f870qy673kqi8ivhmc7c' to '/home/ibizaman/Projects/selfhostblocks/result-vm_authelia_basic'
determining live/dead paths...
[...]

That printed a bunch of paths, good!

$ nix-store --gc
[...]
note: currently hard linking saves 17869.65 MiB
5011 store paths deleted, 3609.95 MiB freed

3Gb saved. Not bad. Let’s try another GHC one.

$ nix-store -q --roots /nix/store/2qqlva2zbkdhbyrz4qyacgq57s8kfy1l-ghc-9.4.8
/home/ibizaman/Projects/selfhostblocks/result-vm_monitoring_auth -> /nix/store/avqv7a5sr382f35z4ahksa8kilbn569g-vm-test-run-monitoring-basic
/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_peerWithoutUser -> /nix/store/6qgznnl4wmg32q96lmcmwqsvzsxrw1q4-vm-test-run-postgresql-peerWithoutUser
/home/ibizaman/Projects/selfhostblocks/result-vm_ssl_test -> /nix/store/x8vprjx1jfbqzis874sc38b6kx5rmqa1-vm-test-run-ssl-test
/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_tcpIPPasswordAuth -> /nix/store/vrs0klwdk7dxqbagbv3qbxwiv9zh5aly-vm-test-run-postgresql-tcpIPPasswordAuth
/home/ibizaman/Projects/selfhostblocks/result-vm_lib_template -> /nix/store/rgbqh4qmkkdml7inrs564w5r8wj35is4-vm-test-run-lib-template
/home/ibizaman/Projects/selfhostblocks/result-vm_nextcloud_basic -> /nix/store/a783xnwswwh8wmnvb9y2axm4cczcmn98-vm-test-run-nextcloud-basic
/home/ibizaman/Projects/selfhostblocks/result-vm_ldap_auth -> /nix/store/swnzf8lsqq8s60g5mcqaxjd1jz8p2z7m-vm-test-run-ldap-auth
/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_peerAuth -> /nix/store/f935av428f1cky6s8mh4yvf36y0m5g9a-vm-test-run-postgresql-peerAuth
/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_tcpIPWithoutPasswordAuth -> /nix/store/6nmzajhcrh8vgd65ck5hawy9bwq0c6z3-vm-test-run-postgresql-tcpIpWithoutPasswordAuth

Okay, I’m fine getting rid of this one. With some bash tricks, I can remove all those result-vm-* links with a one-liner:

$ nix-store -q --roots /nix/store/2qqlva2zbkdhbyrz4qyacgq57s8kfy1l-ghc-9.4.8 \
    | awk -F' -> ' '{print $1}' \
    | xargs rm

$ nix-store --gc --print-dead
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/8df9bml2011d32bilfkga61r332q7kik' to '/home/ibizaman/Projects/selfhostblocks/result-vm_lib_template'
removing stale link from '/nix/var/nix/gcroots/auto/l57ixd12ad2kqh7241bg5i7g05vssxr4' to '/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_peerWithoutUser'
removing stale link from '/nix/var/nix/gcroots/auto/c2ja40qlc5110bd000ws1f14drjbkpvk' to '/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_tcpIPWithoutPasswordAuth'
removing stale link from '/nix/var/nix/gcroots/auto/wkm7xcfg9in3c8pm01zl3d5p72p13f9h' to '/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_tcpIPPasswordAuth'
removing stale link from '/nix/var/nix/gcroots/auto/ax33alfd9hw7xigb20qrpncmgmgn32rp' to '/home/ibizaman/Projects/selfhostblocks/result-vm_postgresql_peerAuth'
removing stale link from '/nix/var/nix/gcroots/auto/6i84xyjq0fq6w45nmrr1bmkr26ji02qj' to '/home/ibizaman/Projects/selfhostblocks/result-vm_ldap_auth'
removing stale link from '/nix/var/nix/gcroots/auto/2dp2ac8vqclzci3mab7pc1xj1k67frvw' to '/home/ibizaman/Projects/selfhostblocks/result-vm_ssl_test'
removing stale link from '/nix/var/nix/gcroots/auto/71r3b6wrbg9vayigzgakq4bpsc0j0s5m' to '/home/ibizaman/Projects/selfhostblocks/result-vm_nextcloud_basic'
removing stale link from '/nix/var/nix/gcroots/auto/qc7czk9rq88iflpry9a8zmjvfrcjsq2s' to '/home/ibizaman/Projects/selfhostblocks/result-vm_monitoring_auth'
determining live/dead paths...

$ nix-store --gc
[...]
note: currently hard linking saves 14657.84 MiB
6299 store paths deleted, 7420.39 MiB freed

Let’s see where we’re at:

$ df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/disk/by-uuid/...  909G  754G  109G  88% /

I continued removing some GHC roots and got up to 120Gb availability.

Analyzing Store Size Some More

I used some pretty crude commands to figure out what roots take space.

$ nix-store --gc --print-roots | grep -v "^/proc"
/home/ibizaman/.cache/nix/flake-registry.json -> /nix/store/5bs7lbd1fk22w7bzdd8d6fvysnyzgw35-flake-registry.json
/home/ibizaman/.local/state/home-manager/gcroots/current-home -> /nix/store/srp7f0jfjr8bq4vf030cw1jmgh2yqz34-home-manager-generation
/home/ibizaman/Projects/blog/.direnv/flake-profile-6-link -> /nix/store/sn3wz8n8md5y31w2505557ylb2ygillk-site-env-env
/home/ibizaman/Projects/blog/result -> /nix/store/ikvnbi9w7ikbwv7501v9hj4ghnc01y9i-site
/home/ibizaman/Projects/esp/.direnv/flake-profile-2-link -> /nix/store/ldbk3nd4hpkj9xiryhjy83inl5yawa9z-nix-shell-env
/home/ibizaman/Projects/nix-config/result -> /nix/store/6dbvmllgim249w566fikwxy6yza37i7g-nixos-24.05.20240421.6143fc5-x86_64-linux.iso

$ du -sch $(nix-store --gc --print-roots | grep -v "^/proc" | awk -F ' -> ' '{ print $2 }')
8.0K    /nix/store/5bs7lbd1fk22w7bzdd8d6fvysnyzgw35-flake-registry.json
40K     /nix/store/srp7f0jfjr8bq4vf030cw1jmgh2yqz34-home-manager-generation
68K     /nix/store/pvxnmywrjn7l3rw35zwd87ixd10lkicj-nix-shell-env
13M     /nix/store/ikvnbi9w7ikbwv7501v9hj4ghnc01y9i-site
80K     /nix/store/ldbk3nd4hpkj9xiryhjy83inl5yawa9z-nix-shell-env
1002M   /nix/store/6dbvmllgim249w566fikwxy6yza37i7g-nixos-24.05.20240421.6143fc5-x86_64-linux.iso

That last one can go away!

$ rm /home/ibizaman/Projects/nix-config/result

$ nix-store --gc
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/8xisdidbngs2s2y8y5hizxpvhydns4df' to '/home/ibizaman/Projects/nix-config/result'
deleting garbage...
[...]
deleting unused links...
note: currently hard linking saves 8562.92 MiB
389 store paths deleted, 1128.68 MiB freed

$ df -h /
Filesystem             Size  Used Avail Use% Mounted on
/dev/disk/by-uuid/...  909G  741G  122G  86% /

Still 741G used?

Let’s step back

I’m not likely to make any progress cleaning up the Nix store anymore. So what’s clogging up the disk space?

$ du -hs /nix/store/
75G     /nix/store/

Not the nix store, that’s for sure.

I ran ncdu on my /home directory and found roughly 150Gb of usage there. That doesn’t explain the used 741Gb. Where does that live?

Well, after searching here and there, I found it. It was the trash! So obvious in hindsight but I never cleaned the KDE trash. 500Gb was living in there.

Lesson: I should’ve taken a step back earlier. On the other hand, this blog post wouldn’t exist if I did.

Improving Git Speed with Large Repositories

2024-07-10T00:00:00Z

Tags: git, performance

It’s no secret that the nixpkgs repo is quite big. Big enough to slow down git operations to noticeable lags.

I use Emacs and Magit and with those, loading the status page takes - I kid you not - 15 seconds. Now, I don’t mind waiting a couple seconds, but here it’s just over my patience threshold.

I must say, the conclusion will be obvious in retrospective but since we use layers upon layers of software - here Magit over Emacs over Git - it’s not always obvious where the fix should happen. And that’s what prompted me to write this blog post, so I can showcase how to debug such a situation.

So, to investigate what’s going on, I toggled verbose logging with M-x magit-toggle-verbose-refresh so that loading the magit-status prints loading time. This is what I get:

Refreshing magit...
Running magit-pre-refresh-hook...done (0.012s)
Refreshing buffer ‘magit: nixpkgs’...
  magit-insert-error-header                0.000006 
  magit-insert-diff-filter-header          0.009790 
  magit-insert-head-branch-header          0.012734 !
  magit-insert-upstream-branch-header      0.000090 
  magit-insert-push-branch-header          0.012852 !
  magit-insert-tags-header                 8.128425 !!
  magit-insert-status-headers              8.173127 !!
  magit-insert-merge-log                   0.004723 
  magit-insert-rebase-sequence             0.000119 
  magit-insert-am-sequence                 0.000066 
  magit-insert-sequencer-sequence          0.000206 
  magit-insert-bisect-output               0.000063 
  magit-insert-bisect-rest                 0.000054 
  magit-insert-bisect-log                  0.000055 
  magit-insert-untracked-files             0.394513 !!
  magit-insert-unstaged-changes            0.055459 !!
  magit-insert-staged-changes              0.036347 !!
  magit-insert-stashes                     0.021789 !
  m..-in..-unpushed-to-pushremote          0.038461 !!
  m..-in..-unpushed-to-upstream-or-recent  0.344540 !!
  m..-in..-unpulled-from-pushremote        0.039289 !!
  magit-insert-unpulled-from-upstream      0.000005 
  magit-insert-local-branches              6.681158 !!
  forge-insert-issues                      0.013407 !
  forge-insert-pullreqs                    0.002832 
Refreshing buffer ‘magit: nixpkgs’...done (15.884s)
Running magit-post-refresh-hook...done (0.009s)
Refreshing magit...done (15.907s, cached 64/93 (69%))

Several steps take a long time. The first being magit-insert-tags-header.

Thanks to the helpful package, I can C-h h magit-insert-tags-header and see the source code of the function directly in the help buffer. And here it is:

(defun magit-insert-tags-header ()
  "Insert a header line about the current and/or next tag."
  (let* ((this-tag (magit-get-current-tag nil t))
         (next-tag (magit-get-next-tag nil t))
         (this-cnt (cadr this-tag))
         (next-cnt (cadr next-tag))
         (this-tag (car this-tag))
         (next-tag (car next-tag))
         (both-tags (and this-tag next-tag t)))
    (when (or this-tag next-tag)
      (magit-insert-section (tag (or this-tag next-tag))
        (insert (format "%-10s" (if both-tags "Tags: " "Tag: ")))
        (cl-flet ((insert-count (tag count face)
                    (insert (concat (propertize tag 'font-lock-face 'magit-tag)
                                    (and (> count 0)
                                         (format " (%s)"
                                                 (propertize
                                                  (format "%s" count)
                                                  'font-lock-face face)))))))
          (when this-tag  (insert-count this-tag this-cnt 'magit-branch-local))
          (when both-tags (insert ", "))
          (when next-tag  (insert-count next-tag next-cnt 'magit-tag)))
        (insert ?\n)))))

From looking at all the functions in there, the two that call to git and are likely the slow ones are in the first two lines: (magit-get-current-tag nil t) and (magit-get-next-tag nil t).

So I copy pasted the two expressions one by one then ran them with Alt-: followed by C-y and Enter. The magit-get-current-tag is the one actually taking those 8 seconds. The second one takes no time to run.

Going one level deeper, C-h h reveals the source code for magit-get-current-tag:

(defun magit-get-current-tag (&optional rev with-distance)
  "Return the closest tag reachable from REV.

If optional REV is nil, then default to `HEAD'.
If optional WITH-DISTANCE is non-nil then return (TAG COMMITS),
if it is `dirty' return (TAG COMMIT DIRTY). COMMITS is the number
of commits in `HEAD' but not in TAG and DIRTY is t if there are
uncommitted changes, nil otherwise."
  (and-let* ((str (magit-git-str "describe" "--long" "--tags"
                                 (and (eq with-distance 'dirty) "--dirty")
                                 rev)))
    (save-match-data
      (string-match
       "\\(.+\\)-\\(?:0[0-9]*\\|\\([0-9]+\\)\\)-g[0-9a-z]+\\(-dirty\\)?$" str)
      (if with-distance
          `(,(match-string 1 str)
            ,(string-to-number (or (match-string 2 str) "0"))
            ,@(and (match-string 3 str) (list t)))
        (match-string 1 str)))))

Here too the probable slow call is easy to spot, it’s in the first line, the call to magit-git-str. I ran the expression and indeed, it’s the slow one.

This means the slowness is with git! I thus ran the command in the terminal directly: git describe --long --tags and indeed, that took roughly 8 seconds to run. I tried without arguments but no improvements.

Next step, how to make git faster? I found this StackOverflow answer and tried it.

git gc --aggressive

But the terminal got killed after running for around 5 minutes because my machine was out of memory!

So, how to make git take less memory? I found this other StackOverflow question with the answer highlighted at the top. I thus added the following snippet to the local .git/config file:

[pack]
    packSizeLimit = 64m
    threads = 6
[gc]
    aggressiveWindow = 150

I bumped pack.threads from 1 as recommended in the post to 6 just to speed things up a bit. I have no idea what the best number would be, I just tried a few. The process now took at most half of CPU time and 70% of 16Gb of memory. At least it was in check and could complete.

Same for the other fields. I must admit I didn’t look up what they meant nor did check if they were good values. I just wanted to fix this quickly.

So with that, I could run git gc --aggressive again which, after about 45 minutes, gave me:

Enumerating objects: 5289057, done.
Counting objects: 100% (5289057/5289057), done.
Delta compression using up to 6 threads
Compressing objects: 100% (4911761/4911761), done.
Writing objects: 100% (5289057/5289057), done.
Total 5289057 (delta 3262154), reused 1315941 (delta 0), pack-reused 0 (from 0)
Checking connectivity: 5289057, done.
Expanding reachable commits in commit graph: 717447, done.
Writing out commit graph in 5 passes: 100% (3587235/3587235), done.

Yes, that took 45 minutes.

But at least, the git describe operation was an order of magnitude faster now:

$ time git describe --long --tags
24.05-pre-59983-g7041e60248e5

real    0m0.621s
user    0m0.521s
sys     0m0.099s

And indeed, refreshing the Magit status buffer takes around 3 seconds only now:

Refreshing buffer ‘magit: nixpkgs’...
  magit-insert-error-header                0.000812 
  magit-insert-diff-filter-header          0.000052 
  magit-insert-head-branch-header          0.022620 !
  magit-insert-upstream-branch-header      0.000064 
  magit-insert-push-branch-header          0.017885 !
  magit-insert-tags-header                 0.660671 !!
  magit-insert-status-headers              0.707210 !!
  magit-insert-merge-log                   0.000586 
  magit-insert-rebase-sequence             0.000657 
  magit-insert-am-sequence                 0.000045 
  magit-insert-sequencer-sequence          0.000126 
  magit-insert-bisect-output               0.000477 
  magit-insert-bisect-rest                 0.000030 
  magit-insert-bisect-log                  0.000024 
  magit-insert-untracked-files             0.329094 !!
  magit-insert-unstaged-changes            0.046189 !!
  magit-insert-staged-changes              0.036653 !!
  magit-insert-stashes                     0.035565 !!
  m..-in..-unpushed-to-pushremote          0.023885 !
  m..-in..-unpushed-to-upstream-or-recent  0.077200 !!
  m..-in..-unpulled-from-pushremote        0.022968 !
  magit-insert-unpulled-from-upstream      0.000074 
  magit-insert-local-branches              1.507013 !!
  forge-insert-issues                      0.024168 !
  forge-insert-pullreqs                    0.014356 !
Refreshing buffer ‘magit: nixpkgs’...done (3.228s)

There could be more optimizations to be done, but that’s enough for me for now.

Conclusion: let’s not forget to run git gc from time to time, especially for big repositories.

Make Good Stubs with Stateful Property Testing

2024-07-08T00:00:00Z

Tags: go, testing

Context
Integration Tests
Where Mocks Fail
Where Stubs Shine
Conclusion

Context

In a previous post, I wrote about how using stateful property testing helped us find test cases that were hard to find by hand. This helped us complete a project in a timely fashion. But some other surprising benefits came from introducing stateful property testing.

To switch from a deprecated MongoDB driver to the official one, we created a regression test suite. We first added stateful property tests, ensuring we covered as much behavior of the deprecated driver as we could. We then could make the new driver behave the same way as the the old one by making it pass those same tests. This gave us the confidence that both drivers behaved the same, even in edge cases, and that we could switch.

Well, having now these stateful property tests acting as a regression test suite, what else can we use them for? Creating stubs.

Integration Tests

Integration tests comprises a broad spectrum of test implementation. On one side, you have the tests using real components, like a full MongoDB or Elasticsearch cluster. Those suffer, from my experience, from flakiness, hard to reproduce differences on local dev and CI and are resource intensive and slow.

import "testing"

func TestUser(t *testing.T) {
  db := CreateMongoDatabase(t)
}

func CreateMongoDatabse(t *testing.T) {
  // ?
}

Also, those are harder to setup. The test there needs a MongoDB database, but who starts the MongoDB instance? Is it done through the test suite, with for example testcontainers? This has advantages, like full isolation between tests, but makes starting tests much slower since they each need to start a container. Another possibility is to start them outside of the test suite, often in a not well loved shell script with its own flaws.

On the other side, there are tests that use fake components - mocks - where the tester decides of the return value of those mocks during testing time. These are fast but suffer from a lack of faithfulness as the bigger the mocked component, the harder it is to reproduce its real behavior. This is even more true in a company with multiple engineering teams where one would mock a component made by another team. Extensive documentation of behavior and edge cases is necessary for the mock to act like the real thing, but that’s a hard to reach goal that’s even harder to justify spending time on.

Where Mocks Fail

I’d like to insist on the team aspect here and let’s take an example for that. I’ll use the same example as in the previous blog post. A Go service that encapsulates CRUD operations on a User struct representing a row in a MongoDB collection. We’ll focus on just one method, GetByNameOrEmail, which, although seemingly simple, already has quite a few edge cases.

package userservice

type UserInterface interface {
  // GetByNameOrEmail retrieves a User from the database
  // by matching either on their name or email field.
  GetByNameOrEmail(nameOrEmail string) (*User, error)
}

// User struct represents a row or record in the
// database.
type User struct{
  ID          bson.ID
  Name        string
  Email       string
  Deleted     bool
  DeletedTime time.Time
}

Let’s imagine this interface is given to us by another team. We don’t have access to the source code. Well, we do, we’re the same company, but maybe the code is too complicated for us and we don’t understand it, maybe it was created a long time ago and no one knows how it works anymore, or maybe we actually can’t read it because it’s in a repo we don’t have access to and getting access will require a few days to process. So for this contrived exercise, we can’t see the code.

Now we must program against this method and test our code. We need to create a test for some code related to billing and we must retrieve a user at some point. The point is, we’re not on the team that maintains this user service. Let’s create a mock:

type MockService() struct {
    GetByNameOrEmailNextCallReturn GetByNameOrEmailReturn
}

type GetByNameOrEmailReturn struct {
    User *User
    Err  error
}

func (ms MockService) GetByNameOrEmail(nameOrEmail string) (*User, error) {
    user, err := ms.GetByNameOrEmailNextCallReturn
    return user, err
}

Here you see how the mock works: at testing time, we instantiate the MockService and then, before calling GetByNameOrEmail, we must ensure we set the GetByNameOrEmailNextCallReturn field to what we want that method to return.

But the follow-up question is what does this function returns? In case of success, we can probably assume it returns a pointer to a filled out User struct and a nil error. But we can’t be sure of it without reading the source code. Also, what if the User is not deleted, is the DeletedTime field set to the zero value of time.Time or to something else?

In case of error, we can probably assume the function returns a nil pointer for User and non-nil error. Or maybe it’s a pointer to a empty User struct? Also, what will be the error? Can we differentiate between a deleted and not deleted user? Does the function return an error if the user is deleted?

So many questions that only the source code can tell us. Actually, even by reading the other team’s source code, we can’t always tell what the error will look like. If the code just returns the code from the MongoDB driver, we then need to go read the driver’s documentation. That’s one good example of leaky abstraction.

Now, the worst part in my opinion is if you went the extra mile and actually managed to encapsulate the original function’s behavior in a mock. And then, the behavior of the real service changes. You will never now and your test will still pass! You will probably learn about it during the next deploy.

Where Stubs Shine

Enter stubs. They are still fast to create as they are just a Go struct. The difference with mocks is that actually do try to encapsulate the behavior of the real object they’re replacing. They do this by managing an internal model that behaves the same way as the real deal. A stub for the User service would look like so:

type StubService() struct {
    Model map[bson.ID]User
}

func NewStubService() StubService {
  return StubService{
    Model: make(map[bson.ID]User),
  }
}

func (sus StubService) GetByNameOrEmail(nameOrEmail string) (*User, error) {
  // Assume we have a getByNameOrEmail function that searches in the map.
  user, ok := getByNameOrEmail(sus.model, nameOrEmail)
  if !ok {
    return nil, errors.New("No user found")
  } else if user.Deleted {
    return &user, errors.New("User is deleted")
  } else {
    return &user, nil
  }
}

Now let’s be clear, the code inside that function does not follow best practices for error handling. But maybe that’s what we got because it matches the behavior of the real service. And that service was made that way because of historical reasons. Changing the service’s behavior is tricky now and will definitely not be done by the team managing the billing system.

Anyway, the important part here is there is no mucking about figuring out what the method should return. We instead insert the User we want in the model, then call the GetByNameOrEmail function. The possibly weird behavior is encapsulated by the stub.

Of course, the issue now is how to create this stub? This is where our earlier state property test acting as a regression test suite comes into play. If the stub passes the same tests as the real service accessing MongoDB, then we can be quite sure it’s acting like the real one!

Ideally, this stub is maintained by the team maintaining the real MongoDB service. The additional work to do this is, by my experience, much smaller than the work needed to setup the stateful property tests anyway. But even if that’s not the case, as long as we have access to the tests, anyone can create a well behaved stub.

This is something we didn’t have with mocks: we now have tests to test the stub. And then we can use the stub in our tests. It seems a bit roundabout but this chain of tests has a very nice property: it’s a chain. If one step in the chain changes, we are guaranteed to know if the rest of the chain breaks or not, before deploying to prod. And if that’s the case, we can assess if we want to pursue that change now or later and communicate about it before breaking prod.

Conclusion

So are stubs always superior to mocks? No, but in my experience, the usefulness of mocks is limited. If you work alone on your own codebase, there are probably no big differences between a stub and a mock. But if you’re talking about multiple teams maintaining and evolving a codebase over multiple years, you want stubs first.

Using an External Git Repo for my Emacs Config with Nix

2024-07-03T00:00:00Z

Tags: nix

Context
Flake Input
Give the Inputs to the NixOS Configuration
Common Config
In Practice

Context

So I have this repository for my Emacs config. How can I make NixOS aware of it and copy the files in the correct location?

Flake Input

First step, putting the repo as a Nix flake input:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

    emacs-conf.url = "github:ibizaman/emacs-conf";
    emacs-conf.flake = false;

    emacs-overlay.url = "https://github.com/nix-community/emacs-overlay/archive/master.tar.gz";
    emacs-overlay.inputs.nixpkgs.follows = "nixpkgs";
  };
}

I’m using the Emacs Overlay because it has quite a lot of goodies, like native compilation of elisp files.

Give the Inputs to the NixOS Configuration

It’s not exactly obvious how to take these inputs and give them to the modules. Here’s how to do it in some common situations.

NixOS with nixos-rebuild

For a NixOS box:

{
  outputs = inputs@{ self, nixpkgs, ... }: {
    nixosConfigurations.machine =
      nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        specialArgs = {
          inherit (inputs) emacs-conf;
        };
        modules = [
          ./machine.nix
        ];
      };
  };
}

Darwin

For a Darwin box, first you need a new input:

{
  inputs = {
    nix-darwin.url = "github:LnL7/nix-darwin/master";
    nix-darwin.inputs.nixpkgs.follows = "nixpkgs";
  };
}

Then you can define it like so.

{
  outputs = inputs@{ self, nixpkgs, nix-darwin, ... }: {
    darwinConfigurations.machine =
      nix-darwin.lib.darwinSystem {
        system = "x86_64-linux";
        specialArgs = {
          inherit (inputs) emacs-conf;
        };
        modules = [
          ./machine.nix
        ];
      };
  };
}

NixOS with colmena

{
  outputs = inputs@{ self, nixpkgs, ... }: {
    colmena = {
      meta = {
        specialArgs = {
          inherit (inputs) emacs-conf;
        };
      };
      machine = {
        imports = [
          ./machine.nix
        ];
      };
    };
  };
}

Common Config

The content of machine.nix is:

{ config, pkgs, lib, inputs, ... }:

let
  emacsWithPackages = pkgs.emacsWithPackagesFromUsePackage {
    config = inputs.emacs-conf + "/init.el";
    defaultInitFile = true;
    package = pkgs.emacs-unstable;
  };
in

{
  config = {
    environment.systemPackages = [
      emacsWithPackages
    ];

    services.emacs = {
      enable = true;
      package = emacsWithPackages;
    };
  };
}

In Practice

To update my Emacs config on my NixOS machine, I:

Push a new commit to the Emacs repo.
Run nix flake lock --update-input emacs-conf
Run nixos-rebuild switch

Assuming I cloned my Emacs repo locally at ~/Projects/emacs-conf, I can also test a change by modifying the init.el file then updating my config with the following command:

nix flake lock \
  --update-input emacs-conf \
  --override-input emacs-conf ~/Projects/emacs-conf

For Emacs configs specifically, I actually also test my config just by loading the code I modified. Emacs being a Lisp machine, updating a function definition is easy.

In an elisp file, I can C-c C-c on a Lisp expression to evaluate it and load it with the following snippet:

(defun eval-point-region-and-deactivate ()
  "Evaluate region or expanded region and deactivates region when done."
  (interactive)
  (use-region-or-expand-region)
  (condition-case-unless-debug err
      (message "%s" (call-interactively 'eval-region))
    (error (deactivate-mark)
           (signal (car err) (cdr err))))
  (deactivate-mark))


(use-package elisp-mode
  :config
  (define-key lisp-mode-map (kbd "C-c C-c")
              #'eval-point-region-and-deactivate))

Recover NixOS System

2024-06-13T00:00:00Z

Tags: nix

Context
Recovery Overview
Takeaway

Context

I managed to brick my NixOS server. The recipe to achieve that is quite simple, just throw all best practices out the window by doing the following three things at the same time:

Make a potentially harmful change.
Make it impossible to recover.
Do not test.

Pretty easy, right?

Potential Harmful Change

My server has its root partition on an encrypted ZFS partition. This means I need to enter the passphrase for my server to boot. To be able to ssh into the server and enter the passphrase remotely, I had the boot.initrd options setup correctly.

But I wasn’t happy with my setup. See, when I sshed in, the prompt to type the passphrase was shown, I could then enter the passphrase and press Enter to mount the root partition, but I then had the prompt still open and I needed to exit the prompt manually. This extra exit step was not to my liking. So I changed:

echo "zfs load-key zroot; killall zfs" \
  >> /root/.profile

to this:

echo "zfs load-key zroot; killall zfs" \
  >> /root/.profile; exit

instead of this:

echo "zfs load-key zroot; killall zfs; exit" \
  >> /root/.profile

See the mistake? The exit was not in the correct location! So instead of showing me the prompt to unencrypt the partition, the startup would fail and I couldn’t do anything about it.

Impossible to Recover

I had been playing around with tweaking the boot initrd options for some time. Each time, it creates a new voluminous file that lines inside /boot. I gave less than 1Gb disk space for that partition so it fills up quickly during my tweaks.

I even made another blog post about how to recover from a filled up /boot partition already.

Anyway, I did what I wrote in that blog post and ran a command to remove old generations:

sudo nix profile wipe-history \
    --profile /nix/var/nix/profiles/system \
    --older-than 14d

But since that was not enough, I removed the older-than argument. This removed all previous generations, as I wanted, but not as I should have done.

Did not Test

nixos-rebuild build-vm is a thing and I should learn to use this tremendously useful feature. I already use NixOS VM tests quite extensively in my Self Host Blocks project so I have no excuse.

Bonus Stress Inducing Factor

Of course, I did this 24 hours before leaving on a trip. Why not?

Recovery Overview

So, how do you recover from this? The quick overview that @k900 from Matrix gave me is:

Boot on a NixOS Recovery USB stick
nixos-enter
change configuration
nixos-rebuild

I knew nixos-enter was a thing from the NixOS installation procedure but never would’ve thought about using it to recover a system!

That being said, because I was using flakes, ZFS for the partitions and colmena to deploy, each step had unforeseen complications. This blog posts goes over those and how I proceeded for each step.

NixOS Recovery USB stick

Also called live CD. But how to make such a thing? The wiki explains how to do this quite well. And it’s so easy, I love nix.

Here’s exactly how I created the live CD:

nixosConfigurations.recovery-iso = let
  inherit (inputs) nixpkgs;
  system = "x86_64-linux";
in
  (nixpkgs.lib.nixosSystem {
    inherit system;
    modules = [
      "${nixpkgs}/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix"
      "${nixpkgs}/nixos/modules/installer/cd-dvd/channel.nix"

      ({ config, pkgs, ... }: {
        boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

        environment.systemPackages = [
          pkgs.colmena
        ];
      })
    ];
  }).config.system.build.isoImage;

Since I wanted to work on ZFS partitions, I changed the kernel for a ZFS compatible one.

I also added the colmena package. You could avoid doing that as long as the machine you want to recover has internet by running nix shell nixpkgs#colmena when booted in the recovery environment.

To build the CD:

nix build .#nixosConfigurations.recovery-iso

And finally to copy it on a USB stick on /dev/sdb:

sudo dd \
  bs=4M \
  status=progress \
  conv=fdatasync \
  if=./result/iso/nixos-24.05.20240421.6143fc5-x86_64-linux.iso \
  of=/dev/sdb

nixos-enter

Again, the wiki explains well what to do in the general case.

For this step, I needed to mount the ZFS partition called zroot and unencrypt it:

sudo zpool import zroot -f
sudo zfs load-key zroot
#

I could then mount the 3 required filesystems. That being said, because I did let ZFS mount them by setting the mountpoint option, I couldn’t just use zfs mount to mount them under the /mnt directory. I needed instead this:

sudo mount -t zfs -o zfsutil zpool/local/root /mnt
sudo mount -t zfs -o zfsutil zpool/local/nix /mnt/nix
sudo mount -t zfs -o zfsutil zpool/safe/home /mnt/home

Now, I could run nixos-enter and get chrooted under /mnt.

flake

At this point, I should have been able to change configuration.nix but there is no such file when using flakes! So instead, I needed a copy of my repository used to deploy this machine.

I could have copied it from my laptop but instead I went back out of the chrooted environment and copied it over from an external hard drive used to store the repositories:

sudo zpool import data -f
zfs get keylocation data
cp /mnt/persist/data_passphrase /persist
sudo zfs load-key data
sudo mount -t zfs -o zfsutil data/nextcloud /mnt/srv/nextcloud

# Repo is now under /mnt/srv/nextcloud/.../nix-config
cp -r /mnt/srv/nextcloud/.../nix-config /mnt/root

The instructions above have additional steps compared to the root partition because those external hard drives use a different passphrase that’s only located on the ZFS root system. So I needed to copy over the key to the location expected by ZFS.

Update Configuration

Now, I could finally make the change to the repository!

nixos-enter
cd /root/nix-config

then change the incriminated line to:

echo "zfs load-key zroot; killall zfs; exit" \
  >> /root/.profile

nixos-rebuild

Since I was not using nixos-rebuild to deploy in the first place, I could not run nixos-rebuild to deploy locally either because there is no nixosConfigurations flake output. When using colmena there is instead a colmena output. This meant I needed to:

Allow colmena to deploy locally this machine by enabling the option:
```
deployment.allowLocalDeployment = true;
```
I had not enabled that option yet for this machine since I never needed to deploy this configuration locally.
Run colmena apply-local --node boot.

But that failed because I did not mount the boot partition!

So, after getting out of the chroot once more, I mounted the correct boot partition:

cat /mnt/etc/fstab | grep boot
sudo mount /dev/disk/by-partlabel/disk-x-ESP /mnt/boot

Then I re-entered the chroot with nixos-enter and ran the following command which finally succeeded.

colmena apply-local --node <mymachine> boot

Unmount Cleanly

But wait! If you reboot now, the system will not be able to mount the ZFS partitions. You first need to export the zpools:

sudo zpool export zroot
sudo zpool export data

If you forget this last step, just reboot on the USB drive and re-import the pools then export them.

Of course I forgot to do this on my first try.

Takeaway

What was the takeaway here? It was never wracking and exhausting. Never again.

Bypass Cached Failure

2024-05-24T00:00:00Z

Tags: nix

When evaluating a nix expression, sometimes you get a failure. But then, on next evaluation, you get this error message:

$ nix build .#checks.x86_64-linux.modules
error: cached failure of attribute 'checks.x86_64-linux.modules'

The failure got cached! How do you see the error message again? By adding --option eval-cache false:

$ nix build .#checks.x86_64-linux.modules --option eval-cache false
...

How to fix /boot volume running out of disk space in NixOS

2024-04-15T00:00:00Z

Tags: nix

After a while without removing old generations, I got into a pickle.

Situation

After running colmena apply to deploy to my server for the Nth time, I got this error:

Activation failed: Child process exited with error code: 1

[ERROR]   stderr) OSError: [Errno 28] No space left on
  device: '/nix/store/cwimj1bg0dgfvngdmbrapkp5ifl3bfgy-initrd-linux-6.6.22/initrd'
  -> '/boot/EFI/nixos/cwimj1bg0dgfvngdmbrapkp5ifl3bfgy-initrd-linux-6.6.22-initrd.efi'

Wait, no space left on the /boot volume? Uh oh. Taking a look, indeed it’s full:

$ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  128M  128M  2.0K 100% /boot

Fix

Now, the trick is two-fold. First, we must delete old generations with old versions of the kernel. As you can see, I didn’t properly clean this up.

$ nix profile history \
    --profile /nix/var/nix/profiles/system
Version 1 (2024-03-31):
  No changes.

Version 2 (2024-03-31) <- 1:
  No changes.

Version 3 (2024-03-31) <- 2:
  No changes.

...

Version 111 (2024-04-11) <- 110:
  No changes.

Version 112 (2024-04-15) <- 111:
  No changes.

To remove old generations and allow the garbage collection to kick in, run:

$ sudo nix profile wipe-history \
    --profile /nix/var/nix/profiles/system \
    --older-than 14d
removing profile version 12
removing profile version 11
removing profile version 10
removing profile version 9
removing profile version 8
removing profile version 7
removing profile version 6
removing profile version 5
removing profile version 4
removing profile version 3
removing profile version 2
removing profile version 1

Followed by the garbage collection:

$ nix store gc
1855 store paths deleted, 1851.81 MiB freed

But that wasn’t enough to get rid of the extraneous kernels.

$ sudo ls -l /boot/EFI/nixos
total 130308
-rwx------ 1 root root 10187264 Apr 15 06:15 0d3nlv97dyyflgq7irn5wy91x2mlszl3-linux-6.6.22-bzImage.efi
-rwx------ 1 root root 27904393 Apr 15 06:15 72p297y5a780hd6r5jwz0zqv4am97vpj-initrd-linux-6.6.22-initrd.efi
-rwx------ 1 root root 19864710 Apr 15 06:15 83i5i813jhsahlf7wmlbn96bngnjinf1-initrd-linux-6.6.22-initrd.efi
-rwx------ 1 root root 28606443 Apr 15 06:15 8bv13658nk7s12qljx0lch4g2bhrkvgr-initrd-linux-6.6.22-initrd.efi
-rwx------ 1 root root  8232960 Apr 15 06:15 cwimj1bg0dgfvngdmbrapkp5ifl3bfgy-initrd-linux-6.6.22-initrd.efi
-rwx------ 1 root root 27909504 Apr 15 06:15 ignpfyssa0ma4xp4asirfaq18f50vz8k-initrd-linux-6.6.22-initrd.efi
-rwx------ 1 root root 10723840 Apr 15 06:15 zqp81gm823adj6d6rk4k04gllhvwz847-linux-6.6.22-bzImage.ef

For that, we actually need to redeploy! So after one more colmena apply, which was successful this time, I got rid of the old kernels:

$ sudo ls -l /boot/EFI/nixos
total 66414
-rwx------ 1 root root 10187264 Apr 15 06:35 0d3nlv97dyyflgq7irn5wy91x2mlszl3-linux-6.6.22-bzImage.efi
-rwx------ 1 root root 28577118 Apr 15 06:35 8bv13658nk7s12qljx0lch4g2bhrkvgr-initrd-linux-6.6.22-initrd.efi
-rwx------ 1 root root 29239767 Apr 15 06:35 cwimj1bg0dgfvngdmbrapkp5ifl3bfgy-initrd-linux-6.6.22-initrd.efi

Jellyfin Streaming Media Center with LibreElec and Orange PI 3

2024-03-22T00:00:00Z

Tags: jellyfin, iot

Libreelec is a Linux distribution optimized to run Kodi. I installed it on an Orange Pi 3 LTS which is connected to my TV. I then installed the Jellyfin plugin and can play all my media from Jellyfin on the TV.

To boost performance, I installed the distribution on the internal flash eMMC storage instead of keeping it on the SD card.

This blog post covers everything I did to set this up.

Install Libreelec on an SD card

Go to libreelec.tv > Downloads > Manual Downloads section > Allwinner > Allwinner H6 > Orange Pi 3 LTS

Of course, that’s if you have an Orange Pi 3 LTS. Otherwise, pick the correct download. To burn on SD card, we will follow the Archlinux wiki.

Put the SD card in your computer.
Use lsblk to identify the path to your SD card.

In the following example, the path is /dev/sda.

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1  29.1G  0 disk 
└─sda1        8:1    1  29.1G  0 part
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0 923.6G  0 part /nix/store
│                                     /
├─nvme0n1p2 259:2    0   7.5G  0 part [SWAP]
└─nvme0n1p3 259:3    0   487M  0 part /boot

Burn Libreelec on the SD card.

Watch out, this will erase anything on the target path. In the following snippet, replace /dev/sdX with the path you got from the lsblk output.

$ gunzip --to-stdout \
    LibreELEC-H6.arm-11.0.3-orangepi-3-lts.img.gz \
  | sudo dd bs=4M conv=fsync oflag=direct \
            status=progress of=/dev/sdX

30+1 records in
30+1 records out
128064485 bytes (128 MB, 122 MiB) copied, 10.7377 s, 11.9 MB/s

Verify the partition layout looks like the following.

Don’t worry if the second partition does not cover the whole of the SD card, Libreelec will resize the partition on first boot.

$ sudo fdisk -l /dev/sda
Disk /dev/sda: 29.13 GiB, 31281119232 bytes, 61095936 sectors
Disk model: USB DISK        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9a55c0c3

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1  *       8192 1056767 1048576  512M  c W95 FAT32 (LBA)
/dev/sda2       1056768 1122303   65536   32M 83 Linux

First boot

Put the SD card in the Orange PI - or any other compatible device you have.

Connect the HDMI to your TV and connect a keyboard. The SD card will only be needed for the first boot, the keyboard for the first and second. You could do without a keyboard by using a remote control but that will be probably hard to type with.

You will then boot into the Jellyfin UI and the installation wizard will popup automatically.

Next page will allow you to choose a hostname. I chose “tvlivingroom”. Do not use special characters nor whitespaces as those are not accepted. The wizard is poorly done here because if you enter a not accepted character, no error message will be shown but you will be shown again the hostname selection screen.

You will need to wait about 10 seconds for the popup to appear to let you change the password. Later on, we will configure only connect with a SSH key and not password.

That should be then end of the wizard.

Now, we need to connect through SSH but to do that, we need to find the IP of this device. To do that, go to the settings page by clicking on the gear.

Then on the settings page, go to the System Information page.

Finally, you will see the IP address of the device. In the following examples, we’ll take 192.168.1.10.

Now, connect through SSH using that IP. If you kept the default password above, use libreelec as the password. Otherwise, use the one you entered.

$ ssh root@192.168.1.10 -o IdentitiesOnly=yes
root@192.168.1.10's password: 
##############################################
#                 LibreELEC                  #
#            https://libreelec.tv            #
##############################################

LibreELEC (official): 11.0.4 (H6.arm)
tvlivingroom:~ #

We will install Libreelec on the eMMC memory. You can skip this step but I recommend going through it as it drastically improves the performance of the UI.

tvlivingroom:~ # install2emmc

===============================
Installing LibreELEC to eMMC
===============================

eMMC found at /dev/mmcblk1


WARNING: ALL DATA ON eMMC WILL BE ERASED! Continue (y/N)?  y
Erasing eMMC ...
Creating partitions
Creating filesystems
Installing bootloader
Copying system files
Adjusting partition UUIDs
Done

Now, shutdown the device.

tvlivingroom:~ # shutdown -hP now

Second boot

Remove SD card then start the device by pressing the button a couple seconds.

Now, the not fun part is you will need to redo the whole wizard setup. Go back to the previous section if you need a refresher but skip the part where you install to the eMMC memory. Continue from here when that’s done.

SSH access

Okay, on your laptop, generate an ssh key. I use an SSH agent to remember the passphrase of my SSH keys so I do use a passphrase. If you don’t want to use a passphrase, just leave it empty when you get prompted.

Generate a random passphrase

$ openssl rand -hex 32
XYZ...

Generate an SSH key.

$ ssh-keygen -t ed25519 -f ~/.ssh/tvlivingroom
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again:
[...]

Add the key to the agent.

$ ssh-add ~/.ssh/tvlivingroom

Copy the key to the device.

$ ssh-copy-id -i ~/.ssh/tvlivingroom.pub -o IdentitiesOnly=yes root@192.168.1.20

You could SSH to the device with the following command, but it’s quite long, so skip to the next step.

$ ssh -i ~/.ssh/tvlivingroom.pub root@192.168.1.10

Create a match block in ~/.ssh/config:

Host salon
  User root
  HostName 192.168.50.201
  IdentityFile /home/timi/.ssh/salon

Or, if you’re using Home Manager, this snippet would do the trick:

home-manager.users.me = {
  programs.ssh = {
    matchBlocks = {
      "tvlivingroom" = {
        user = "root";
        identityFile = "/home/me/.ssh/tvlivingroom";
        hostname = "192.168.1.10";
      };
    };
  };
};

Success, we can now connect with:

$ ssh tvlivingroom

Harden device

Now, we can change the root password and restrict access to ssh key only.

I generate a new login/password combo with Bitwarden, then ssh on the Orange Pi and change the password with:

$ ssh tvlivingroom
tvlivingroom:~ # passwd
Changing password for root
New password: 
Retype password: 
passwd: password for root changed by root

Now, we can remove password login through ssh by going through the UI:

Setup Jellyfin on Kodi

Finally, let’s go to the fun part and connect to our Jellyfin server thanks to the jellyfin-kodi add-on.

I copied the steps here from the official instructions, specifically the embedded devices intructions the the install Jellyfin for Kodi instructions and adding my screenshots. We will install the Jellyfin for Kodi add-on, not the JellyCon one as I find the former much better integrated in the UI.

Install add-on repository.

” item." />

Then press OK. Now, we can actually install the add-on.

Install the Jellyfin for Kodi add-on.

Alright, now that the Jellyfin repository is installed, we can finally install the Jellyfin add-on!

After a few seconds, a popup will appear to choose which Jellyfin server to connect to. Assuming one is discoverable locally, you will see the following.

And you’re done! You can now use your Jellyfin server on the TV. A few ideas for what to do next:

Automate Jellyfin/Kodi with Home Assistant. For example, I pause the room speakers when playback starts.
Synchronize pictures to use as screensavers on the TV

Finally, I rebooted just to make sure everything worked as intended.

Use Case for Stateful Property Testing in Go

2024-02-27T00:00:00Z

Tags: go, testing

Our Initial Problem - A Driver Migration
Business Logic Led To Combinatory Explosion When Testing
Property Testing
Stateful Property Testing
Stateful Property Testing to Tame The Combinatory Explosion
Great Success

(Part 2 of this blog post is posted here)

In this blog post, I’ll explain what problem I tried to solve by introducing stateful property testing at work, what is stateful property testing, how well that worked out and what other extra benefits came along with stateful property testing.

Our Initial Problem - A Driver Migration

We wanted to update our MongoDB database version but the driver we used did not support the new version, was deprecated and did not get any updates. We needed to switch to the official MongoDB driver first.

Switching drivers should not be too hard. Types and methods are pretty similar, the update is quite mechanical. Of course, the devil is in the details and it appears there is an incompatible representation of the ID type between both drivers. This caused issues during our first deploy where a bug sneaked in spite of our existing tests.

We tried to add more tests but we couldn’t come up with extensive enough tests even while knowing about this incompatibility issue.

This ultimately let to multiple failed deploys. We could revert them easily and quickly but this experience led to diminishing confidence in our capacity to deploy without issue.

Business Logic Led To Combinatory Explosion When Testing

Before understanding how stateful property testing helped us to tame our codebase, we must first review how our codebase is structured for accessing external dependencies.

Access to practically all external resources - like databases, internal and external services - is done through a dedicated Go struct and methods.

For example, accessing a hypothetical “user” database table would be done through the following code. I completely invented this example for the blog post but it’s close to what we have.

package userservice

// User struct represents a row or record in the
// database.
type User struct{
  ID          bson.ID
  name        string
  email       string
  deleted     bool
  deletedTime time.Time
}

// NewUser struct with only those fields relevant to
// creating or updating a new user.
type NewUser struct{
  name  string
  email string
}

// Service handles reading from and writing to
// the database.
type Service struct{
  dbHandle mongo.db
}

// New creates a Service.
func New(dbHandle) Service {
  return Service{dbHandle: dbHandle}
}

// Insert a user in the database.
func (us Service) Insert(u NewUser) (bson.ID, error)

// GetByNameOrEmail retrieves a User from the database
// by matching either on their name or email field.
func (us Service) GetByNameOrEmail(nameOrEmail string) (User, error)

// Update a user in the database. Fails if the ID does
// not correspond to an existing record.
func (us Service) Update(ID bson.ID, u NewUser) error

// SoftDelete a user by toggling its "deleted" flag to
// True.
func (us Service) SoftDelete(ID bson.ID) error

// Delete a user's record from the database. This will
// fail if the user was soft-deleted less than a week
// ago.
func (us Service) Delete(ID bson.ID) error

I left out the implementation as it does not matter for the purpose of the post. You can begin to see why we had a hard time covering all the edge cases in our manual test suite. Some methods there toggle flags that influence the expected result of others. For example in the above Service, the GetByNameOrEmail method should only return matching users that had not been soft deleted previously.

Now, we have about 25 services like the UserService that each handle accessing one table and each of those have between 5 and 25 methods. This quickly leads to an unmanageable combinatory explosion of test cases to cover.

We did have actual integration tests, using a real instance of a MongoDB to exercise those methods. So we caught a lot of bugs that were introduced when migrating over to the new driver, but not all of them.

Property Testing

When testing a function, property testing is close to what we call in Go fuzzy testing. The idea is to generate random inputs and assert that a certain property of the function we test is never violated.

For example, let’s test a Go function to deduplicate items from a slice. There are a few properties we can test for:

The output’s length has to be smaller than the input’s length.
The output must only contain items that are in the input.
The input must only contain items that are in the output.
The output cannot contain twice the same item.

Some of those properties can overlap with others but it’s good to be thorough.

Caveat #1 of property testing: coming up with properties to test is not easy.

The library we’re using is called rapid. Here is how testing the first properties would look like, assuming the deduplicating function is defined elsewere:

import "testing"
import "flyingmutant/rapid"

func deduplicate[T comparable]([]T) []T

func TestingLength(t *testing.T) {
  rapid.Check(t, func (t *rapid.T) {
    input := rapid.SliceOf(rapid.Int()).Draw(t, "input slice")

    output := deduplicate(input)

    if len(output) > len(input) {
      t.Fatal("property violated: %d !<= %d", len(output), len(input))
    }
  })
}

Note that the version used in the snippets here is for version v0.7.2. The interface changed a bit since then but nothing that impacts what’s discussed here.

The rapid library will run the func (*rapid.T) function a bunch of times with different inputs. If an error is encountered, the library will reduce the input to the minimum input still showcasing the error. In the example above, it will try to find the minimum input slice that still fails the property. This is super useful for us humans to have as few noise as possible when trying to understand where the error comes from.

Caveat #2 of property testing: how can you be sure that an input with two or more duplicate items got generated? In a finite amount of time and tries, you can’t with the test as written above.

We need to engineer the input to have a few duplicates. There are multiple ways to do that and I usually choose the one that can be implemented with the less edge cases, in other words the one with the less if clauses. Usually, you want to avoid needing to test your tests.

To generate a slice with randomly selected duplicates, I would do something like this:

// This creates a slice generator. Use it like so:
//
//   inputSlice[int](100, 3).Draw(t, "slice with repeated items")
//
func inputSlice[T any](maxSize int, maxRepeat int) *rapid.Generator[[]T] {
  return rapid.Custom(func(t *rapid.T) []T {
    // Randomly draw size of input
    inputSize := rapid.IntRange(0, maxSize).Draw(t, "input slice length")
    input := make([]T, 0, inputSize)

    // For each item, generate 1 or more duplicate.
    for i := 0; i < inputSize; i++ {
      item := rapid.Make[T]().Draw(t, "item")
      repeat := rapid.IntRange(1, maxRepeat).Draw(t, "item repeat count")
      for r := 0; r < repeat; r++ {
        input = append(input, item)
      }
    }

    // Shuffle the items.
    rand.Shuffle(len(input), func(i, j int) {
      input[i], input[j] = input[j], input[i]
    })

    return input
  })
}

This will pick up to N randomly selected items, for each item repeat it up to M times and finally return the randomly shuffled resulting slice.

As you can see, generating an input that covers all edge cases we want in a finite amount of tries is hard but important. Alas, looking at that function, I would still probably need a test for it.

Stateful Property Testing

Stateful property testing uses the same idea as property testing but applied to a go struct with multiple methods where some of those methods can modify the struct’s state.

As a first example, let’s imagine the User struct - the one representing a row in our table - has methods we want to test.

// User struct we defined earlier. To simplify this
// section of the post, let's assume we only have a
// name and deleted field.
type User struct{
  name        string
  deleted     bool
}

func (u *User) SetName(name string) {
  u.name = true
}

func (u *User) GetName() string {
  return u.name
}

// SetDeleted modifies a flag on the User.
func (u *User) SetDeleted() {
  u.deleted = true
}

// IsDeleted returns the state of the flag.
func (u User) IsDeleted() bool {
  return u.deleted
}

Testing this using this rapid library looks like so. Since the User struct has a state - it remembers it’s name and if it’s deleted - we want to check that this state behaves as we expect.

To do that, we create a model of this behavior - the userStateMachine struct in the following example. There, we model a state machine where the delete state starts at False and the name starts as the empty string. Calling SetDeleted() will transition the delete state to True and SetName() will transition the name to the given value.

We must instruct rapid about this. We do it by creating this userStateMachine struct and adding a few methods. We create one method for each relevant state we want to transition, here we create SetName() and Delete() - the name itself do not matter - which will call the relevant method on both the model and the real User struct we want to test. We then create a third method Check() that is used to verify the real User struct’s state matches the model.

As long as the Check() function succeeds, rapid will randomly pick between the SetName() and Delete() method, execute it then call Check() again. It does that cycle about a 100 times by default. If any method fails, it stops there and tries to replay the failure with smaller and smaller inputs and prints the smallest input that still fails the test. For rapid, the input includes the name given to SetName() but also the number of calls to SetName() and Delete(). And that’s really cool IMO.

func TestUser(t *testing.T) {
  // Incantation needed by the rapid library.
  rapid.Check(t, func (t *rapid.T) {
    t.Repeat(rapid.StateMachineActions(&userStateMachine{}))
  })
}

// userStateMachine holds a User we want to test
// and a model of how the User should behave.
type userStateMachine struct {
  u User

  name      string
  isDeleted bool
}

// SetName modifies the state of the User and of
// the model.
func (usm *userStateMachine) SetName(t *rapid.T) {
  newName := rapid.String().Draw(t, "new name")

  // Call the method we want to test
  usm.u.SetName(newName)

  // Update our internal model
  usm.name = newName
}

// Delete modifies the state of the User and of
// the model.
func (usm *userStateMachine) Delete(t *rapid.T) {
  // Call the method we want to test
  usm.u.SetDeleted()

  // Update our internal model
  usm.isDeleted = true
}

// Check verifies that the User and the model agree on
// the state.
func (usm *userStateMachine) Check(t *rapid.T) {
  if usm.u.GetName() != usm.name {
    t.Fatalf("name: got (%s) != want (%s)",
             usm.u.GetName(), usm.name)
  }
  if usm.u.IsDeleted() != usm.isDeleted {
    t.Fatalf("deleted: got (%t) != want (%t)",
             usm.u.IsDeleted(), usm.isDeleted)
  }
}

Stateful Property Testing to Tame The Combinatory Explosion

Now, let’s apply what we just saw to the Service we defined earlier. Since the Service handles access to a MongoDB collection, we will model the state with a map of IDs to User struct.

// User struct we defined earlier.
type User struct{
  ID          bson.ID
  name        string
  email       string
  deleted     bool
  deletedTime time.Time
}

// NewUser struct with only those fields relevant to
// creating a new user.
type NewUser struct{
  name  string
  email string
}

func TestService(t *testing.T) {
  rapid.Check(t, func (t *rapid.T) {
    t.Repeat(rapid.StateMachineActions(&serviceStateMachine{
      // Give the initialized Service to the state machine.
      us: New(/* setup db access code */),

      // Start with an empty state.
      state: make(map[bson.ID]User),
    }))
  })
}

// serviceStateMachine holds a User we want to test
// and a model of how the User should behave.
type serviceStateMachine struct {
  // We test the Service.
  us Service

  // Against a model.
  state map[bson.ID]User
}

// Insert User, expects the insertion to always succeed
// because MongoDB chooses the ID and there are no unique
// contraint.
func (ssm serviceStateMachine) Insert(t *rapid.T) {
  // Generate a NewUser with random name and email.
  newUser := rapid.Make[NewUser]().Draw(t, "new user")

  // Insert the new User in the database.
  newID, err := ssm.us.Insert(newUser)
  if err != nil {
    t.Fatal(err)
  }

  // Add the new User to the state.
  ssm.state[newId] = User{
    ID: newId,
    name: newUser.name,
    email: newUser.email,
    deleted: false,
    deletedTime: time.Time{},
  }
}

// GetByNameOrEmailExisting tries to find a User we know
// exists either by its name or its email.
func (ssm serviceStateMachine) GetByNameOrEmailExisting(t *rapid.T) {
  // We purposely pick the user ID and not the User
  // struct here because then the only thing the rapid
  // library will log is the ID and not the full User
  // struct. It doesn't seem important at first, but
  // trust me when the User struct gets more fields
  // and you need to navigate through a lot of log
  // lines, reducing the log lines is a substantial
  // quality of life improvement.
  existingID := rapid.SampleFrom(lo.Keys(ssm.state)).Draw(t, "user ID")
  existingUser := ssm.state[existingID]
  nameOrEmail := rapid.OneOf(
    rapid.Just(existingUser.name),
    rapid.Just(existingUser.email),
  ).Draw(t, "name or email")

  // Get the User from the database.
  gotUser, err := ssm.us.GetByNameOrEmail(nameOrEmail)
  if err != nil {
    t.Fatal(err)
  }

  // Assert the user we got is the one we want.
  if diff := cmp.Diff(existingUser, gotUser); diff != "" {
    t.Fatalf("(-want, +got):\n%s", diff)
  }
}

// GetByNameOrEmailNotExisting tries to find a User using
// a random string and expects to fail.
func (ssm serviceStateMachine) GetByNameOrEmailNotExisting(t *rapid.T) {
  nameOrEmail := rapid.String().Draw(t, "random name or email")

  // Get the User from the database and assert we didn't
  // find it.
  _, err := ssm.us.GetByNameOrEmail(nameOrEmail)
  if err != NotFoundError {
    t.Fatal(err)
  }
}

// UpdateExisting tries to Update an existing User and
// expects to succeed.
func (ssm servicesStateMachine) UpdateExisting(t *rapid.T) {
  if len(ssm.state) == 0 {
    t.Skip("No existing user.")
  }
  existingID := rapid.SampleFrom(lo.Keys(ssm.state)).Draw(t, "user ID")
  existingUser := ssm.state[existingID]

  // Generate a NewUser with either an existing or randomly picked
  // name and email.
  updatedUser := NewUser{
    name = rapid.OneOf(rapid.String(), rapid.Just(existingUser.name)).Draw(t, "new user")
    email = rapid.OneOf(rapid.String(), rapid.Just(existingUser.email)).Draw(t, "new email")
  }

  // Update the User in the database.
  err := ssm.us.Update(existingID, updatedUser)
  if err != nil {
    t.Fatal(err)
  }

  // Update the state.
  u := ssm.state[existingID]
  u.name = updatedUser.name
  u.email = updatedUser.email
}

// UpdateNotExisting tries to Update an User that does not
// exist and expects to fail.
func (ssm servicesStateMachine) UpdateNotExisting(t *rapid.T) {
  randomID := rapid.Make[bson.ID]().Draw(t, "user ID")

  // Generate a NewUser with randomly picked name and email.
  updatedUser := NewUser{
    name = rapid.String().Draw(t, "new user")
    email = rapid.String().Draw(t, "new email")
  }

  // Update the User in the database.
  err := ssm.us.Update(randomID, updatedUser)
  if err == nil {
    t.Fatal("Expected error while updating the user")
  }
}

// SoftDeleteExisting tries to soft delete an existing User and
// expects to succeed.
func (ssm servicesStateMachine) SoftDeleteExisting(t *rapid.T) {
  if len(ssm.state) == 0 {
    t.Skip("No existing user.")
  }
  existingID := rapid.SampleFrom(lo.Keys(ssm.state)).Draw(t, "user ID")

  // Update the User in the database.
  err := ssm.us.SoftDelete(existingID)
  if err != nil {
    t.Fatal(err)
  }

  // Update the state. For the model, soft deleting is equivalent
  // to a plain delete.
  delete(ssm.state, existingID)
}

// SoftDeleteNotExisting tries to soft delete an User that does
// not exist and expects to fail.
func (ssm servicesStateMachine) SoftDeleteNotExisting(t *rapid.T) {
  randomID := rapid.Make[bson.ID]().Draw(t, "user ID")

  // Update the User in the database.
  err := ssm.us.SoftDelete(randomID)
  if err == nil {
    t.Fatal("Expected error while deleting the user")
  }
}

// DeleteExisting tries to delete an existing User and expects
// to succeed.
func (ssm servicesStateMachine) DeleteExisting(t *rapid.T) {
  if len(ssm.state) == 0 {
    t.Skip("No existing user.")
  }
  existingID := rapid.SampleFrom(lo.Keys(ssm.state)).Draw(t, "user ID")

  // Update the User in the database.
  err := ssm.us.Delete(existingID)
  if err != nil {
    t.Fatal(err)
  }

  // Update the state.
  delete(ssm.state, existingID)
}

// DeleteNotExisting tries to soft delete an User that does
// not exist and expects to fail.
func (ssm servicesStateMachine) DeleteNotExisting(t *rapid.T) {
  randomID := rapid.Make[bson.ID]().Draw(t, "user ID")

  // Update the User in the database.
  err := ssm.us.Delete(randomID)
  if err == nil {
    t.Fatal("Expected error while deleting the user")
  }
}

As you can see from the snippets above, we still need to define, for each method we test, every possible edge case. Usually for services that access databases, a common edge case is if a user exists or not. Depending on that, the test expects the function call to succeed or not.

I like the snippet above because it shows that although we call different methods on the Service for deletion and soft deletion, we expect the behavior to be the same in both cases - the affected User should not be found anymore. The model reflects that as in both cases we delete the User.

Then, stateful property testing kicks in by randomly trying combinations of the edge cases we defined. The good news is we can focus on each method in isolation, just thinking about how each of the User field interact with the expected result of the methods on the Service struct. The rapid library takes care of taming the combinatory explosion.

There’s a few more things I’d like to call out from the snippet above.

This is a lot of code. More than me and my team were initially comfortable to write for tests. In the end, it was worth it.
In the long run, logging too much becomes annoying as it is just noise. That’s why when randomly choosing a User that exists, it’s better to pick the ID than the User struct directly, otherwise you print fields that you don’t care about.
We usually split the various edge cases in multiple methods. In the snippet, we have the Existing and NotExisting suffixed methods. You could do the same in one function but then you need a way to know if you picked an existing user or not and you begin to fiddle with if clauses everywhere. In the end, splitting into multiple methods makes the test code easier to understand.
Since we only have a finite time to run the test, the more edge cases you have, the less chance you’ll have to actually exercise all the edge case combinations in one test run. There are two ways to go about that: input generation engineering and classification. We covered the former above.

Classification means we add a label to each edge case. Each time an edge case gets reached, we increment the related label’s counter by 1. At the end of the test, we generate a report that shows the distribution of edge case. We can then analyze this distribution to ensure all the edge cases got reached a sufficient amount of times. If not, we need to either run the tests longer or to either change how we generate the inputs. Classification is in general useful when we have a lot of edge cases.

Great Success

Setting up the stateful property tests to cover most if not all of the edge cases we could think of was no easy task, but it was really rewarding as it discovered all remaining incompatibility issues. Although the team was not enthusiastic in introducing that kind of testing in the first place - and rightfully, it’s quite a lot of code - the result put everyone on board and boosted the confidence in deploying our changes.

In a following post, I will talk about an additional benefit of the stateful property tests as we wrote them: they allowed to switch from mocks to stubs. We will see how they helped and just why switching to stub is a good idea in the first place. This next blog post is now posted here.

Mounting Webdav Folder in NixOS

2024-02-23T00:00:00Z

Tags: nix, nextcloud

I want Digikam to access Nextcloud storage using WebDav. But Digikam needs this folder to be actually mounted on the filesystem, so the setup implies some fiddling.

Internet Search

Here are all the interesting links that helped me figure out how this should work.

From the Digikam manual:

Collections on Network Shares: these are root album folders stored remote file systems as Samba or NFS and mounted as native on your system.

Clearly, this means the folder must be mounted.

Searching on mounting a webdav folder on linux poped up the Archlinux wiki on davfs2, giving me one new keyword to search for.

In the NixOS options search, there is already a davfs2 service, which is implemented here and sets up a configuration file, user and (un)mount.davfs2 programs. Perfect!

A last search returned the Nextcloud manual which shows how to manage the davfs2 secrets. This is not handled by the NixOS options above.

Let’s try the existing service

On my NixOS machine, I just added the following line and deployed.

services.davfs2.enable = true;

Let’s see what we gained:

$ mount.davfs --help
Usage:
    mount.davfs -V,--version   : print version string
    mount.davfs -h,--help      : print this message

To mount a WebDAV-resource don't call mount.davfs directly, but use
`mount' instead.
    mount <mountpoint>  : or
    mount <server-url>  : mount the WebDAV-resource as specified in
                          /etc/fstab.
    mount -t davfs <server-url> <mountpoint> [-o options]
                        : mount the WebDAV-resource <server-url>
                          on mountpoint <mountpoint>. Only root
                          is allowed to do this. options is a
                          comma separated list of options.

Recognised options:
    conf=        : absolute path of user configuration file
    uid=         : owner of the filesystem (username or numeric id)
    gid=         : group of the filesystem (group name or numeric id)
    file_mode=   : default file mode (octal)
    dir_mode=    : default directory mode (octal)
    ro           : mount read-only
    rw           : mount read-write
    [no]exec     : (don't) allow execution of binaries
    [no]suid     : (don't) allow suid and sgid bits to take effect
    [no]grpid    : new files (don't) get the group id of the directory
                   in which they are created.
    [no]_netdev  : (no) network connection needed

The exact command I tried that works with Nextcloud is:

$ sudo mount \
      -t davfs \
      -o uid=1000 \
      https://$fqdn/remote.php/dav/files/$myuser \
      mnt

Please enter the username to authenticate with server
https://$fqdn/remote.php/dav/files/$myuser or hit enter for none.
  Username: $username
Please enter the password to authenticate user timi with server
https://$fqdn/remote.php/dav/files/$myuser or hit enter for none.
  Password: $password
warning: the server does not support locks

I just needed to enter my Nextcloud username and password when prompted.

Create a secret file

I highly recommend creating an App password for mounting the webdav folder, this is much faster than using your main account password as can be seen in this benchmark.

davfs2 reads secrets from a file called /etc/davfs2/secrets. Using sops-nix, you can create this file with:

sops.secrets."webdav/nextcloud" = {
  sopsFile = ./secrets.yaml;
  mode = "0600";
  path = "/etc/davfs2/secrets";
};

The secret in the sops file must follow a specific format, one line per webdav folder to mount.

From our example above, we would have something like this:

https://$fqdn/remote.php/dav/files/$myuser $username $password

Make a Systemd service out of it

systemd.mounts = [
  {
    enable = true;
    description = "Webdav mount point";
    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
  
    what = "https://$fqdn/remote.php/dav/files/$myuser";
    where = "/mnt/nextcloud";
    options = uid=1000,gid=1000,file_mode=0664,dir_mode=2775
    type = "davfs";
    mountConfig.TimeoutSec = 15;
  }
];

ZFS on NixOS

2024-02-09T00:00:00Z

Tags: nix, zfs

I resisted long enough. Having bought two new hard drives, I had a good enough reason to switch over from LVM + mdadm.

This blog post is as much a memo for myself than a terse guide to people wanting to try ZFS.

Partitioning

From https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#performance-considerations

Create your pool using whole disks: When running zpool create use whole disk names. This will allow ZFS to automatically partition the disk to ensure correct alignment. It will also improve interoperability with other OpenZFS implementations which honor the wholedisk property.

From https://wiki.archlinux.org/title/ZFS#Storage_pools there’s mixed messaging.

It is not necessary to partition the drives before creating the ZFS filesystem. It is recommended to point ZFS at an entire disk (ie. /dev/sdx rather than /dev/sdx1), which will automatically create a GPT (GUID Partition Table) and add an 8 MB reserved partition at the end of the disk for legacy bootloaders.

while a bit later:

The OS does not generate bogus partition numbers from whatever unpredictable data ZFS has written to the partition sector, and if desired, you can easily over provision SSD drives, and slightly over provision spindle drives to ensure that different models with slightly different sector counts can zpool replace into your mirrors.

From https://forums.freebsd.org/threads/zfs-whole-disk-vs-gpt-slice.62855/

That user posted some good links. It seems using partitions is totally fine. That anyway, giving a full disk to ZFS on Linux will create a GPT partition anyway.

I will try with partitions and by naming them with GPT labels. Labels are great because they are unique and can convey meaning.

$ nix shell nixpkgs#gptfdisk

$ sudo fdisk -l

...

Disk /dev/sdb: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors
Disk model: G
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors
Disk model: G
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

$ sudo gdisk -l /dev/sdb
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-23437770718, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-23437770718, default = 23437768703) or {+-}size{KMGTP}: -1G
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300):

Command (? for help): c
Using 1
Enter name: ZFSmirror1

Command (? for help): p
Disk /dev/sdb: 23437770752 sectors, 10.9 TiB
Model: G               
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): BEA1BE33-E007-4582-B29F-7952D9EE8E26
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 23437770718
Partitions will be aligned on 2048-sector boundaries
Total free space is 2099166 sectors (1025.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048     23435673566   10.9 TiB    8300  ZFS data mirror 1

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.

I then did the same with /dev/sdc.

I chose a label name with no space otherwise spaces gets replaced with \x20 which gives names like: 'ZFS\x20data\x20mirror\x201'. No thanks.

$ ls -1 /dev/disk/by-partlabel/
ZFSmirror1
ZFSmirror2

Enable ZFS on the machine

From https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/ :

boot.supportedFilesystems = [ "zfs" ];

boot.zfs.forceImportRoot is enabled by default for backwards compatibility purposes, but it is highly recommended to disable this option, as it bypasses some of the safeguards ZFS uses to protect your ZFS pools.

boot.zfs.forceImportRoot = false;

networking.hostId should be unique per machine. The primary use case is to ensure when using ZFS that a pool isn’t imported accidentally on a wrong machine.

networking.hostId = "3d6f479a";

Create zpool

From https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#performance-considerations I got that a good argument is:

-o ashift=12

Also:

Have enough memory: A minimum of 2GB of memory is recommended for ZFS. Additional memory is strongly recommended when the compression and deduplication features are enabled.

From https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/, I got:

-o ashift=12
-o xattr=sa
-o compression=lz4
-o atime=off
-o recordsize=1M

I won’t be using a SLOG as this zpool will hold pictures and documents, so large files. Also no L2ARV for me as I don’t have that much RAM available.

From https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html

-o ashift=12
-o autotrim=on
-O acltype=posixacl
-O canmount=off
-O dnodesize=auto
-O normalization=formD
-O relatime=on
-O xattr=sa
-O mountpoint=none

About dnodesize:

Consider setting dnodesize to auto if the dataset uses the xattr=sa property setting and the workload makes heavy use of extended attributes. This may be applicable to SELinux-enabled systems, Lustre servers, and Samba servers, for example. Literal values are supported for cases where the optimal size is known in advance and for performance testing.

Leave dnodesize set to legacy if you need to receive a send stream of this dataset on a pool that doesn’t enable the large_dnode feature, or if you need to import this pool on a system that doesn’t support the large_dnode feature.

My final command was:

$ nix shell nixpkgs#gptfdisk
$ zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -O encryption=on \
    -O keyformat=passphrase \
    -O keylocation=prompt \
    -O compression=lz4 \
    -O canmount=off \
    -O mountpoint=/srv \
    -O xattr=sa \
    -O atime=off \
    -O acltype=posixacl \
    -O recordsize=1M \
    data \
    mirror \
    /dev/disk/by-partlabel/ZFSmirror1 \
    /dev/disk/by-partlabel/ZFSmirror2

This command will ask you for a passphrase.

$ zpool status -v
  pool: data
 state: ONLINE
config:

        NAME          STATE     READ WRITE CKSUM
        data          ONLINE       0     0     0
          ZFSmirror1  ONLINE       0     0     0
          ZFSmirror2  ONLINE       0     0     0

errors: No known data errors

I chose legacy mountpoint just so I could mount them manually in a temporary directory and transfer over what needed to be.

$ sudo zfs create -o mountpoint=legacy data/nextcloud
$ sudo zfs create -o mountpoint=legacy data/git

Quotas

Nothing fancy here but I wanted to put some.

From https://docs.oracle.com/cd/E23823_01/html/819-5461/gazvb.html:

$ sudo zfs set quota=4T data/nextcloud
$ sudo zfs set quota=40G data/git

It can also be done at creation by using -o quote=.

Reservation for Performance

From https://web.archive.org/web/20161028084224/http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations :

Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-96% utilization range. Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer.

For my workload, I chose roughly 90%, so I’ll reserve 1Tb from the ~12Tb available.

From https://docs.oracle.com/cd/E23823_01/html/819-5461/gazvb.html :

$ sudo zfs set reservation=1T data

Transfer data

Mount new pool to a temporary directory under a temporary root /pool directory, then copy over with rsync.

$ sudo mount -t zfs data/nextcloud /pool/nextcloud
$ sudo rsync -aHP /srv/data/nextcloud/ /pool/nextcloud

Mount in correct location

$ sudo umount /srv/nextcloud
$ sudo zfs set mountpoint=/srv/nextcloud data/nextcloud

Setting the mountpoint this way will let ZFS mount the pool.

Reboot

Because we encrypted the zpool, On reboot, you need to run:

$ sudo zpool import data
$ sudo zfs load-key data
$ sudo zfs mount -a

There are ways to automate this like remote unlocking.

Next will be snapshots and automating mounting after reboots.

Automated Flake Lock Update Pull Requests and Merging

2023-12-25T00:00:00Z

Tags: nix, ci

This is pretty cool. Thanks to two GitHub workflows, I managed to automate updating the flake.lock of my project.

Every day at midnight, the first workflow will run, try to update the flake.lock file and if there’s an update, it will create a Pull Request with the automerge label. The second workflow runs every time a Pull Request gets updated (opened, labeled, etc.) and enabled auto-merging of the Pull Request if the automerge label is set.

Create Your Personal Access Token

First thing, you will need to create a Personal Access Token (PAT) to enable both workflow to work.

Go to your PAT settings and create a Fine Grained PAT. Select which repository the PAT should be granted for then set Read and Write access for both Contents and Pull Requests permissions. The resulting PAT should look like this:

PAT with Read and Write access set for Contents (code) and Pull Requests permissions.

The Contents permission is needed to create a branch and push commits to it and the Pull Requests one to allow to create a pull request. Both are needed for the first workflow but admittedly only the latter is needed for the second workflow. You could create a second PAT with only the Pull Requests permission for the second workflow but I didn’t go that far.

Now, add the PAT as a repository secret named GH_TOKEN_FOR_UPDATES in the secrets page of your repo, at https://github.com///settings/secrets/actions:

GH_TOKEN_FOR_UPDATES repository secret.

Workflow 1: Create PR with Updated `flake.lock`

The workflow uses the update-nix-flake-lock action from Determinate Systems:

name: Update Flake Lock

on:
  workflow_dispatch:
  schedule:
    - cron: '0 * * * 0' # runs daily at 00:00

jobs:
  lockfile:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@main
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          extra-conf: "system-features = nixos-test benchmark big-parallel kvm"
      - name: Update flake.lock
        uses: DeterminateSystems/update-flake-lock@main
        with:
          token: ${{ secrets.GH_TOKEN_FOR_UPDATES }}
          pr-labels: |
            automerge

As stated in the introduction, this workflow runs daily at midnight, calls the DeterminateSystems/update-flake-lock action that updates the flake.lock file and creates a Pull Request with the automerge label.

Although not necessary, I kept the workflow_dispatch trigger as this allows me to run the action manually. I wasn’t going to wait for midnight to test the workflow! :D

By the way, the PAT was necessary to trick GitHub in thinking I created the Pull Request. Otherwise, GitHub actions are not run if a Pull Request gets created from another action. See this section of the README for more details.

Workflow 2: Auto-Merge PR

You cannot (yet?) enable auto-merging by default in GitHub, so we need instead to enable it ourselves whenever a Pull Request gets created or updated.

Well, “ourselves” is by using the auto-merge-pull-request action:

name: Auto Merge

on:
  # Try enabling auto-merge for a pull request when a draft is marked as “ready for review”, when
  # a required label is applied or when a “do not merge” label is removed, or when a pull request
  # is updated in any way (opened, synchronized, reopened, edited).
  pull_request_target:
    types:
      - opened
      - synchronize
      - reopened
      - edited
      - labeled
      - unlabeled
      - ready_for_review

  # Try enabling auto-merge for the specified pull request or all open pull requests if none is
  # specified.
  workflow_dispatch:
    inputs:
      pull-request:
        description: Pull Request Number
        required: false

jobs:
  automerge:
    runs-on: ubuntu-latest
    steps:
      - uses: reitermarkus/automerge@v2
        with:
          token: ${{ secrets.GH_TOKEN_FOR_UPDATES }}
          merge-method: rebase
          do-not-merge-labels: never-merge
          required-labels: automerge
          pull-request: ${{ github.event.inputs.pull-request }}
          review: ${{ github.event.inputs.review }}
          dry-run: false

Again, I left the workflow_dispatch trigger for testing, although it wasn’t necessary in the end since I could just toggle the automerge label on a Pull Request.

Result

This Pull Request was created and merged automatically.

Investigate Nextcloud Slowness Part 2

2023-12-11T00:00:00Z

Tags: nextcloud, postgresql, performance

My Nextcloud instance feels… sluggish, for lack of better term. It’s working, but it definitely feels slow. Pages take seconds to load. Let’s continue our previous investigation of Nextcloud slowness and try to fix this.

I will warn you, not all investigations did lead to an optimization, most were dead ends. But I think the process is still valuable as I will show you interesting techniques. So it’s worth to read along.

If you had to take out one thing from this post, it should be know your whole system. As you’ll see, I dived deep into profiling which was helpful, but in the end what helped the most was a macro view of my server.

Finding Interesting Profiles

Last time, we enabled debugging in Nextcloud by having PHP write a profile for every request having the appropriate header. At the time, we used curl to issue select requests to Nextcloud.

This time, we want to do it with a more realistic load through the browser by simply loading various pages and seeing what requests take a long time to run, then check their profiles.

For Firefox, I use the extension BrianGilbert/xdebug-helper-for-firefox which adds the required trigger header to outgoing requests. In the extension’s settings page we set the “Trace Trigger Value” and “Profile Trigger Value” to debug me which must be the value we set for the xdebug.trigger_value PHP option.

For reference, here are the PHP option we set in the previous blog post (in nix syntax):

services.nextcloud.phpOptions = {
  "xdebug.trigger_value" = "debug_me";
  
  "xdebug.mode" = "profile";
  "xdebug.output_dir" = "/var/log/xdebug";
  "xdebug.start_with_request" = "trigger";
};

A good candidate for interesting profiles is the Photos main page under /apps/photos/. We go to that page then enable profiling in the extension by clicking on the little icon in the URL bar:

Now, we open the Web Developer Tools and go to the Network tab and reload the page. We can see all the requests being made. To pick an interesting one, we can simply sort by duration:

Ah! The main request to /apps/photos is second on the list, let’s start with that one. To know what the corresponding profile is, we can just look at the x-xdebug-profile-filename response header:

Finally, let’s download the profiles locally and open them with kachegrind:

rsync -va --delete myserver:/var/log/xdebug .

Another request that was very slow - 12 seconds! - was accessing a folder through webdav:

We will check the profile for that one later.

Analyzing the First Profile - It’s the Database!

Spoiler: result of this analysis is mixed.

Let’s analyze the profile with KCacheGrind:

nix run nixpkgs#kcachegrind xdebug/cachegrind.out.43532

With KCacheGrind open, we can then sort all the function calls by the self parameter. This will show us where most of the time is spent.

Function calls sorted by self.

And we see the culprit: calls to the database. This line is very promising as it shows the query was I/O bound (because call to database) and it has all the attributes of something we can optimize:

We have knobs to tune the database. Changing the Nextcloud code would’ve been harder.
There are only 20 calls that take 42.29% of the time. We know from looking at the network tab in the browser that the whole query took 920ms. So that’s 920ms * 42.29/100 / 20 calls = 19.45 ms per call.

I’m not sure if 20 ms per call is slow or not, but loading the page feels sluggish for sure so we’ll see if there’s anything we can do.

Diving Into the Database Call

There are several ways we can log slow queries. Here, we will print all queries that take longer than one second by setting the log_min_duration parameter to 1000 (1 second).

There are also multiple ways to set this value. Here, I’ll do it through SQL:

$ sudo -u nextcloud psql
psql (14.10)
Type "help" for help.

UPDATE pg_settings
  SET setting = 1000
  WHERE name = 'log_min_duration_statement';

Now, after reloading a few times the page at /apps/photos, I see sometimes printed in the logs a query fetching from the oc_filecache table (formatting mine):

[35320] LOG:  duration: 4894.961 ms execute <unnamed>:
  SELECT "file"."fileid", "storage",
    "path", "path_hash", "file"."parent", "file"."name",
    "mimetype", "mimepart", "size", "mtime",
    "storage_mtime", "encrypted", "etag", "permissions",
    "checksum", "unencrypted_size"
  FROM "oc_filecache" "file"
  WHERE (("file"."name" = $1) OR ("file"."name" = $2))
    AND (("storage" = $3)
    AND (("path" = $4) OR ("path" LIKE $5)))
[35320] DETAIL: parameters:
  $1 = '.nomedia',
  $2 = '.noimage',
  $3 = '5',
  $4 = 'files',
  $5 = 'files/%'

This is super useful because we can run an EXPLAIN ANALYZE on it. But before that, we will enable track_io_timing which will print “time spent reading and writing data file blocks and temporary file blocks (in milliseconds)”:

UPDATE pg_settings
  SET setting = 'on'
  WHERE name = 'track_io_timing';

Now, the explain:

nextcloud=> EXPLAIN (ANALYZE, BUFFERS)
  SELECT "file"."fileid", "storage",
    "path", "path_hash", "file"."parent", "file"."name",
    "mimetype", "mimepart", "size", "mtime",
    "storage_mtime", "encrypted", "etag", "permissions",
    "checksum", "unencrypted_size"
  FROM "oc_filecache" "file"
  WHERE (("file"."name" = '.nomedia')
         OR ("file"."name" = '.noimage'))
    AND (("storage" = 5)
    AND (("path" = 'files') OR ("path" LIKE 'files/%')));

Which outputs:

Gather  (cost=9808.72..112624.78 rows=7 width=211)
        (actual time=101.721..150.161 rows=5 loops=1)
  Workers Planned: 2
  Workers Launched: 2
  Buffers: shared hit=49589 dirtied=65
  ->  Parallel Bitmap Heap Scan on oc_filecache file
      (cost=8808.72..111624.08 rows=3 width=211)
      (actual time=94.107..125.883 rows=2 loops=3)
    Recheck Cond: (storage = 5)
    Filter: ((((name)::text = '.nomedia'::text) [...]
    Rows Removed by Filter: 193922
    Heap Blocks: exact=16025
    Buffers: shared hit=49589 dirtied=65
    ->  Bitmap Index Scan on fs_storage_mimepart
        (cost=0.00..8808.72 rows=580305 width=0)
        (actual time=30.398..30.398 rows=622607 loops=1)
      Index Cond: (storage = 5)
      Buffers: shared hit=1036
Planning:
  Buffers: shared hit=315
Planning Time: 1.686 ms
Execution Time: 150.286 ms
(17 rows)

The execution took this time only 150ms! It also uses indices as we would want. Was the slow request a fluke?

A Database Call That’s Slow

The storage parameter intrigued me though so I thought I’d peek into how many rows there are per storage value:

nextcloud=> SELECT storage, COUNT(*)
            FROM oc_filecache
            GROUP BY storage;

 storage |  count
---------+---------
       1 |       4
       2 | 1604311
       5 |  583191
       6 |       4
       7 |    1071
       8 |       9
       9 |  149197
(7 rows)

Earlier, we ran the explain with WHERE storage=5. Let’s try storage=2 which has more rows:

Gather  (cost=1000.00..120228.77 rows=20 width=211)
        (actual time=4971.440..4995.646 rows=0 loops=1)
  Workers Planned: 2
  Workers Launched: 2
  Buffers: shared hit=49096 read=48279 dirtied=53
  I/O Timings: read=13205.555
  ->  Parallel Seq Scan on oc_filecache file
        (cost=0.00..119226.77 rows=8 width=211)
        (actual time=4962.747..4962.748 rows=0 loops=3)
    Filter: (((name)::text = '.nomedia'::text) [...]
    Rows Removed by Filter: 779248
    Buffers: shared hit=49096 read=48279 dirtied=53
    I/O Timings: read=13205.555
Planning:
  Buffers: shared hit=315
Planning Time: 3.013 ms
Execution Time: 4995.789 ms
(14 rows)

This time, the query took 5 seconds to run! It’s doing a sequential scan and clearly doing some disk I/O.

Usually, you’ll want to run a VACUUM on a table to make sure PostgresSQL can optimize the plan:

nextcloud=> VACUUM ANALYZE oc_filecache;
VACUUM

That didn’t change the explain significantly.

By the way, we’re printing the BUFFERS values because that will show us all buffer and I/O timings. From the documentation:

Include the number of shared blocks hit, read, dirtied, and written, the number of local blocks hit, read, dirtied, and written, the number of temp blocks read and written, and the time spent reading and writing data file blocks and temporary file blocks (in milliseconds) if track_io_timing is enabled. (we did enable it!)

A hit means that a read was avoided because the block was found already in cache when needed. Shared blocks contain data from regular tables and indexes; local blocks contain data from temporary tables and indexes; while temporary blocks contain short-term working data used in sorts, hashes, Materialize plan nodes, and similar cases.

The fast query from above, the one hitting storage 5, has only hits! No wonder it’s fast. On the other hand, the slow one hitting storage 2 has a lot of reads.

The number of blocks dirtied indicates the number of previously unmodified blocks that were changed by this query; while the number of blocks written indicates the number of previously-dirtied blocks evicted from cache by this backend during query processing.

There’s not a significant difference in the dirtied buffers between both slow and fast queries, so we won’t be looking too much into that. Let’s instead see how to reduce shared buffer reads.

An Insight About Caching

To make sure the explain plan wasn’t a fluke, I ran it a few times in a row to try to load the results in the cache. That actually made an impact. At one point, the query with storage = 2 managed to run in 300ms, instead of the 5 seconds we saw earlier. What changed wasn’t the shared buffers read value but the I/O timing! I suppose this means the OS itself or the disk was caching the results but I don’t know how to confirm this. Let’s try to tune the shared buffers to improve caching.

A few things to note before we continue. The drive where the PostgreSQL database is located is an NVMe drive. This is pretty fast already.

$ df /var/lib/postgresql
Filesystem                                             1K-blocks     Used Available Use% Mounted on
/dev/disk/by-uuid/6da26333-4d11-4b29-9509-6dcc8151013b 114304164 51399260  57052388  48% /

$ readlink /dev/disk/by-uuid/6da26333-4d11-4b29-9509-6dcc8151013b
../../sda1

$ sudo fdisk -l /dev/sda
Disk /dev/sda: 119.24 GiB, 128035676160 bytes, 250069680 sectors
Disk model: SK hynix SC311 S

Not the fastest NVMe hard drive but it will have to do.

The other thing to note is the results do not stay in the cache very long. If I wait even just 10 seconds and re-issue the explain, the query already takes more than a second again from the 300ms. I got the feeling optimizing this is going to be tricky.

I found the following website that allows you to input your PostgreSQL machine specs and outputs what settings you should set. https://pgtune.leopard.in.ua/. My server’s spec are:

Intel(R) Core(TM) i5-6600T CPU @ 2.70GHz (2 core with hyper-threading, so 4 in total)
7 GB + 512 MB memory

But since I know this server is shared by quite a few services, I only allotted 2Gb of memory and 1 CPU core. Specifically, I inputted:

DB Version = 14
OS Type = "linux"
DB Type = "web"
Total Memory (RAM) = "2 GB"
CPUs num = 1
Connections num = 100
Data Storage = "sdd"

I used the web application (web) profile as the description matched the most.

This gave me:

max_connections = "100";
shared_buffers = "512MB";
effective_cache_size = "1536MB";
maintenance_work_mem = "128MB";
checkpoint_completion_target = "0.9";
wal_buffers = "16MB";
default_statistics_target = "100";
random_page_cost = "1.1";
effective_io_concurrency = "200";
work_mem = "2621kB";
huge_pages = "off";
min_wal_size = "1GB";
max_wal_size = "4GB";

That didn’t change how long the query stays cached. But that’s when I found the extension pg_prewarm. This allows you to manually cache a whole table into RAM!

Let’s try it out:

nextcloud=> SELECT pg_prewarm('oc_filecache');
 pg_prewarm 
------------
      97375
(1 row)

 Seq Scan on oc_filecache file  (cost=0.00..138449.77 rows=423562 width=211) (actual time=372.843..372.844 rows=0 loops=1)
   Filter: ((storage = 2) AND (((path)::text = 'files'::text) OR ((path)::text ~~ 'files/%'::text)))
   Rows Removed by Filter: 2338703
   Buffers: shared hit=97375
 Planning Time: 0.286 ms
 Execution Time: 372.879 ms
(6 rows)

Very promising although I’m reluctant to have this table be forced into memory by a cron job running pg_prewarm. It feels heavy handed and there’s probably a reason why Postgres doesn’t provide a way to do this out of the box. Are there any knobs I could tweak though? I’m not sure how to progress here to be honest.

Statistical Analysis

The query above is just one query though. Is continuing to investigate this query even worth it? To be sure, let’s enable the pg_stats_statement extension with:

nextcloud=> CREATE EXTENSION pg_stats_statement;

We can then issue the following query that is inspired by this blog post. I made the it so only the query’s verb (SELECT, INSERT, …) and table name are shown. Also, I order by mean_exec_time.

SELECT
  regexp_replace(
    regexp_replace(
      query,
      '^([^ ]+).*FROM ([^ ]+).*',
      '\1 \2'
    ),
    '(UPDATE|INSERT INTO) ([^ ]+).*$',
    '\1 \2'
  ) AS short_query,
  round(total_exec_time::numeric, 2) AS total_exec_time,
  calls,
  round(mean_exec_time::numeric, 2) AS mean,
  round((100 * total_exec_time /
  sum(total_exec_time::numeric) OVER ())::numeric, 2) AS percentage_cpu
from pg_stat_statements
WHERE query NOT LIKE '%ANALYZE%'
  AND query NOT LIKE '%EXPLAIN%'
  AND query NOT LIKE '%pg_stat_statements%'
ORDER BY mean DESC
LIMIT 20;

Before running the above query, I created the extension and used Nextcloud for about a week.

Times are in milliseconds. This clearly tells me that indeed this table’s worth optimizing for as it’s taking most of the places in the top 20:

             short_query              | total_exec_time | calls |  mean   | percentage_cpu 
--------------------------------------+-----------------+-------+---------+----------------
 SELECT "oc_filecache"                |      1782003.82 |   296 | 6020.28 |          36.60
 SELECT "oc_filecache"                |         4758.20 |     2 | 2379.10 |           0.10
 <insufficient privilege>             |         4297.53 |     3 | 1432.51 |           0.09
 SELECT "oc_filecache"                |        13057.10 |    10 | 1305.71 |           0.27
 SELECT oc_filecache                  |         1171.26 |     1 | 1171.26 |           0.02
 SELECT "oc_filecache"                |         2088.51 |     2 | 1044.25 |           0.04
 SELECT "oc_filecache"                |        21261.02 |    24 |  885.88 |           0.44
 SELECT "oc_filecache"                |         1905.76 |     4 |  476.44 |           0.04
 <insufficient privilege>             |          458.04 |     1 |  458.04 |           0.01
 SELECT "oc_filecache"                |         5969.22 |    14 |  426.37 |           0.12
 SELECT "oc_systemtag_object_mapping" |       131973.41 |   316 |  417.64 |           2.71
 SELECT oc_filecache                  |         1035.19 |     3 |  345.06 |           0.02
 DELETE "oc_activity"                 |         4890.14 |    16 |  305.63 |           0.10
 SELECT "oc_filecache"                |         1345.88 |     5 |  269.18 |           0.03
 UPDATE "oc_filecache"                |       361959.17 |  1739 |  208.14 |           7.43
 SELECT "oc_file_metadata"            |          130.37 |     1 |  130.37 |           0.00
 SELECT "oc_file_metadata"            |          122.03 |     1 |  122.03 |           0.00
 SELECT "oc_activity"                 |       123446.91 |  1247 |   99.00 |           2.54
 <insufficient privilege>             |          187.31 |     2 |   93.65 |           0.00
 <insufficient privilege>             |          175.60 |     2 |   87.80 |           0.00

But to be honest, I don’t know how to continue here. If anyone reading this has an idea, feel free to contact me :D

Let’s switch to another request, maybe there’s one that I can actually optimize.

A Profile That’s Harder to Optimize

This is an example of request that’s harder to optimize. Indeed, it is CPU bound as we can see because most of the time is spent cycling (in a loop):

We’ll skip investigating this one as reducing the time this one takes likely means modifying Nextcloud code or from one of its dependencies.

Analyzing a Second Interesting Profile

To find another one, I just clicked in the UI until I got a request that took longer than 5 seconds.

Unfortunately, I found one. 12 seconds! That’s a good candidate for optimization.

The profile doesn’t show much, just that PHP is doing some curl request, so we’ll need to look at the callers:

The caller stack is fully linear which is great news because there’s no loop involved at all.

We can see Nextcloud is asking for the preview of a file which gets forwarded to the OnlyOffice plugin which in turns makes an HTTP call to presumably the OnlyOffice service running on my server.

Or is it? Actually, the third to last caller is DnsPinMiddleware from lines 109-155 (lines elided in the screenshot). The related code can be seen on GitHub. This is clearly some DNS resolving code. I know I already mis-configured some things here so let’s review my configuration and see if we can spot some mistakes.

Analyzing DNS Config

I did set up the integration in Nextcloud to access my OnlyOffice service at onlyoffice.mydomain.com:

My OnlyOffice server is accessible at onlyoffice.mydomain.com (it’s not really mydomain.com of course). Let’s try a dig (I obfuscated the actual IP address and domain name):

$ nix run nixpkgs#dig onlyoffice.mydomain.com

; <<>> DiG 9.18.20 <<>> onlyoffice.mydomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7924
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;onlyoffice.mydomain.com.        IN  A

;; ANSWER SECTION:
onlyoffice.mydomain.com.  19778  IN  A  WWW.XXX.YYY.ZZZ

;; Query time: 0 msec
;; SERVER: 192.168.1.60#53(192.168.1.60) (UDP)
;; WHEN: Thu Dec 14 06:01:53 UTC 2023
;; MSG SIZE  rcvd: 60

I have my own dnsmasq server listening at 192.168.1.60 and who answered us:

;; SERVER: 192.168.1.60#53(192.168.1.60) (UDP)

It told us that the onlyoffice.mydomain.com address can be found at my public IP (obfuscated here for reasons).

onlyoffice.mydomain.com.  19778  IN  A  WWW.XXX.YYY.ZZZ

This is clearly not a good thing because it means my DNS server is answering with the public IP of my router. It should instead answer with the private IP of my server.

This has two main implications. Firstly, it means my DNS server doesn’t know this onlyoffice.mydomain.com fqdn and forwards the question to a public DNS. This is bad for performance reasons but also for privacy reasons as it’s leaking DNS calls that should be internal only. Also, this won’t work if my internet is disconnected.

Secondly, the subsequent HTTP call (which we didn’t see in the profile) will try to contact my OnlyOffice service using my public IP address instead of the private IP directly. This means at least more hops and so an increased response time. Actually, the router is smart and realizes the IP I want to reach is its own public IP so the request won’t actually go out to the internet thanks to NAT hairpinning. Here it’s helping us a bit but it bit me previously.

Anyway, the correct fix here is to make my dnsmasq service aware of OnlyOffice and answer to the DNS queries with the internal IP of my server which is 192.168.1.10:

services.dnsmasq.settings.address = [
  "/onlyoffice.mydomain.com/192.168.1.10"
];

This is the resulting change in the dnsmasq config file:

address=/onlyoffice.mydomain.com/192.168.1.10

After deploying this, dig returns the internal IP:

$ nix run nixpkgs#dig onlyoffice.mydomain.com

; <<>> DiG 9.18.20 <<>> onlyoffice.mydomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45810
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;onlyoffice.mydomain.com.     IN  A

;; ANSWER SECTION:
onlyoffice.mydomain.com.   0  IN  A  192.168.1.10

;; Query time: 0 msec
;; SERVER: 192.168.1.60#53(192.168.1.60) (UDP)
;; WHEN: Sat Dec 16 17:59:15 UTC 2023
;; MSG SIZE  rcvd: 60

Great, with that fixed, let’s reload the UI and look at remaining long running requests.

Remaining Optimizations

I’m just reloading the /apps/files/?dir=/ endpoint and there are quite a few indeed:

I see files, files, files. I bet this oc_filecache table has something to do with all these. Actually, no! It’s Redis, another dependency of Nextcloud used for caching.

All the profiles have one thing in common, they spend a lot of time calling to Redis. Sometimes 30% of the time as can be seen in the following table which I compiled from 4 profiles:

Time (ms)	# Calls	Per Call (ms)	% of Request
286	89	3.2	30.4
273	86	3.2	13.7
529	89	5.9	25
486	86	5.7	24.3

3 milliseconds per call is not much, but it adds up.

The documentation gives us a good hint as to why Redis could be slow, it could be Redis swapping to disk. To check that, we can issue the following command:

$ sudo -u nextcloud redis-cli -s /run/redis-nextcloud/redis.sock info | grep process_id
process_id:1424342

$ cd /proc/1424342

$ sudo cat smaps | egrep '^(Swap|Size)'
[...]
Size:   364 kB
Swap:   364 kB
Size:   8192 kB
Swap:   1296 kB
Size:   24576 kB
Swap:   19004 kB

Indeed, a few memory maps are pretty heavily swapped. Not good, I suppose?

That made me think, could this be the case also for PostgreSQL? I issued the same commands as above, using systemctl cat postgresql to get the PID. And Indeed, that was the case.

I don’t have a permanent fix against swapping but I could already test if swapping has an impact simply by restarting Redis and PostgreSQL. I did that, re-ran the smaps command above, verified nothing was swapped and reloaded the web UI. But that didn’t help, the request was still spending the same amount of time on Redis calls and PostgreSQL calls.

Return to the Basics

After this very disappointing optimization tour, I thought maybe I should check the overall pressure on the system.

$ free -m
      total  used  free   shared  buff/cache  available
Mem:   7663  5328   209      334        2125       1703
Swap:  7628  7271   357

There’s 1.7Gb of available memory, so that’s good, not all RAM is used. But 7.2Gb of SWAP is used! That’s really not good.

Oh well, time to understand why so much SWAP is used. I found this script in the comments of a StackOverflow answer which prints the base command followed by the PID and the used SWAP space.

$ for file in /proc/*/status ; do \
    awk '/Tgid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; \
  done \
  | grep kB \
  | sort -k 3 -n \
  | tail -n 20
.php-fpm-wrappe 2608 50092 kB
.php-fpm-wrappe 2626 55092 kB
hledger-web 945 65476 kB
dotnet 1140 69844 kB
piper 2203 81592 kB
piper 2212 81600 kB
.php-fpm-wrappe 2515 102124 kB
.php-fpm-wrappe 2344 102152 kB
.php-fpm-wrappe 2463 103080 kB
.php-fpm-wrappe 2577 103212 kB
.php-fpm-wrappe 3559 103900 kB
.php-fpm-wrappe 2359 104004 kB
mono 1155 117608 kB
dotnet 1200 126696 kB
python3.11 1681 149296 kB
.wyoming-openwa 2086 155984 kB
.hass-wrapped 2141 239184 kB
jellyfin 33101 266688 kB
.wyoming-faster 2085 808604 kB
.wyoming-faster 2083 808608 kB

Wow there’s a lot. There’s no silver bullet to fix this but I will in the meantime stop the Home Assistant voice pipeline (wyoming/piper) and see how free swap looks like.

After about one hour of waiting, the used SWAP did decrease a bit:

$ free -m
      total  used  free   shared  buff/cache  available
Mem:   7663  4837   132      332        2693       2196
Swap:  7628  6030   159

After a couple days, free SWAP is at 785.

I think there’s no secret anymore. I need more RAM :’(

Deploy to NixOS With Colmena and Without Root User

2023-12-03T00:00:00Z

Tags: nix

By default, you use the root user to deploy to a target machine. If you want to avoid that, you can create another user but getting all the configuration right is not obvious. So here’s a rundown in 3 steps.

1. Create User

First, you need to create the user and make it able to run sudo without requiring a password. Let’s pick the username nixos. The hammer way of doing it is allowing all commands to be ran with sudo without password.

users.users.nixos = {
  isNormalUser = true;
};

security.sudo.extraRules = [
  { users = [ "nixos" ];
    commands = [
      { command = "ALL";
        options = [ "NOPASSWD" ];
      }
    ];
  }
];

If you don’t do that, you’ll stumble in the following error and the build fails right afterwards:

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper

2. User Must be Trusted by Nix

Second, you must make the user a trusted user. On the target machine’s configuration, add:

nix.settings.trusted-users = [ "nixos" ];

Otherwise, although you will be able to copy derivations over to the target machine, you won’t be allowed to talk to the nix daemon to add them to the nix store:

error: cannot add path '/nix/store/00yiiplzcqzmqaw10cghbxlb4l4xibc0-i3lock-color.pam' because it lacks a signature by a trusted key

3. Make Deploy System Use New User

This step will depend on which system you use. For Colmena, add the following option in the target’s configuration:

deployment.targetUser = "nixos";

Nix Log a Flake Output

2023-12-03T00:00:00Z

Tags: nix

After running a nix build command and getting an error, the output usually prompts you to run nix log like so:

For full logs, run 'nix log /nix/store/iyp0l0h9ik0zkmmg8ryxmb8y15a32apz-self-host-blocks-manual.drv'

This is great but a bit heavy on copy/pasting. Also, if you update your code and build again, the hash will change as expected but that requires you to copy/paste again, you can’t use your terminal’s history to quickly check the logs.

One solution is to add -L (--print-build-log) cli argument to the nix build command. I use it when the output stays small but for long running output I find it just clutters the terminal output.

I just found out that you can use nix log on the same flake output you use in nix build and get to see the logs too! An example:

nix build .#manualHtml
nix log .#manualHtml

No copy/pasting involved!

Flakify This Blog

2023-12-02T00:00:00Z

Tags: nix, blog

Motivation

I wanted to use flakes to build this blog since quite a while but never got to it.

The last nudge needed was I was fed up with the current deploy process which involved a second repository (see a previous blog post for more details). It was really hindering my motivation to write blog posts. I wanted instead to deploy by just pushing to the main branch thanks to a GitHub action, like any sane person.

First Try

In one of my other projects, that’s exactly what I did thanks to a GitHub action, so I copied that action and just updated the part on how to build the docs.

name: Deploy blog

on:
  # Runs on pushes targeting the default branch
  push:
    branches: ["main"]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  # Single deploy job since we're just deploying
  deploy:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}

    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Install nix
        uses: cachix/install-nix-action@v20

      - name: Build docs
        run: |
          nix-build -v

          ./result/bin/site build

          mkdir -p ibizaman.github.io
          ./result/bin/site deploy

          # see https://github.com/actions/deploy-pages/issues/58
          cp \
            --recursive \
            --dereference \
            --no-preserve=mode,ownership \
            ibizaman.github.io \
            public

      - name: Setup Pages
        uses: actions/configure-pages@v3

      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: ./public

      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2

But nix-build failed with:

file 'nixpkgs' was not found in the Nix search path
(add it using $NIX_PATH or -I)

This is because the release.nix file uses the bracket syntax to find nixpkgs in the $NIX_PATH.

Now, cachix/install-nix-action does have a way to add nixpkgs to the nix path, so I could’ve done that. Instead, I took the time to switch to flakes to avoid needing to change the $NIX_PATH at all.

Use Flakes to Build Hakyll

Now, I tried a few variations of a flake.nix file. First, copying the release.nix and just adding an inputs section. Then, I tried the template from haskell.nix using haskell.nix. In both cases, I was recompiling a lot of Haskell packages.

Finally, I found hakyll-flakes. My flake.nix file is now:

{
  description = "ibizaman's blog";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    hakyll-flakes.url = "github:Radvendii/hakyll-flakes";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { nixpkgs, hakyll-flakes, flake-utils, ... }:
    let
      supportedSystems = [
        "x86_64-linux"
        "x86_64-darwin"
        "aarch64-linux"
        "aarch64-darwin"
      ];
    in
      flake-utils.lib.eachSystem supportedSystems (system:
        hakyll-flakes.lib.mkAllOutputs {
          inherit system;
          name = "site";
          src = ./.;
          websiteBuildInputs = with nixpkgs.legacyPackages.${system}; [
            # rubber
            # texlive.combined.scheme-full
            # poppler_utils
          ];
        });
}

Building the project is done with nix build .#website which generates the html files under result/ folder:

$ ls -1 result
about.html
archive.html
atom.xml
contact.html
css
images
index.html
posts
tags
tags.html

Also, nix run . watch is very useful.

Deploy with GitHub Action

I just needed to update the GitHub action:

- name: Build docs
  run: |
    nix \
      --print-build-logs \
      --show-trace #\
      build .#website

    # see https://github.com/actions/deploy-pages/issues/58
    cp \
      --recursive \
      --dereference \
      --no-preserve=mode,ownership \
      result \
      public

Finally, I needed to configure the Settings > Environments > Configure github-pages to allow deploying from my main branch.

The two relevant commits are 6969b9 and 4f3e33.

The only thing to remember now is to correctly commit the blog post, which I already managed to forget once :D

Addendum

I needed to disable the pages integration on my second - now useless - repo so that I could add the correct custom domain to deploy to to my current repo. I just went and archive that second repo altogether.

Add Patches to Nixpkgs

2023-11-26T00:00:00Z

Tags: nix

edit 2024-06-15: fix snippet

Following the trail from discourse to a closed GitHub pull request to a comment in a second GitHub pull request :) I’m happy to share that we can now easily apply patches to nixpkgs itself.

This is pretty neat because now, if you want to apply an open pull request immediately to your project, you can easily do it. Well, the code is not particularly elegant but it’s easier than cloning nixpkgs and merging the PR yourself…

This is what applying a patch looks like now:

let
  system = "x86_64-linux";
  originPkgs = nixpkgs.legacyPackages.${system};

  patches = [
    (originPkgs.fetchpatch {
      url = "https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/315018.patch";
      hash = "sha256-8jcGyO/d+htfv/ZajxXh89S3OiDZAr7/fsWC1JpGczM=";
    })
  ];
  patchedNixpkgs = originPkgs.applyPatches {
    name = "nixpkgs-patched";
    src = nixpkgs;
    inherit patches;
  };
in
  {
    nixpkgs = import patchednixpkgs { inherit system; };
  };

On GitHub, to get a patch from a PR, you must go to the PR (for example https://github.com/NixOS/nixpkgs/pull/268168), edit the URL to add a .patch suffix (https://github.com/NixOS/nixpkgs/pull/268168.patch) and that will redirect you to the final URL (the one in the snippet above for this example).

Switch to Colmena for Local Deploys

2023-11-08T00:00:00Z

Tags: nix

The Incentive section explains why I even wanted to do this. Feel free to skip to the second paragraph.

Incentive
Switching From Nixos-Rebuild to Colmena

Incentive

I’ve been deploying to my two server boxes with Colmena since a few months now and I’m very happy.

My laptop is using NixOS and I’ve been “deploying” to it using nixos-rebuild switch since the start.

Recently, I wanted to mount a webdav folder from one of my servers to my laptop, to do that I created a NixOS module (more on that in a later post) that sets up the following systemd mount:

$ systemctl cat home-me-mymount.mount 
# /etc/systemd/system/home-me-mymount.mount

[Unit]
After=network-online.target
Description=Webdav mount point
Wants=network-online.target

[Mount]
Options=uid=1000
TimeoutSec=15
Type=davfs
What=https://mydomain.com/webdav/mount
Where=/home/me/mymount

I then tried to start the mount:

$ sudo systemctl start home-me-mymount.mount 
systemd[1]: Mounting Webdav mount point...
mount.davfs[1231360]: davfs2 1.7.0
mount.davfs[1231360]: opening /etc/davfs2/secrets failed
systemd[1]: home-me-mymount.mount: Mount process exited, code=exited, status=255/EXCEPTION
systemd[1]: home-me-mymount.mount: Failed with result 'exit-code'.
systemd[1]: Failed to mount Webdav mount point.

Right, we need to create a file at /etc/davfs2/secrets with the password needed to access the Webdav folder.

I obviously didn’t want to create this file by hand. Also, I want this secret to be stored encrypted and be deployed in the correct place. We’re using Nix after all and those are good practices.

I have been using sops-nix to deploy secrets with great success and declaring this secrets looks like so using sops-nix:

sops.secrets."webdav" = {
  sopsFile = ./secrets.yaml;
  mode = "0600";
  path = "/etc/davfs2/secrets";
};

The snippet above is only half of the job though, the remaining part is actually creating the secrets.yaml and encrypting it. I’ll leave that out of this blog post as it’s not relevant here. Also, everything is explained in the sops-nix repo’s readme file.

The issue here is we can’t use nixos-rebuild to deploy keys with sops-nix or any other key management system. You need to switch to nixops, colmena or any other such deploy system listed here.

Switching From Nixos-Rebuild to Colmena

Colmena allows you to use to deploy locally and switching to it is pretty easy in my case. This was the flake output related to my laptop before:

- nixosConfigurations.mylaptop = nixpkgs.lib.nixosSystem {
-   inherit system;
-   specialArgs = {
-     inherit nixpkgs;
-   };
-   modules = [
-     inputs.home-manager.nixosModules.default
-     ./machines/laspin-configuration.nix
-     ./machines/laspin-home.nix
-   ];
- };

I removed those lines and created the following ones:

colmena = {
  meta = {
    nixpkgs = import inputs.nixpkgs {
      system = "x86_64-linux";
    };
    specialArgs = inputs;
  };
+  mylaptop = {
+    deployment = {
+      # Allow local deployment with `colmena apply-local`
+      allowLocalDeployment = true;
+
+      # Disable SSH deployment. This node will be skipped in a
+      # normal `colmena apply`.
+      targetHost = null;
+    };
+
+    imports = [
+      inputs.home-manager.nixosModules.default
+      inputs.sops-nix.nixosModules.default
+      ./machines/laspin-configuration.nix
+      ./machines/laspin-home.nix
+    ];
+  };
};

I just copy pasted the modules field to the imports field, added the deployment field to tell colmena to deploy locally, added the sops-nix module because of my use case and… That was it!

Note that for Colmena matches the hostname with the attribute name mylaptop. I actually have networking.hostName = "mylaptop"; set in my config.

To deploy on my laptop, I switched from:

$ sudo nixos-rebuild switch

to:

$ colmena apply-local --sudo

Which created the secret file I needed in the first place:

$ readlink /etc/davfs2/secrets 
/run/secrets/webdav

Migrate Nextcloud User To LDAP Account

2023-10-26T00:00:00Z

Tags:

The Issue
Prerequisite
The CLI Way to Migrate Data
The Web UI Way to Migrate Data
Conclusion

The Issue

When setting up Nextcloud for the first time, I created some users. Everything was fine.

But then, I wanted to integrate with LLDAP to manage the users through a LDAP server. The issue arose from the fact that the user created from Nextcloud is not the same as the one created from the LDAP server.

This means the new user has none of the data of the old user. We need to migrate data to the new user and then delete the old one.

For example, with a hypothetical user called “bob”:

Created from Nextcloud:
- Display name: bob
- Username: bob
- Path on the filesystem: $nextcloud_root/data/bob
Created from LLDAP:
- Display name: bob
- Username: e7749dfe-9740-440d-b857-0c0c508c6876
- Path on the filesystem: $nextcloud_root/data/e7749dfe-9740-440d-b857-0c0c508c6876

As you can see, the username ends up being a UUID. For Nextcloud, everything about these two users is different.

You can see the display name and username of a user by going to the /settings/users endpoint.

Prerequisite

Before continuing on and (spoiler) migrating data from one user to the other, you first need to have created both users on Nextcloud. I assume the old Nextcloud user was already created, otherwise you have no reason to read this post. But the new LDAP user is probably not yet created.

To do that, you have to login into Nextcloud with that new LDAP user. Nextcloud is smart enough to handle having two users with the same display name but two different username like in the example above.

To connect as the old bob user, just use bob and the old password. To connect as the new bob user, use bob and the new password.

The CLI Way to Migrate Data

CLI transfer ownership is done with the following command:

occ files:transfer-ownership \
  [options] [--] \
  <source-user> <destination-user>

Running occ files:transfer-ownership --help tells us:

All files and folders are moved to another user - outgoing shares and incoming user file shares (optionally) are moved as well.

There’s a --move option but I could never use it successfully. I had assume without setting this option that the folders would be somehow copied but no, they are effectively transferred. Every time I tried to use it, I get the following error:

Destination path does not exists or is not empty

Anyway, with our hypothetical “bob” user, the command becomes:

occ files:transfer-ownership \
  bob e7749dfe-9740-440d-b857-0c0c508c6876

Note the ownership transfer is executed right away.

The Web UI Way to Migrate Data

Here, we go in the UI to transfer ownership of folders. Later, the nextcloud cron job will take care of actually moving the folders.

In the old account you want to move files from, go to Settings > Sharing
In the Files section, click on Choose file or folder to transfer.

Since I had more than 10 folders to transfer, I tried playing it smart and selecting “no” folder, assuming this would transfer all the folders. But no, although the UI lets you do that, the cron job fails later on.

The trick here was just to move all folders I wanted to transfer into a new folder and transfer that folder.
Choose the new owner.
Click on “Transfer”.
Log out then log in with the new user and accept the ownership transfer.
Wait for the cron job to kick in.

This schedules the transfer through the nextcloud cron job. So to see if it works, you’ll need to monitor that systemd service. In my case, it failed because the job couldn’t access the perl binary. I fixed that by making it available to the cron job.

To avoid waiting for the cron job to kick in on its own, just start it manually with systemctl start nextcloud-cron.service.

Conclusion

Now you can transfer all data from legacy accounts to new LDAP managed accounts using the CLI or the Web UI. Of course, this can be used to transfer from any user to any user.

Remember to test the transfer by monitoring the nextcloud cron job if you’re using the Web UI way.

What's up with Nextcloud webdav slowness?

2023-08-12T00:00:00Z

Tags: nextcloud, performance

Accessing Nextcloud’s web UI, using http, seems reasonably fast but accessing the webdav interface is painfully slow. Like, it doesn’t even seem to work at all slow.

What’s up with that?

Let’s embark in a journey on how to profile Nextcloud.

Did anyone stumbled onto this issue already?

Searching for slow Nextcloud webdav brought up this post. It details what the OP tried to fix their issue but with a not satisfying conclusion. Anyway, they provide very useful curl commands to test timing:

Http timing:

time curl \
  -H 'Cookie:$SESSION' \
  'https://$MYINSTANCE/apps/dashboard/#/'

$MYINSTANCE would for example be nextcloud.domain.com, which I’m going to use from now on.

Webdav timing:

time curl \
  -X 'PROPFIND' \
  -H 'Depth: 1' \
  -u '$USER:$PASSWORD' \
  'https://$MYINSTANCE/remote.php/dav/files/$USER/'

I advice instead to leave out the $PASSWORD as curl will ask for it. Otherwise, the password will appear at least in your shell history.

Try to replicate

I tweaked the above commands by adding the -I flag so I could see the headers but not the response:

time curl -I ...

Http timing is not bad:

$ 
real    0m0.464s
user    0m0.086s
sys     0m0.012s

Webdav timing is abysmal:

real    0m26.931s
user    0m0.095s
sys     0m0.011s

Is the issue in the Nextcloud code or elsewhere?

I need to know first if the issue lies in the Nextcloud php code, in the phpfpm server serving the php code or in the nginx proxy that’s serving Nextcloud publicly.

To know that, I enabled the access log in nginx with the following snippet. The convoluted string actually produces a valid json format which makes it super easy to investigate.

services.nginx.logError = "stderr warn";
services.nginx.appendHttpConfig = ''
  log_format apm
    '{'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"time_local":"$time_local",'
    '"request":"$request",'
    '"request_length":"$request_length",'
    '"server_name":"$server_name",'
    '"status":"$status",'
    '"bytes_sent":"$bytes_sent",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"referrer":"$http_referrer",'
    '"user_agent":"$http_user_agent",'
    '"gzip_ration":"$gzip_ratio",'
    '"post":"$request_body",'
    '"upstream_addr":"$upstream_addr",'
    '"upstream_status":"$upstream_status",'
    '"request_time":"$request_time",'
    '"upstream_response_time":"$upstream_response_time",'
    '"upstream_connect_time":"$upstream_connect_time",'
    '"upstream_header_time":"$upstream_header_time"'
    '}';

  access_log syslog:server=unix:/dev/log apm;
  '';

Take a look in the nginx manual to see what the upstream_* variables mean.

This is what I see for the curl request above. I formatted the json output for easier reading.

Sep 17 21:08:45 myserver nginx[287808]: myserver nginx: 
{
  "remote_addr": "$ADDR",
  "remote_user": "$USER",
  "time_local": "17/Sep/2023:21:08:45 +0000",
  "request": "PROPFIND https://nextcloud.domain.com/remote.php/dav/files/$USER/ HTTP/2.0",
  "request_length": "185",
  "server_name": "$INSTANCE",
  "status": "207",
  "bytes_sent": "1401",
  "body_bytes_sent": "633",
  "referrer": "-",
  "user_agent": "curl/8.2.1",
  "gzip_ration": "-",
  "post": "-",
  "upstream_addr": "unix:/run/phpfpm/nextcloud.sock",
  "upstream_status": "207",
  "request_time": "26.058",
  "upstream_response_time": "26.057",
  "upstream_connect_time": "0.000",
  "upstream_header_time": "26.057"
}

This indicates the problem lies in the Nextcloud code as the time spent in the “upstream” part is the same as the total request time.

Let’s profile Nextcloud

We need to enable the xdebug extension and add some lines to the php.ini config. In NixOS, that’s done quite easily by adding:

services.nextcloud = {
  # Disable the minifier and outputs some additional
  # debug information.
  extraOptions = {
    "debug" = true;
    "filelocking.debug" = true;
  };

  # Enable profiling xdebug mode and save file only
  # if the trigger_value is given in the request.
  phpOptions = {
    "xdebug.mode" = "profile";
    "xdebug.trigger_value" = "debug_me";
    "xdebug.output_dir" = "/var/log/xdebug";
    "xdebug.start_with_request" = "trigger";
  };
  
  # Adds xdebug extension
  phpExtraExtensions = all: [ all.xdebug ];
};

# Create output folder with correct permission.
systemd.services.phpfpm-nextcloud.preStart = ''
  mkdir -p /var/log/xdebug
  chown -R nextcloud: /var/log/xdebug
'';

This will create a cachegrind.out.XXXXXX file under the /var/log/xdebug directory for each request having the cookie XDEBUG_PROFILE=debug_me set.

You could leave out the start_with_request option but I wouldn’t advice doing that as normal usage of the Nextcloud instance will also produce a lot of cachegrind.out.XXXXXX files which are just noise.

Anyway, the full curl request to time webdav becomes:

time curl -I -s \
  -X 'PROPFIND' \
  -H 'Depth: 1' \
  -u '$USER' \
  --cookie 'XDEBUG_PROFILE=debug_me' \
  'https://$MYINSTANCE/remote.php/dav/files/$USER/'

This is the full output of the response:

Enter host password for user '$USER':
HTTP/2 207 
server: nginx
date: Sun, 17 Sep 2023 21:08:45 GMT
content-type: application/xml; charset=utf-8
x-xdebug-profile-filename: /var/log/xdebug/cachegrind.out.552470
set-cookie: oc_sessionPassphrase=XYZ; path=/; secure; HttpOnly; SameSite=Lax
content-security-policy: default-src 'none';
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
set-cookie: occXYZ=XYZ; path=/; secure; HttpOnly; SameSite=Lax
set-cookie: cookie_test=test; expires=Sun, 17 Sep 2023 22:08:44 GMT; Max-Age=3600
vary: Brief,Prefer
dav: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nextcloud-checksum-update, nc-calendar-search, nc-enable-birthday-calendar
x-request-id: 9Yjd8BK6bGf92s7qXpWW
x-debug-token: 9Yjd8BK6bGf92s7qXpWW
strict-transport-security: max-age=31536000; includeSubDomains


real    0m28.410s
user    0m0.050s
sys     0m0.007s

Again that abysmal execution time. Here, we’re interested in the x-xdebug-profile-filename response header which shows us the corresponding profiling file.

I will open that file in KCacheGrind. I downloaded that file on my laptop then started KCacheGrind with nix run nixpkgs#kcachegrind.

Here is what I saw, and my jaw dropped (open image in a new tab to be able to zoom):

I’m not a KCacheGrind expert but let me give you a primer. Each line shows:

a function call in the Function column,
how many times that function got called in the Called column,
how long was spent somewhere inside that function and any of its descendants in the Incl column,
and how long was spent somewhere inside that function and none of its descendants in the Self column.

Here we can see the overwhelming majority of the time is spent inside 1 call to the php::usleep function which is called once by the OC\Security\Bruteforce\Throttler->getDelay function.

All the slowness was thus due to Nextcloud’s bruteforce avoidance feature. Somehow it thought someone was trying to bruteforce access to the server. But why did it activate at all as I’m the only user?

The issue was seemingly unrelated

Looking back in the logs I could see that the remote_addr of the request was pointing to my router instance and not to the user’s real IP address.

Thanks to @bb77 over in the Nextcloud forum, I learned my issue was due to NAT hairpinning. This allows, I quote wikipedia, “a machine on the LAN is able to access another machine on the LAN via the external IP address of the LAN/router”.

This is all great but this means the Nextcloud server was seeing all internal requests as coming from the same IP address, the one of my router, leading to more bruteforce mitigation than necessary.

I circumvented by using split DNS by adding nextcloud.domain.com to my internal dns server with:

services.dnsmasq = {
  extraConfig = ''
    address=/$MYISNTANCE/192.168.1.10
  '';

Where 192.168.1.10 is the internal IP of the server hosting Nextcloud.

Now, the webdav request finishes in 8.188s and the remote_addr shows correctly the one from my laptop. It’s still slow and more profiling showed the issue lies now in ldap calls. But I’ll leave that for a later blog post.

Backup Home-Assistant with and without Nix

2023-07-02T00:00:00Z

Tags: nix, server

This post will show one way to backup Home Assistant automatically. I’ll use Nix to set things up but I will also show the resulting configuration files so you can follow along even if you do not use Nix.

The idea is to have home assistant itself create a backup using an automation that is scheduled to run on a regular basis. Then, you can use whatever method to store that backup in a secure location.

Backup Automation

Create the following shell command (excerpt from configuration.nix):

shell_command:
  delete_backups: find /var/lib/hass/backups -type f -delete

And create the following automation:

alias: Create Backup on Schedule
mode: single
trigger:
- minutes: '5'
  platform: time_pattern
action:
- data: {}
  service: shell_command.delete_backups
- data: {}
  service: backup.create

This script will run on the 5th minute of every hour to 1/ delete old backups 2/ create a new backup. The backup will live inside the Home Assistant’s backup/ folder.

In Nix, you would do:

services.home-assistant = {
  config = {
    "automation manual" = [
      {
        alias = "Create Backup on Schedule";
        trigger = [
          {
            platform = "time_pattern";
            minutes = "5";
          }
        ];
        action = [
          {
            service = "shell_command.delete_backups";
            data = {};
          }
          {
            service = "backup.create";
            data = {};
          }
        ];
        mode = "single";
      }
    ];

    shell_command = {
      delete_backups = "find ${config.services.home-assistant.configDir}/backups -type f -delete";
    };
  };
};

Note that I combine declarative and UI defined automations.

Allow backup user to access Home Assistant backup folder

Create a backup user and make it member of the Home Assistant group:

useradd backup --system --groups hass

In nix:

users.groups.hass = {
  members = [ "backup" ];
};

Now, we need to ensure the backup user has access to the Home Assistant folder by enabling the “read” and “execute” group bits. First, by editing the Home Assistant service config and updating the UMask settings:

systemctl edit home-assistant.service

Then enter:

UMask=0027

Finally, restart the service.

In nix:

users.users.hass.homeMode = "0750";

systemd.services.home-assistant.serviceConfig = {
  UMask = lib.mkForce "0027";
};

Second, you need to update the files already created by Home Assistant:

sudo find /var/lib/hass -type d -exec chmod -r g+rx '{}' ';'
sudo find /var/lib/hass -type f -exec chmod -r g+r '{}' ';'

Display Counter Increase in Grafana

2023-06-25T00:00:00Z

Tags: grafana

I had this problem where in grafana I wanted to show the increase of a gauge or counter over time. I didn’t want to show the absolute value, but the increase.

There is no builtin function to handle showing increases where a time serie possibly has null values and drops, like for example after a deploy where it starts from 0. So here are the steps to achieve this.

Say you have this time series:

and you want to produce this:

We go from 7 to 8 at the start, then go from 0 to 5 at the end, that makes a total increase of 6.

First, you need to duplicate the time series with an offset of $__interval, here I plot both without (A) and with offset (B):

A:  sum (my_gauge)
B:  sum (my_gauge offset $__interval)

A naive A-B operation does not produce a good result: the total is wrong and we can see negative values

A:  sum (my_gauge)
B:  sum (my_gauge offset $__interval)
C:  $A - $B

let’s first get rid of negative values:

C:  ($A - $B) >= 0

The total is better, but it’s not 6 just yet.

You can see where the vertical bar is, we do not register the jump from 0 to 1, because there is no data point. To add data points, we need to do this:

A:  sum (my_gauge) or vector (0)
B:  sum (my_gauge offset $__interval) or vector (0)

If we again print the difference, we can see the total is accurate:

C:  ($A - $B) >= 0

But there’s a not pleasing gap there when the values drop. The final calculation is:

C:  ($A - $B) >= 0 or vector (0)

The actual calculation I have is this ~~beautiful~~ horrendous formula:

(
    (sum (my_gauge)                    or vector (0))
  - (sum (my_gauge offset $__interval) or vector (0))
) >= 0 or vector (0)

which gives:

Declarative WiFi with Encrypted Secret on NixOS

2023-05-15T00:00:00Z

Tags: server, nix

This blog post recaps how to install NixOS with only a WiFi connection, how to manage the WiFi connection declaratively and how to encrypt the passphrase during deploy.

This is only really useful for a machine that will not move much. If you’re trying to configure WiFi on a laptop, I would recommend using services.networkmanager and configure the connection manually.

Why Encrypt the Passphrase
Enable WiFi Manually
Initial Install WiFi Configuration
Deploy with Encrypted Secret

Why Encrypt the Passphrase

Like for any non-encrypted secret in NixOS, two issues arise if you put the passphrase in clear in the configuration:

the passphrase will be stored in the nix store of both the build machine and the target machine
and the passphrase will be stored in the git repo you use to manage your deploy.

The solution to the first issue is to use something like the deployment.keys option that is supported by most deployment tools.

The solution to the second issue is to encrypt the secret in the repo, and decrypt it on the target machine, after deploy. Multiple tools exist to handle this.

In this post, we will use sops-nix which provides a solution for both issues.

Enable WiFi Manually

A little aside before we start. If something goes wrong, you can always setup WiFi manually with the commands from the wpa_supplicant and dhcpcd Arch Linux wiki. The only difference with the wiki is you do not need to install the commands as they come with NixOS.

sudo wpa_supplicant -B \
     -i wlan0 \
     -c <(wpa_passphrase SSID PASSPHRASE)

touch dhcpcd.conf && \
    sudo dhcpcd --config dhcpcd.conf

Initial Install WiFi Configuration

Assuming you just booted on NixOS for the first time on the target machine and you made some edits to configuration.nix, your next step is to run nixos-rebuild switch. But first, you need a working internet connection.

What is following looks like a convoluted way to set WiFi up but it sets us up nicely to be able to declaratively set the WiFi connection with an encrypted password later on.

In the machine’s configuration.nix, add:

networking.wireless = {
  enable = true;
  environmentFile = "/run/secrets/MY_SSID_PSK";
  networks = {
    "MY_SSID" = {
      psk = "@MY_SSID_PSK@";
    };
  };
};

Replace MY_SSID with the name of the SSID you will be connecting to.

Then create the file /run/secrets/MY_SSID_PSK with the following content:

MY_SSID_PSK=theactualpassphrase

Deploy with Encrypted Secret

Like we said earlier, we will use nix-sops to encrypt the secret at rest and during deploy.

A few prerequisites: - You copied over the machine’s configuration.nix locally which includes the networking.wireless section we added earlier. - You created a public/private key pair that allows you to ssh into the target machine.

Now, to actually encrypt the secret, we will follow the nix-sops readme file. The gist is:

Install the necessary packages to run the commands:
```
nix shell nixpkgs#ssh-to-age nixpkgs#sops
```
You need the latest ssh-to-age binary as the one provided in 21.11 does not have all the necessary arguments.
Create an age secret from that public/private key pair used to connect to the target machine.
```
ssh-to-age -private-key \
           -i ~/.ssh/TARGET_HOSTNAME \
           -o ~/.config/sops/age/TARGET_HOSTNAME.txt
age-keygen -y ~/.config/sops/age/TARGET_HOSTNAME.txt
```
Use the output of that last command for admin_nixos later on.

Also, replace TARGET_HOSTNAME with the actual hostname of the target machine.

If the private key uses a passphrase, you’ll first need to export an environment variable with the passphrase:
```
read -s SSH_TO_AGE_PASSPHRASE
export SSH_TO_AGE_PASSPHRASE
```
Get the age secret from the target machine
```
ssh-keyscan -t ed25519 TARGET_MACHINE_IP | \
    ssh-to-age
```
Use the output of that command for server_TARGET_HOSTNAME later on.

Note here I am using the IP of the target machine as ssh-keyscan was failing to retrieve anything with the hostname. I do not know why.
Then fill in .sops.yaml with:
```
keys:
  - &admin_nixos age1...
  - &server_TARGET_HOSTNAME age1...
creation_rules:
  - path_regex: secrets/[^/]+\.yaml$
    key_groups:
    - age:
      - *admin_nixos
      - *server_TARGET_HOSTNAME
```
That file should be living in your repository used for deploys.

A few replacements are needed in the file:
- Replace age1... string for admin_nixos with the value we obtained at step 2.
- Replace age1... string for server_TARGET_HOSTNAME with the value we obtained at step 3.
- Replace TARGET_HOSTNAME with the actual hostname of the target server.
Create the encrypted secret file:

mkdir -p secrets
sops secrets/secrets.yaml

The content of the file should be the content of the file in /run/secrets we created earlier: MY_SSID_PSK=theactualpassphrase

Wire up SOPS in the configuration.nix:

sops = {
  defaultSopsFile = ./secrets/secrets.yaml;
  age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  age.keyFile = "/home/ME/.config/sops/age/TARGET_HOSTNAME.txt";
  secrets."MY_SSID_PSK" = {};
};

Now, next time you will deploy, sops will use the secret file, send it over to the target machine when deploying, decrypt the file and populate the content of /run/secrets/MY_SSID_PSK.

Install NixOS on a Raspberry PI

2023-05-12T00:00:00Z

Tags: server, nix

I am writing this blog post because, although all the documentation is available online, it is not always obvious how all the parts should fit together.

Install NixOS
- Aside on WiFi
Activate the System Manually
Configure SSH Public Key Access
Remove SSH Password Access
Deploy to the Raspberry PI with Colmena
Conclusion

Install NixOS

Follow the Official NixOS on Raspberry Pi wiki. You should be able to download a pre-built image through Hydra which was nice. The guide will walk you through writing the image to a SD card and booting up the PI for the first time.

After configuration.nix file gets generated on first boot, modify it to:

Change the hostname.
```
networking.hostName = "RPI_HOSTNAME";
```
Enable ssh daemon, which automatically opens the port 22 in the firewall.
```
services.openssh.enable = true;
```

Add a nixos user that can sudo that will be used to deploy.

users.users.nixos = {
  isNormalUser = true;
  # Enable ‘sudo’ for the user:
  extraGroups = [ "wheel" ];
  packages = with pkgs; [
    # ...
  ];
};

And enable sudo without entering the password.

security.sudo.extraRules = [
  { users = [ "nixos" ];
    options = [ "NOPASSWD" ];
  }
];

You can choose another user name here, no need to stick to nixos.

You should not need to modify the hardware-configuration.nix file.

Aside on WiFi

If you need WiFi, check out my blog post on using WiFi with NixOS. Like for this post, all the documentation is out there but setting up WiFi declaratively with an encrypted secret is not trivial.

If you connected the Raspberry PI with an Ethernet cable to the router, it should have automatically setup access to the internet.

Activate the System Manually

With a functioning internet connection, run:

sudo nixos-rebuild switch

Also, set a password for the user we just created.

passwd nixos

Note that if you chose another user name than nixos, the nixos user we were using until now does not exist anymore. You will need to exit the session and login as the user we configured earlier.

In the rest of the post, I will assume you did stick to nixos so every time you see nixos, replace it by the user name you chose.

Configure SSH Public Key Access

Now, you can access the Raspberry PI using its IP address and the nixos user’s password. We will configure a private-public key pair instead and disable ssh password access.

On your laptop - not the Raspberry PI - run:

ssh-keygen -t ed25519 \
           -f ~/.ssh/RPI_HOSTNAME \
           -C "nixos@LAPTOP_HOSTNAME"

Personally, I do use a passphrase when generating the key and add the key to ssh-agent. I will add a post about that later.

Now, copy over the public key with:

ssh-copy-id -i ~/.ssh/RPI_HOSTNAME RPI_IP_ADDRESS

And update your laptop’s ssh config. Somewhere in your NixOS config you should add:

program.ssh.matchBlocks = {
    "RPI_HOSTNAME" = {
        user = "nixos";
        iidentityFile = "/home/ME/.ssh/RPI_HOSTNAME";
    };
};

Now, you can ssh in with ssh RPI_HOSTNAME.

Remove SSH Password Access

It is not ideal to leave this, so add the following to the Raspberry PI’s configuration.nix.

services.openssh.permitRootLogin = "no";
services.openssh.passwordAuthentication = false;

Deploy to the Raspberry PI with Colmena

Colmena is one of the existing ways to deploy a configuration to a NixOS machine.

Add the machine to a flake.nix:

{
  outputs = {
    colmena = {
      meta = {
        nixpkgs = import nixpkgs {
          system = "x86_64-linux";
        };
      };

      RPI_HOSTNAME = { name, nodes, ... }: {
        deployment = {
          # Must correspond to the hostname in the
          # ssh config.
          targetHost = "RPI_HOSTNAME";
          # Needed to build on the Raspberry PI
          # because the laptop is a x86 architecture.
          buildOnTarget = true;
          # The user that can do password-less sudo.
          targetUser = "nixos";
        };
        networking.hostName = name;
      
        # The configuration.nix file copied from the
        # Raspberry PI.
        imports = [
          ./machines/RPI_HOSTNAME-configuration.nix
        ];
      };
    };
  };
};

Copy over the Rasbpberry PI’s configuration.nix and hardware-configuration.nix files to a local machines/ folder and reference the former in the import above. You should have the following files:

flakes.nix that imports
machines/RPI_HOSTNAME-configuration.nix that imports
machines/RPI_HOSTNAME-hardware-configuration.nix

Finally, run:

colmena apply

It should work for a few minutes - around 15 the first time in my case - then say “activation successful”.

Conclusion

We just installed NixOS on a Raspberry PI and were able to deploy to it.

Quickly create a post from Emacs

2023-05-09T00:00:00Z

Tags: emacs, hakyll

With the following snippet, I can call M-x hakyll-blog-new-post, fill in the title and get started writing right away.

(defgroup hakyll-blog nil
  "Hakyll Blog."
  :group 'applications)

(defcustom hakyll-blog-dir
  "~/blog"
  "Hakyll blog directory."
  :type 'string
  :group 'hakyll-blog)

(defcustom hakyll-blog-file-time-format
  "%Y-%m-%d"
  "Hakyll blog post filename time format."
  :type 'string
  :group 'hakyll-blog)

(defun hakyll-blog-new-post (title)
  "Create new blog post under `hakyll-blog-dir' with given TITLE."
  (interactive "sBlog post title: ")
  (find-file (hakyll-blog--file-format title))
  (insert (format "---\ntitle: %s\ntags: \n---\n\n" title)))

(defun hakyll-blog--file-format (title)
  "File name for TITLE post."
  (format "%s/posts/%s-%s.markdown"
          (expand-file-name hakyll-blog-dir)
          (format-time-string hakyll-blog-file-time-format)
          (s-dashed-words title)))

Launch Capture Window in KDE

2023-05-08T00:00:00Z

Tags: emacs

I created a custom shortcut Meta-T in KDE with the following trigger:

emacsclient -e "(org-capture)" --create-frame

I went with calling org-capture directly as that shows a buffer where I can choose which template to use. If I was using the org-protocol feature, I’d need to setup one KDE shortcut per template, the maintenance of which would annoy me to no end, on top of filling up my global shortcuts.

But this did not raise the new frame in all cases. For example, if I had my browser up and pressed Meta-T, the new Emacs frame would be created but would not be focused.

To make that work, I needed the shortcut trigger to be:

emacsclient -e \
  "(progn (x-focus-frame nil) (org-capture))" \
  --create-frame

and I needed to add the following snippet to my Emacs config:

(defun my/focus-new-client-frame ()
  (select-frame-set-input-focus (selected-frame)))

(add-hook 'server-after-make-frame-hook
  #'my/focus-new-client-frame)

I got that snippet from this reddit post.

Now, when pressing Meta-T anywhere will create a new frame and focus it.

Sops on NixOps 2

2023-04-21T00:00:00Z

Tags: nix, server

Setting Sops is done by following the steps in the README file. But I struggled on step “Get a public key for your target machine”. This is what it tells us to do:

nix-shell -p ssh-to-age \
  --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'

But you need to ssh into the machine to get the public key. How to do that? Well, this “little” one liner does the trick:

nixops export --network dev | \
  jq '..|."virtualbox.publicHostKey"? | select(. != null)' -r

The .. is a recursive descent on all JSON object fields.
We’re searching for the field names virtualbox.publicHostKey. We need the double quotes because the dot is actually part of the field name.
We then select the non null items, effectively keeping the only match.

This outputs something like ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJhZqrj2+idV2uZXHUp2Q4sJ8SzRWGYz0nHSKuiW5oo3 NixOps auto-generated key.

You can then pipe that to ssh-to-age and go to the next step.

OS Install

2021-10-09T00:00:00Z

Tags: server

Note: since I wrote this blog post, I switched to using NixOS. I’m leaving this post up because it could still hold valuable info.

This is obviously the first step, we will install the OS. I chose Archlinux. Everything here should be adaptable to other linux distributions but the locations of files could be different.

We’ll create a live USB, use it to install Arch on another server and we’ll finish by SSHing to the server using SSH keys for authentication.

Preliminaries

We need to choose a few things before starting the install. Here is what I’ll stick to in this blog post:

hostname of the server: $server
hostname of the laptop you’re currently using: $laptop
user I’ll use to connect to it: $user

I confess, there’s nothing original here but I feel like it’s easier to follow along this way. I use the convention of bash variables with the dollar prefix in the following.

Live USB

First step is to create a bootable USB that we’ll use to install Arch on the server. We’ll create two partitions, one for the OS, one for storing some files like a SSH public key that will allow us to SSH into the server without password.

Detailed instructions can be found in the Arch wiki.

Download the Arch ISO

From https://archlinux.org/download/.

Find the USB drive path

$ lsblk -p -d -o NAME,MODEL,SIZE,TRAN \
    | grep 'NAME\|usb'

Example output:

NAME         MODEL               SIZE TRAN
/dev/sda     ST9500325ASG      465.8G sata
/dev/nvme0n1 SPCC M.2 PCIe SSD 953.9G nvme

In the following, I’ll use /dev/sdX as the path. Replace it by the one you got from the command above.

Partition and format the USB drive

Create two partitions, the first one must be fat32, the second ext4.

$ fdisk /dev/sdX <


Write OS and files to USB drive
Mount the two partitions and the Arch ISO.
$ mkdir -p mnt/root mnt/data mnt/iso
$ mount -o rw /dev/sdX1 mnt/root
$ mount -o rw /dev/sdX2 mnt/data
$ mount -o loop arch.iso mnt/iso
Copy the files to the root partition.
$ rsync -a \
    --info=progress2 \
    --human-readable \
    --no-inc-recursive \
    mnt/iso \
    mnt/root
Create a SSH private key and public key pair and copy it to the USB
key. I use Password Store as my password manager to generate and
store the passphrase.
$ pass generate --clip sshkey-passphrase/$laptop/$server

$ ssh-keygen -b 4096 -i ~/.ssh/$server -N $(pass show sshkey-passphrase/$laptop/$server)
$ cp ~/.ssh/$server.pub mnt/data
I use the convention sshkey-passphrase/$laptop/$server for the
location of the passphrase in my password manager.
Update the syslinux install to instruct the OS where the root
partition is located.
$ uuid=$(blkid -o value -s UUID "/dev/sdX1")
$ sed -i -e \
    "s|archisolabel=.*$|archisodevice=/dev/disk/by-uuid/$uuid|" \
    "mnt/root/arch/boot/syslinux/archiso_sys.cfg"
$ syslinux-install_update -iam
Don’t forget to sync to actually write the files. This step can take a
while.
$ sync
Now, let’s unmount and we can then remove the USB drive.
$ umount mnt/*
Install OS on the server
Let’s put the USB key in the server’s USB port and boot the server.
I’ll assume you have brand new drives or are happy to wipe them clean.
TODO
GnuPG setup for ssh agent



Server setup - Introduction
2021-10-05T00:00:00Z

    
        Posted on October  5, 2021
        
    
	
	  
	  Tags: server
	  
	
    
        Note: since I wrote this blog post, I switched to using NixOS. I’m leaving this post up because
it could still hold valuable info.
This is the introductory post in the let’s build a server series. This
series goes through all the steps needed to setup the same server I
have at home. Along the way, I’ll explain everything I know or thought
of while setting up this server.
We’ll build a box with a media server, automated movies and series
downloader protected by VPN, a file server and synchronizer,
integration with dropbox, a code server, dynamic IP support,
monitoring, RSS support, SSO, a dashboard, backup and recovery.
We’ll also talk about how your laptop and phone will interact with the
server, so essentially file synchronization, picture viewing and
backup.


Our dashboard to be

We will install and configure on an Archlinux box:

Bazarr
Borgmatic
Caddy
Deluge through VPN
DynDNS through Godaddy
Ersatztv
Gitlab
GnuPG
Haproxy
Homer
Jackett
Jellyfin
Keycloak
LVM
MDADM
Nextcloud
Oauth2-Proxy
OpenVPN
Password Store
PostgreSQL
Radarr
Rclone
Redis
Sonarr
TinyTinyRSS
PHP-FPM

You’ll notice I left out an automation tool like Docker or Ansible
which we won’t use here. The goal of this serie is to empower you and
to make you understand what’s going on under the hood. That’s how I
like to build the things I use.
I assume you are familiar with the command line, that’s about it.
Hardware and Distro

CPU: Intel i5 760 2.80GHz, 4 cores
RAM: 16GB
Graphics Card: GeForce GTS 250
Disks:

Samsung SSD 950 PRO (256GB)
TOSHIBA HDWD110 (1 TB)
WDC WD10EZEX-60W (1 TB)
ST2000DM006-2DM1 (1.82 TB) x2

Distro: Archlinux 5.13.13-arch1-1

In short, it’s an old desktop repurposed in a server with some new
hard drives. The drives are in a RAID 1 setup so I have a total of
2.82 TB of available storage. The SSD is used to store /var to speed
up writing logs and access to PostgreSQL.
    




Add Tags to a Hakyll Site
2020-10-22T00:00:00Z

    
        Posted on October 22, 2020
        
    
	
	  
	  Tags: hakyll
	  
	
    
        Let’s add some tags to each blog post. The tags are added in the
header section of each post. For example:
---
title: Deploy the Blog to GitHub Pages
tags: hakyll, haskell, nix
---
This will produce a blog post with 3 tags. Clicking on a tag will
bring you to a page with all blog posts tagged with that tag.
You can also access to all tags from a special Tags page.
Let’s set this all up. Everything outlined here can be found in this
commit.
I was heavily inspired by this blog
post
by Javran but I diverged in the way I wanted the tags to be shown. I
also added a Tags page.
Show Tags in Post
First, let’s show the tags on each post page. This will be the result:

We first need to add a section to templates/post.html:
 class="tags">
  $if(tags)$
  Tags: $tags$
  $endif$

Note that the $if()$ clause only checks if a field exists. So the
Tags: line will be printed also if the $tags$ field is empty.
We then need to update site.hs to populate the $tags$ field:
main = hakyllWith conf $ do

  -- [...]

+  tags <- buildTags "posts/*" (fromCapture "tags/*.html")
+  let postCtxWithTags = postCtx tags

  -- [...]

   match "posts/*" $ do
     route $ setExtension "html"
     compile
       $   pandocCompiler
-      >>= loadAndApplyTemplate "templates/post.html"    postCtx
-      >>= loadAndApplyTemplate "templates/default.html" postCtx
+      >>= loadAndApplyTemplate "templates/post.html"    postCtxWithTags
+      >>= loadAndApplyTemplate "templates/default.html" postCtxWithTags
       >>= relativizeUrls


-postCtx :: Context String
-postCtx = dateField "date" "%B %e, %Y" `mappend` defaultContext
+postCtx :: Tags -> Context String
+postCtx tags =
+  tagsField "tags" tags
+    `mappend` dateField "date" "%B %e, %Y"
+    `mappend` defaultContext
The match "posts/*" block existed already but we changed the
postCtx to postCtxWithTags. Actually, we changed all occurrences
of postCtx with postCtxWithTags. We also added a "tags" field in
the postCtx function.
Let’s analyze two lines which took me some time to understand.
Build Tags
buildTags :: MonadMetadata m
          => Pattern
          -> (String -> Identifier)
          -> m Tags
We use it like so:
buildTags "posts/*" (fromCapture "tags/*.html")
In this case, buildTags will parse each file in the posts/
directory and extract all tags in the header. We will later generate
an HTML page per tag and that’s when the fromCapture function will
be used.
fromCapture :: Pattern -> String -> Identifier
Pattern is a simple glob pattern:
(toFilePath $ fromCapture "tags/*.html" "foo") == "tags/foo.html"
Tags Field
tagsField :: String -> Tags -> Context a
We use it like so:
tagsField "tags" tags
This adds a field named "tags" that can be accessed in the template
as $tags$. Context a is what the template has access to.
Here, the default rendering for $tags$ (comma separated list of
urls) suits us but we will see later how to make something custom.
Tag Page
Let’s create an HTML page per tag that will show all posts that has
the tag. It will look like this:

We create a template for the tag page in templates/tag.html. Here,
we can reuse the post-list.html template:
$partial("templates/post-list.html")$
In templates/post-list.html, we added the tags div:
 
     $for(posts)$
         
             $title$ - $date$
+            $tags$
         
     $endfor$
 
With some accompanying css in css/default.css:
article .tags {
  font-size: 1.4rem;
  font-style: italic;
  color: #555;
}

div.tags {
  text-align: right;
}
In site.hs, we add a new block to the main function which takes
care of generating one HTML page per tag.
main = hakyllWith conf $ do

+  tagsRules tags $ \tag pattern -> do
+    let title = "Posts tagged \"" ++ tag ++ "\""
+    route idRoute
+    compile $ do
+      posts <- recentFirst =<< loadAll pattern
+      let ctx =
+            constField "title" title
+              `mappend` listField "posts" postCtxWithTags (return posts)
+              `mappend` defaultContext
+
+      makeItem ""
+        >>= loadAndApplyTemplate "templates/tag.html"     ctx
+        >>= loadAndApplyTemplate "templates/default.html" ctx
+        >>= relativizeUrls
Where tags is the variable we added earlier with the buildTags
function.
Page with all Tags
Last piece of the puzzle, let’s add a page with all tags. Clicking on
a tag will forward to the corresponding tag page, the ones we created
above. It will look like this:

In site.hs, we add a new block:
main = hakyllWith conf $ do

+  create ["tags.html"] $ do
+    route idRoute
+    compile $ do
+      tags' <- tagsMetadata tags
+      let tagsCtx =
+            listField
+                "tags"
+                (  field "name"  (return . tagName . itemBody)
+                <> field "url"   (return . tagUrl . itemBody)
+                <> field "count" (return . show . tagCount . itemBody)
+                )
+                (sequence $ map makeItem $ tags')
+              `mappend` defaultContext
+
+      getResourceBody
+        >>= applyAsTemplate tagsCtx
+        >>= loadAndApplyTemplate "templates/default.html" tagsCtx
+        >>= relativizeUrls
+
+data TagMetadata = TagMetadata
+         { tagName :: String
+         , tagUrl :: String
+         , tagCount :: Int
+         }
+
+tagsMetadata :: Tags -> Compiler [TagMetadata]
+tagsMetadata tags = do
+  let tagsList = map fst $ tagsMap tags
+  forM (tagsMap tags) $ \(tag, ids) -> do
+    route' <- getRoute $ tagsMakeId tags tag
+    return $ TagMetadata tag (fromMaybe "/" route') (length ids)
There is a lot going on here, let’s analyze it step by step.
Tags.html Page
The create function will create the tags.html page. The
getResourceBody will load a template html page of the same name.
Let’s create it:
---
title: Tags
---

Here you can find all the tags set on the various posts:


    $for(tags)$
        
             href="$url$">$name$ ($count$)
        
    $endfor$

It needs a $tags$ list whose items contain three fields: the $url$
of the tag page, the $name$ of the tag and how many posts are tagged
with it in $count$.
Tags Metadata
The tagsMetadata function creates a list of TagMetadata out of the
Tags. I copied it from the renderTags source
code
and only added the use of TagMetadata.
TagMetadata is a helper type I introduced to make passing data to
the template’s context explicit.
Tags Context
tagsCtx is a Context having a $tags$ field which is a list. Each
item has the $url$, $name$ and $count$ fields which are needed
by the template. Those fields come from the TagMetadata type.
Tags Nav
Finally, we can add a link to the /tags.html page to the nav bar in
the templates/default.html file:
    Contact
+   Tags
    Archive
Conclusion
Although that was not straightforward and I ran into a lot of issues
while figuring things out, once more Haskell shines because most of
the issues were resolved at compile time. The only things I needed to
figure out at runtime were related to styling and cosmetics.
    




Unexpected Writing Perk
2020-10-20T00:00:00Z

    
        Posted on October 20, 2020
        
    
	
	  
	  Tags: misc
	  
	
    
        It’s kind of obvious in retrospective, but I try to pay attention when
I’m writing something on the internet.
For example, sharing my Emacs configuration made me realize I had a
few hacks in place and I felt dirty sharing those. I took some time to
rework the hacks and that resulted in improving my configuration and
making it more maintainable.
The same thing happens when I try to explain an idea to someone or
when I make a presentation at work. I try to explain my ideas in the
most intelligible way possible and that’s often when I realize I don’t
grasp the idea fully and am unable to explain at least some part of
it.
In other words, writing is like having a rubber
duck.
Thank you for being my rubber duck.

I hope you will gain something from reading my blog posts. If you do,
or do not, I’d love to hear about it!
    




Configure Emacs to Write this Blog With Nix
2020-10-17T00:00:00Z

    
        Posted on October 17, 2020
        
    
	
	  
	  Tags: hakyll, haskell, nix, emacs, lsp, systemd
	  
	
    
        We need to make Emacs find Haskell executables through Nix. It will
use LSP with
the Ghcide LSP server as basis
to parse the source code,
Flycheck for live errors in the
buffer and Brittany to
format the code. Ghcide and Brittany are installed with
Nix.
To make all this work, we will use the following Emacs packages:

nix-sandbox provides us with helper functions to get the current project’s nix sandbox.
nix-mode makes it easy to edix .nix files as well as provides the nix-build function.
lsp-mode handles talking to a LSP server, here Ghcide.
lsp-ui shows LSP actions and various infos.
company-lsp provides autocompletion based on the LSP server.
lsp-haskell is used by lsp-mode to talk to Ghcide.
haskell-mode provides syntax highlighting, hoogle integration and much more for editing Haskell files.
flycheck provides syntax check highlighting.

By the way, I’m using
use-package with
straight to configure
Emacs.
Finally, to make this work with Emacs running as a daemon, there will
be some systemd-fu required.
Emacs Setup
Nix Sandbox and Helper
We add nix-sandbox and nix-mode:
(use-package nix-sandbox
  :straight t)

(use-package nix-mode
  :straight t
  :mode "\\.nix\\'"
  :init
  (require 'nix-build))
nix-sandbox defines nix-current-sandbox which returns the path to
shell.nix if it exists, or falls back to default.nix if it exists
or nil if none exist. It also defines nix-shell-command which,
from the docs:

(defun nix-shell-command (sandbox &rest args)
  "Assemble a command from ARGS that can be executed in the specified SANDBOX."
  ...

Super useful for running brittany or ghcide inside our Nix
environment. We won’t need to run those manually though.
Another useful function is nix-compile which interactively asks for
a sandbox and a command to run.
We add nix-mode mostly for its nix-build function. This allows us
to build the site executable from Emacs.
lsp-mode, lsp-ui, company-lsp
These are fairly standard configuration whenever you use LSP in Emacs.
I reproduce them here for completeness.
(use-package lsp-mode
  :straight t
  :commands lsp
  :init
  (defun my/lsp-format-buffer-silent ()
    "Silence errors from `lsp-format-buffer'."
    (ignore-errors (lsp-format-buffer)))
  :hook ((sh-mode . lsp-deferred)
         (javascript-mode . lsp-deferred)
         (html-mode . lsp-deferred)
         (before-save . my/lsp-format-buffer-silent))
  :config
  (setq lsp-signature-auto-activate t)
  (lsp-lens-mode t))

(use-package lsp-ui
  :straight t
  :hook (lsp-mode-hook . lsp-ui-mode)
  :commands lsp-ui-mode
  :config
  (setq lsp-ui-flycheck-enable t
        lsp-ui-flycheck-live-reporting nil))

(use-package company-lsp
  :straight t
  :commands company-lsp
  :config
  (push 'company-lsp company-backends))
Haskell-mode
Haskell-mode is used to edit haskell source code.
(use-package haskell-mode
  :straight t
  :after nix-sandbox

  :init

  (defun my/haskell-set-stylish ()
	(if-let* ((sandbox (nix-current-sandbox))
			  (fullcmd (nix-shell-command sandbox "brittany"))
			  (path (car fullcmd))
			  (args (cdr fullcmd)))
	  (setq-local haskell-mode-stylish-haskell-path path
				  haskell-mode-stylish-haskell-args args)))

  (defun my/haskell-set-hoogle ()
	(if-let* ((sandbox (nix-current-sandbox)))
		(setq-local haskell-hoogle-command (nix-shell-string sandbox "hoogle"))))

  :hook ((haskell-mode . capitalized-words-mode)
		 (haskell-mode . haskell-decl-scan-mode)
		 (haskell-mode . haskell-indent-mode)
		 (haskell-mode . haskell-indentation-mode)
		 (haskell-mode . my/haskell-set-stylish)
		 (haskell-mode . my/haskell-set-hoogle)
		 (haskell-mode . lsp-deferred)
		 (haskell-mode . haskell-auto-insert-module-template))

  :config

  (defun my/haskell-hoogle--server-command (port)
	(if-let* ((hooglecmd `("hoogle" "serve" "--local" "-p" ,(number-to-string port)))
			  (sandbox (nix-current-sandbox)))
		(apply 'nix-shell-command sandbox hooglecmd)
	  hooglecmd))

  (setq haskell-hoogle-server-command 'my/haskell-hoogle--server-command
		haskell-stylish-on-save t))
The advanced configuration is for handling hoogle and brittany
inside Nix. In both cases, we use buffer-local variables through the
haskell-mode hook and always check if we are in a Nix environment.
If we are, that is if (nix-current-sandbox) is not nil, then we
use nix-sandbox’s helpers to wrap around the hoogle and brittany
executables. If not, we directly call hoogle and brittany.
To search in the local database, we can then use
haskell-hoogle-lookup-from-local. On first call, it will start the
local server.
brittany will be called every time we save a buffer to format it.
lsp-haskell
Now we setup the package that talks to Ghcide, our LSP server.
(use-package lsp-haskell
  :straight t
  :after nix-sandbox

  :init
  (setq lsp-prefer-flymake nil)
  (require 'lsp-haskell)

  :config

  ;; from https://github.com/travisbhartwell/nix-emacs#haskell-mode
  (defun my/nix--lsp-haskell-wrapper (args)
	(if-let ((sandbox (nix-current-sandbox)))
		(apply 'nix-shell-command sandbox args)
	  args))

  (setq lsp-haskell-process-path-hie "ghcide"
		lsp-haskell-process-args-hie '()
		lsp-haskell-process-wrapper-function 'my/nix--lsp-haskell-wrapper))
Like for haskell-mode, the advanced configuration is focused on
wrapping the various commands with nix-shell-command and
nix-shell-string, with the wrapper being a pass-through if
(nix-current-sandbox) returns nil.
Flycheck
Finally, we configure flycheck itself. Like for lsp-haskell, we
make it so flycheck can talk to Ghcide through Nix.
(use-package flycheck
  :straight t
  :after nix-sandbox

  :init
  (add-hook 'after-init-hook 'global-flycheck-mode)

  ;; from https://github.com/travisbhartwell/nix-emacs#flycheck
  (defun my/nix--flycheck-command-wrapper (command)
	(if-let ((sandbox (nix-current-sandbox)))
		(apply 'nix-shell-command (nix-current-sandbox) command)
	  command))
  (defun my/nix--flycheck-executable-find (cmd)
	(if-let ((sandbox (nix-current-sandbox)))
		(nix-executable-find (nix-current-sandbox) cmd)
	  (flycheck-default-executable-find cmd)))

  :config
  (setq flycheck-check-syntax-automatically '(save idle-change)
		flycheck-relevant-error-other-file-show nil
		flycheck-command-wrapper-function 'my/nix--flycheck-command-wrapper
		flycheck-executable-find 'my/nix--flycheck-executable-find)

  (add-to-list 'display-buffer-alist
			   `(,(rx bos "*Flycheck errors*" eos)
				 (display-buffer-reuse-window
				  display-buffer-in-side-window)
				 (side            . bottom)
				 (reusable-frames . visible)
				 (window-height   . 0.33))))
Systemd-fu for Emacs Daemon
I will assume you use systemd to start emacs in daemon mode. If so,
create or update the ~/.config/systemd/user/emacs.service file with
the same NIX_* and PATH Environment fields:
[Unit]
Description=Emacs text editor
Documentation=info:emacs man:emacs(1) https://gnu.org/software/emacs/

[Service]
Type=forking
ExecStart=/usr/bin/emacs --daemon
ExecStop=/usr/bin/emacsclient --eval "(kill-emacs)"
Environment=NIX_PROFILES=/nix/var/nix/profiles/default %h/.nix-profile
Environment=NIX_PATH=%h/.nix-defexpr/channels
Environment=NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
Environment=PATH=%h/.nix-profile/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:
Restart=on-failure

[Install]
WantedBy=default.target
Conclusion
That’s it, you should be well equipped for editing your site.
Actually, everything we saw here is transferable to any Haskell code
running through Nix.
    




Add Syntax Highlighting
2020-10-16T00:00:00Z

    
        Posted on October 16, 2020
        
    
	
	  
	  Tags: hakyll
	  
	
    
        This is “just” a matter of creating a CSS file and adding it to the
repository. The hard part is to have the patience to create the CSS
file, at least for me, because the css classes are not descriptive
(ex: what the hell are class="ot" and class="op"?). I tried
finding ready to use ones but I could only find some on
tejasbubane/hakyll-css. I
chose the
zenburn
one.
Next steps are to copy paste its content to css/syntax.css then add
a link to the css file in templates/default.html:
 rel="stylesheet" href="/css/syntax.css" />
    




Deploy the Blog to GitHub Pages
2020-10-16T00:00:00Z

    
        Posted on October 16, 2020
        
    
	
	  
	  Tags: hakyll, haskell, nix
	  
	
    
        I want to use GitHub Pages because it is a very simple to use static
site server and it is readily available when using GitHub. We could
try to use GitHub CI to deploy to GitHub Pages but I do not want to
deal with this complexity just yet.
The method I will use for now is manual and requires two repositories:
one that holds the source code and the other that holds the generated
static files. The easiest way to handle this is to use a git
submodule. (I say easiest but I still spent a few hours setting this
up correctly…)
In my case, the repository holding the static files is
https://github.com/ibizaman/ibizaman.github.io
and the one holding the source code is
https://github.com/ibizaman/blog.
In the latter, we will create a submodule pointing to the former. In
other words, the submodule will be in the repository holding the
source code.
Setup a Deploy Command
In essence, we need to build the site and copy the resulting files to
the submodule’s path. We then commit and push the submodule to its
repository.
I tried first to simply use the _site/ directory, output of the
./result/bin/site build command, as the submodule. The issue is the
./result/bin/site rebuild command completely wipes the _site/,
including the submodule’s .git directory. This puts the submodule in
a bad state.
Next solution is to generate the files in _site/ then copy them to
the submodule’s directory. Usually, that’s where I start putting
commands in a Makefile so I don’t forget how to handle this. But
actually, Hakyll allows you to run a custom command when running
./result/bin/site deploy. This is configured in site.hs. We will
use that as a self-contained Makefile. Documentation can be found on
Hackage.
In site.hs, we change:
+conf :: Configuration
+conf = defaultConfiguration {
+    deployCommand = "rm -rf ibizaman.github.io/* && cp -r _site/* ibizaman.github.io"
+  }

 main :: IO ()
-main = hakyll $ do
+main = hakyllWith conf $ do
This is tailored to the names of my repos, of course. It corresponds
to this
commit.
Actually Deploy
These are the, arguably manual, steps to deploy this blog:
$ nix-build
$ ./result/bin/site rebuild
$ ./result/bin/site deploy
$ (cd ibizaman.github.io && git commit -m 'MESSAGE' && git tag TAG && git push)
$ git commit -m 'MESSAGE' && git push
You often don’t need to do the first two steps as ./result/bin/watch
takes care of it.
I’m sure I will get annoyed of running these commands at some point in
the future but it’s good enough for now.
    




Setup a Hakyll Blog with Nix
2020-10-15T00:00:00Z

    
        Posted on October 15, 2020
        
    
	
	  
	  Tags: hakyll, haskell, nix
	  
	
    
        This blog post explains how to setup this site. You can find the
source code on GitHub. We will not
publish yet, it will be all local. Publishing is for the following
post.
The stack we’ll be using is Hakyll on
top of Haskell and
Nix. I use
Emacs as my editor so there
will be some later blog posts explaining how to set that up.
We will start by writing Nix files and for learning Nix, I recommend
the Nix pills. I
thought I could simply find example snippets online and learn from
that but the language and the conventions were too alien for me to
understand anything.
I followed this blog
post
from Robert Pearce to set things up. I removed the niv part as I am
not using it.
There was an encoding bug as I’m using UTF-8 but by default Hakyll
only understands ASCII encoding. To solve that one, I followed this
blog
post
from Ondřej Slámečka. There is a fix in the Hakyll’s
FAQ,
but the post from Ondřej is tailored for nix so I followed it.
I recommend reading both blog posts as they explain things well. That
said, here is my version.
Initial Nix Files
Corresponds to this commit.
There are 3 nix files to create, then we will be able to use
hakyll-init to generate the base site.
./default.nix
(import ./release.nix { }).project
./shell.nix
(import ./release.nix { }).shell
./release.nix
# https://robertwpearce.com/hakyll-pt-6-pure-builds-with-nix.html
{ compiler ? "ghc883"
, pkgs ? import  {}
}:

let
  inherit (pkgs.lib.trivial) flip pipe;
  inherit (pkgs.haskell.lib) appendConfigureFlags;

  haskellPackages = pkgs.haskell.packages.${compiler}.override {
    overrides = hpNew: hpOld: {
      hakyll =
        pipe
           hpOld.hakyll
           [ (flip appendConfigureFlags [ "-f" "watchServer" "-f" "previewServer" ])
           ];

      # https://www.slamecka.cz/posts/2020-06-08-encoding-issues-with-nix-hakyll/
      hakyll-blog = (hpNew.callCabal2nix "blog" ./. { }).overrideAttrs (old: {
        nativeBuildInputs = old.nativeBuildInputs or [] ++ [
          pkgs.makeWrapper
        ];
        postInstall = old.postInstall or "" + ''
        wrapProgram $out/bin/site \
          --set LANG "en_US.UTF-8" \
          --set LOCALE_ARCHIVE "${pkgs.glibcLocales}/lib/locale/locale-archive"
        '';
      });
    };
  };

  project = haskellPackages.hakyll-blog;
in
{
  project = project;

  shell = haskellPackages.shellFor {
    packages = p: with p; [
      project
    ];
    buildInputs = with haskellPackages; [
      ghcid
      ghcide
      brittany
      hlint
    ];
    withHoogle = true;
  };
}
The first two files are using the release.nix one so let’s dive into
that one.
First, we set some defaults and import some functions:
{ compiler ? "ghc883"
, pkgs ? import  {}
}:

let
  inherit (pkgs.lib.trivial) flip pipe;
  inherit (pkgs.haskell.lib) appendConfigureFlags;
Then we set some overrides. To know what an override is, check the
Nix pills.
haskellPackages = pkgs.haskell.packages.${compiler}.override {
  overrides = hpNew: hpOld: {
First for Hakyll, we configure it with watchServer and
previewServer options. Both options are super useful when developing
as the produced ./result/bin/site executable will watch for file
changes and rebuild the blog post when needed.
hakyll =
  pipe
     hpOld.hakyll
     [ (flip appendConfigureFlags [ "-f" "watchServer" "-f" "previewServer" ])
     ];
Second, we create a variable hakyll-blog to compile the blog. The
name "blog" must correspond to the name of the cabal file, which we
will generate later. We also set some flags to make Hakyll able to
read UTF-8 files.
hakyll-blog = (hpNew.callCabal2nix "blog" ./. { }).overrideAttrs (old: {
  nativeBuildInputs = old.nativeBuildInputs or [] ++ [
    pkgs.makeWrapper
  ];
  postInstall = old.postInstall or "" + ''
  wrapProgram $out/bin/site \
    --set LANG "en_US.UTF-8" \
    --set LOCALE_ARCHIVE "${pkgs.glibcLocales}/lib/locale/locale-archive"
  '';
});
We then set the project variable used in the default.nix file:
  project = haskellPackages.hakyll-blog;
in
{
  project = project;
Finally, we set the shell variable and provide some packages useful
for editors, namely ghcide, brittany, hlint and a local hoogle
with withHoogle:
shell = haskellPackages.shellFor {
  packages = p: with p; [
    project
  ];
  buildInputs = with haskellPackages; [
    ghcid
    ghcide
    brittany
    hlint
  ];
  withHoogle = true;
};
Generate the Initial Template Blog
Corresponds to this commit.
Next step will take some time because we will run Nix for the first
time. It will fetch and compile every dependencies. It took roughly 1
hour on my laptop.
$ nix-shell --pure -p haskellPackages.hakyll --run "hakyll-init ."
This will generate a cabal file named after the git repo’s directory. Here is an example, with the following directory structure:
./blog
./blog/.git
Running the hakyll-init . command in the blog directory will create the following files:
[nix-shell:~/blog]$ hakyll-init .
Creating ./posts/2015-11-28-carpe-diem.markdown
Creating ./posts/2015-10-07-rosa-rosa-rosam.markdown
Creating ./posts/2015-12-07-tu-quoque.markdown
Creating ./posts/2015-08-12-spqr.markdown
Creating ./site.hs
Creating ./images/haskell-logo.png
Creating ./templates/post-list.html
Creating ./templates/default.html
Creating ./templates/archive.html
Creating ./templates/post.html
Creating ./css/default.css
Creating ./index.html
Creating ./about.rst
Creating ./contact.markdown
Creating ./blog.cabal
You get a bunch of files and 4 example blog posts under the posts/
directory. Also, the Cabal file blog.cabal is named from the
directory and must be named that way for the release.nix file to
work correctly, as we established previously.
Then, we can build the ./result/bin/site blog executable:
$ nix-build --show-trace
Finally, we can run the executable to compile the site, including all
4 example blog posts:
./result/bin/site build
This will create the _cache and _site directories. The latter is
where the generated files will be located.
Serve and Watch For Changes
./result/bin/site watch
Your blog will be up and running at http://localhost:8000.
You can change posts, add or remove posts, change the css and other
files and the site executable will see those changes and rebuild the
site. You just need to reload the site in your browser.
Gitignore
Corresponds to this commit.
I can’t live with temporary and generated files like that. Let’s add a
.gitignore:
*#
*~
_cache/
_site/
result
The first two lines are for Emacs’ temporary files, you could omit
them.
Your First Post
I just deleted all 4 example blog posts and created another
file.
Conclusion
That’s it, we created our blog with Hakyll and Nix and we can see it
locally. Next up, let’s publish it.
    




First Post
2020-10-15T00:00:00Z

    
        Posted on October 15, 2020
        
    
	
	  
	  Tags: misc
	  
	
    
        I wanted to start writing a blog since a long time. I like sharing my
knowledge so it seemed like a good idea. Time will tell! I needed a
way to write blog posts and publish them so I searched for a tool that
could suite me.
There are a few subjects I want to share about but being a programmer,
talking about code and sharing code snippets seemed natural. This is
often the missing feature for blogging tools or services. Also, I’m an
engineer at heart so I like to build things myself, or at least know
I’m able to modify the tools I use to my heart’s content. I searched
for some DIY alternatives to online services and found Hakyll. It
seemed like the perfect fix because I wanted to write about a language
I fell in love with, Haskell.
So here we are, this is my first blog post, what can I write about?
Let’s talk about how to setup a blog similar to this one. This is my
freshest project anyway.
First caveat, I cannot yet publish this blog post because I’ve got
nowhere to publish it! It’s all local. Next blog post will explain how
to setup a Hakyll blog using nix… locally. The blog post after that
will finally talk about how to publish. And then you’ll be able to
read this, hooray.
The stack used to for this blog is Hakyll on top of Haskell and nix. I
really like how Nix makes development environments fully reproducible.
It’s not easy to setup nor understand so it’s for sure a big
investment.

ibizaman's Blog

Connect Home Assistant to Nextcloud Calendar

Locate calendar in Nextcloud

Add calendar to Home Assistant

Conclusion

Testing LDAP and SSO Integrations in NixOS with Playwright

Automated Updates

NixOS VM Tests

Playwright Tests

Declarative LDAP

Declarative SSO

Declarative Module Integration

Testing LDAP and SSO Integrations

Conclusion?

Debugging

Bonus Test

Conclusion

ESPSomfy on Atom Lite with NixOS

Installation

1. Plug the Atom Lite in the computer

2. Find the Atom Lite

3. Enter shell with needed tools

4. Check Atom Lite is flashable

5. Erase the Atom Lite

6. Download the firmware

7. Flash the Atom Lite

8. Verify Installation

Hardware

1. Connect to Atom Lite’s WiFi AP

2. Load ESPSomfyRTS web interface

3. Configure device

4. Wiring

5. Test

6. Soldering

Next Steps

Bisect Experiment at the PR Level

First Automation Attempt

Naive Commit Selection

Smarter Commit Selection

Manual Run

Links

Possible Improvement

Upstreaming

Use Bitwarden as auth source in Emacs

1. Setup Emacs

2. Setup nix config

3. Configure secret

Misc

Clear auth-sources cache

Self-hosted Bitwarden

Log into vault

Unlock vault

Sync vault

Qemu tips for cdrom, ide and nvme hard drives

Nix

CDROM (sr0)

Iso file in Nix

EFI

SATA (sdX) drive

NVMe (nvmeX) drive

Epilogue

Access a Host Through SSH on NixOS

Scaffolding

Generate a SSH keypair

Configuration for the Server Host

Configuration for the Client Host

Cleanup

Conclusion

Enable Hibernation on NixOS

Parental Control with Jellyfin and Radarr/Sonarr

*arr Configuration

Jellyfin Configuration

Pre-Seeded Known Host to Access New Server

Why Skipping this Step is a Good Idea

Why Does Host Key Verification Exist

Pre-Seed the Host Key

Display Pictures from Nextcloud to a Kodi Photo Frame

Wanted User Experience

Setup

On Nextcloud