Skip to content

Set up systemd-oomd and limit gitlab memory usage

Benjamin Lee requested to merge systemd-oomd into main

Current memory usage of gitlab services:

gitlab: 3.8G (12%)
gitlab-pages: 19.7M (~0%)
gitlab-workhorse: 46.4M (~0%)
gitlab-runner: 58.9M (~0%)
gitlab-sidekiq: 1.0G (~3%)
total: (16%)

We might want to test that memory usage doesn't go over 50% during a CI job before deploying this.

I haven't tested this config at all, although I did test a similar system in a VM and it had reasonable behavior:

  systemd.services = let
    names = [ "hog-1" "hog-2" ];
  in builtins.listToAttrs (builtins.map (name: {
      name = name;
      value = {
        serviceConfig = {
          ExecStart = "${pkgs.stress-ng}/bin/stress-ng --vm-bytes 256M -m 1";
          Restart = "on-failure";
          Slice = "system-hogs.slice";
        };
      };
    }) names);

  systemd.slices."system-hogs" = {
    sliceConfig = {
      MemoryHigh = "300M";
      MemoryMax = "500M";
    };
  };

  systemd.oomd.enable = true;

  systemd.slices."system" = {
    sliceConfig = {
      ManagedOOMMemoryPressure = "kill";
      ManagedOOMMemoryPressureLimit = "60%";
    };
  };

  systemd.slices."user" = {
    sliceConfig = {
      ManagedOOMMemoryPressure = "kill";
      ManagedOOMMemoryPressureLimit = "40%";
    };
  };

  systemd.slices."root" = {
    sliceConfig = {
      MemoryHigh = "95%";
    };
  };

Merge request reports