Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SystemD dropins for minimal sys-net and sys-usb #540

Merged
merged 4 commits into from
Feb 13, 2025
Merged

Conversation

Guiiix
Copy link
Member

@Guiiix Guiiix commented Dec 21, 2024

Some unnecessary services are running on sys-net and sys-usb which consumes RAM for nothing. SystemD dropins are added to automatically disable identified unecessary services by enabling minimal-netvm or minimal-usbvm service on the sys-vms with qvm-service.

@marmarek
Copy link
Member

packaging need an update, both RPM and DEB. See CI logs, for example:

14:34:27 [qb.build_rpm.core-agent-linux.vm-fc40.build] DEBUG:     Installed (but unpackaged) file(s) found:
14:34:27 [qb.build_rpm.core-agent-linux.vm-fc40.build] DEBUG:    /usr/lib/systemd/user/at-spi-dbus-bus.service.d/30_qubes.conf
14:34:27 [qb.build_rpm.core-agent-linux.vm-fc40.build] DEBUG:    /usr/lib/systemd/user/gvfs-daemon.service.d/30_qubes.conf
14:34:27 [qb.build_rpm.core-agent-linux.vm-fc40.build] DEBUG:    /usr/lib/systemd/user/pipewire.service.d/30_qubes.conf
14:34:27 [qb.build_rpm.core-agent-linux.vm-fc40.build] DEBUG:    /usr/lib/systemd/user/wireplumber.service.d/30_qubes.conf

@marmarek
Copy link
Member

PipelineRetry

@Guiiix Guiiix force-pushed the main branch 4 times, most recently from b66e7fa to 20f59e3 Compare December 27, 2024 09:19
Copy link

codecov bot commented Dec 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.14%. Comparing base (c534919) to head (6591aa0).
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #540   +/-   ##
=======================================
  Coverage   70.14%   70.14%           
=======================================
  Files           3        3           
  Lines         469      469           
=======================================
  Hits          329      329           
  Misses        140      140           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@qubesos-bot
Copy link

qubesos-bot commented Dec 27, 2024

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025021202-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025020404-4.3&flavor=update

Failed tests

51 failures

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/127852#dependencies

30 fixed
  • system_tests_suspend

    • mount_and_boot_options: unnamed test (unknown)
    • mount_and_boot_options: Failed (test died)
      # Test died: no candidate needle with tag(s) 'x11' matched...
  • system_tests_dispvm

    • TC_20_DispVM_fedora-41-xfce: test_100_open_in_dispvm (failure)
      AssertionError: './open-file test.txt' failed with ./open-file test...
  • system_tests_devices

    • TC_00_List_whonix-gateway-17: test_000_list_loop (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_001_list_loop_mounted (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_010_list_dm (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_011_list_dm_mounted (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_012_list_dm_delayed (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_013_list_dm_removed (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_020_list_loop_partition (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-gateway-17: test_021_list_loop_partition_mounted (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_000_list_loop (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_001_list_loop_mounted (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_010_list_dm (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_011_list_dm_mounted (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_012_list_dm_delayed (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_013_list_dm_removed (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_020_list_loop_partition (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_00_List_whonix-workstation-17: test_021_list_loop_partition_mounted (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_10_Attach_whonix-gateway-17: test_000_attach_reattach (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

    • TC_10_Attach_whonix-workstation-17: test_000_attach_reattach (error)
      subprocess.CalledProcessError: Command 'set -e;truncate -s 128M /tm...

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-vm-firefox' matc...
  • system_tests_audio

  • system_tests_qrexec_perf@hw1

    • TC_00_QrexecPerf_debian-12-xfce: test_110_simple_data_duplex (failure)
      AssertionError: '/usr/lib/qubes/tests/qrexec_perf.py --vm1=test-ins...
  • system_tests_storage_perf@hw1

    • integ: storage_perf (error)
      ModuleNotFoundError: No module named 'qubes.tests.integ.storage_perf'
  • system_tests_basic_vm_qrexec_gui_ext4

    • switch_pool: Failed (test died)
      # Test died: command 'printf "label: gpt\n,,L" | sfdisk /dev/sdb' f...
  • system_tests_backup

    • TC_10_BackupVM_whonix-gateway-17: test_110_send_to_vm_no_space (error)
      subprocess.CalledProcessError: Command 'mknod /dev/loop0 b 7 0;trun...

    • TC_10_BackupVM_whonix-workstation-17: test_110_send_to_vm_no_space (error)
      subprocess.CalledProcessError: Command 'mknod /dev/loop0 b 7 0;trun...

Unstable tests

## Performance Tests ### Performance degradation: No issues ### Remaining performance tests:
52 tests
  • debian-12-xfce_exec: 8.32
  • debian-12-xfce_exec-root: 29.00
  • debian-12-xfce_socket: 8.68
  • debian-12-xfce_socket-root: 8.45
  • debian-12-xfce_exec-data-simplex: 43.35
  • debian-12-xfce_exec-data-duplex: 48.60
  • debian-12-xfce_exec-data-duplex-root: 65.68
  • debian-12-xfce_socket-data-duplex: 77.61
  • fedora-41-xfce_exec: 9.04
  • fedora-41-xfce_exec-root: 69.49
  • fedora-41-xfce_socket: 8.58
  • fedora-41-xfce_socket-root: 8.64
  • fedora-41-xfce_exec-data-simplex: 50.97
  • fedora-41-xfce_exec-data-duplex: 50.58
  • fedora-41-xfce_exec-data-duplex-root: 82.83
  • fedora-41-xfce_socket-data-duplex: 74.02
  • whonix-gateway-17_exec: 6.17
  • whonix-gateway-17_exec-root: 38.73
  • whonix-gateway-17_socket: 7.88
  • whonix-gateway-17_socket-root: 7.57
  • whonix-gateway-17_exec-data-simplex: 48.12
  • whonix-gateway-17_exec-data-duplex: 49.30
  • whonix-gateway-17_exec-data-duplex-root: 71.98
  • whonix-gateway-17_socket-data-duplex: 84.49
  • whonix-workstation-17_exec: 8.25
  • whonix-workstation-17_exec-root: 53.38
  • whonix-workstation-17_socket: 8.00
  • whonix-workstation-17_socket-root: 8.17
  • whonix-workstation-17_exec-data-simplex: 44.65
  • whonix-workstation-17_exec-data-duplex: 46.51
  • whonix-workstation-17_exec-data-duplex-root: 79.11
  • whonix-workstation-17_socket-data-duplex: 83.74
  • dom0_root_rand-read 3:read_bandwidth_kb: 10011.00
  • dom0_root_rand-write 3:write_bandwidth_kb: 13604.00
  • dom0_root_seq-read 3:read_bandwidth_kb: 426924.00
  • dom0_root_seq-write 3:write_bandwidth_kb: 185752.00
  • dom0_varlibqubes_rand-read 3:read_bandwidth_kb: 13427.00
  • dom0_varlibqubes_rand-write 3:write_bandwidth_kb: 23004.00
  • dom0_varlibqubes_seq-read 3:read_bandwidth_kb: 527677.00
  • dom0_varlibqubes_seq-write 3:write_bandwidth_kb: 223784.00
  • fedora-41-xfce_root_rand-read 3:read_bandwidth_kb: 9363.00
  • fedora-41-xfce_root_rand-write 3:write_bandwidth_kb: 14741.00
  • fedora-41-xfce_root_seq-read 3:read_bandwidth_kb: 413697.00
  • fedora-41-xfce_root_seq-write 3:write_bandwidth_kb: 160415.00
  • fedora-41-xfce_private_rand-read 3:read_bandwidth_kb: 9115.00
  • fedora-41-xfce_private_rand-write 3:write_bandwidth_kb: 15506.00
  • fedora-41-xfce_private_seq-read 3:read_bandwidth_kb: 412866.00
  • fedora-41-xfce_private_seq-write 3:write_bandwidth_kb: 92761.00
  • fedora-41-xfce_volatile_rand-read 3:read_bandwidth_kb: 9011.00
  • fedora-41-xfce_volatile_rand-write 3:write_bandwidth_kb: 13249.00
  • fedora-41-xfce_volatile_seq-read 3:read_bandwidth_kb: 415243.00
  • fedora-41-xfce_volatile_seq-write 3:write_bandwidth_kb: 67795.00

@marmarek
Copy link
Member

Pipewire drop-in needs a different name:

fedora-41-xfce:err: Transaction test error:
fedora-41-xfce:err:   file /usr/lib/systemd/user/pipewire.service.d/30_qubes.conf conflicts between attempted installs of pipewire-qubes-4.3.4-1.fc41.x86_64 and qubes-core-agent-systemd-4.3.14-1.3.fc41.x86_64

@marmarek
Copy link
Member

Without polkit(?), the right click nm-applet checkboxes ("enable networking", "enable wifi") don't work...

@Guiiix
Copy link
Member Author

Guiiix commented Dec 28, 2024

@marmarek this is a mistake. Polkit should be disabled only for minimal sys-usb. Same for Bluetooth.

@marmarek
Copy link
Member

marmarek commented Jan 3, 2025

I've tested it a bit on Novacustom V56 laptop, and with sys-net set to 300MB wifi quite often crashes (page allocation failure in dmesg) after system suspend. In fact, it happens also quite ofen with 350MB too...

@Guiiix Guiiix marked this pull request as ready for review February 9, 2025 09:49
Comment on lines 8 to 9
ConditionPathExists=/var/run/qubes-service/minimal-netvm
ConditionPathExists=/var/run/qubes/this-is-netvm
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the intended effect? Right now it enables it when both conditions are true - so, if it's netvm and the minimal-netvm service is enabled. If you meant to automatically enable it in netvm but also in potentially other if requested, then it should have | prefix. But otherwise, I'd prefer a single condition - either just "minimal-netvm" service, or automatic in an netvm.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would ensure that minimal-netvm has not been activated on anything else than the sys-net VM but maybe this is irrelevant

@marmarek
Copy link
Member

And please announce availability of minimal-netvm and minimal-usbvm services, see qubes-rpc/post-install.d/10-qubes-core-agent-features.sh

@marmarek
Copy link
Member

I can confirm that now sys-net with 325MB no longer runs into driver failures due to too little memory :) Wifi automatically re-connects after suspend etc.

@Guiiix
Copy link
Member Author

Guiiix commented Feb 11, 2025

I can confirm that now sys-net with 325MB no longer runs into driver failures due to too little memory :) Wifi automatically re-connects after suspend etc.

I've been running my sys-net with 300MB for a couple of days and it works too. NM applet is a bit slow but internet connection is stable and I don't see any failure in logs 🙂

@marmarek marmarek merged commit 6591aa0 into QubesOS:main Feb 13, 2025
5 checks passed
@marmarek
Copy link
Member

marmarek commented Feb 19, 2025

Uhm, sys-net just got too aggressive OOM killer: https://openqa.qubes-os.org/tests/129158:

[2025-02-19 05:55:58] [    4.549894] EXT4-fs (xvdb): mounted filesystem cc0de015-4576-48b6-92fd-e92dbc733321 r/w with ordered data mode. Quota mode: none.
[2025-02-19 05:55:58] [    4.694852] cp invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
[2025-02-19 05:55:58] [    4.694892] CPU: 1 PID: 427 Comm: cp Not tainted 6.6.77-1.qubes.fc41.x86_64 #1
[2025-02-19 05:55:58] [    4.694914] Hardware name: Xen HVM domU, BIOS 4.19.1 02/16/2025
[2025-02-19 05:55:58] [    4.694933] Call Trace:
[2025-02-19 05:55:58] [    4.694943]  <TASK>
[2025-02-19 05:55:58] [    4.694955]  dump_stack_lvl+0x4d/0x70
[2025-02-19 05:55:58] [    4.694972]  dump_header+0x4a/0x240
[2025-02-19 05:55:58] [    4.694988]  oom_kill_process+0xf7/0x190
[2025-02-19 05:55:58] [    4.695002]  out_of_memory+0xf4/0x330
[2025-02-19 05:55:58] [    4.695015]  __alloc_pages_slowpath.constprop.0+0x55e/0xbc0
[2025-02-19 05:55:58] [    4.695032]  ? __do_sys_brk+0x290/0x3e0
[2025-02-19 05:55:58] [    4.695046]  __alloc_pages+0x320/0x340
[2025-02-19 05:55:58] [    4.695059]  __folio_alloc+0x15/0x30
[2025-02-19 05:55:58] [    4.695071]  vma_alloc_folio+0x9e/0x380
[2025-02-19 05:55:58] [    4.695085]  do_anonymous_page+0x6c/0x3b0
[2025-02-19 05:55:58] [    4.695103]  __handle_mm_fault+0x39b/0x6d0
[2025-02-19 05:55:58] [    4.695117]  handle_mm_fault+0xdd/0x350
[2025-02-19 05:55:58] [    4.695130]  do_user_addr_fault+0x15b/0x630
[2025-02-19 05:55:58] [    4.695145]  exc_page_fault+0x7f/0x180
[2025-02-19 05:55:58] [    4.695159]  asm_exc_page_fault+0x26/0x30
[2025-02-19 05:55:58] [    4.695173] RIP: 0033:0x7b9ebf676fb3
[2025-02-19 05:55:58] [    4.695187] Code: 34 19 49 39 d4 49 89 74 24 60 0f 95 c2 48 29 d8 48 83 c1 10 0f b6 d2 48 83 c8 01 48 c1 e2 02 48 09 da 48 83 ca 01 48 89 51 f8 <48> 89 46 08 e9 aa fe ff ff 48 8d 0d 2d 80 13 00 ba 64 10 00 00 48
[2025-02-19 05:55:58] [    4.695232] RSP: 002b:00007ffccca7b030 EFLAGS: 00010206
[2025-02-19 05:55:58] [    4.695248] RAX: 0000000000006fc1 RBX: 00000000000000d0 RCX: 00005997ce4a8f80
[2025-02-19 05:55:58] [    4.695269] RDX: 00000000000000d1 RSI: 00005997ce4a9040 RDI: 0000000000000000
[2025-02-19 05:55:58] [    4.695289] RBP: 00007ffccca7b080 R08: 00000000000000d0 R09: 0000000000000001
[2025-02-19 05:55:58] [    4.695310] R10: 0000000000000004 R11: 0000000000000000 R12: 00007b9ebf7dfac0
[2025-02-19 05:55:58] [    4.695330] R13: 00000000000000c7 R14: 00000000000000d0 R15: fffffffffffffed8
[2025-02-19 05:55:58] [    4.695354]  </TASK>
[2025-02-19 05:55:58] [    4.695391] Mem-Info:
[2025-02-19 05:55:58] [    4.695401] active_anon:16039 inactive_anon:163 isolated_anon:0
[2025-02-19 05:55:58] [    4.695401]  active_file:9440 inactive_file:5929 isolated_file:0
[2025-02-19 05:55:58] [    4.695401]  unevictable:0 dirty:1002 writeback:1
[2025-02-19 05:55:58] [    4.695401]  slab_reclaimable:2144 slab_unreclaimable:7346
[2025-02-19 05:55:58] [    4.695401]  mapped:5560 shmem:155 pagetables:587
[2025-02-19 05:55:58] [    4.695401]  sec_pagetables:0 bounce:0
[2025-02-19 05:55:58] [    4.695401]  kernel_misc_reclaimable:0
[2025-02-19 05:55:58] [    4.695401]  free:1343 free_pcp:36 free_cma:0
[2025-02-19 05:55:58] [    4.695488] Node 0 active_anon:64156kB inactive_anon:652kB active_file:37760kB inactive_file:23716kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:22240kB dirty:4008kB writeback:4kB shmem:620kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:1836kB pagetables:2348kB sec_pagetables:0kB all_unreclaimable? no
[2025-02-19 05:55:58] [    4.695561] Node 0 DMA free:1736kB boost:2048kB min:2176kB low:2208kB high:2240kB reserved_highatomic:0KB active_anon:8404kB inactive_anon:64kB active_file:3208kB inactive_file:740kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[2025-02-19 05:55:58] [    4.695625] lowmem_reserve[]: 0 201 201 201 201
[2025-02-19 05:55:58] [    4.695641] Node 0 DMA32 free:3636kB boost:2048kB min:3796kB low:4232kB high:4668kB reserved_highatomic:0KB active_anon:55588kB inactive_anon:588kB active_file:34680kB inactive_file:22828kB unevictable:0kB writepending:4012kB present:274428kB managed:205392kB mlocked:0kB bounce:0kB free_pcp:144kB local_pcp:112kB free_cma:0kB
[2025-02-19 05:55:58] [    4.695708] lowmem_reserve[]: 0 0 0 0 0
[2025-02-19 05:55:58] [    4.695721] Node 0 DMA: 3*4kB (UM) 0*8kB 0*16kB 1*32kB (U) 0*64kB 1*128kB (U) 0*256kB 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 1708kB
[2025-02-19 05:55:58] [    4.695757] Node 0 DMA32: 231*4kB (UE) 92*8kB (U) 40*16kB (UME) 8*32kB (U) 3*64kB (UE) 1*128kB (M) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 3644kB
[2025-02-19 05:55:58] [    4.695801] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[2025-02-19 05:55:58] [    4.695825] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[2025-02-19 05:55:58] [    4.695848] 15543 total pagecache pages
[2025-02-19 05:55:58] [    4.695860] 0 pages in swap cache
[2025-02-19 05:55:58] [    4.695870] Free swap  = 1048316kB
[2025-02-19 05:55:58] [    4.695881] Total swap = 1048572kB
[2025-02-19 05:55:58] [    4.695894] 72605 pages RAM
[2025-02-19 05:55:58] [    4.695902] 0 pages HighMem/MovableOnly
[2025-02-19 05:55:58] [    4.695912] 17417 pages reserved
[2025-02-19 05:55:58] [    4.695923] 0 pages cma reserved
[2025-02-19 05:55:58] [    4.695934] 0 pages hwpoisoned
[2025-02-19 05:55:58] [    4.695945] Tasks state (memory values in pages):
[2025-02-19 05:55:58] [    4.695958] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[2025-02-19 05:55:58] [    4.695987] [    279]     0   279     3853     1568    73728        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696012] [    283]     0   283     3837     1536    69632        0             0 systemd-userdbd
[2025-02-19 05:55:58] [    4.696036] [    284]     0   284    17460      704    57344        0             0 qubesdb-daemon
[2025-02-19 05:55:58] [    4.696060] [    286]     0   286     3976     1696    73728        0             0 systemd-userwor
[2025-02-19 05:55:58] [    4.696084] [    287]     0   287     3976     1696    73728        0             0 systemd-userwor
[2025-02-19 05:55:58] [    4.696683] [    288]     0   288     3976     1696    69632        0             0 systemd-userwor
[2025-02-19 05:55:58] [    4.696713] [    290]     0   290     3914     1632    73728        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696737] [    291]     0   291     3914     1632    73728        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696762] [    292]     0   292     3914     1632    73728        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696786] [    293]     0   293     3914     1664    73728        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696810] [    294]     0   294     3914     1632    77824        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696835] [    295]     0   295     3976     1696    73728        0             0 systemd-userwor
[2025-02-19 05:55:58] [    4.696859] [    298]     0   298     3914     1664    69632        0             0 systemd-nsresou
[2025-02-19 05:55:58] [    4.696884] [    308]     0   308     3976     1696    69632        0             0 systemd-userwor
[2025-02-19 05:55:58] [    4.696909] [    325]     0   325     2090     1219    49152        0             0 haveged
[2025-02-19 05:55:58] [    4.696932] [    326]     0   326     1171      832    57344        0             0 mount-dirs.sh
[2025-02-19 05:55:58] [    4.696958] [    328]     0   328     8010     2912    86016        0          -250 systemd-journal
[2025-02-19 05:55:58] [    4.696982] [    331]     0   331     9171     3155    94208        0         -1000 systemd-udevd
[2025-02-19 05:55:58] [    4.697007] [    387]     0   387    10900     2894   118784        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697031] [    388]     0   388    10576     2542   110592        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697055] [    389]     0   389    10109     2731   110592        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697080] [    390]     0   390    10576     2574   114688        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697133] [    391]     0   391    10006     2822   110592        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697157] [    392]     0   392    10897     3014   118784        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697182] [    393]     0   393    10895     2789   114688        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697205] [    394]     0   394    10606     2576   114688        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697230] [    395]     0   395    10789     2693   118784        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697253] [    396]     0   396    10608     2513   110592        0             0 (udev-worker)
[2025-02-19 05:55:58] [    4.697278] [    407]     0   407     1138      864    53248        0             0 setup-rw.sh
[2025-02-19 05:55:58] [    4.697302] [    427]     0   427    14388     4030   163840        0             0 cp
[2025-02-19 05:55:58] [    4.697329] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/qubes-mount-dirs.service,task=cp,pid=427,uid=0
[2025-02-19 05:55:58] [    4.697378] Out of memory: Killed process 427 (cp) total-vm:57552kB, anon-rss:13048kB, file-rss:3072kB, shmem-rss:0kB, UID:0 pgtables:160kB oom_score_adj:0
[2025-02-19 05:55:58] [�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mqubes-mount-dirs.s…�[0mInitialize and mount /rw and /home.

[2025-02-19 05:55:58] See 'systemctl status qubes-mount-dirs.service' for details.

That's during VM startup, with 300MB.
I have two ideas:

  • I see a bunch of systemd-nsresourced processes, based on its man page it looks like something needed only for containers - can you check what uses it and if it can be disabled?
  • set memory to 325MB

@Guiiix
Copy link
Member Author

Guiiix commented Feb 20, 2025

I updated my net-vm to r4.3-testing and indeed, new services are present. I tried to disable systemd-nsresourced.service but it is started by systemd-nsresourced.socket and even with ConditionalPathExists and both units, it's starting anyway.

@marmarek
Copy link
Member

ConditionalPathExists

typo: ConditionPathExists

see journalctl, maybe it will tell you why it ignored the setting
alternatively, for testing you can mask either or both units and see what happens; I expect something to fail, but it's interesting to learn what, as maybe that other thing isn't needed either ;)

@Guiiix
Copy link
Member Author

Guiiix commented Feb 20, 2025

Oops, sorry for the typo.
Well, the reason was stupid, systemd-nsresourced.socket was starting before qubes-sysinit.service... So /var/run/qubes-service/minimal-netvm was not present. 😅
I added an After statement in the dropins and now it's not running anymore.
Everything seem to be fine, no service failed, no strange error in the journal...

@Guiiix
Copy link
Member Author

Guiiix commented Feb 20, 2025

When masking the service:

  • socket fails to start before the service couldn't be loaded

When masking the socket or both

  • no service failed nor error in the journal

@Guiiix
Copy link
Member Author

Guiiix commented Feb 20, 2025

systemd-userdbd doesn't seem necessary too

@ben-grande
Copy link
Contributor

Can you please document this feature in https://github.com/QubesOS/qubes-core-admin-client/blob/main/doc/manpages/qvm-service.rst

@Guiiix
Copy link
Member Author

Guiiix commented Feb 28, 2025

Got an error about conntrack tables full today with 300MB RAM...

@marmarek
Copy link
Member

Hm, I guess the default size is chosen based on the memory mount. But it should be configurable. But still, even if we increase the limit, the question is whether there is enough RAM to store it.
Can you check /proc/sys/net/nf_conntrack_max? There is also /proc/sys/net/netfilter/nf_conntrack_max, not sure what is the relation between them - I see the same value in both. FWIW I have 2560 there on R4.2 with 400MB for sys-net.

@Guiiix
Copy link
Member Author

Guiiix commented Feb 28, 2025

Yes I guess this is configurable but it might be hard to find a suitable value not too high as we don't have enough memory and not too low as we can have a lot of connections when running multiple qubes with network access.

On my sys-net (R4.3-current-testing repos, 325MB RAM) , I have the same value for both variables: 2048.

@marmarek
Copy link
Member

So, it's not too different from my value, yet I haven't run into such issues. Is it possible that some of your applications opens more than usual connections? You can also list what is in the conntrack table with the conntrack -L call, maybe that will hint what is going on (it will probably make more sens in sys-firewall, as that will see which AppVM that is, sys-net will have all connections attributed to sys-firewall's IP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants