Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel 6.11 "Allocator stuck? Waited for 30 seconds" #749

Open
truongsinh opened this issue Sep 22, 2024 · 13 comments
Open

Kernel 6.11 "Allocator stuck? Waited for 30 seconds" #749

truongsinh opened this issue Sep 22, 2024 · 13 comments

Comments

@truongsinh
Copy link

I'm running 6.11.0-061100rc7daily20240915-generic, is this problem the lock contention that was rejected during 6.11 RC window?

Allocator stuck? Waited for 30 seconds
[12833.040409] Allocator stuck? Waited for 30 seconds
[12833.040420] Allocator debug:
[12833.040421]   capacity8344943698
[12833.040422]   reserved             725647278
[12833.040505]   hidden               16229376
[12833.040506]   btree                55331840
[12833.040508]   data                 7198854472
[12833.040509]   cached               0
[12833.040510]   reserved             0
[12833.040511]   online_reserved      33344
[12833.040512]   nr_inodes            0
[12833.040514]   
[12833.040515]   freelist_wait        waiting
[12833.040516]   open buckets allocated11
[12833.040518]   open buckets total   1024
[12833.040519]   open_buckets_wait    empty
[12833.040520]   open_buckets_btree   2
[12833.040521]   open_buckets_user    8
[12833.040522]   btree reserve cache  1
[12833.040567] 
[12833.040569] Dev 0:
[12833.040570]                      buckets         sectors      fragmented
[12833.040571]   free                 67502               0               0
[12833.040572]   sb                      13            6152             504
[12833.040573]   journal               8192         4194304               0
[12833.040574]   btree                    0               0               0
[12833.040575]   user               4240325      2171043656            2744
[12833.040576]   cached                   0               0               0
[12833.040577]   parity                   0               0               0
[12833.040578]   stripe                   0               0               0
[12833.040579]   need_gc_gens             0               0               0
[12833.040580]   need_discard             0               0               0
[12833.040601]   unstriped                0               0               0
[12833.040603]   capacity           4316032
[12833.040603]   
[12833.040604]   reserves:
[12833.040605]   stripe              134932
[12833.040606]   normal               67494
[12833.040607]   copygc                  56
[12833.040608]   btree                   28
[12833.040608]   btree_copygc             0
[12833.040609]   reclaim                  0
[12833.040610]   interior_updates         0
[12833.040611]   
[12833.040612]   open buckets             8
[12833.040613]   buckets to invalidate    0
[12833.040614] 
[12833.040615] Dev 1:
[12833.040616]                      buckets         sectors      fragmented
[12833.040617]   free                 15180               0               0
[12833.040627]   sb                      13            6152             504
[12833.040628]   journal               7631         3907072               0
[12833.040630]   btree                26798        13720576               0
[12833.040631]   user                927166       474708992               0
[12833.040632]   cached                   0               0               0
[12833.040634]   parity                   0               0               0
[12833.040635]   stripe                   0               0               0
[12833.040636]   need_gc_gens             0               0               0
[12833.040637]   need_discard             0               0               0
[12833.040639]   unstriped                0               0               0
[12833.040640]   capacity            976788
[12833.040641]   
[12833.040641]   reserves:
[12833.040643]   stripe               30580
[12833.040644]   normal               15318
[12833.040644]   copygc                  56
[12833.040645]   btree                   28
[12833.040646]   btree_copygc             0
[12833.040647]   reclaim                  0
[12833.040648]   interior_updates         0
[12833.040649]   
[12833.040650]   open buckets             1
[12833.040651]   buckets to invalidate    0
[12833.040652] 
[12833.040653] Dev 2:
[12833.040654]                      buckets         sectors      fragmented
[12833.040655]   free                 15191               0               0
[12833.040657]   sb                      13            6152             504
[12833.040658]   journal               7631         3907072               0
[12833.040659]   btree                27455        14056960               0
[12833.040660]   user                926498       474366976               0
[12833.040662]   cached                   0               0               0
[12833.040663]   parity                   0               0               0
[12833.040664]   stripe                   0               0               0
[12833.040665]   need_gc_gens             0               0               0
[12833.040666]   need_discard             0               0               0
[12833.040667]   unstriped                0               0               0
[12833.040669]   capacity            976788
[12833.040669]   
[12833.040670]   reserves:
[12833.040671]   stripe               30580
[12833.040673]   normal               15318
[12833.040674]   copygc                  56
[12833.040674]   btree                   28
[12833.040675]   btree_copygc             0
[12833.040676]   reclaim                  0
[12833.040677]   interior_updates         0
[12833.040686]   
[12833.040686]   open buckets             0
[12833.040688]   buckets to invalidate    0
[12833.040689] 
[12833.040690] Dev 3:
[12833.040691]                      buckets         sectors      fragmented
[12833.040692]   free                119199               0               0
[12833.040693]   sb                      13            6152             504
[12833.040694]   journal               8192         4194304               0
[12833.040695]   btree                53817        27554304               0
[12833.040697]   user               7449691      3814241792               0
[12833.040698]   cached                   0               0               0
[12833.040699]   parity                   0               0               0
[12833.040700]   stripe                   0               0               0
[12833.040700]   need_gc_gens             0               0               0
[12833.040701]   need_discard             0               0               0
[12833.040702]   unstriped                0               0               0
[12833.040710]   capacity           7630912
[12833.040711]   
[12833.040711]   reserves:
[12833.040712]   stripe              238522
[12833.040713]   normal              119289
[12833.040714]   copygc                  56
[12833.040716]   btree                   28
[12833.040716]   btree_copygc             0
[12833.040717]   reclaim                  0
[12833.040718]   interior_updates         0
[12833.040718]   
[12833.040719]   open buckets             1
[12833.040720]   buckets to invalidate    0
[12833.040721] 
[12833.040722] Dev 4:
[12833.040723]                      buckets         sectors      fragmented
[12833.040724]   free     18446744073709293322            0               0
[12833.040725]   sb                       0               0               0
[12833.040726]   journal                  0               0               0
[12833.040727]   btree                    0               0               0
[12833.040728]   user                258294       264493056               0
[12833.040729]   cached                   0               0               0
[12833.040730]   parity                   0               0               0
[12833.040731]   stripe                   0               0               0
[12833.040732]   need_gc_gens             0               0               0
[12833.040739]   need_discard             0               0               0
[12833.040745]   unstriped                0               0               0
[12833.040746]   capacity           1907739
[12833.040747]   
[12833.040753]   reserves:
[12833.040754]   stripe               59644
[12833.040755]   normal               29836
[12833.040756]   copygc                  28
[12833.040757]   btree                   14
[12833.040758]   btree_copygc             0
[12833.040758]   reclaim                  0
[12833.040759]   interior_updates         0
[12833.040760]   
[12833.040761]   open buckets             0
[12833.040762]   buckets to invalidate    0
[12833.040763] 
[12833.040763] Copygc debug:
[12833.040765]   running: 0
[12833.040765]   copygc_wait:0
[12833.040766]   copygc_wait_at:0
[12833.040767]   Currently waiting for:0 B
[12833.040767]   Currently waiting since:5.60 TiB
[12833.040768]   Currently calculated wait:0 B
[12833.040769] 
[12833.040770] Journal debug:
[12833.040770]   flags:                   replay_done,running,may_skip_flush
[12833.040771]   dirty journal entries:   0/32768
[12833.040772]   seq:                     1031358
[12833.040773]   seq_ondisk:              1031358
[12833.040774]   last_seq:                1031359
[12833.040774]   last_seq_ondisk:         1031358
[12833.040775]   flushed_seq_ondisk:      1031358
[12833.040776]   watermark:               stripe
[12833.040777]   each entry reserved:     321
[12833.040778]   nr flush writes:         12091
[12833.040779]   nr noflush writes:       3033
[12833.040780]   average write size:      69.7 KiB
[12833.040781]   nr direct reclaim:       0
[12833.040781]   nr background reclaim:   408443
[12833.040782]   reclaim kicked:          0
[12833.040783]   reclaim runs in:         0 ms
[12833.040784]   blocked:                 0
[12833.040784]   current entry sectors:   512
[12833.040785]   current entry error:     ok
[12833.040786]   current entry:           closed
[12833.040786]   unwritten entries:
[12833.040787]   last buf closed
[12833.040788]   space:
[12833.040788]     discarded              512:4193792
[12833.040789]     clean ondisk           512:4193792
[12833.040790]     clean                  512:4193792
[12833.040805]     total                  512:4194304
[12833.040806]   dev 0:
[12833.040807]     nr                     8192
[12833.040807]     bucket size            512
[12833.040809]     available              8190:512
[12833.040809]     discard_idx            0
[12833.040810]     dirty_ondisk           0 (seq 0)
[12833.040811]     dirty_idx              0 (seq 0)
[12833.040812]     cur_idx                0 (seq 0)
[12833.040813]   dev 1:
[12833.040814]     nr                     7631
[12833.040815]     bucket size            512
[12833.040816]     available              7629:0
[12833.040817]     discard_idx            6033
[12833.040818]     dirty_ondisk           6033 (seq 1031335)
[12833.040825]     dirty_idx              6033 (seq 1031335)
[12833.040826]     cur_idx                6033 (seq 1031335)
[12833.040828]   dev 2:
[12833.040828]     nr                     7631
[12833.040830]     bucket size            512
[12833.040831]     available              7629:240
[12833.040832]     discard_idx            6636
[12833.040833]     dirty_ondisk           6636 (seq 1031358)
[12833.040834]     dirty_idx              6636 (seq 1031358)
[12833.040835]     cur_idx                6636 (seq 1031358)
[12833.040836]   dev 3:
[12833.040837]     nr                     8192
[12833.040837]     bucket size            512
[12833.040839]     available              8190:240
[12833.040840]     discard_idx            1574
[12833.040841]     dirty_ondisk           1574 (seq 1031358)
[12833.040842]     dirty_idx              1574 (seq 1031358)
[12833.040842]     cur_idx                1574 (seq 1031358)
[12833.040843]   dev 4:
[12833.040844]     nr                     8192
[12833.040845]     bucket size            1024
[12833.040846]     available              8190:1024
[12833.040847]     discard_idx            0
[12833.040847]     dirty_ondisk           0 (seq 0)
[12833.040848]     dirty_idx              0 (seq 0)
[12833.040849]     cur_idx                0 (seq 0)
[12833.040850] 
[12833.623726] Allocator stuck? Waited for 30 seconds
[12833.623736] Allocator debug:
[12833.623737]   capacity8344943698
[12833.623739]   reserved             725647278
[12833.623778]   hidden               16229376
[12833.623779]   btree                55331840
[12833.623780]   data                 7198854472
[12833.623781]   cached               0
[12833.623782]   reserved             0
[12833.623783]   online_reserved      33344
[12833.623783]   nr_inodes            0
[12833.623785]   
[12833.623785]   freelist_wait        waiting
[12833.623786]   open buckets allocated11
[12833.623788]   open buckets total   1024
[12833.623788]   open_buckets_wait    empty
[12833.623789]   open_buckets_btree   2
[12833.623790]   open_buckets_user    8
[12833.623791]   btree reserve cache  1
[12833.623792] 
[12833.623793] Dev 0:
[12833.623794]                      buckets         sectors      fragmented
[12833.623795]   free                 67502               0               0
[12833.623796]   sb                      13            6152             504
[12833.623797]   journal               8192         4194304               0
[12833.623798]   btree                    0               0               0
[12833.623799]   user               4240325      2171043656            2744
[12833.623800]   cached                   0               0               0
[12833.623801]   parity                   0               0               0
[12833.623802]   stripe                   0               0               0
[12833.623803]   need_gc_gens             0               0               0
[12833.623804]   need_discard             0               0               0
[12833.623805]   unstriped                0               0               0
[12833.623806]   capacity           4316032
[12833.623807]   
[12833.623808]   reserves:
[12833.623809]   stripe              134932
[12833.623810]   normal               67494
[12833.623810]   copygc                  56
[12833.623811]   btree                   28
[12833.623812]   btree_copygc             0
[12833.623813]   reclaim                  0
[12833.623814]   interior_updates         0
[12833.623815]   
[12833.623816]   open buckets             8
[12833.623816]   buckets to invalidate    0
[12833.623817] 
[12833.623818] Dev 1:
[12833.623819]                      buckets         sectors      fragmented
[12833.623820]   free                 15180               0               0
[12833.623821]   sb                      13            6152             504
[12833.623832]   journal               7631         3907072               0
[12833.623833]   btree                26798        13720576               0
[12833.623834]   user                927166       474708992               0
[12833.623835]   cached                   0               0               0
[12833.623836]   parity                   0               0               0
[12833.623837]   stripe                   0               0               0
[12833.623852]   need_gc_gens             0               0               0
[12833.623853]   need_discard             0               0               0
[12833.623854]   unstriped                0               0               0
[12833.623855]   capacity            976788
[12833.623856]   
[12833.623857]   reserves:
[12833.623858]   stripe               30580
[12833.623859]   normal               15318
[12833.623860]   copygc                  56
[12833.623862]   btree                   28
[12833.623864]   btree_copygc             0
[12833.623867]   reclaim                  0
[12833.623869]   interior_updates         0
[12833.623869]   
[12833.623870]   open buckets             1
[12833.623871]   buckets to invalidate    0
[12833.623873] 
[12833.623875] Dev 2:
[12833.623876]                      buckets         sectors      fragmented
[12833.623877]   free                 15191               0               0
[12833.623879]   sb                      13            6152             504
[12833.623887]   journal               7631         3907072               0
[12833.623888]   btree                27455        14056960               0
[12833.623888]   user                926498       474366976               0
[12833.623890]   cached                   0               0               0
[12833.623891]   parity                   0               0               0
[12833.623892]   stripe                   0               0               0
[12833.623893]   need_gc_gens             0               0               0
[12833.623894]   need_discard             0               0               0
[12833.623895]   unstriped                0               0               0
[12833.623896]   capacity            976788
[12833.623897]   
[12833.623898]   reserves:
[12833.623899]   stripe               30580
[12833.623900]   normal               15318
[12833.623900]   copygc                  56
[12833.623901]   btree                   28
[12833.623902]   btree_copygc             0
[12833.623903]   reclaim                  0
[12833.623904]   interior_updates         0
[12833.623911]   
[12833.623911]   open buckets             0
[12833.623912]   buckets to invalidate    0
[12833.623913] 
[12833.623914] Dev 3:
[12833.623915]                      buckets         sectors      fragmented
[12833.623916]   free                119199               0               0
[12833.623919]   sb                      13            6152             504
[12833.623921]   journal               8192         4194304               0
[12833.623922]   btree                53817        27554304               0
[12833.623923]   user               7449691      3814241792               0
[12833.623924]   cached                   0               0               0
[12833.623925]   parity                   0               0               0
[12833.623926]   stripe                   0               0               0
[12833.623927]   need_gc_gens             0               0               0
[12833.623928]   need_discard             0               0               0
[12833.623929]   unstriped                0               0               0
[12833.623930]   capacity           7630912
[12833.623931]   
[12833.623932]   reserves:
[12833.623933]   stripe              238522
[12833.623935]   normal              119289
[12833.623935]   copygc                  56
[12833.623936]   btree                   28
[12833.623937]   btree_copygc             0
[12833.623938]   reclaim                  0
[12833.623939]   interior_updates         0
[12833.623940]   
[12833.623941]   open buckets             1
[12833.623942]   buckets to invalidate    0
[12833.623943] 
[12833.623944] Dev 4:
[12833.623944]                      buckets         sectors      fragmented
[12833.623945]   free     18446744073709293322            0               0
[12833.623946]   sb                       0               0               0
[12833.623948]   journal                  0               0               0
[12833.623949]   btree                    0               0               0
[12833.623950]   user                258294       264493056               0
[12833.623951]   cached                   0               0               0
[12833.623952]   parity                   0               0               0
[12833.623953]   stripe                   0               0               0
[12833.623960]   need_gc_gens             0               0               0
[12833.623962]   need_discard             0               0               0
[12833.623963]   unstriped                0               0               0
[12833.623964]   capacity           1907739
[12833.623965]   
[12833.623966]   reserves:
[12833.623972]   stripe               59644
[12833.623973]   normal               29836
[12833.623974]   copygc                  28
[12833.623974]   btree                   14
[12833.623975]   btree_copygc             0
[12833.623977]   reclaim                  0
[12833.623977]   interior_updates         0
[12833.623978]   
[12833.623979]   open buckets             0
[12833.623980]   buckets to invalidate    0
[12833.623981] 
[12833.623982] Copygc debug:
[12833.623983]   running: 0
[12833.623984]   copygc_wait:0
[12833.623985]   copygc_wait_at:0
[12833.623985]   Currently waiting for:0 B
[12833.623987]   Currently waiting since:5.60 TiB
[12833.623988]   Currently calculated wait:0 B
[12833.623991] 
[12833.623993] Journal debug:
[12833.623993]   flags:                   replay_done,running,may_skip_flush
[12833.623994]   dirty journal entries:   0/32768
[12833.623995]   seq:                     1031358
[12833.623996]   seq_ondisk:              1031358
[12833.623997]   last_seq:                1031359
[12833.623998]   last_seq_ondisk:         1031358
[12833.624000]   flushed_seq_ondisk:      1031358
[12833.624001]   watermark:               stripe
[12833.624002]   each entry reserved:     321
[12833.624003]   nr flush writes:         12091
[12833.624004]   nr noflush writes:       3033
[12833.624005]   average write size:      69.7 KiB
[12833.624006]   nr direct reclaim:       0
[12833.624007]   nr background reclaim:   408443
[12833.624008]   reclaim kicked:          0
[12833.624009]   reclaim runs in:         0 ms
[12833.624010]   blocked:                 0
[12833.624011]   current entry sectors:   512
[12833.624012]   current entry error:     ok
[12833.624013]   current entry:           closed
[12833.624014]   unwritten entries:
[12833.624015]   last buf closed
[12833.624016]   space:
[12833.624016]     discarded              512:4193792
[12833.624018]     clean ondisk           512:4193792
[12833.624019]     clean                  512:4193792
[12833.624026]     total                  512:4194304
[12833.624026]   dev 0:
[12833.624027]     nr                     8192
[12833.624028]     bucket size            512
[12833.624029]     available              8190:512
[12833.624030]     discard_idx            0
[12833.624031]     dirty_ondisk           0 (seq 0)
[12833.624032]     dirty_idx              0 (seq 0)
[12833.624033]     cur_idx                0 (seq 0)
[12833.624034]   dev 1:
[12833.624035]     nr                     7631
[12833.624036]     bucket size            512
[12833.624036]     available              7629:0
[12833.624037]     discard_idx            6033
[12833.624038]     dirty_ondisk           6033 (seq 1031335)
[12833.624039]     dirty_idx              6033 (seq 1031335)
[12833.624040]     cur_idx                6033 (seq 1031335)
[12833.624041]   dev 2:
[12833.624042]     nr                     7631
[12833.624043]     bucket size            512
[12833.624044]     available              7629:240
[12833.624045]     discard_idx            6636
[12833.624045]     dirty_ondisk           6636 (seq 1031358)
[0] 0:bash*                                             "qemuvm" 03:17 22-Sep-24
[12833.624047]     cur_idx                6636 (seq 1031358)
[12833.624048]   dev 3:
[12833.624049]     nr                     8192
[12833.624050]     bucket size            512
[12833.624051]     available              8190:240
[12833.624052]     discard_idx            1574
[12833.624053]     dirty_ondisk           1574 (seq 1031358)
[12833.624054]     dirty_idx              1574 (seq 1031358)
[12833.624055]     cur_idx                1574 (seq 1031358)
[12833.624056]   dev 4:
[12833.624056]     nr                     8192
[12833.624057]     bucket size            1024
[12833.624058]     available              8190:1024
[12833.624059]     discard_idx            0
[12833.624060]     dirty_ondisk           0 (seq 0)
[12833.624067]     dirty_idx              0 (seq 0)
[12833.624068]     cur_idx                0 (seq 0)
[12833.624069] 

[0] 0:bash*                                             "qemuvm" 03:16 22
@truongsinh
Copy link
Author

This time, it's kernel task of rebalance:

root@qemuvm:~# [  861.228704] INFO: task bch-rebalance/1:837 blocked for more than 122 seconds.
[  861.236360]       Not tainted 6.11.0-061100rc7daily20240915-generic #202409150203
[  861.250846] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  987.179736] Allocator stuck? Waited for 30 seconds
[  987.179752] Allocator debug:
[  987.179757]   capacity8344943698
[  987.179761]   reserved             725647278
[  987.179766]   hidden               16231424
[  987.179770]   btree                55331840
[  987.179774]   data                 7198854472
[  987.179779]   cached               0
[  987.179785]   reserved             0
[  987.179788]   online_reserved      0
[  987.179792]   nr_inodes            0
[  987.179796]   
[  987.179800]   freelist_wait        waiting
[  987.179805]   open buckets allocated4
[  987.179809]   open buckets total   1024
[  987.179814]   open_buckets_wait    empty
[  987.179818]   open_buckets_btree   2
[  987.179822]   open_buckets_user    1
[  987.179826]   btree reserve cache  1
[  987.179830] 
[  987.179834] Dev 0:
[  987.179838]                      buckets         sectors      fragmented
[  987.179843]   free                 67509               0               0
[  987.179925]   sb                      13            6152             504
[  987.179931]   journal               8192         4194304               0
[  987.179936]   btree                    0               0               0
[  987.179941]   user               4240318      2171042816               0
[  987.179945]   cached                   0               0               0
[  987.179950]   parity                   0               0               0
[  987.179954]   stripe                   0               0               0
[  987.179959]   need_gc_gens             0               0               0
[  987.179964]   need_discard             0               0               0
[  987.179968]   unstriped                0               0               0
[  987.179973]   capacity           4316032
[  987.179977]   
[  987.179981]   reserves:
[  987.179984]   stripe              134932
[  987.179990]   normal               67494
[  987.179994]   copygc                  56
[  987.179999]   btree                   28
[  987.180002]   btree_copygc             0
[  987.180007]   reclaim                  0
[  987.180011]   interior_updates         0
[  987.180014]   
[  987.180018]   open buckets             0
[  987.180023]   buckets to invalidate    0
[  987.180026] 
[  987.180030] Dev 1:
[  987.180034]                      buckets         sectors      fragmented
[  987.180039]   free                 15179               0               0
[  987.180044]   sb                      13            6152             504
[  987.180048]   journal               7631         3907072               0
[  987.180053]   btree                26798        13720576               0
[  987.180058]   user                927167       474709504               0
[  987.180062]   cached                   0               0               0
[  987.180067]   parity                   0               0               0
[  987.180071]   stripe                   0               0               0
[  987.180076]   need_gc_gens             0               0               0
[  987.180081]   need_discard             0               0               0
[  987.180085]   unstriped                0               0               0
[  987.180090]   capacity            976788
[  987.180094]   
[  987.180097]   reserves:
[  987.180101]   stripe               30580
[  987.180105]   normal               15318
[  987.180109]   copygc                  56
[  987.180114]   btree                   28
[  987.180118]   btree_copygc             0
[  987.180121]   reclaim                  0
[  987.180126]   interior_updates         0
[  987.180129]   
[  987.180133]   open buckets             1
[  987.180137]   buckets to invalidate    0
[  987.180141] 
[  987.180144] Dev 2:
[  987.180148]                      buckets         sectors      fragmented
[  987.180153]   free                 15190               0               0
[  987.180158]   sb                      13            6152             504
[  987.180162]   journal               7631         3907072               0
[  987.180167]   btree                27455        14056960               0
[  987.180172]   user                926499       474367304             184
[  987.180176]   cached                   0               0               0
[  987.180181]   parity                   0               0               0
[  987.180185]   stripe                   0               0               0
[  987.180190]   need_gc_gens             0               0               0
[  987.180194]   need_discard             0               0               0
[  987.180199]   unstriped                0               0               0
[  987.180203]   capacity            976788
[  987.180207]   
[  987.180211]   reserves:
[  987.180215]   stripe               30580
[  987.180219]   normal               15318
[  987.180223]   copygc                  56
[  987.180227]   btree                   28
[  987.180231]   btree_copygc             0
[  987.180235]   reclaim                  0
[  987.180239]   interior_updates         0
[  987.180242]   
[  987.180246]   open buckets             2
[  987.180250]   buckets to invalidate    0
[  987.180254] 
[  987.180257] Dev 3:
[  987.180261]                      buckets         sectors      fragmented
[  987.180266]   free                119199               0               0
[  987.180270]   sb                      13            6152             504
[  987.180275]   journal               8192         4194304               0
[  987.180280]   btree                53817        27554304               0
[  987.180284]   user               7449691      3814241792               0
[  987.180289]   cached                   0               0               0
[  987.180294]   parity                   0               0               0
[  987.180298]   stripe                   0               0               0
[  987.180303]   need_gc_gens             0               0               0
[  987.180307]   need_discard             0               0               0
[  987.180312]   unstriped                0               0               0
[  987.180316]   capacity           7630912
[  987.180320]   
[  987.180324]   reserves:
[  987.180328]   stripe              238522
[  987.180332]   normal              119289
[  987.180336]   copygc                  56
[  987.180340]   btree                   28
[  987.180344]   btree_copygc             0
[  987.180347]   reclaim                  0
[  987.180352]   interior_updates         0
[  987.180355]   
[  987.180359]   open buckets             0
[  987.180363]   buckets to invalidate    0
[  987.180367] 
[  987.180370] Dev 4:
[  987.180374]                      buckets         sectors      fragmented
[  987.180379]   free     18446744073709294247            0               0
[  987.180384]   sb                       1            1024               0
[  987.180388]   journal                  1            1024               0
[  987.180393]   btree                    0               0               0
[  987.180397]   user                258294       264493056               0
[  987.180402]   cached                   0               0               0
[  987.180406]   parity                   0               0               0
[  987.180411]   stripe                   0               0               0
[  987.180416]   need_gc_gens             0               0               0
[  987.180420]   need_discard             0               0               0
[  987.180425]   unstriped                0               0               0
[  987.180429]   capacity           1907739
[  987.180433]   
[  987.180437]   reserves:
[  987.180441]   stripe               59644
[  987.180445]   normal               29836
[  987.180449]   copygc                  28
[  987.180453]   btree                   14
[  987.180457]   btree_copygc             0
[  987.180461]   reclaim                  0
[  987.180465]   interior_updates         0
[  987.180469]   
[  987.180472]   open buckets             0
[  987.180476]   buckets to invalidate    0
[  987.180480] 
[  987.180483] Copygc debug:
[  987.180487]   running: 0
[  987.180491]   copygc_wait:0
[  987.180495]   copygc_wait_at:0
[  987.180499]   Currently waiting for:0 B
[  987.180503]   Currently waiting since:5.60 TiB
[  987.180507]   Currently calculated wait:0 B
[  987.180511] 
[  987.180515] Journal debug:
[  987.180519]   flags:                   replay_done,running,may_skip_flush
[  987.180524]   dirty journal entries:   0/32768
[  987.180528]   seq:                     1031382
[  987.180532]   seq_ondisk:              1031382
[  987.180536]   last_seq:                1031383
[  987.180540]   last_seq_ondisk:         1031382
[  987.180544]   flushed_seq_ondisk:      1031382
[  987.180548]   watermark:               stripe
[  987.180553]   each entry reserved:     321
[  987.180557]   nr flush writes:         15
[  987.180561]   nr noflush writes:       0
[  987.180565]   average write size:      6.69 KiB
[  987.180569]   nr direct reclaim:       0
[  987.180573]   nr background reclaim:   145
[  987.180577]   reclaim kicked:          0
[  987.180581]   reclaim runs in:         0 ms
[  987.180585]   blocked:                 0
[  987.180589]   current entry sectors:   512
[  987.180593]   current entry error:     ok
[  987.180597]   current entry:           closed
[  987.180602]   unwritten entries:
[  987.180606]   last buf closed
[  987.180609]   space:
[  987.180613]     discarded              512:4193792
[  987.180618]     clean ondisk           512:4193792
[  987.180622]     clean                  512:4193792
[  987.180626]     total                  512:4194304
[  987.180630]   dev 0:
[  987.180634]     nr                     8192
[  987.180638]     bucket size            512
[  987.180642]     available              8190:512
[  987.180646]     discard_idx            0
[  987.180651]     dirty_ondisk           0 (seq 0)
[  987.180655]     dirty_idx              0 (seq 0)
[  987.180659]     cur_idx                0 (seq 0)
[  987.180663]   dev 1:
[  987.180667]     nr                     7631
[  987.180671]     bucket size            512
[  987.180675]     available              7629:336
[  987.180679]     discard_idx            0
[  987.180683]     dirty_ondisk           0 (seq 1031381)
[  987.180688]     dirty_idx              0 (seq 1031381)
[  987.180693]     cur_idx                0 (seq 1031381)
[  987.180697]   dev 2:
[  987.180700]     nr                     7631
[  987.180704]     bucket size            512
[  987.180709]     available              7629:48
[  987.180713]     discard_idx            6636
[  987.180717]     dirty_ondisk           6636 (seq 1031382)
[  987.180721]     dirty_idx              6636 (seq 1031382)
[  987.180726]     cur_idx                6636 (seq 1031382)
[  987.180730]   dev 3:
[  987.180734]     nr                     8192
[  987.180738]     bucket size            512
[  987.180742]     available              8190:96
[  987.180746]     discard_idx            1574
[  987.180750]     dirty_ondisk           1574 (seq 1031382)
[  987.180755]     dirty_idx              1574 (seq 1031382)
[  987.180759]     cur_idx                1574 (seq 1031382)
[  987.180763]   dev 4:
[  987.180767]     nr                     8192
[  987.180771]     bucket size            1024
[  987.180775]     available              8190:1024
[  987.180779]     discard_idx            0
[  987.180783]     dirty_ondisk           0 (seq 0)
[  987.180787]     dirty_idx              0 (seq 0)
[  987.180791]     cur_idx                0 (seq 0)
[  987.180795] 

@koverstreet
Copy link
Owner

your accounting is messed up - run fsck

@gkmcd
Copy link

gkmcd commented Oct 17, 2024

I'm getting the same output in dmesg. This is after running mount with fsck,fix_errors, bcachefs fsck & bcachesfs data rereplicate.

####@ganymede ~> bcachefs version
1.12.0
####@ganymede ~> uname -a
Linux ganymede 6.11.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Oct 2024 20:11:06 +0000
x86_64 GNU/Linux
[211626.209127] Allocator stuck? Waited for 30 seconds
[211626.209133] Allocator debug:
[211626.209134]   capacity77280750838
[211626.209136]   reserved             6720065290
[211626.209138]   hidden               92328960
[211626.209139]   btree                404880384
[211626.209141]   data                 69549192929
[211626.209142]   cached               6582460531
[211626.209144]   reserved             245823744
[211626.209145]   online_reserved      36155632
[211626.209147]   nr_inodes            0
[211626.209148]   
[211626.209150]   freelist_wait        waiting
[211626.209151]   open buckets allocated96
[211626.209153]   open buckets total   1024
[211626.209154]   open_buckets_wait    empty
[211626.209155]   open_buckets_btree   4
[211626.209157]   open_buckets_user    91
[211626.209158]   btree reserve cache  1

[211626.209160] Dev 0:
[211626.209162]                      buckets         sectors      fragmented
[211626.209164]   free                178560               0               0
[211626.209165]   sb                       7            6152            1016
[211626.209167]   journal               8192         8388608               0
[211626.209169]   btree                 2159         2161664           49152
[211626.209170]   user              11257415     11527606528              72
[211626.209172]   cached                   0               0               0
[211626.209173]   parity                   0               0               0
[211626.209175]   stripe                   0               0               0
[211626.209176]   need_gc_gens             0               0               0
[211626.209178]   need_discard             0               0               0
[211626.209179]   unstriped                0               0               0
[211626.209181]   capacity          11446333
[211626.209182]   
[211626.209184]   reserves:
[211626.209185]   stripe              357724
[211626.209186]   normal              178876
[211626.209188]   copygc                  28
[211626.209189]   btree                   14
[211626.209190]   btree_copygc             0
[211626.209192]   reclaim                  0
[211626.209193]   interior_updates         0
[211626.209194]   
[211626.209196]   open buckets             1
[211626.209197]   buckets to invalidate    0

[211626.209199] Dev 1:
[211626.209201]                      buckets         sectors      fragmented
[211626.209202]   free                178546               0               0
[211626.209204]   sb                       7            6152            1016
[211626.209205]   journal               8192         8388608               0
[211626.209207]   btree                 2163         2168832           46080
[211626.209208]   user              11257425     11527616583             104
[211626.209210]   cached                   0               0               0
[211626.209211]   parity                   0               0               0
[211626.209213]   stripe                   0               0               0
[211626.209215]   need_gc_gens             0               0               0
[211626.209216]   need_discard             0               0               0
[211626.209218]   unstriped                0               0               0
[211626.209219]   capacity          11446333
[211626.209220]   
[211626.209221]   reserves:
[211626.209223]   stripe              357724
[211626.209224]   normal              178876
[211626.209225]   copygc                  28
[211626.209227]   btree                   14
[211626.209228]   btree_copygc             0
[211626.209229]   reclaim                  0
[211626.209230]   interior_updates         0
[211626.209232]   
[211626.209233]   open buckets             0
[211626.209234]   buckets to invalidate    0

[211626.209236] Dev 2:
[211626.209238]                      buckets         sectors      fragmented
[211626.209239]   free                 89127               0               0
[211626.209241]   sb                       4            6152            2040
[211626.209242]   journal               8192        16777216               0
[211626.209244]   btree                 8554        17350144          168448
[211626.209245]   user               5617289     11504227436              56
[211626.209247]   cached                   0               0               0
[211626.209248]   parity                   0               0               0
[211626.209250]   stripe                   0               0               0
[211626.209251]   need_gc_gens             0               0               0
[211626.209253]   need_discard             0               0               0
[211626.209254]   unstriped                0               0               0
[211626.209256]   capacity           5723166
[211626.209257]   
[211626.209258]   reserves:
[211626.209259]   stripe              178862
[211626.209260]   normal               89438
[211626.209262]   copygc                  14
[211626.209263]   btree                    7
[211626.209264]   btree_copygc             0
[211626.209266]   reclaim                  0
[211626.209267]   interior_updates         0
[211626.209268]   
[211626.209269]   open buckets             1
[211626.209270]   buckets to invalidate    0

[211626.209273] Dev 3:
[211626.209274]                      buckets         sectors      fragmented
[211626.209276]   free                 74547               0               0
[211626.209277]   sb                       7            6152            1016
[211626.209279]   journal               8192         8388608               0
[211626.209280]   btree                    0               0               0
[211626.209282]   user                     0               0               0
[211626.209283]   cached             1824993      1860332358         8460684
[211626.209285]   parity                   0               0               0
[211626.209286]   stripe                   0               0               0
[211626.209288]   need_gc_gens             0               0               0
[211626.209289]   need_discard             0               0               0
[211626.209291]   unstriped                0               0               0
[211626.209292]   capacity           1907739
[211626.209293]   
[211626.209294]   reserves:
[211626.209296]   stripe               59644
[211626.209297]   normal               29836
[211626.209298]   copygc                  28
[211626.209300]   btree                   14
[211626.209301]   btree_copygc             0
[211626.209302]   reclaim                  0
[211626.209304]   interior_updates         0
[211626.209305]   
[211626.209306]   open buckets            32
[211626.209307]   buckets to invalidate    1

[211626.209310] Dev 4:
[211626.209311]                      buckets         sectors      fragmented
[211626.209312]   free                119171               0               0
[211626.209314]   sb                       4            6152            2040
[211626.209316]   journal               8192        16777216               0
[211626.209317]   btree                50248       102650368          257536
[211626.209319]   user               7453269     15264308874           24984
[211626.209320]   cached                   0               0               0
[211626.209322]   parity                   0               0               0
[211626.209323]   stripe                   0               0               0
[211626.209325]   need_gc_gens             0               0               0
[211626.209327]   need_discard             1               0               0
[211626.209328]   unstriped                0               0               0
[211626.209330]   capacity           7630885
[211626.209331]   
[211626.209332]   reserves:
[211626.209333]   stripe              238478
[211626.209335]   normal              119246
[211626.209336]   copygc                  14
[211626.209337]   btree                    7
[211626.209339]   btree_copygc             0
[211626.209340]   reclaim                  0
[211626.209341]   interior_updates         0
[211626.209343]   
[211626.209344]   open buckets            12
[211626.209345]   buckets to invalidate    0

[211626.209347] Dev 6:
[211626.209349]                      buckets         sectors      fragmented
[211626.209350]   free                119156               0               0
[211626.209352]   sb                       4            6152            2040
[211626.209353]   journal               8192        16777216               0
[211626.209355]   btree                57500       117491200          268800
[211626.209356]   user               7446032     15249483790           25664
[211626.209358]   cached                   0               0               0
[211626.209359]   parity                   0               0               0
[211626.209361]   stripe                   0               0               0
[211626.209362]   need_gc_gens             0               0               0
[211626.209364]   need_discard             1               0               0
[211626.209365]   unstriped                0               0               0
[211626.209367]   capacity           7630885
[211626.209368]   
[211626.209369]   reserves:
[211626.209370]   stripe              238478
[211626.209372]   normal              119246
[211626.209373]   copygc                  14
[211626.209374]   btree                    7
[211626.209376]   btree_copygc             0
[211626.209377]   reclaim                  0
[211626.209378]   interior_updates         0
[211626.209380]   
[211626.209381]   open buckets            13
[211626.209382]   buckets to invalidate    0

[211626.209384] Dev 7:
[211626.209386]                      buckets         sectors      fragmented
[211626.209387]   free               2869734               0               0
[211626.209389]   sb                       4            6152            2040
[211626.209390]   journal               8192        16777216               0
[211626.209392]   btree                80041       163058176          865792
[211626.209393]   user               2185546      4475949718           51824
[211626.209395]   cached             2487368      4722084501       372045177
[211626.209397]   parity                   0               0               0
[211626.209398]   stripe                   0               0               0
[211626.209400]   need_gc_gens             0               0               0
[211626.209401]   need_discard             0               0               0
[211626.209403]   unstriped                0               0               0
[211626.209404]   capacity           7630885
[211626.209406]   
[211626.209407]   reserves:
[211626.209408]   stripe              238478
[211626.209410]   normal              119246
[211626.209411]   copygc                  14
[211626.209412]   btree                    7
[211626.209414]   btree_copygc             0
[211626.209415]   reclaim                  0
[211626.209416]   interior_updates         0
[211626.209417]   
[211626.209418]   open buckets            36
[211626.209420]   buckets to invalidate    0

[211626.209422] Copygc debug:
[211626.209423]   running: 0
[211626.209424]   copygc_wait:0
[211626.209426]   copygc_wait_at:0
[211626.209427]   Currently waiting for:0 B
[211626.209428]   Currently waiting since:38.7 TiB
[211626.209430]   Currently calculated wait:0 B

[211626.209432] Journal debug:
[211626.209433]   flags:                   replay_done,running,may_skip_flush
[211626.209435]   dirty journal entries:   0/32768
[211626.209436]   seq:                     14508129
[211626.209438]   seq_ondisk:              14508129
[211626.209439]   last_seq:                14508130
[211626.209440]   last_seq_ondisk:         14508129
[211626.209442]   flushed_seq_ondisk:      14508129
[211626.209443]   watermark:               stripe
[211626.209444]   each entry reserved:     321
[211626.209446]   nr flush writes:         106294
[211626.209447]   nr noflush writes:       8185
[211626.209448]   average write size:      89.8 KiB
[211626.209450]   nr direct reclaim:       0
[211626.209451]   nr background reclaim:   2414979
[211626.209452]   reclaim kicked:          0
[211626.209454]   reclaim runs in:         0 ms
[211626.209455]   blocked:                 0
[211626.209456]   current entry sectors:   2048
[211626.209458]   current entry error:     ok
[211626.209459]   current entry:           closed
[211626.209460]   unwritten entries:
[211626.209462]   last buf closed
[211626.209463]   space:
[211626.209464]     discarded              2048:16773120
[211626.209466]     clean ondisk           2048:16773120
[211626.209467]     clean                  2048:16773120
[211626.209469]     total                  2048:16777216
[211626.209470]   dev 0:
[211626.209471]     nr                     8192
[211626.209473]     bucket size            1024
[211626.209474]     available              8190:0
[211626.209475]     discard_idx            4087
[211626.209477]     dirty_ondisk           4087 (seq 14508045)
[211626.209478]     dirty_idx              4087 (seq 14508045)
[211626.209480]     cur_idx                4087 (seq 14508045)
[211626.209481]   dev 1:
[211626.209482]     nr                     8192
[211626.209483]     bucket size            1024
[211626.209485]     available              8190:0
[211626.209486]     discard_idx            3608
[211626.209488]     dirty_ondisk           3608 (seq 14508043)
[211626.209489]     dirty_idx              3608 (seq 14508043)
[211626.209490]     cur_idx                3608 (seq 14508043)
[211626.209492]   dev 2:
[211626.209493]     nr                     8192
[211626.209494]     bucket size            2048
[211626.209496]     available              8190:424
[211626.209497]     discard_idx            4685
[211626.209498]     dirty_ondisk           4685 (seq 14508129)
[211626.209500]     dirty_idx              4685 (seq 14508129)
[211626.209501]     cur_idx                4685 (seq 14508129)
[211626.209502]   dev 3:
[211626.209504]     nr                     8192
[211626.209505]     bucket size            1024
[211626.209506]     available              8190:1024
[211626.209508]     discard_idx            0
[211626.209509]     dirty_ondisk           0 (seq 0)
[211626.209510]     dirty_idx              0 (seq 0)
[211626.209512]     cur_idx                0 (seq 0)
[211626.209513]   dev 4:
[211626.209514]     nr                     8192
[211626.209515]     bucket size            2048
[211626.209517]     available              8190:504
[211626.209518]     discard_idx            2223
[211626.209520]     dirty_ondisk           2223 (seq 14508128)
[211626.209521]     dirty_idx              2223 (seq 14508128)
[211626.209522]     cur_idx                2223 (seq 14508128)
[211626.209524]   dev 6:
[211626.209525]     nr                     8192
[211626.209526]     bucket size            2048
[211626.209528]     available              8190:592
[211626.209529]     discard_idx            1636
[211626.209531]     dirty_ondisk           1636 (seq 14508129)
[211626.209532]     dirty_idx              1636 (seq 14508129)
[211626.209533]     cur_idx                1636 (seq 14508129)
[211626.209535]   dev 7:
[211626.209536]     nr                     8192
[211626.209537]     bucket size            2048
[211626.209539]     available              8190:0
[211626.209540]     discard_idx            593
[211626.209542]     dirty_ondisk           593 (seq 14507898)
[211626.209543]     dirty_idx              593 (seq 14507898)
[211626.209545]     cur_idx                593 (seq 14507898)

[211626.298375] Allocator stuck? Waited for 30 seconds
[211626.298380] Allocator debug:
[211626.298382]   capacity77280750838
[211626.298383]   reserved             6720065290
[211626.298385]   hidden               92328960
[211626.298386]   btree                404880384
[211626.298388]   data                 69549192929
[211626.298390]   cached               6582460531
[211626.298391]   reserved             245823744
[211626.298393]   online_reserved      36155632
[211626.298394]   nr_inodes            0
[211626.298395]   
[211626.298397]   freelist_wait        waiting
[211626.298398]   open buckets allocated96
[211626.298400]   open buckets total   1024
[211626.298401]   open_buckets_wait    empty
[211626.298402]   open_buckets_btree   4
[211626.298404]   open_buckets_user    91
[211626.298405]   btree reserve cache  1

[211626.298407] Dev 0:
[211626.298409]                      buckets         sectors      fragmented
[211626.298410]   free                178560               0               0
[211626.298412]   sb                       7            6152            1016
[211626.298413]   journal               8192         8388608               0
[211626.298415]   btree                 2159         2161664           49152
[211626.298416]   user              11257415     11527606528              72
[211626.298418]   cached                   0               0               0
[211626.298419]   parity                   0               0               0
[211626.298421]   stripe                   0               0               0
[211626.298423]   need_gc_gens             0               0               0
[211626.298424]   need_discard             0               0               0
[211626.298426]   unstriped                0               0               0
[211626.298427]   capacity          11446333
[211626.298429]   
[211626.298430]   reserves:
[211626.298431]   stripe              357724
[211626.298432]   normal              178876
[211626.298434]   copygc                  28
[211626.298435]   btree                   14
[211626.298436]   btree_copygc             0
[211626.298438]   reclaim                  0
[211626.298439]   interior_updates         0
[211626.298440]   
[211626.298441]   open buckets             1
[211626.298443]   buckets to invalidate    0

[211626.298445] Dev 1:
[211626.298446]                      buckets         sectors      fragmented
[211626.298448]   free                178546               0               0
[211626.298450]   sb                       7            6152            1016
[211626.298451]   journal               8192         8388608               0
[211626.298453]   btree                 2163         2168832           46080
[211626.298454]   user              11257425     11527616583             104
[211626.298456]   cached                   0               0               0
[211626.298457]   parity                   0               0               0
[211626.298459]   stripe                   0               0               0
[211626.298460]   need_gc_gens             0               0               0
[211626.298462]   need_discard             0               0               0
[211626.298463]   unstriped                0               0               0
[211626.298465]   capacity          11446333
[211626.298466]   
[211626.298467]   reserves:
[211626.298469]   stripe              357724
[211626.298470]   normal              178876
[211626.298471]   copygc                  28
[211626.298473]   btree                   14
[211626.298474]   btree_copygc             0
[211626.298475]   reclaim                  0
[211626.298476]   interior_updates         0
[211626.298477]   
[211626.298479]   open buckets             0
[211626.298480]   buckets to invalidate    0

[211626.298482] Dev 2:
[211626.298484]                      buckets         sectors      fragmented
[211626.298485]   free                 89127               0               0
[211626.298487]   sb                       4            6152            2040
[211626.298489]   journal               8192        16777216               0
[211626.298490]   btree                 8554        17350144          168448
[211626.298492]   user               5617289     11504227436              56
[211626.298493]   cached                   0               0               0
[211626.298495]   parity                   0               0               0
[211626.298496]   stripe                   0               0               0
[211626.298498]   need_gc_gens             0               0               0
[211626.298499]   need_discard             0               0               0
[211626.298501]   unstriped                0               0               0
[211626.298502]   capacity           5723166
[211626.298503]   
[211626.298505]   reserves:
[211626.298506]   stripe              178862
[211626.298507]   normal               89438
[211626.298508]   copygc                  14
[211626.298510]   btree                    7
[211626.298511]   btree_copygc             0
[211626.298512]   reclaim                  0
[211626.298513]   interior_updates         0
[211626.298515]   
[211626.298516]   open buckets             1
[211626.298517]   buckets to invalidate    0

[211626.298519] Dev 3:
[211626.298520]                      buckets         sectors      fragmented
[211626.298522]   free                 74547               0               0
[211626.298523]   sb                       7            6152            1016
[211626.298525]   journal               8192         8388608               0
[211626.298526]   btree                    0               0               0
[211626.298528]   user                     0               0               0
[211626.298529]   cached             1824993      1860332358         8460684
[211626.298531]   parity                   0               0               0
[211626.298532]   stripe                   0               0               0
[211626.298534]   need_gc_gens             0               0               0
[211626.298535]   need_discard             0               0               0
[211626.298537]   unstriped                0               0               0
[211626.298538]   capacity           1907739
[211626.298540]   
[211626.298541]   reserves:
[211626.298542]   stripe               59644
[211626.298543]   normal               29836
[211626.298545]   copygc                  28
[211626.298546]   btree                   14
[211626.298547]   btree_copygc             0
[211626.298548]   reclaim                  0
[211626.298550]   interior_updates         0
[211626.298551]   
[211626.298552]   open buckets            32
[211626.298553]   buckets to invalidate    1

[211626.298556] Dev 4:
[211626.298557]                      buckets         sectors      fragmented
[211626.298558]   free                119171               0               0
[211626.298560]   sb                       4            6152            2040
[211626.298561]   journal               8192        16777216               0
[211626.298563]   btree                50248       102650368          257536
[211626.298564]   user               7453269     15264308874           24984
[211626.298566]   cached                   0               0               0
[211626.298567]   parity                   0               0               0
[211626.298569]   stripe                   0               0               0
[211626.298570]   need_gc_gens             0               0               0
[211626.298572]   need_discard             1               0               0
[211626.298573]   unstriped                0               0               0
[211626.298575]   capacity           7630885
[211626.298576]   
[211626.298577]   reserves:
[211626.298578]   stripe              238478
[211626.298580]   normal              119246
[211626.298581]   copygc                  14
[211626.298582]   btree                    7
[211626.298583]   btree_copygc             0
[211626.298585]   reclaim                  0
[211626.298586]   interior_updates         0
[211626.298587]   
[211626.298588]   open buckets            12
[211626.298589]   buckets to invalidate    0

[211626.298592] Dev 6:
[211626.298593]                      buckets         sectors      fragmented
[211626.298594]   free                119156               0               0
[211626.298596]   sb                       4            6152            2040
[211626.298598]   journal               8192        16777216               0
[211626.298599]   btree                57500       117491200          268800
[211626.298601]   user               7446032     15249483790           25664
[211626.298602]   cached                   0               0               0
[211626.298604]   parity                   0               0               0
[211626.298605]   stripe                   0               0               0
[211626.298607]   need_gc_gens             0               0               0
[211626.298608]   need_discard             1               0               0
[211626.298610]   unstriped                0               0               0
[211626.298611]   capacity           7630885
[211626.298613]   
[211626.298614]   reserves:
[211626.298615]   stripe              238478
[211626.298617]   normal              119246
[211626.298618]   copygc                  14
[211626.298619]   btree                    7
[211626.298621]   btree_copygc             0
[211626.298622]   reclaim                  0
[211626.298623]   interior_updates         0
[211626.298625]   
[211626.298626]   open buckets            13
[211626.298627]   buckets to invalidate    0

[211626.298629] Dev 7:
[211626.298631]                      buckets         sectors      fragmented
[211626.298632]   free               2869734               0               0
[211626.298634]   sb                       4            6152            2040
[211626.298635]   journal               8192        16777216               0
[211626.298637]   btree                80041       163058176          865792
[211626.298638]   user               2185546      4475949718           51824
[211626.298640]   cached             2487368      4722084501       372045177
[211626.298641]   parity                   0               0               0
[211626.298643]   stripe                   0               0               0
[211626.298644]   need_gc_gens             0               0               0
[211626.298646]   need_discard             0               0               0
[211626.298648]   unstriped                0               0               0
[211626.298649]   capacity           7630885
[211626.298650]   
[211626.298651]   reserves:
[211626.298652]   stripe              238478
[211626.298654]   normal              119246
[211626.298655]   copygc                  14
[211626.298656]   btree                    7
[211626.298658]   btree_copygc             0
[211626.298659]   reclaim                  0
[211626.298660]   interior_updates         0
[211626.298661]   
[211626.298662]   open buckets            36
[211626.298664]   buckets to invalidate    0

[211626.298666] Copygc debug:
[211626.298667]   running: 0
[211626.298668]   copygc_wait:0
[211626.298670]   copygc_wait_at:0
[211626.298671]   Currently waiting for:0 B
[211626.298672]   Currently waiting since:38.7 TiB
[211626.298674]   Currently calculated wait:0 B

[211626.298676] Journal debug:
[211626.298678]   flags:                   replay_done,running,may_skip_flush
[211626.298679]   dirty journal entries:   0/32768
[211626.298681]   seq:                     14508129
[211626.298682]   seq_ondisk:              14508129
[211626.298684]   last_seq:                14508130
[211626.298685]   last_seq_ondisk:         14508129
[211626.298686]   flushed_seq_ondisk:      14508129
[211626.298688]   watermark:               stripe
[211626.298689]   each entry reserved:     321
[211626.298691]   nr flush writes:         106294
[211626.298692]   nr noflush writes:       8185
[211626.298693]   average write size:      89.8 KiB
[211626.298695]   nr direct reclaim:       0
[211626.298696]   nr background reclaim:   2414979
[211626.298697]   reclaim kicked:          0
[211626.298699]   reclaim runs in:         0 ms
[211626.298700]   blocked:                 0
[211626.298701]   current entry sectors:   2048
[211626.298703]   current entry error:     ok
[211626.298704]   current entry:           closed
[211626.298706]   unwritten entries:
[211626.298707]   last buf closed
[211626.298708]   space:
[211626.298710]     discarded              2048:16773120
[211626.298711]     clean ondisk           2048:16773120
[211626.298712]     clean                  2048:16773120
[211626.298714]     total                  2048:16777216
[211626.298715]   dev 0:
[211626.298716]     nr                     8192
[211626.298718]     bucket size            1024
[211626.298719]     available              8190:0
[211626.298720]     discard_idx            4087
[211626.298722]     dirty_ondisk           4087 (seq 14508045)
[211626.298723]     dirty_idx              4087 (seq 14508045)
[211626.298725]     cur_idx                4087 (seq 14508045)
[211626.298726]   dev 1:
[211626.298727]     nr                     8192
[211626.298729]     bucket size            1024
[211626.298730]     available              8190:0
[211626.298731]     discard_idx            3608
[211626.298733]     dirty_ondisk           3608 (seq 14508043)
[211626.298734]     dirty_idx              3608 (seq 14508043)
[211626.298735]     cur_idx                3608 (seq 14508043)
[211626.298737]   dev 2:
[211626.298738]     nr                     8192
[211626.298759]     bucket size            2048
[211626.298761]     available              8190:424
[211626.298762]     discard_idx            4685
[211626.298763]     dirty_ondisk           4685 (seq 14508129)
[211626.298765]     dirty_idx              4685 (seq 14508129)
[211626.298766]     cur_idx                4685 (seq 14508129)
[211626.298768]   dev 3:
[211626.298769]     nr                     8192
[211626.298770]     bucket size            1024
[211626.298772]     available              8190:1024
[211626.298773]     discard_idx            0
[211626.298774]     dirty_ondisk           0 (seq 0)
[211626.298776]     dirty_idx              0 (seq 0)
[211626.298777]     cur_idx                0 (seq 0)
[211626.298778]   dev 4:
[211626.298780]     nr                     8192
[211626.298781]     bucket size            2048
[211626.298782]     available              8190:504
[211626.298784]     discard_idx            2223
[211626.298785]     dirty_ondisk           2223 (seq 14508128)
[211626.298787]     dirty_idx              2223 (seq 14508128)
[211626.298788]     cur_idx                2223 (seq 14508128)
[211626.298790]   dev 6:
[211626.298791]     nr                     8192
[211626.298792]     bucket size            2048
[211626.298794]     available              8190:592
[211626.298795]     discard_idx            1636
[211626.298796]     dirty_ondisk           1636 (seq 14508129)
[211626.298798]     dirty_idx              1636 (seq 14508129)
[211626.298799]     cur_idx                1636 (seq 14508129)
[211626.298801]   dev 7:
[211626.298802]     nr                     8192
[211626.298803]     bucket size            2048
[211626.298804]     available              8190:0
[211626.298806]     discard_idx            593
[211626.298807]     dirty_ondisk           593 (seq 14507898)
[211626.298809]     dirty_idx              593 (seq 14507898)
[211626.298810]     cur_idx                593 (seq 14507898)
Filesystem: 32b2df91-1683-4288-9ebf-e60d104cc069
Size:                       36.0 TiB
Used:                       32.7 TiB
Online reserved:            17.2 GiB

Data type       Required/total  Durability    Devices
reserved:       1/2                [] 58.6 GiB
btree:          1/2             2             [sde sdf]           75.0 MiB
btree:          1/2             2             [sde sda]            214 MiB
btree:          1/2             2             [sde sdb]           75.0 MiB
btree:          1/2             2             [sde sdc]           31.0 MiB
btree:          1/2             2             [sde sdd]           1.68 GiB
btree:          1/2             2             [sdf sda]            209 MiB
btree:          1/2             2             [sdf sdb]           73.0 MiB
btree:          1/2             2             [sdf sdc]           34.0 MiB
btree:          1/2             2             [sdf sdd]           1.69 GiB
btree:          1/2             2             [sda sdb]           1.49 GiB
btree:          1/2             2             [sda sdc]           1.49 GiB
btree:          1/2             2             [sda sdd]           13.1 GiB
btree:          1/2             2             [sdb sdc]           33.9 GiB
btree:          1/2             2             [sdb sdd]           62.4 GiB
btree:          1/2             2             [sdc sdd]           76.6 GiB
user:           1/2             2             [sde sdf]           8.15 TiB
user:           1/2             2             [sde sda]           2.13 TiB
user:           1/2             2             [sde sdb]            203 GiB
user:           1/2             2             [sde sdc]            168 GiB
user:           1/2             2             [sde sdd]           93.8 GiB
user:           1/2             2             [sdf sda]           2.23 TiB
user:           1/2             2             [sdf sdb]            165 GiB
user:           1/2             2             [sdf sdc]            108 GiB
user:           1/2             2             [sdf sdd]           85.8 GiB
user:           1/2             2             [sda sdb]           3.10 TiB
user:           1/2             2             [sda sdc]           2.79 TiB
user:           1/2             2             [sda sdd]            474 GiB
user:           1/2             2             [sdb sdc]           9.19 TiB
user:           1/2             2             [sdb sdd]           1.57 TiB
user:           1/2             2             [sdc sdd]           1.96 TiB
cached:         1/1             0             [sdg]                887 GiB
cached:         1/1             1             [sdb]               2.66 MiB
cached:         1/1             1             [sdd]               2.20 TiB

Compression:
type              compressed    uncompressed     average extent size
zstd                 305 GiB         417 GiB                 134 KiB
incompressible      35.4 TiB        35.4 TiB                 115 KiB

Btree usage:
extents:            61.1 GiB
inodes:              130 MiB
dirents:            56.0 MiB
xattrs:              512 KiB
alloc:              12.3 GiB
reflink:            17.2 GiB
subvolumes:          512 KiB
snapshots:           512 KiB
lru:                 360 MiB
freespace:          9.50 MiB
need_discard:       1.00 MiB
backpointers:        102 GiB
bucket_gens:         210 MiB
snapshot_trees:      512 KiB
deleted_inodes:      512 KiB
logged_ops:         1.00 MiB
rebalance_work:      512 KiB
accounting:         5.00 MiB

Pending rebalance work:
144 KiB

hdd.hdd1 (device 0):             sde              rw
                                data         buckets    fragmented
  free:                     87.2 GiB          178560
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    1.03 GiB            2159      24.0 MiB
  user:                     5.37 TiB        11257415      36.0 KiB
  cached:                        0 B               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  unstriped:                     0 B               0
  capacity:                 5.46 TiB        11446333

hdd.hdd2 (device 1):             sdf              rw
                                data         buckets    fragmented
  free:                     87.2 GiB          178546
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    1.03 GiB            2163      22.5 MiB
  user:                     5.37 TiB        11257425      52.0 KiB
  cached:                        0 B               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  unstriped:                     0 B               0
  capacity:                 5.46 TiB        11446333

hdd.hdd3 (device 2):             sda              rw
                                data         buckets    fragmented
  free:                     87.0 GiB           89127
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                    8.27 GiB            8554      82.3 MiB
  user:                     5.36 TiB         5617289      28.0 KiB
  cached:                        0 B               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  unstriped:                     0 B               0
  capacity:                 5.46 TiB         5723166

hdd.hdd6 (device 6):             sdc              rw
                                data         buckets    fragmented
  free:                      116 GiB          119156
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                    56.0 GiB           57500       131 MiB
  user:                     7.10 TiB         7446032      12.5 MiB
  cached:                        0 B               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:             1.00 MiB               1
  unstriped:                     0 B               0
  capacity:                 7.28 TiB         7630885

hdd.hdd7 (device 4):             sdb              rw
                                data         buckets    fragmented
  free:                      116 GiB          119171
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                    48.9 GiB           50248       126 MiB
  user:                     7.11 TiB         7453269      12.2 MiB
  cached:                        0 B               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:             1.00 MiB               1
  unstriped:                     0 B               0
  capacity:                 7.28 TiB         7630885
  
hdd.hdd8 (device 7):             sdd              rw
                                data         buckets    fragmented
  free:                     2.74 TiB         2869734
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                    77.8 GiB           80041       423 MiB
  user:                     2.08 TiB         2185546      25.3 MiB
  cached:                   2.20 TiB         2487368       177 GiB
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  unstriped:                     0 B               0
  capacity:                 7.28 TiB         7630885

ssd.ssd1 (device 3):             sdg              rw
                                data         buckets    fragmented
  free:                     36.4 GiB           74547
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                          0 B               0
  cached:                    887 GiB         1824993      4.03 GiB
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  unstriped:                     0 B               0
  capacity:                  932 GiB         1907739

@koverstreet
Copy link
Owner

Can you check the contents of the freespace btree, via debugfs?

@gkmcd
Copy link

gkmcd commented Oct 17, 2024

Let me know if this is not what you need:

[root@ganymede ####]# ls -al /sys/kernel/debug/bcachefs/32b2df91-1683-4288-9ebf-e60d1
04cc069/btrees/freespace/
total 0
drwxr-xr-x  2 root root 0 Oct 15 18:38 .
drwxr-xr-x 23 root root 0 Oct 15 18:38 ..
-r--------  1 root root 0 Oct 15 18:38 bfloat-failed
-r--------  1 root root 0 Oct 15 18:38 formats
-r--------  1 root root 0 Oct 15 18:38 keys
[root@ganymede ####]# cat /sys/kernel/debug/bcachefs/32b2df91-1683-4288-9ebf-e60d104c
c069/btrees/freespace/bfloat-failed

l 0 POS_MIN - 0:10278384:0:
    ptrs: seq 20773ba5a5e6b274 written 296 min_key POS_MIN durability: 2 ptr: 7:78491
1:1024 gen 13 ptr: 4:3545057:0 gen 3
    format: u64s 1 fields 0:0, 24:0, 0:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 126000/261984 (48% full)
    sib u64s: 15591, 15591 (merge threshold 10916)
    nr packed keys 15750
    nr unpacked keys 0
    floats 491
    failed unpacked 0
l 0 0:10278384:1 - 0:11231622:0:
    ptrs: seq 9d525ef20db01156 written 240 min_key 0:10278384:1 durability: 2 ptr: 7:
3002606:0 gen 0 ptr: 4:772921:0 gen 2
    format: u64s 1 fields 0:0, 24:10278384, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 109872/261984 (41% full)
    sib u64s: 13734, 13734 (merge threshold 10916)
    nr packed keys 13734
    nr unpacked keys 0
    floats 428
    failed unpacked 0
l 0 0:11231622:1 - 1:7385314:0:
    ptrs: seq 36fa97be327797b5 written 280 min_key 0:11231622:1 durability: 2 ptr: 7:
782558:1024 gen 10 ptr: 4:3539422:1024 gen 3
    format: u64s 2 fields 8:0, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 121344/261984 (46% full)
    sib u64s: 14506, 14506 (merge threshold 10916)
    nr packed keys 7584
    nr unpacked keys 0
    floats 473
    failed unpacked 0
l 0 1:7385314:1 - 1:8433105:0:
    ptrs: seq a464c06168acef97 written 120 min_key 1:7385314:1 durability: 2 ptr: 7:2
993551:0 gen 2 ptr: 6:1486767:0 gen 13
    format: u64s 1 fields 0:1, 24:7385314, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 39472/261984 (15% full)
    sib u64s: 4934, 4934 (merge threshold 10916)
    nr packed keys 4934
    nr unpacked keys 0
    floats 153
    failed unpacked 0
l 0 1:8433105:1 - 1:10823583:0:
    ptrs: seq 668a750ff97e78fe written 112 min_key 1:8433105:1 durability: 2 ptr: 7:2
59310:1024 gen 9 ptr: 4:61626:1024 gen 9
    format: u64s 1 fields 0:1, 24:8433105, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 57040/261984 (21% full)
    sib u64s: 7130, 7130 (merge threshold 10916)
    nr packed keys 7130
    nr unpacked keys 0
    floats 221
    failed unpacked 0
l 0 1:10823583:1 - 1:11149353:0:
    ptrs: seq 72e9786bf5743bda written 200 min_key 1:10823583:1 durability: 2 ptr: 7:
613864:512 gen 2 ptr: 4:2462393:512 gen 3
    format: u64s 1 fields 0:1, 24:10823583, 1:0, 11:0, 0:0, 0:0    unpack fn len: 0
bytes used 98240/261984 (37% full)
    sib u64s: 12280, 12280 (merge threshold 10916)
    nr packed keys 12280
    nr unpacked keys 0
    floats 382
    failed unpacked 0
l 0 1:11149353:1 - 2:889910:0:
    ptrs: seq 3f7aef87a7839a99 written 296 min_key 1:11149353:1 durability: 2 ptr: 7:
772608:512 gen 12 ptr: 6:3083433:512 gen 2
    format: u64s 2 fields 8:1, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 94624/261984 (36% full)
    sib u64s: 11268, 11268 (merge threshold 10916)
    nr packed keys 5914
    nr unpacked keys 0
    floats 368
    failed unpacked 0
l 0 2:889910:1 - 2:2700756:0:
    ptrs: seq b6549b39fc87d733 written 96 min_key 2:889910:1 durability: 2 ptr: 7:748
658:1536 gen 11 ptr: 6:3076326:512 gen 4
    format: u64s 1 fields 0:2, 24:889910, 1:0, 9:0, 0:0, 0:0    unpack fn len: 0
    bytes used 20632/261984 (7% full)
    sib u64s: 14985, 16914 (merge threshold 10916)
    nr packed keys 2579
    nr unpacked keys 0
    floats 79
    failed unpacked 0
l 0 2:2700756:1 - 2:5393910:0:
    ptrs: seq dbaaafc0960bb6c7 written 296 min_key 2:2700756:1 durability: 2 ptr: 7:8
61817:512 gen 3 ptr: 0:1366967:512 gen 24
    format: u64s 1 fields 0:2, 24:2700756, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 146640/261984 (55% full)
    sib u64s: 18320, 18320 (merge threshold 10916)
    nr packed keys 18330
    nr unpacked keys 0
    floats 571
    failed unpacked 0
l 0 2:5393910:1 - 2:72057594040566946:0:
    ptrs: seq 10bd91317c854fc9 written 288 min_key 2:5393910:1 durability: 2 ptr: 7:2
44150:0 gen 13 ptr: 6:56500:0 gen 16
    format: u64s 2 fields 0:2, 56:5393910, 8:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 104656/261984 (39% full)
    sib u64s: 13080, 13080 (merge threshold 10916)
    nr packed keys 6541
    nr unpacked keys 0
    floats 407
    failed unpacked 0
l 0 2:72057594040566946:1 - 4:2396724:0:
    ptrs: seq f8d0c6c4ab8d25f0 written 200 min_key 2:72057594040566946:1 durability:
2 ptr: 7:863385:1536 gen 6 ptr: 6:3205166:512 gen 2
    format: u64s 2 fields 8:2, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 82896/261984 (31% full)
 sib u64s: 13412, 11214 (merge threshold 10916)
    nr packed keys 5181
    nr unpacked keys 0
    floats 322
    failed unpacked 0
l 0 4:2396724:1 - 4:3741594:0:
    ptrs: seq 51bde2f34119d28e written 136 min_key 4:2396724:1 durability: 2 ptr: 7:8
65919:0 gen 3 ptr: 2:2687830:0 gen 17
    format: u64s 1 fields 0:4, 24:2396724, 1:0, 15:0, 0:0, 0:0    unpack fn len: 0
    bytes used 43264/261984 (16% full)
    sib u64s: 16359, 18087 (merge threshold 10916)
    nr packed keys 5408
    nr unpacked keys 0
    floats 168
    failed unpacked 0
l 0 4:3741594:1 - 4:5569109:0:
    ptrs: seq fc9d5df197b2a64c written 320 min_key 4:3741594:1 durability: 2 ptr: 7:1
78821:1024 gen 4 ptr: 4:1929477:1024 gen 24
    format: u64s 2 fields 0:4, 24:3741594, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 110480/261984 (42% full)
    sib u64s: 13802, 13802 (merge threshold 10916)
    nr packed keys 6905
    nr unpacked keys 0
    floats 430
    failed unpacked 0
l 0 4:5569109:1 - 6:3190557:0:
    ptrs: seq 1624c0b22505d959 written 88 min_key 4:5569109:1 durability: 2 ptr: 7:33
9449:1536 gen 23 ptr: 4:3524179:512 gen 3
    format: u64s 2 fields 8:4, 64:0, 8:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 26048/261984 (9% full)
    sib u64s: 12836, 19944 (merge threshold 10916)
    nr packed keys 1628
    nr unpacked keys 0
    floats 100
    failed unpacked 0
l 0 6:3190557:1 - 6:3514378:0:
    ptrs: seq f82831eff32a0c3a written 256 min_key 6:3190557:1 durability: 2 ptr: 6:3
083467:1536 gen 5 ptr: 7:218271:512 gen 17
    format: u64s 1 fields 0:6, 19:3190557, 1:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 111824/261984 (42% full)
    sib u64s: 13632, 13632 (merge threshold 10916)
    nr packed keys 13978
    nr unpacked keys 0
    floats 435
    failed unpacked 0
l 0 6:3514378:1 - 7:893855:0:
    ptrs: seq aa30e637cef1788d written 168 min_key 6:3514378:1 durability: 2 ptr: 7:2
49608:0 gen 20 ptr: 0:1367232:0 gen 16
    format: u64s 2 fields 8:6, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 47968/261984 (18% full)
    sib u64s: 23073, 14845 (merge threshold 10916)
    nr packed keys 2998
    nr unpacked keys 0
    floats 186
    failed unpacked 0
l 0 7:893855:1 - 7:72057594038265451:0:
    ptrs: seq f7ec093b638bb54f written 208 min_key 7:893855:1 durability: 2 ptr: 7:85
9239:0 gen 9 ptr: 4:3544723:0 gen 2
    format: u64s 2 fields 0:7, 56:893855, 8:0, 32:0, 0:0, 0:0    unpack fn len: 0
    bytes used 92016/261984 (35% full)
    sib u64s: 15849, 18538 (merge threshold 10916)
    nr packed keys 5751
    nr unpacked keys 0
    floats 358
    failed unpacked 0
l 0 7:72057594038265451:1 - SPOS_MAX:
    ptrs: seq 45798adc8678cbb2 written 72 min_key 7:72057594038265451:1 durability: 2
 ptr: 7:734981:1024 gen 23 ptr: 6:2881433:1024 gen 2
    format: u64s 3 fields 64:0, 64:0, 32:0, 0:0, 0:0, 0:0    unpack fn len: 0
    bytes used 5200/261984 (1% full)
    sib u64s: 18351, 65535 (merge threshold 10916)
    nr packed keys 0
    nr unpacked keys 130
    floats 19
    failed unpacked 19
    failed unpacked at depth 4
        7:72057594038523189:0
    failed unpacked at depth 3
        7:72057594038649256:0
    failed unpacked at depth 4
        7:72057594038664626:0
    failed unpacked at depth 2
        7:72057594038674280:0
    failed unpacked at depth 4
        7:72057594038677150:0
    failed unpacked at depth 3
        7:72057594038677973:0
    failed unpacked at depth 4
        7:72057594038694402:0
    failed unpacked at depth 1
        7:72057594038700009:0
    failed unpacked at depth 3
        7:72057594038701152:0
    failed unpacked at depth 2
        7:72057594038703253:0
    failed unpacked at depth 3
        7:72057594038705845:0
    failed unpacked at depth 0
        7:72057594038707566:0
    failed unpacked at depth 3
        7:72057594038708053:0
    failed unpacked at depth 2
        7:72057594038708412:0
    failed unpacked at depth 3
        7:72057594038709016:0
    failed unpacked at depth 1
        7:72057594038709817:0
    failed unpacked at depth 3
        7:72057594038710743:0
    failed unpacked at depth 2
        7:72057594038713357:0
    failed unpacked at depth 3
        7:72057594038714130:0

[root@ganymede ####]# cat /sys/kernel/debug/bcachefs/32b2df91-1683-4288-9ebf-e60d104c
c069/btrees/freespace/formats

l 0 POS_MIN - 0:10278384:0:
    ptrs: seq 20773ba5a5e6b274 written 296 min_key POS_MIN durability: 2 ptr: 7:78491
1:1024 gen 13 ptr: 4:3545057:0 gen 3
    format: u64s 1 fields 0:0, 24:0, 0:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 126000/261984 (48% full)
    sib u64s: 15591, 15591 (merge threshold 10916)
    nr packed keys 15750
    nr unpacked keys 0
    floats 491
    failed unpacked 0
l 0 0:10278384:1 - 0:11231622:0:
    ptrs: seq 9d525ef20db01156 written 240 min_key 0:10278384:1 durability: 2 ptr: 7:
3002606:0 gen 0 ptr: 4:772921:0 gen 2
    format: u64s 1 fields 0:0, 24:10278384, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 109872/261984 (41% full)
    sib u64s: 13734, 13734 (merge threshold 10916)
    nr packed keys 13734
    nr unpacked keys 0
    floats 428
    failed unpacked 0
l 0 0:11231622:1 - 1:7385314:0:
    ptrs: seq 36fa97be327797b5 written 280 min_key 0:11231622:1 durability: 2 ptr: 7:
782558:1024 gen 10 ptr: 4:3539422:1024 gen 3
    format: u64s 2 fields 8:0, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 121344/261984 (46% full)
    sib u64s: 14506, 14506 (merge threshold 10916)
    nr packed keys 7584
    nr unpacked keys 0
    floats 473
    failed unpacked 0
l 0 1:7385314:1 - 1:8433105:0:
    ptrs: seq a464c06168acef97 written 120 min_key 1:7385314:1 durability: 2 ptr: 7:2
993551:0 gen 2 ptr: 6:1486767:0 gen 13
    format: u64s 1 fields 0:1, 24:7385314, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 39472/261984 (15% full)
    sib u64s: 4934, 4934 (merge threshold 10916)
    nr packed keys 4934
    nr unpacked keys 0
    floats 153
    failed unpacked 0
l 0 1:8433105:1 - 1:10823583:0:
    ptrs: seq 668a750ff97e78fe written 112 min_key 1:8433105:1 durability: 2 ptr: 7:2
59310:1024 gen 9 ptr: 4:61626:1024 gen 9
    format: u64s 1 fields 0:1, 24:8433105, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 57040/261984 (21% full)
    sib u64s: 7130, 7130 (merge threshold 10916)
    nr packed keys 7130
    nr unpacked keys 0
    floats 221
    failed unpacked 0
l 0 1:10823583:1 - 1:11149353:0:
    ptrs: seq 72e9786bf5743bda written 200 min_key 1:10823583:1 durability: 2 ptr: 7:
613864:512 gen 2 ptr: 4:2462393:512 gen 3
    format: u64s 1 fields 0:1, 24:10823583, 1:0, 11:0, 0:0, 0:0    unpack fn len: 0
    bytes used 98240/261984 (37% full)
    sib u64s: 12280, 12280 (merge threshold 10916)
    nr packed keys 12280
    nr unpacked keys 0
    floats 382
    failed unpacked 0
l 0 1:11149353:1 - 2:889910:0:
    ptrs: seq 3f7aef87a7839a99 written 296 min_key 1:11149353:1 durability: 2 ptr: 7:
772608:512 gen 12 ptr: 6:3083433:512 gen 2
    format: u64s 2 fields 8:1, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 94624/261984 (36% full)
    sib u64s: 11268, 11268 (merge threshold 10916)
    nr packed keys 5914
    nr unpacked keys 0
    floats 368
    failed unpacked 0
l 0 2:889910:1 - 2:2700756:0:
    ptrs: seq b6549b39fc87d733 written 96 min_key 2:889910:1 durability: 2 ptr: 7:748
658:1536 gen 11 ptr: 6:3076326:512 gen 4
    format: u64s 1 fields 0:2, 24:889910, 1:0, 9:0, 0:0, 0:0    unpack fn len: 0
    bytes used 20632/261984 (7% full)
    sib u64s: 14985, 16914 (merge threshold 10916)
    nr packed keys 2579
    nr unpacked keys 0
    floats 79
    failed unpacked 0
l 0 2:2700756:1 - 2:5393910:0:
    ptrs: seq dbaaafc0960bb6c7 written 296 min_key 2:2700756:1 durability: 2 ptr: 7:8
61817:512 gen 3 ptr: 0:1366967:512 gen 24
    format: u64s 1 fields 0:2, 24:2700756, 1:0, 12:0, 0:0, 0:0    unpack fn len: 0
    bytes used 146640/261984 (55% full)
    sib u64s: 18320, 18320 (merge threshold 10916)
    nr packed keys 18330
    nr unpacked keys 0
    floats 571
    failed unpacked 0
l 0 2:5393910:1 - 2:72057594040566946:0:
    ptrs: seq 10bd91317c854fc9 written 288 min_key 2:5393910:1 durability: 2 ptr: 7:2
44150:0 gen 13 ptr: 6:56500:0 gen 16
    format: u64s 2 fields 0:2, 56:5393910, 8:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 104656/261984 (39% full)
    sib u64s: 13080, 13080 (merge threshold 10916)
    nr packed keys 6541
    nr unpacked keys 0
    floats 407
    failed unpacked 0
l 0 2:72057594040566946:1 - 4:2396724:0:
    ptrs: seq f8d0c6c4ab8d25f0 written 200 min_key 2:72057594040566946:1 durability:
2 ptr: 7:863385:1536 gen 6 ptr: 6:3205166:512 gen 2
    format: u64s 2 fields 8:2, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 82896/261984 (31% full)
    sib u64s: 13412, 11214 (merge threshold 10916)
    nr packed keys 5181
    nr unpacked keys 0
    floats 322
    failed unpacked 0
l 0 4:2396724:1 - 4:3741594:0:
    ptrs: seq 51bde2f34119d28e written 136 min_key 4:2396724:1 durability: 2 ptr: 7:8
65919:0 gen 3 ptr: 2:2687830:0 gen 17
    format: u64s 1 fields 0:4, 24:2396724, 1:0, 15:0, 0:0, 0:0    unpack fn len: 0
    bytes used 43264/261984 (16% full)
    sib u64s: 16359, 18087 (merge threshold 10916)
    nr packed keys 5408
    nr unpacked keys 0
    floats 168
    failed unpacked 0
l 0 4:3741594:1 - 4:5569109:0:
    ptrs: seq fc9d5df197b2a64c written 320 min_key 4:3741594:1 durability: 2 ptr: 7:1
78821:1024 gen 4 ptr: 4:1929477:1024 gen 24
    format: u64s 2 fields 0:4, 24:3741594, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 110480/261984 (42% full)
    sib u64s: 13802, 13802 (merge threshold 10916)
    nr packed keys 6905
    nr unpacked keys 0
    floats 430
    failed unpacked 0
l 0 4:5569109:1 - 6:3190557:0:
    ptrs: seq 1624c0b22505d959 written 88 min_key 4:5569109:1 durability: 2 ptr: 7:33
9449:1536 gen 23 ptr: 4:3524179:512 gen 3
    format: u64s 2 fields 8:4, 64:0, 8:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 26048/261984 (9% full)
    sib u64s: 12836, 19944 (merge threshold 10916)
    nr packed keys 1628
    nr unpacked keys 0
    floats 100
    failed unpacked 0
l 0 6:3190557:1 - 6:3514378:0:
    ptrs: seq f82831eff32a0c3a written 256 min_key 6:3190557:1 durability: 2 ptr: 6:3
083467:1536 gen 5 ptr: 7:218271:512 gen 17
    format: u64s 1 fields 0:6, 19:3190557, 1:0, 16:0, 0:0, 0:0    unpack fn len: 0
    bytes used 111824/261984 (42% full)
    sib u64s: 13632, 13632 (merge threshold 10916)
    nr packed keys 13978
    nr unpacked keys 0
    floats 435
    failed unpacked 0
l 0 6:3514378:1 - 7:893855:0:
    ptrs: seq aa30e637cef1788d written 168 min_key 6:3514378:1 durability: 2 ptr: 7:2
49608:0 gen 20 ptr: 0:1367232:0 gen 16
    format: u64s 2 fields 8:6, 64:0, 8:0, 24:0, 0:0, 0:0    unpack fn len: 0
    bytes used 47968/261984 (18% full)
    sib u64s: 23073, 14845 (merge threshold 10916)
    nr packed keys 2998
    nr unpacked keys 0
    floats 186
    failed unpacked 0
l 0 7:893855:1 - 7:72057594038265451:0:
    ptrs: seq f7ec093b638bb54f written 208 min_key 7:893855:1 durability: 2 ptr: 7:85
9239:0 gen 9 ptr: 4:3544723:0 gen 2
    format: u64s 2 fields 0:7, 56:893855, 8:0, 32:0, 0:0, 0:0    unpack fn len: 0
    bytes used 92016/261984 (35% full)
    sib u64s: 15849, 18538 (merge threshold 10916)
    nr packed keys 5751
    nr unpacked keys 0
    floats 358
    failed unpacked 0
l 0 7:72057594038265451:1 - SPOS_MAX:
    ptrs: seq 45798adc8678cbb2 written 72 min_key 7:72057594038265451:1 durability: 2
 ptr: 7:734981:1024 gen 23 ptr: 6:2881433:1024 gen 2
    format: u64s 3 fields 64:0, 64:0, 32:0, 0:0, 0:0, 0:0    unpack fn len: 0
    bytes used 5200/261984 (1% full)
    sib u64s: 18351, 65535 (merge threshold 10916)
    nr packed keys 0
    nr unpacked keys 130
    floats 19
    failed unpacked 19
l 1 POS_MIN - SPOS_MAX:
    ptrs: seq f9fddba3c79cdb written 128 min_key POS_MIN durability: 2 ptr: 7:858890:
512 gen 5 ptr: 2:2687884:512 gen 18
    format: u64s 3 fields 64:0, 64:0, 32:0, 0:0, 0:0, 0:0    unpack fn len: 0
    bytes used 1440/261984 (0% full)
    sib u64s: 180, 180 (merge threshold 10916)
    nr packed keys 18
    nr unpacked keys 0
    floats 4
    failed unpacked 0

@koverstreet
Copy link
Owner

koverstreet commented Oct 17, 2024 via email

@gkmcd
Copy link

gkmcd commented Oct 17, 2024

keys.txt

@gkmcd
Copy link

gkmcd commented Oct 19, 2024

Tried to unmount, umount again reported device busy, but umount -l did the trick. Tried to remount with -o fsck,fix_errors, but this produced no discernible activity at all (no output in dmesg or drive activity). On shutdown, systemd complained about a bunch of stuck processes related to disk IO. Rebooting triggered the unclean shutdown process, which completed successfully (no ERESTARTSYS this time). Things seemed normal for a while, but after a period of uptime dmesg is again full of the above.

@koverstreet
Copy link
Owner

Ok, I looked at the output more closely: copygc should be running, but it's not.

I have a fix that adds a direct wakeup to copygc when we aren't able to allocate, can you try my bcachefs-testing branch?

@gkmcd
Copy link

gkmcd commented Oct 21, 2024

OK, awesome. I've done a lot of yak-shaving with not much spare time but still haven't managed to boot the testing kernel yet.

@gkmcd
Copy link

gkmcd commented Oct 21, 2024

With bcachefs-testing, mounting with -o fsck,fixerrors

[   59.010244] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): starting version 1.12: rebalance_work_acct_fix opts=metadata_replicas=2,data_replicas=2,compression=zstd,foreground_target=ssd,background_target=hdd,promote_target=ssd,nopromote_whole_extents,fsck,fix_errors=yes
[   59.010251] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): recovering from unclean shutdown
[   59.010256] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): Doing compatible version upgrade from 1.12: rebalance_work_acct_fix to 1.13: inode_has_child_snapshots
                 running recovery passes: check_inodes
[  104.429973] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): journal read done, replaying entries 14591810-14591810
[  104.845556] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): accounting_read...
[  104.866946] invalid bkey u64s 6 type accounting 0:0:774 len 0 ver 0: btree btree=xattrs 1024
[  104.866950]   accounting key with version=0: delete?, fixing
[  104.867271] invalid bkey u64s 6 type accounting 0:0:2054 len 0 ver 0: btree btree=subvolumes 1024
[  104.867274]   accounting key with version=0: delete?, fixing
[  104.867442] invalid bkey u64s 6 type accounting 0:0:2310 len 0 ver 0: btree btree=snapshots 1024
[  104.867444]   accounting key with version=0: delete?, fixing
[  104.867611] invalid bkey u64s 6 type accounting 0:0:3846 len 0 ver 0: btree btree=snapshot_trees 1024
[  104.867613]   accounting key with version=0: delete?, fixing
[  104.867782] invalid bkey u64s 6 type accounting 0:0:4614 len 0 ver 0: btree btree=rebalance_work 1024
[  104.867784]   accounting key with version=0: delete?, fixing
[  104.867956] invalid bkey u64s 8 type accounting 0:0:65539 len 0 ver 0: dev_data_type dev=0 data_type=sb 7 6152 1016
[  104.867958]   accounting key with version=0: delete?, fixing
[  104.868139] invalid bkey u64s 8 type accounting 0:0:65795 len 0 ver 0: dev_data_type dev=1 data_type=sb 7 6152 1016
[  104.868141]   accounting key with version=0: delete?, fixing
[  104.868323] invalid bkey u64s 8 type accounting 0:0:66051 len 0 ver 0: dev_data_type dev=2 data_type=sb 4 6152 2040
[  104.868324]   accounting key with version=0: delete?, fixing
[  104.868505] invalid bkey u64s 8 type accounting 0:0:66307 len 0 ver 0: dev_data_type dev=3 data_type=sb 7 6152 1016
[  104.868507]   accounting key with version=0: delete?, fixing
[  104.868788] invalid bkey u64s 8 type accounting 0:0:66563 len 0 ver 0: dev_data_type dev=4 data_type=sb 4 6152 2040
[  104.868790]   accounting key with version=0: delete?, fixing
[  104.869091] invalid bkey u64s 8 type accounting 0:0:67075 len 0 ver 0: dev_data_type dev=6 data_type=sb 4 6152 2040
[  104.869093]   accounting key with version=0: delete?, fixing
[  104.869278] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): Ratelimiting new instances of previous error
[  104.922399]  done
[  104.922406] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): alloc_read... done
[  106.627489] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): stripes_read... done
[  106.627502] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): snapshots_read... done
[  106.627514] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_allocations...
[ 1891.284576] accounting mismatch for btree btree=xattrs: got 0 should be 1024, fixing
[ 1891.284781] accounting mismatch for compression zstd: got 3274757 876768560 639835816 should be 3274227 862892808 631802328, fixing
[ 1891.284855] accounting mismatch for compression incompressible: got 329607065 76128010616 76128010616 should be 329606178 75589504064 75589504064, fixing
[ 1891.284938] accounting mismatch for btree btree=subvolumes: got 0 should be 1024, fixing
[ 1891.284988] accounting mismatch for btree btree=snapshots: got 0 should be 1024, fixing
[ 1891.285040] accounting mismatch for btree btree=snapshot_trees: got 0 should be 1024, fixing
[ 1891.285092] accounting mismatch for btree btree=rebalance_work: got 0 should be 1024, fixing
[ 1891.285147] accounting mismatch for dev_data_type dev=0 data_type=sb: got 0 0 0 should be 7 6152 1016, fixing
[ 1891.285207] accounting mismatch for dev_data_type dev=1 data_type=sb: got 0 0 0 should be 7 6152 1016, fixing
[ 1891.285268] accounting mismatch for dev_data_type dev=2 data_type=sb: got 0 0 0 should be 4 6152 2040, fixing
[ 1891.285328] accounting mismatch for dev_data_type dev=3 data_type=sb: got 0 0 0 should be 7 6152 1016, fixing
[ 1891.285381] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): Ratelimiting new instances of previous error
[ 2008.877819]  done
[ 2008.985483] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): going read-write
[ 2009.009621] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): journal_replay... done
[ 2009.213511] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_alloc_info... done
[ 2050.013288] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_lrus... done

and then later

[16643.069278] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 48%, done 156598/320952 nodes, at reflink:0:7386981912:0
[16653.069233] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 49%, done 159615/320952 nodes, at reflink:0:8052291040:0
[16658.807443]  done
[16658.807448] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_alloc_to_lru_refs... done
[16774.236175] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_snapshot_trees... done
[16774.236438] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_snapshots... done
[16774.236664] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_subvols... done
[16774.244960] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_subvol_children... done
[16774.244965] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): delete_dead_snapshots... done
[16774.244967] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_inodes... done
[16776.826289] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_extents... done
[16975.317709] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_indirect_extents... done
[17079.724673] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_dirents... done
[17080.086817] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_xattrs... done
[17080.087284] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_root... done
[17080.087450] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_unreachable_inodes... done
[17080.185157] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_subvolume_structure... done
[17080.185164] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_directory_structure... done
[17080.374203] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_nlinks... done
[17080.603806] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): resume_logged_ops... done
[17080.616734] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): delete_dead_inodes... done
[17080.887369] invalid bkey u64s 14 type reflink_v 0:6881236752:0 len 16 ver 0: refcount: 2 durability: 2 crc: c_size 120 size 128 offset 112 nonce 0 csum crc32c 0:355a6fca  compress zstd ptr: 4:6247691:1336 gen 0 crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c 0:664da0b4  compress none ptr: 3:255480:0 gen 28 d=0 cached rebalance: target hdd compression zstd crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c 0:664da0b4  compress incompressible ptr: 7:4785695:0 gen 0 (invalid extent entry 0000000000000000)
[17080.887374]   invalid extent entry type (got 6, shutting down
[17080.887828] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): inconsistency detected - emergency read only at journal seq 14591831
[17080.887921] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): invalid bkey on insert from bch2_moving_ctxt_init -> get_update_rebalance_opts [bcachefs]

[17080.888441] 4 transaction updates for bch2_moving_ctxt_init journal seq 14591831
[17080.888442]   update: btree=alloc cached=1 bch2_trans_start_alloc_update+0x154/0x2e0 [bcachefs]
[17080.888442]     old u64s 13 type alloc_v4 3:255480:0 len 0 ver 0: 
[17080.888443]     gen 28 oldest_gen 13 data_type cached
[17080.888444]     journal_seq       14097560
[17080.888444]     need_discard      1
[17080.888445]     need_inc_gen      1
[17080.888445]     dirty_sectors     0
[17080.888445]     stripe_sectors    0
[17080.888446]     cached_sectors    1024
[17080.888446]     stripe            0
[17080.888447]     stripe_redundancy 0
[17080.888447]     io_time[READ]     60381996880
[17080.888448]     io_time[WRITE]    81741345976
[17080.888448]     fragmentation     0
[17080.888449]     bp_start          8
[17080.888449]   
[17080.888450]     new u64s 13 type alloc_v4 3:255480:0 len 0 ver 0: 
[17080.888450]     gen 28 oldest_gen 13 data_type cached
[17080.888451]     journal_seq       14097560
[17080.888451]     need_discard      1
[17080.888452]     need_inc_gen      1
[17080.888452]     dirty_sectors     0
[17080.888452]     stripe_sectors    0
[17080.888453]     cached_sectors    1024
[17080.888453]     stripe            0
[17080.888454]     stripe_redundancy 0
[17080.888454]     io_time[READ]     60381996880
[17080.888454]     io_time[WRITE]    81741345976
[17080.888455]     fragmentation     0
[17080.888455]     bp_start          8
[17080.888456]   
[17080.888456]   update: btree=alloc cached=1 bch2_trans_start_alloc_update+0x154/0x2e0 [bcachefs]
[17080.888457]     old u64s 13 type alloc_v4 4:6247691:0 len 0 ver 0: 
[17080.888457]     gen 0 oldest_gen 0 data_type user
[17080.888458]     journal_seq       11085573
[17080.888458]     need_discard      1
[17080.888459]     need_inc_gen      1
[17080.888459]     dirty_sectors     2049
[17080.888460]     stripe_sectors    0
[17080.888460]     cached_sectors    0
[17080.888460]     stripe            0
[17080.888461]     stripe_redundancy 0
[17080.888461]     io_time[READ]     39587651312
[17080.888462]     io_time[WRITE]    62450769440
[17080.888462]     fragmentation     0
[17080.888462]     bp_start          8
[17080.888463]   
[17080.888463]     new u64s 13 type alloc_v4 4:6247691:0 len 0 ver 0: 
[17080.888464]     gen 0 oldest_gen 0 data_type user
[17080.888464]     journal_seq       11085573
[17080.888465]     need_discard      1
[17080.888465]     need_inc_gen      1
[17080.888465]     dirty_sectors     2049
[17080.888466]     stripe_sectors    0
[17080.888466]     cached_sectors    0
[17080.888467]     stripe            0
[17080.888467]     stripe_redundancy 0
[17080.888468]     io_time[READ]     39587651312
[17080.888468]     io_time[WRITE]    62450769440
[17080.888468]     fragmentation     0
[17080.888469]     bp_start          8
[17080.888469]   
[17080.888470]   update: btree=alloc cached=1 bch2_trans_start_alloc_update+0x154/0x2e0 [bcachefs]
[17080.888470]     old u64s 13 type alloc_v4 7:4785695:0 len 0 ver 0: 
[17080.888471]     gen 0 oldest_gen 0 data_type user
[17080.888471]     journal_seq       14510169
[17080.888471]     need_discard      1
[17080.888472]     need_inc_gen      1
[17080.888472]     dirty_sectors     16
[17080.888473]     stripe_sectors    0
[17080.888473]     cached_sectors    0
[17080.888473]     stripe            0
[17080.888474]     stripe_redundancy 0
[17080.888474]     io_time[READ]     61169962240
[17080.888475]     io_time[WRITE]    83046136272
[17080.888475]     fragmentation     16777216
[17080.888476]     bp_start          8
[17080.888476]   
[17080.888476]     new u64s 13 type alloc_v4 7:4785695:0 len 0 ver 0: 
[17080.888477]     gen 0 oldest_gen 0 data_type user
[17080.888477]     journal_seq       14510169
[17080.888478]     need_discard      1
[17080.888478]     need_inc_gen      1
[17080.888479]     dirty_sectors     16
[17080.888479]     stripe_sectors    0
[17080.888479]     cached_sectors    0
[17080.888480]     stripe            0
[17080.888480]     stripe_redundancy 0
[17080.888481]     io_time[READ]     61169962240
[17080.888481]     io_time[WRITE]    83046136272
[17080.888481]     fragmentation     16777216
[17080.888482]     bp_start          8
[17080.888482]   
[17080.888483]   update: btree=reflink cached=0 get_update_rebalance_opts+0x16a/0x2e0 [bcachefs]
[17080.888483]     old u64s 13 type reflink_v 0:6881236752:0 len 16 ver 0: refcount: 2 durability: 2 crc: c_size 120 size 128 offset 112 nonce 0 csum crc32c 0:355a6fca  compress zstd ptr: 4:6247691:1336 gen 0 crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c 0:664da0b4  compress none ptr: 3:255480:0 gen 28 d=0 cached rebalance: target hdd compression zstd crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c 0:664da0b4  compress incompressible ptr: 7:4785695:0 gen 0
[17080.888484]     new u64s 14 type reflink_v 0:6881236752:0 len 16 ver 0: refcount: 2 durability: 2 crc: c_size 120 size 128 offset 112 nonce 0 csum crc32c 0:355a6fca  compress zstd ptr: 4:6247691:1336 gen 0 crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c 0:664da0b4  compress none ptr: 3:255480:0 gen 28 d=0 cached rebalance: target hdd compression zstd crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c 0:664da0b4  compress incompressible ptr: 7:4785695:0 gen 0 (invalid extent entry 0000000000000000)
[17080.888486]   write_buffer_keys: btree=backpointers level=0 u64s 5 type deleted 4:13102359044208:0 len 0 ver 0write_buffer_keys: btree=accounting level=0 u64s 6 type accounting 0:3:16844034 len 0 ver 62671436933758988: replicas cached: 1/1 [3] -16write_buffer_keys: btree=accounting level=0 u64s 8 type accounting 0:0:1028 len 0 ver 62671436933758995: compression zstd -1 -128 -120write_buffer_keys: btree=backpointers level=0 u64s 5 type deleted 7:10036329840640:0 len 0 ver 0write_buffer_keys: btree=accounting level=0 u64s 6 type accounting 0:1796:16909314 len 0 ver 62671436933759010: replicas user: 1/2 [4 7] -31write_buffer_keys: btree=accounting level=0 u64s 8 type accounting 0:0:1284 len 0 ver 62671436933759017: compression incompressible -1 -16 -16write_buffer_keys: btree=accounting level=0 u64s 8 type accounting 0:0:8 len 0 ver 62671436933759026: inum  -1 -16 -31write_buffer_keys: btree=backpointers level=0 u64s 9 type backpointer 4:13102359044208:0 len 0 ver 0: bucket=4:6247691:0 btree=reflink level=0 offset=1336:1
[17081.027146] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): bch2_trans_commit_write_locked(): fatal error fatal error in transaction commit: fsck_delete_bkey
[17081.064411] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): unshutdown complete, journal seq 14591831

Fatal error in transaction commit doesn't sound good. No longer (yet) seeing "Allocator stuck? Waited for 30 seconds" though. It seems like write speeds are very slow, will see if I can check that.

Looks like the FS is read-only? All writes are failing, nothing in dmesg though

@gkmcd
Copy link

gkmcd commented Oct 21, 2024

Unmounted and remount with fsck,fixerrors, fsck failed at the same point. Perhaps the "allocator failed" issue above has caused some kind of corruption fsck can't repair?

@gkmcd
Copy link

gkmcd commented Oct 28, 2024

Pulled and built from from master yesterday, fsck now crashes with kernel BUG at fs/bcachefs/move.c:388! Also see another Allocator stuck? Waited for 30 seconds after this.

Should I create a new issue for this?

...
[16496.369502] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 36%, done 116056/320952 nodes, at extents:1476395207:5095936:U32_MAX
[16506.369448] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 38%, done 123984/320952 nodes, at extents:1476411043:32687192:U32_MAX
[16516.369411] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 41%, done 133157/320952 nodes, at reflink:0:1917425344:0
[16526.369375] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 43%, done 140407/320952 nodes, at reflink:0:3749891600:0
[16536.369340] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 45%, done 145687/320952 nodes, at reflink:0:4991503688:0
[16546.369306] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 46%, done 149974/320952 nodes, at reflink:0:5931661968:0
[16556.369269] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 48%, done 154102/320952 nodes, at reflink:0:6841711536:0
[16566.369233] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): extents_to_backpointers: 49%, done 158384/320952 nodes, at reflink:0:7777373520:0
[16574.143210]  done
[16574.143215] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_alloc_to_lru_refs... done
[16691.505420] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_snapshot_trees... done
[16691.505701] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_snapshots... done
[16691.506038] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_subvols... done
[16691.520472] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_subvol_children... done
[16691.520477] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): delete_dead_snapshots... done
[16691.520479] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_inodes... done
[16694.296226] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_extents... done
[16796.313980] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_indirect_extents... done
[16811.278631] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_dirents... done
[16811.592094] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_xattrs... done
[16811.592447] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_root... done
[16811.592578] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_unreachable_inodes... done
[16811.686463] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_subvolume_structure... done
[16811.686468] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_directory_structure... done
[16811.862039] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): check_nlinks... done
[16812.079761] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): resume_logged_ops... done
[16812.092576] bcachefs (32b2df91-1683-4288-9ebf-e60d104cc069): delete_dead_inodes... done
[16822.088210] ------------[ cut here ]------------
[16822.088216] kernel BUG at fs/bcachefs/move.c:388!
[16822.090748] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
[16822.093078] CPU: 7 UID: 0 PID: 4670 Comm: bch-rebalance/3 Not tainted 6.12.0-rc1bcachefs-00167-gb78da43137b1 #1 152162feaea44eb31f2d86917e9ff8fdec5c5942
[16822.093086] Hardware name: Hewlett-Packard HP Z440 Workstation/212B, BIOS M60 v02.62 01/04/2024
[16822.093088] RIP: 0010:get_update_rebalance_opts.isra.0+0x210/0x230 [bcachefs]
[16822.093347] Code: 56 0e 0f b6 50 01 83 e2 04 0f 84 43 fe ff ff 0f b6 40 03 41 80 4e 14 01 c0 e8 07 0f b6 c0 66 41 89 46 10 e9 2a fe ff ff 0f 0b <0f> 0b 4c 89 ef e8 f6 58 f6 ff 48 89 c2 e9 97 fe ff ff 66 66 2e 0f
[16822.093350] RSP: 0018:ffffb52fcbc0b908 EFLAGS: 00010246
[16822.093355] RAX: ffffb52fcbc0bce0 RBX: ffffb52fcbc0bb10 RCX: ffffb52fcbc0bd10
[16822.093358] RDX: 0000000000000000 RSI: ffffb52fcbc0bb10 RDI: ffff9fc6302a0000
[16822.093360] RBP: ffffb52fcbc0baa0 R08: ffff9fc88b21d480 R09: 0000000000000000
[16822.093362] R10: 0000000000000105 R11: 0000000000000002 R12: ffffb52fcbc0bd10
[16822.093364] R13: ffff9fc6302a0000 R14: ffff9fbb68c40000 R15: ffff9fbb68c40000
[16822.093367] FS:  0000000000000000(0000) GS:ffff9fca2fb80000(0000) knlGS:0000000000000000
[16822.093370] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[16822.093373] CR2: 00007ff71000f0f0 CR3: 00000001f4220001 CR4: 00000000001706f0
[16822.093376] Call Trace:
[16822.093382]  <TASK>
[16822.093386]  ? __die_body.cold+0x19/0x27
[16822.093399]  ? die+0x2e/0x50
[16822.093408]  ? do_trap+0xca/0x110
[16822.093414]  ? do_error_trap+0x6a/0x90
[16822.093418]  ? get_update_rebalance_opts.isra.0+0x210/0x230 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.093588]  ? exc_invalid_op+0x50/0x70
[16822.093597]  ? get_update_rebalance_opts.isra.0+0x210/0x230 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.093762]  ? asm_exc_invalid_op+0x1a/0x20
[16822.093771]  ? get_update_rebalance_opts.isra.0+0x210/0x230 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.093932]  bch2_move_get_io_opts_one+0x20c/0x270 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.094097]  ? bch2_btree_iter_peek_slot+0x2e8/0x770 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.094229]  ? btree_path_alloc.cold+0x17/0x1c [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.094388]  ? bch2_path_get+0x18f/0x3b0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.094523]  ? do_rebalance_extent+0x148/0x7a0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.094812]  ? psi_group_change+0x12a/0x300
[16822.094822]  ? do_rebalance_extent+0x194/0x7a0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.094996]  do_rebalance_extent+0x194/0x7a0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.095175]  ? do_rebalance+0x304/0x8d0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.095350]  do_rebalance+0x304/0x8d0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.095525]  ? lruvec_stat_mod_folio.constprop.0+0x1c/0x30
[16822.095537]  ? local_clock_noinstr+0xd/0xd0
[16822.095544]  ? local_clock+0x15/0x30
[16822.095550]  ? __bch2_trans_get+0x152/0x300 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.095691]  ? __pfx_bch2_rebalance_thread+0x10/0x10 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.095873]  bch2_rebalance_thread+0x66/0xb0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.096048]  ? bch2_rebalance_thread+0x5c/0xb0 [bcachefs fb49aab32f9f6d3bf3f9e53db628b3b0b0d17793]
[16822.096223]  kthread+0xd2/0x100
[16822.096233]  ? __pfx_kthread+0x10/0x10
[16822.096239]  ret_from_fork+0x34/0x50
[16822.096245]  ? __pfx_kthread+0x10/0x10
[16822.096251]  ret_from_fork_asm+0x1a/0x30
[16822.096262]  </TASK>
[16822.096263] Modules linked in: bcachefs lz4_compress lz4hc_compress snd_seq_dummy snd_hrtimer snd_seq snd_seq_device tun mei_hdcp mei_pxp mei_gsc xe overlay snd_hda_codec_hdmi drm_ttm_helper gpu_sched drm_suballoc_helper drm_gpuvm drm_exec intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp i915 coretemp crct10dif_pclmul crc32_pclmul polyval_clmulni snd_hda_codec_realtek polyval_generic ghash_clmulni_intel snd_hda_codec_generic sha512_ssse3 snd_hda_scodec_component sha1_ssse3 aesni_intel snd_hda_intel gf128mul snd_intel_dspcfg crypto_simd cryptd snd_hda_codec snd_hwdep i2c_algo_bit snd_hda_core drm_buddy vfat video e1000e fat ttm hp_wmi rapl snd_pcm sparse_keymap mei_wdt intel_cstate iTCO_wdt platform_profile intel_pmc_bxt psmouse intel_gtt rfkill wmi_bmof iTCO_vendor_support mei_me snd_timer drm_display_helper intel_uncore snd pcspkr i2c_i801 ptp mei i2c_smbus pps_core lpc_ich cec soundcore wmi tpm_infineon mac_hid dm_mod crypto_user loop nfnetlink zram ip_tables x_tables hid_generic usbhid
[16822.096364]  uas usb_storage serio_raw atkbd mpt3sas libps2 nvme vivaldi_fmap nvme_core sr_mod sha256_ssse3 cdrom raid_class i8042 scsi_transport_sas nvme_auth serio btrfs blake2b_generic libcrc32c crc32c_generic crc32c_intel xor raid6_pq
[16822.096528] ---[ end trace 0000000000000000 ]---
[16822.096721] RIP: 0010:get_update_rebalance_opts.isra.0+0x210/0x230 [bcachefs]
[16822.096895] Code: 56 0e 0f b6 50 01 83 e2 04 0f 84 43 fe ff ff 0f b6 40 03 41 80 4e 14 01 c0 e8 07 0f b6 c0 66 41 89 46 10 e9 2a fe ff ff 0f 0b <0f> 0b 4c 89 ef e8 f6 58 f6 ff 48 89 c2 e9 97 fe ff ff 66 66 2e 0f
[16822.096898] RSP: 0018:ffffb52fcbc0b908 EFLAGS: 00010246
[16822.096902] RAX: ffffb52fcbc0bce0 RBX: ffffb52fcbc0bb10 RCX: ffffb52fcbc0bd10
[16822.096905] RDX: 0000000000000000 RSI: ffffb52fcbc0bb10 RDI: ffff9fc6302a0000
[16822.096907] RBP: ffffb52fcbc0baa0 R08: ffff9fc88b21d480 R09: 0000000000000000
[16822.096910] R10: 0000000000000105 R11: 0000000000000002 R12: ffffb52fcbc0bd10
[16822.096912] R13: ffff9fc6302a0000 R14: ffff9fbb68c40000 R15: ffff9fbb68c40000
[16822.096914] FS:  0000000000000000(0000) GS:ffff9fca2fb80000(0000) knlGS:0000000000000000
[16822.096918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[16822.096920] CR2: 00007ff71000f0f0 CR3: 00000001f4220001 CR4: 00000000001706f0
[63648.391988] Allocator stuck? Waited for 30 seconds
[63648.391994] Allocator debug:
[63648.391995]   capacity77280750838
[63648.391997]   reserved             6720065290
[63648.391998]   hidden               92328960
[63648.392000]   btree                405056512
[63648.392001]   data                 69555749434
[63648.392002]   cached               6632541366
[63648.392004]   reserved             345689648
[63648.392005]   online_reserved      112
[63648.392007]   nr_inodes            0
[63648.392008]   
[63648.392009]   freelist_wait        waiting
[63648.392011]   open buckets allocated87
[63648.392012]   open buckets total   1024
[63648.392013]   open_buckets_wait    empty
[63648.392015]   open_buckets_btree   4
[63648.392016]   open_buckets_user    82
[63648.392017]   btree reserve cache  1

[63648.392020] Dev 0:
[63648.392021]                      buckets         sectors      fragmented
[63648.392023]   free                178438               0               0
[63648.392024]   sb                       7            6152            1016
[63648.392026]   journal               8192         8388608               0
[63648.392027]   btree                 2444         2496512            6144
[63648.392029]   user              11257252     11527439673              24
[63648.392030]   cached                   0               0               0
[63648.392031]   parity                   0               0               0
[63648.392033]   stripe                   0               0               0
[63648.392034]   need_gc_gens             0               0               0
[63648.392036]   need_discard             0               0               0
[63648.392037]   unstriped                0               0               0
[63648.392039]   capacity          11446333
[63648.392040]   
[63648.392041]   reserves:
[63648.392042]   stripe              357724
[63648.392044]   normal              178876
[63648.392045]   copygc                  28
[63648.392046]   btree                   14
[63648.392047]   btree_copygc             0
[63648.392048]   reclaim                  0
[63648.392049]   interior_updates         0
[63648.392050]   
[63648.392052]   open buckets             1
[63648.392053]   buckets to invalidate    0

[63648.392055] Dev 1:
[63648.392056]                      buckets         sectors      fragmented
[63648.392058]   free                178430               0               0
[63648.392059]   sb                       7            6152            1016
[63648.392061]   journal               8192         8388608               0
[63648.392062]   btree                 2440         2493440            5120
[63648.392064]   user              11257264     11527451757              80
[63648.392065]   cached                   0               0               0
[63648.392067]   parity                   0               0               0
[63648.392068]   stripe                   0               0               0
[63648.392069]   need_gc_gens             0               0               0
[63648.392071]   need_discard             0               0               0
[63648.392072]   unstriped                0               0               0
[63648.392074]   capacity          11446333
[63648.392075]   
[63648.392076]   reserves:
[63648.392077]   stripe              357724
[63648.392078]   normal              178876
[63648.392080]   copygc                  28
[63648.392081]   btree                   14
[63648.392082]   btree_copygc             0
[63648.392083]   reclaim                  0
[63648.392085]   interior_updates         0
[63648.392086]   
[63648.392087]   open buckets             0
[63648.392088]   buckets to invalidate    0

[63648.392090] Dev 2:
[63648.392091]                      buckets         sectors      fragmented
[63648.392093]   free                 89337               0               0
[63648.392094]   sb                       4            6152            2040
[63648.392096]   journal               8192        16777216               0
[63648.392097]   btree                 8830        18068480           15360
[63648.392098]   user               5616803     11503232128               0
[63648.392100]   cached                   0               0               0
[63648.392101]   parity                   0               0               0
[63648.392103]   stripe                   0               0               0
[63648.392104]   need_gc_gens             0               0               0
[63648.392106]   need_discard             0               0               0
[63648.392107]   unstriped                0               0               0
[63648.392108]   capacity           5723166
[63648.392110]   
[63648.392111]   reserves:
[63648.392112]   stripe              178862
[63648.392113]   normal               89438
[63648.392114]   copygc                  14
[63648.392116]   btree                    7
[63648.392117]   btree_copygc             0
[63648.392118]   reclaim                  0
[63648.392119]   interior_updates         0
[63648.392120]   
[63648.392121]   open buckets             0
[63648.392122]   buckets to invalidate    0

[63648.392125] Dev 3:
[63648.392126]                      buckets         sectors      fragmented
[63648.392127]   free                 74548               0               0
[63648.392129]   sb                       7            6152            1016
[63648.392130]   journal               8192         8388608               0
[63648.392132]   btree                    0               0               0
[63648.392133]   user                     0               0               0
[63648.392135]   cached             1824992      1860444703         8347296
[63648.392136]   parity                   0               0               0
[63648.392138]   stripe                   0               0               0
[63648.392139]   need_gc_gens             0               0               0
[63648.392140]   need_discard             0               0               0
[63648.392142]   unstriped                0               0               0
[63648.392143]   capacity           1907739
[63648.392144]   
[63648.392145]   reserves:
[63648.392147]   stripe               59644
[63648.392148]   normal               29836
[63648.392149]   copygc                  28
[63648.392150]   btree                   14
[63648.392151]   btree_copygc             0
[63648.392153]   reclaim                  0
[63648.392154]   interior_updates         0
[63648.392155]   
[63648.392156]   open buckets            32
[63648.392157]   buckets to invalidate    0

[63648.392159] Dev 4:
[63648.392161]                      buckets         sectors      fragmented
[63648.392162]   free                119242               0               0
[63648.392163]   sb                       4            6152            2040
[63648.392165]   journal               8192        16777216               0
[63648.392166]   btree                49828       102022656           25088
[63648.392168]   user               7453619     15265040017           10672
[63648.392169]   cached                   0               0               0
[63648.392171]   parity                   0               0               0
[63648.392172]   stripe                   0               0               0
[63648.392173]   need_gc_gens             0               0               0
[63648.392175]   need_discard             0               0               0
[63648.392176]   unstriped                0               0               0
[63648.392178]   capacity           7630885
[63648.392179]   
[63648.392180]   reserves:
[63648.392181]   stripe              238478
[63648.392182]   normal              119246
[63648.392183]   copygc                  14
[63648.392185]   btree                    7
[63648.392186]   btree_copygc             0
[63648.392187]   reclaim                  0
[63648.392188]   interior_updates         0
[63648.392190]   
[63648.392191]   open buckets             6
[63648.392192]   buckets to invalidate    0

[63648.392194] Dev 6:
[63648.392195]                      buckets         sectors      fragmented
[63648.392197]   free                119257               0               0
[63648.392198]   sb                       4            6152            2040
[63648.392200]   journal               8192        16777216               0
[63648.392201]   btree                56871       116430336           41472
[63648.392203]   user               7446561     15250568258           24608
[63648.392204]   cached                   0               0               0
[63648.392206]   parity                   0               0               0
[63648.392207]   stripe                   0               0               0
[63648.392209]   need_gc_gens             0               0               0
[63648.392210]   need_discard             0               0               0
[63648.392212]   unstriped                0               0               0
[63648.392213]   capacity           7630885
[63648.392214]   
[63648.392215]   reserves:
[63648.392216]   stripe              238478
[63648.392218]   normal              119246
[63648.392219]   copygc                  14
[63648.392220]   btree                    7
[63648.392222]   btree_copygc             0
[63648.392223]   reclaim                  0
[63648.392224]   interior_updates         0
[63648.392225]   
[63648.392226]   open buckets            13
[63648.392227]   buckets to invalidate    0

[63648.392230] Dev 7:
[63648.392231]                      buckets         sectors      fragmented
[63648.392232]   free               2842000               0               0
[63648.392234]   sb                       4            6152            2040
[63648.392235]   journal               8192        16777216               0
[63648.392237]   btree                79902       163545088           94208
[63648.392238]   user               2188506      4482017601           46024
[63648.392239]   cached             2512281      4772096663       373054839
[63648.392241]   parity                   0               0               0
[63648.392242]   stripe                   0               0               0
[63648.392244]   need_gc_gens             0               0               0
[63648.392245]   need_discard             0               0               0
[63648.392246]   unstriped                0               0               0
[63648.392248]   capacity           7630885
[63648.392249]   
[63648.392250]   reserves:
[63648.392251]   stripe              238478
[63648.392252]   normal              119246
[63648.392254]   copygc                  14
[63648.392255]   btree                    7
[63648.392256]   btree_copygc             0
[63648.392257]   reclaim                  0
[63648.392259]   interior_updates         0
[63648.392260]   
[63648.392261]   open buckets            34
[63648.392262]   buckets to invalidate    0

[63648.392264] Copygc debug:
[63648.392265]   running: 0
[63648.392266]   copygc_wait:0
[63648.392268]   copygc_wait_at:0
[63648.392269]   Currently waiting for:0 B
[63648.392270]   Currently waiting since:38.7 TiB
[63648.392271]   Currently calculated wait:0 B

[63648.392274] Journal debug:
[63648.392275]   flags:                   replay_done,running,may_skip_flush
[63648.392276]   dirty journal entries:   0/32768
[63648.392278]   seq:                     14593793
[63648.392279]   seq_ondisk:              14593793
[63648.392280]   last_seq:                14593794
[63648.392281]   last_seq_ondisk:         14593793
[63648.392283]   flushed_seq_ondisk:      14593793
[63648.392284]   watermark:               stripe
[63648.392285]   each entry reserved:     321
[63648.392287]   nr flush writes:         1943
[63648.392288]   nr noflush writes:       11
[63648.392289]   average write size:      12.9 KiB
[63648.392290]   nr direct reclaim:       0
[63648.392292]   nr background reclaim:   22172
[63648.392293]   reclaim kicked:          0
[63648.392294]   reclaim runs in:         0 ms
[63648.392295]   blocked:                 0
[63648.392297]   current entry sectors:   2048
[63648.392298]   current entry error:     ok
[63648.392299]   current entry:           closed
[63648.392300]   unwritten entries:
[63648.392301]   last buf closed
[63648.392303]   space:
[63648.392304]     discarded              2048:16773120
[63648.392305]     clean ondisk           2048:16773120
[63648.392307]     clean                  2048:16773120
[63648.392308]     total                  2048:16777216
[63648.392309]   dev 0:
[63648.392310]     nr                     8192
[63648.392312]     bucket size            1024
[63648.392313]     available              8190:72
[63648.392314]     discard_idx            4712
[63648.392316]     dirty_ondisk           4712 (seq 14593792)
[63648.392317]     dirty_idx              4712 (seq 14593792)
[63648.392318]     cur_idx                4712 (seq 14593792)
[63648.392320]   dev 1:
[63648.392321]     nr                     8192
[63648.392322]     bucket size            1024
[63648.392323]     available              8190:304
[63648.392325]     discard_idx            4230
[63648.392326]     dirty_ondisk           4230 (seq 14593793)
[63648.392327]     dirty_idx              4230 (seq 14593793)
[63648.392329]     cur_idx                4230 (seq 14593793)
[63648.392330]   dev 2:
[63648.392331]     nr                     8192
[63648.392332]     bucket size            2048
[63648.392334]     available              8190:1624
[63648.392335]     discard_idx            7128
[63648.392336]     dirty_ondisk           7128 (seq 14593791)
[63648.392337]     dirty_idx              7128 (seq 14593791)
[63648.392339]     cur_idx                7128 (seq 14593791)
[63648.392340]   dev 3:
[63648.392341]     nr                     8192
[63648.392342]     bucket size            1024
[63648.392344]     available              8190:1024
[63648.392345]     discard_idx            0
[63648.392346]     dirty_ondisk           0 (seq 0)
[63648.392347]     dirty_idx              0 (seq 0)
[63648.392349]     cur_idx                0 (seq 0)
[63648.392350]   dev 4:
[63648.392351]     nr                     8192
[63648.392352]     bucket size            2048
[63648.392354]     available              8190:1560
[63648.392355]     discard_idx            4684
[63648.392356]     dirty_ondisk           4684 (seq 14593792)
[63648.392357]     dirty_idx              4684 (seq 14593792)
[63648.392359]     cur_idx                4684 (seq 14593792)
[63648.392360]   dev 6:
[63648.392361]     nr                     8192
[63648.392362]     bucket size            2048
[63648.392364]     available              8190:1336
[63648.392365]     discard_idx            4092
[63648.392366]     dirty_ondisk           4092 (seq 14593793)
[63648.392367]     dirty_idx              4092 (seq 14593793)
[63648.392369]     cur_idx                4092 (seq 14593793)
[63648.392370]   dev 7:
[63648.392371]     nr                     8192
[63648.392372]     bucket size            2048
[63648.392374]     available              8190:0
[63648.392375]     discard_idx            3111
[63648.392376]     dirty_ondisk           3111 (seq 14593753)
[63648.392378]     dirty_idx              3111 (seq 14593753)
[63648.392379]     cur_idx                3111 (seq 14593753)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants