Fallback MarcRecord query in NormalizedMarcRecordReader causes "cannot allocate memory" error

[corylown]

This isn't an urgent problem because the postgres optimized query works fine. However, the fallback query appears to result in a "cannot allocate memory" error when using the NormalizedMarcRecordReader to iterate over a large set of records.

Query is here:

aggregator/app/services/normalized_marc_record_reader.rb

Line 50 in 5bc18b5

MarcRecord.select(:marc001, :id).where(upload: uploads).order(file_id: :asc, id: :asc).each do |record|

In normal POD operation this error has only occurred when running GenerateFullDumpJob. It can be replicated on the Rails console:

uploads = Upload.active.where(stream: Organization.find_by(slug: 'duke').default_stream)

NormalizedMarcRecordReader.new(uploads).each_slice(5000) do |records|
  records.each { |rec| puts rec.marc001 }
end

Example of error:

3.3.0 :001 > org = Organization.find_by(slug: 'duke')
 => 
#<Organization:0x00007f15bc5a5ba8
... 
3.3.0 :002 > GenerateFullDumpJob.perform_now(org)
I, [2025-01-22T07:27:07.215906 #3658443]  INFO -- : [ActiveJob] [GenerateFullDumpJob] [ea413fa1-bc65-4bea-9350-23f174799b20] Performing GenerateFullDumpJob (Job ID: ea413fa1-bc65-4bea-9350-23f174799b20) from Sidekiq(default) with arguments: #<GlobalID:0x00007f15bcad9d68 @uri=#<URI::GID gid://aggregator/Organization/5>>
/opt/app/pod/aggregator/releases/20250121160921/app/services/oai_marc_record_writer_service.rb:59: [BUG] heap_page_body_free: munmap failed
ruby 3.3.1 (2024-04-23 revision c56cd86388) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0078 p:---- s:0451 e:000450 CFUNC  :write
c:0077 p:0062 s:0446 e:000444 METHOD /opt/app/pod/aggregator/releases/20250121160921/app/services/oai_marc_record_writer_service.rb:59
c:0076 p:0021 s:0437 e:000436 METHOD /opt/app/pod/aggregator/releases/20250121160921/app/services/oai_marc_record_writer_service.rb:12
c:0075 p:0030 s:0429 e:000428 BLOCK  /opt/app/pod/aggregator/releases/20250121160921/app/jobs/generate_full_dump_job.rb:46 [FINISH]
c:0074 p:---- s:0425 e:000424 CFUNC  :each
c:0073 p:0019 s:0421 e:000420 BLOCK  /opt/app/pod/aggregator/releases/20250121160921/app/jobs/generate_full_dump_job.rb:41 [FINISH]

...

-- Threading information ---------------------------------------------------
Total ractor count: 1
Ruby thread count for this ractor: 12

-- C level backtrace information -------------------------------------------
mmap: Cannot allocate memory
/usr/local/rvm/rubies/ruby-3.3.1/bin/../lib/libruby.so.3.3(0x7f15db444607) [0x7f15db444607]
[0x7f15db23f88f]
[0x7f15db1754c3]
[0x7f15db175a7e]
[0x7f15db263f87]
[0x7f15db26d26e]