Skip to content

Commit

Permalink
Default to Lua table if tds not installed
Browse files Browse the repository at this point in the history
If tds is not installed because e.g. Torch with installed with torch-nv deb package then default to using a regular Lua table to store LMDB keys.
There is a limit of 2GB for heap memory in Luajit so we might not be able to train on some large datasets without tds.
  • Loading branch information
gheinrich committed Apr 19, 2016
1 parent 7e9acc4 commit c96d52d
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions tools/torch/data.lua
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ package.path = debug.getinfo(1, "S").source:match[[^@?(.*[\/])[^\/]-$]] .."?.lua
require 'logmessage'
local ffi = require 'ffi'

local tds = check_require 'tds'
local tdsIsInstalled, tds = pcall(function() return check_require 'tds' end)

-- enable shared serialization to speed up Tensor passing between threads
threads.Threads.serialization('threads.sharedserialize')
Expand Down Expand Up @@ -322,7 +322,14 @@ end

-- Derived class method lmdb_getKeys
function DBSource:lmdb_getKeys ()
local Keys = tds.Vec()
local Keys
if tdsIsInstalled then
-- use tds.Vec() to allocate memory outside of Lua heap
Keys = tds.Vec()
else
-- if tds is not installed, use regular table (and Lua memory allocator)
Keys = {}
end
local i=0
local key=nil
for k,v in all_keys(self.lmdb_data.c,nil,self.lightningmdb.MDB_NEXT) do
Expand Down

0 comments on commit c96d52d

Please sign in to comment.