How to split a Torch class into several files in a Lua rock

In my recently aided in the development of a Dataframe package for Torch. As the code base has quickly doubled there is a need to split the class into several sections for better organization and follow-up (issue #8).

A simple test-class would be a test.lua file in the root folder of the test-package:

test = torch.class('test')
function test:__init()
  self.data = {}
end

function test:a()
  print("a")
end

function test:b()
  print("b")
end

Now the rockspec for this would simply be:

package = "torch-test"
 version = "0.1-1"
 source = {
    url = "..."
 }
 description = {
    summary = "A test class",
    detailed = [[
       Just an example
    ]],
    license = "MIT/X11",
    maintainer = "Jon Doe"
 }
 dependencies = {
    "lua ~> 5.1",
    "torch >= 7.0",
 }
 build = {
  type = 'builtin',
  modules = {
      ["test"] = 'test.lua',
  }
 }

Solution

In order to get multiple files to work for a single class it is necessary to return the class object initially created and pass it to the subsections. The above example can be put into the file structure:

\init.lua
\main.lua
\test-0.1-1.rockspec
\Extensions\a.lua
\Extensions\b.lua

The luarocks install/make copies the files according to 'require' syntax where each . signifies a directory and the .lua is left out, i.e. we need to change the rockspec to:

package = "torch-test"
 version = "0.1-1"
 source = {
    url = "..."
 }
 description = {
    summary = "A test class",
    detailed = [[
       Just an example
    ]],
    license = "MIT/X11",
    maintainer = "Jon Doe"
 }
 dependencies = {
    "lua ~> 5.1",
    "torch >= 7.0",
 }
 build = {
  type = 'builtin',
  modules = {
      ["test.init"] = 'init.lua',
      ["test.main"] = 'main.lua',
      ["test.Extensions.a"] = 'a.lua',
      ["test.Extensions.b"] = 'b.lua'
    }
  }

The above will thus create a test-folder where the packages reside together with the files and subdirectories. The class initialization now resides in the init.lua that returns the class object:

test = torch.class('test')
function test:__init()
  self.data = {}
end
return test

The subclass-files now need to pickup the class object that is passed using loadfile() (see init.lua file below). The a.lua should now look like this:

local params = {...}
local test = params[1]
function test:a()
  print("a")
end

and similar addition for the b.lua:

local params = {...}
local test = params[1]
function test:b()
  print("b")
end

In order to glue everything together we have the init.lua file. The following is probably a little over-complicated but it takes care of:

Finding all extensions available and loading them (Note: requires lua filesystem that you should add to the rockspec and you still need to add each file into the rockspec or it won't be in the Extensions folder)
Identifies the paths folder
Loads the main.lua
Works in a pure testing environment without the package installed

The code for init.lua:

require 'lfs'

local file_exists = function(name)
   local f=io.open(name,"r")
   if f~=nil then io.close(f) return true else return false end
end

-- If we're in development mode the default path should be the current
local test_path = "./?.lua"
local search_4_file = "Extensions/load_batch"
if (not file_exists(string.gsub(test_path, "?", search_4_file))) then
  -- split all paths according to ;
  for path in string.gmatch(package.path, "[^;]+;") do
    -- remove trailing ;
    path = string.sub(path, 1, string.len(path) - 1)
    if (file_exists(string.gsub(path, "?", "test/" .. search_4_file))) then
      test_path = string.gsub(path, "?", "test/?")
      break;
    end
  end
  if (test_path == nil) then
    error("Can't find package files in search path: " .. tostring(package.path))
  end
end

local main_file = string.gsub(test_path,"?", "main")
local test = assert(loadfile(main_file))()

-- Load all extensions, i.e. .lua files in Extensions directory
ext_path = string.gsub(test_path, "[^/]+$", "") .. "Extensions/"
for extension_file,_ in lfs.dir (ext_path) do
  if (string.match(extension_file, "[.]lua$")) then
    local file = ext_path .. extension_file
    assert(loadfile(file))(test)
  end
end

return test

I hope this helps if you run into the same problem and find the documentation a little too sparse. If you happen to know a better solution, please share.