inets httpd: Can't get simple esi script to work

Here's the proplist_file that I use to configure the httpd server:

[
  {modules, [
    mod_alias,
    mod_actions,
    mod_cgi,
    mod_get,
    mod_esi,
    mod_log
  ]},
  {bind_address, "localhost"}, 
  {port,0},
  {server_name,"httpd_test"},
  {server_root,"/Users/7stud/erlang_programs/inets_proj"},
  {document_root,"./htdocs"},
  {script_alias, {"/cgi-bin/", "/Users/7stud/erlang_programs/inets_proj/cgi-bin/"} },
  {erl_script_alias, {"/cgi-bin/example", [httpd_example]} },
  {erl_script_nocache, true},
  {error_log, "./errors.log"},
  {transfer_log, "./requests.log"}
].

For esi scripts the key property is:

  {erl_script_alias, {"/cgi-bin/example", [httpd_example]} }

According to the inets docs:

ESI Properties - Requires mod_esi

{erl_script_alias, {URLPath, [AllowedModule]}}
URLPath = string() and AllowedModule = atom(). erl_script_alias marks all URLs matching url-path as erl scheme scripts. A matching URL is mapped into a specific module and function, for example:
 {erl_script_alias, {"/cgi-bin/example", [httpd_example]}}
A request to http://your.server.org/cgi-bin/example/httpd_example:yahoo would refer to httpd_example:yahoo/3 or, if that does not exist, httpd_example:yahoo/2 and http://your.server.org/cgi-bin/example/other:yahoo would not be allowed to execute.

My directory structure is:

~/erlang_programs$ tree inets_proj/
inets_proj/
├── cgi-bin
│   ├── 1.pl
│   ├── example
│   │   ├── httpd_example.beam
│   │   └── httpd_example.erl
│   ├── httpd_example.beam
│   └── httpd_example.erl
├── cl.beam
├── cl.erl
├── errors.log
├── htdocs
│   └── file1.txt
├── httpd_example.beam
├── httpd_example.erl
├── mylog.log
├── requests.log
├── s.beam
├── s.erl
└── server.conf

I wasn't sure where to put the httpd_example module, so I put it in several places.

esi_mod.erl:

-module(esi_mod).
-compile(export_all).

log(Data) ->
    {ok, IoDevice} = file:open(
        "/Users/7stud/erlang_programs/inets_proj/mylog.log",
        [append]
    ),

    ok = file:write(IoDevice, Data),
    file:close(IoDevice).

get_data(SessionID, _Env, _Input) ->
    Headers = "Content-Type: text/html\r\n\r\n",
    Data = "Hello, esi",

    log(["--Inside esi_mod:get_data() ~n"]),

    ok = mod_esi:deliver(SessionID, Headers),  %Headers must be a string.
    ok = mod_esi:deliver(SessionID, Data).     %Data can be an iolist.

Here is the server info as reported in the shell:

$ erl
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V9.2  (abort with ^G)

1> c(s).         
s.erl:2: Warning: export_all flag enabled - all functions will be exported
{ok,s}
2> S = s:start().
<0.86.0>
3> httpd:info(S).
[{mime_types,[{"htm","text/html"},{"html","text/html"}]},
 {server_name,"httpd_test"},
 {erl_script_nocache,true},
 {script_alias,{"/cgi-bin/",
                "/Users/7stud/erlang_programs/inets_proj/cgi-bin/"}},
 {bind_address,{127,0,0,1}},
 {modules,[mod_alias,mod_actions,mod_cgi,mod_get,mod_esi,
           mod_log]},
 {server_root,"/Users/7stud/erlang_programs/inets_proj"},
 {erl_script_alias,{"/cgi-bin/example",[httpd_example]}},
 {port,64470},
 {transfer_log,<0.93.0>},
 {error_log,<0.92.0>},
 {document_root,"./htdocs"}]
4>

Here's my request:

~$ curl -vv "http://localhost:64470/cgi-bin/example/httpd_example:get_data"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 64470 (#0)
> GET /cgi-bin/example/httpd_example:get_data HTTP/1.1
> Host: localhost:64470
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 404 Object Not Found
< Date: Wed, 28 Feb 2018 10:22:38 GMT
< Server: inets/6.4.5
< Content-Type: text/html
< Content-Length: 245
< 
<HTML>
       <HEAD>
           <TITLE>Object Not Found</TITLE>
      </HEAD>
      <BODY>
      <H1>Object Not Found</H1>
The requested URL &#47;cgi-bin&#47;example&#47;httpd_example:get_data was not found on this server.
</BODY>
      </HTML>
* Connection #0 to host localhost left intact
~$

errors.log:

[Timestamp] access to /cgi-bin/example/httpd_example:get_data failed for 127.0.0.1, reason: "httpd_file: Can't open ./htdocs/cgi-bin/example/httpd_example:get_data: File not found"

According to the error message, the request path:

/cgi-bin/example/httpd_example:get_data

got converted to:

./htdocs/cgi-bin/example/httpd_example:get_data

which means the request path was tacked onto ./htdocs. Huh??

Solution

Okay, I got things working. First, while I was searching around I happened across:

10 Essential Erlang Tools for Erlang Developers

and therein lies the single greatest erlang "tool" that I've used to date:

Command history that isn't wiped out when you exit the erlang shell.

You can install/enable it by following the instructions here: https://github.com/ferd/erlang-history

Back to the problem at hand:

1) I found that the order of the httpd modules is important. mod_esi needs to be before mod_get:

server.conf:

[
  {modules, [
    mod_alias,
    mod_actions,
    mod_esi,
    mod_cgi,
    mod_get,
    mod_log
  ]},
  {bind_address, "localhost"}, 
  {port,0},
  {server_name,"httpd_test"},
  {server_root,"/Users/7stud/erlang_programs/inets_proj"},
  {document_root,"./htdocs"},
  {script_alias, {"/cgi-bin/", "/Users/7stud/erlang_programs/inets_proj/cgi-bin/"} },
  {erl_script_alias, {"/erl", [mymod]} },
  {erl_script_nocache, true},
  {error_log, "./errors.log"},
  {transfer_log, "./requests.log"}
].

After getting the module order correct, I stopped getting the "File not found" errors due to the strange converted paths.

2) The code for the esi module has to be in the server_root directory. My esi module's name is mymod.erl:

~/erlang_programs$ tree inets_proj/
inets_proj/
├── cgi-bin
│   ├── 1.pl
│   ├── example
│   │   ├── httpd_example.beam
│   │   └── httpd_example.erl
│   ├── httpd_example.beam
│   └── httpd_example.erl
├── cl.beam
├── cl.erl
├── errors.log
├── htdocs
│   └── file1.txt
├── mylog.log
├── mymod.beam
├── mymod.erl
├── requests.log
├── s.beam
├── s.erl
├── server.conf
├── xhttpd_example.beam
└── xhttpd_example.erl

3) Because I specified:

{erl_script_alias, {"/erl", [mymod]} }

the url I need to use is:

http://localhost:57867/erl/mymod:get_data

The port has to match the server's port. The proper path is whatever path you specify in the erl_script_alias property plus /modulename:funcname or /modulename/funcname.

Here's mymod.erl:

-module(mymod).
-export([get_data/3]).

log(Data) ->
    {ok, IoDevice} = file:open(
        "/Users/7stud/erlang_programs/inets_proj/mylog.log",
        [append]
    ),

    file:write(IoDevice, Data),
    file:close(IoDevice).

get_data(SessionID, Env, Input) ->
    Headers = "Content-Type: text/html\r\n\r\n",
    Data = [
        <<"Hello, ">>, 
        "esi!\n"
    ],

    log(io_lib:format(
        "Inside mymod:get_data()\nSessionId=~p\nEnv=~p\nInput=~p\n", 
        [SessionID, Env, Input]
    )),

    mod_esi:deliver(SessionID, Headers),  %Headers must be a string.
    mod_esi:deliver(SessionID, Data).     %Data can be an iolist.

According to the mod_esi docs:

mod_esi:deliver/2 shall be used to generate the response to the client and SessionID is an identifier that shall by used when calling this function, do not assume anything about the datatype. This function may be called several times to chunk the response data. Notice that the first chunk of data sent to the client must at least contain all HTTP header fields that the response will generate. If the first chunk does not contain the end of HTTP header, that is, "\r\n\r\n", the server assumes that no HTTP header fields will be generated.

4) Compile mymod.erl:

~/erlang_programs/inets_proj$ erlc mymod.erl 
~/erlang_programs/inets_proj$

You have to recompile after every change you make to mymod.erl, then restart the server. It would be simpler if this would work:

5> httpd:reload_config("server.conf", disturbing).
{error,{missing_property,server_name}}

but even though my config file does specify a server_name property I get that error.

5) I suggest you do error logging by including mod_log in the list of modules and specifying the property:

{error_log, "./errors.log"}

Then check that file for any feedback on what happened when a request fails.

6) When I called my custom log() method (in order to write some info to a file) unbeknownst to me the log() method was causing an exception, which caused the server to reject the request and enter a module traverse failed message in error.log:

[Timestamp], module traverse failed: mod_esi:do => 
   Error Type:  exit
   Error:       {mod_esi_linked_process_died,<0.97.0>,normal}
   Stack trace: [{mod_esi,receive_headers,1,[{file,"mod_esi.erl"},{line,428}]},
                 {mod_esi,deliver_webpage_chunk,3,
                     [{file,"mod_esi.erl"},{line,389}]},
                 {mod_esi,erl_scheme_webpage_chunk,5,
                     [{file,"mod_esi.erl"},{line,380}]},
                 {mod_esi,generate_webpage,7,
                     [{file,"mod_esi.erl"},{line,314}]},
                 {httpd_response,traverse_modules,2,
                     [{file,"httpd_response.erl"},{line,77}]},
                 {httpd_response,generate_and_send_response,1,
                     [{file,"httpd_response.erl"},{line,44}]},
                 {httpd_request_handler,handle_response,1,
                     [{file,"httpd_request_handler.erl"},{line,655}]},
                 {gen_server,try_dispatch,4,
                     [{file,"gen_server.erl"},{line,616}]}]

7) Here's the module I used to start the server:

-module(s).
-compile(export_all).

%Need to look up port with httpd:info(Server)

ensure_inets_start() ->
    case inets:start() of
        ok -> ok;
        {error,{already_started,inets}} -> ok
    end.


start() ->
    ok = s:ensure_inets_start(),

    {ok, Server} = inets:start(httpd, 
        [{proplist_file, "./server.conf"}]
    ),
    Server.


stop(Server) ->
    ok = inets:stop(httpd, Server).

8) Here are a few sample requests using curl...

Server info in the shell:

~/erlang_programs/inets_proj$ erl
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.2  (abort with ^G)
1> c(s).         
s.erl:2: Warning: export_all flag enabled - all functions will be exported
{ok,s}
2> S = s:start().
<0.86.0>
3> httpd:info(S).
[{mime_types,[{"htm","text/html"},{"html","text/html"}]},
 {server_name,"httpd_test"},
 {erl_script_nocache,true},
 {script_alias,{"/cgi-bin/",
                "/Users/7stud/erlang_programs/inets_proj/cgi-bin/"}},
 {bind_address,{127,0,0,1}},
 {modules,[mod_alias,mod_actions,mod_esi,mod_cgi,mod_get,
           mod_log]},
 {server_root,"/Users/7stud/erlang_programs/inets_proj"},
 {erl_script_alias,{"/erl",[mymod]}},
 {port,59202},
 {transfer_log,<0.93.0>},
 {error_log,<0.92.0>},
 {document_root,"./htdocs"}]
4>

Request 1 (esi get request):

~$  curl -v "http://localhost:57867/erl/mymod:get_data"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 57867 (#0)
> GET /erl/mymod:get_data HTTP/1.1
> Host: localhost:57867
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2018 13:28:09 GMT
< Server: inets/6.4.5
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: Wed, 28 Feb 2018 13:28:09 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< 
Hello, esi!
* Connection #0 to host localhost left intact
~$

mylog.log:

~/erlang_programs/inets_proj$ cat mylog.log
...
...
Inside mymod:get_data()
SessionId=<0.99.0>
Env=[{server_software,"inets/6.4.5"},
     {server_name,"httpd_test"},
     {host_name,"ChristophersMBP"},
     {gateway_interface,"CGI/1.1"},
     {server_protocol,"HTTP/1.1"},
     {server_port,59202},
     {request_method,"GET"},
     {remote_addr,"127.0.0.1"},
     {peer_cert,undefined},
     {script_name,"/erl/mymod:get_data"},
     {http_host,"localhost:59202"},
     {http_user_agent,"curl/7.58.0"},
     {http_accept,"*/*"}]
Input=[]
-------

Request 2 (esi get request with query string):

~$  curl -v "http://localhost:59202/erl/mymod:get_data?a=1&b=2"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 59202 (#0)
> GET /erl/mymod:get_data?a=1&b=2 HTTP/1.1
> Host: localhost:59202
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2018 13:47:41 GMT
< Server: inets/6.4.5
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: Wed, 28 Feb 2018 13:47:41 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< 
Hello, esi!
* Connection #0 to host localhost left intact

mylog.log:

~/erlang_programs/inets_proj$ cat mylog.log
...
...
Inside mymod:get_data()
SessionId=<0.105.0>
Env=[{server_software,"inets/6.4.5"},
     {server_name,"httpd_test"},
     {host_name,"ChristophersMBP"},
     {gateway_interface,"CGI/1.1"},
     {server_protocol,"HTTP/1.1"},
     {server_port,59202},
     {request_method,"GET"},
     {remote_addr,"127.0.0.1"},
     {peer_cert,undefined},
     {script_name,"/erl/mymod:get_data?a=1&b=2"},
     {http_host,"localhost:59202"},
     {http_user_agent,"curl/7.58.0"},
     {http_accept,"*/*"},
     {query_string,"a=1&b=2"}]
Input="a=1&b=2"

Request 3 (esi post request):

~$  curl -v --data "a=1&b=2" "http://localhost:59202/erl/mymod:get_data"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 59202 (#0)
> POST /erl/mymod:get_data HTTP/1.1
> Host: localhost:59202
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 7
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 7 out of 7 bytes
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2018 13:51:44 GMT
< Server: inets/6.4.5
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: Wed, 28 Feb 2018 13:51:44 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html
< 
Hello, esi!
* Connection #0 to host localhost left intact

mylog.log:

Inside mymod:get_data()
SessionId=<0.108.0>
Env=[{server_software,"inets/6.4.5"},
     {server_name,"httpd_test"},
     {host_name,"ChristophersMBP"},
     {gateway_interface,"CGI/1.1"},
     {server_protocol,"HTTP/1.1"},
     {server_port,59202},
     {request_method,"POST"},
     {remote_addr,"127.0.0.1"},
     {peer_cert,undefined},
     {script_name,"/erl/mymod:get_data"},
     {http_host,"localhost:59202"},
     {http_user_agent,"curl/7.58.0"},
     {http_accept,"*/*"},
     {http_content_length,"7"},
     {http_content_type,"application/x-www-form-urlencoded"}]
Input="a=1&b=2"
-------

Request 4 (cgi get request):

~$  curl -v "http://localhost:59202/cgi-bin/1.pl"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 59202 (#0)
> GET /cgi-bin/1.pl HTTP/1.1
> Host: localhost:59202
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2018 13:41:43 GMT
< Server: inets/6.4.5
< Transfer-Encoding: chunked
< Content-Type: text/html
< 
Hello, Perl.
* Connection #0 to host localhost left intact

Request 5 (get request for regular file from document_root directory):

~$  curl -v "http://localhost:59202/file1.txt"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 59202 (#0)
> GET /file1.txt HTTP/1.1
> Host: localhost:59202
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2018 13:42:15 GMT
< Server: inets/6.4.5
< Content-Type: text/plain
< Etag: nCZT0114
< Content-Length: 14
< Last-Modified: Mon, 26 Feb 2018 02:51:52 GMT
< 
Hello, world!
* Connection #0 to host localhost left intact