Lua - Concatenation of variables (pairs) escaping special characters - Openresty Nginx

Question

I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.

Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.

This answer shows how to concatenate headers into a string $request_headers:

set_by_lua $request_headers '
  local h = ngx.req.get_headers()
  local request_headers_all = ""
  for k, v in pairs(h) do
    request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
  end
  return request_headers_all
';

I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.

Log format:

log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**

Host: "192.168.1.4", Via : "-", 
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one

When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.

I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?

What is the "..variablename.." doing, is this how variables are always used inside Lua strings?

How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?

Answer 1

The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).

Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.

I will attempt to update the title to reflect more specific requirements

TLDR

0. Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
1. Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
2. Update nginx configuration with your custom/existing log_format and add the switch escape=none.

Full Explanation

Investigating the provided link answer set_by_lua function documentation :

NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.

So from the original set_by_lua from the link:

set_by_lua $request_headers '
  return 'stringvalue'
';

I changed to set_by_lua_block function:

this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)

set_by_lua_block $request_headers{
  local h = ngx.req.get_headers()
  local request_headers_all = ""
  for k, v in pairs(h) do
    local rowtext = ""
    rowtext = string.format("[%s %s]\n", k, v)
    request_headers_all = request_headers_all .. rowtext

  end
  return request_headers_all
}

The important part is that this _block {} function escapes the special characters correctly.

After that I received output in the log files as : x0A (newline character literal).

The final step then is to update the nginx.conf file with the custom log_format and add escape=none:

log_format log_realip escape=none "$nginx_variables"