openresty 1.15.8.1 ngx.re can't work

[vinsonzou]

• Kernel Version: 2.6.32-358.el6.x86_64
• build_info:

nginx version: openresty/1.15.8.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.1rc1 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.15 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.7 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module

=== TEST 1:
--- config
    location = /re {
        content_by_lua_block {
            local ngx_re = require "ngx.re"

            local res, err = ngx_re.split("a,b,c,d", ",")
            if err then
                ngx.log(ngx.ERR, "failed: ", err)
                return
            end

            for i = 1, #res do
                ngx.say(res[i])
            end
        }
    }

--- Error log
2019/05/19 17:56:34 [error] 11838#0: *85 lua entry thread aborted: runtime error: /usr/local/openresty//lualib/ngx/re.lua:47: missing declaration for symbol 'ngx_http_lua_ffi_exec_regex'
stack traceback:
coroutine 0:
	[C]: in function 'require'
	content_by_lua(test.conf:22):2: in main chunk, client: 127.0.0.1, server: 127.0.0.1, request: "GET /re HTTP/1.1", host: "127.0.0.1:8080"

[agentzh]

@vinsonzou Seems like your nginx build lacks PCRE? Try the command to check:

ldd /usr/local/openresty/nginx/nginx/sbin/nginx | grep -i pcre

Also, it's recommended to use our official binary packages of OpenResty:

https://openresty.org/en/linux-packages.html

[vinsonzou]

@agentzh tks. I can't build with --with-pcre .

[agentzh]

@vinsonzou You don't have to build with --with-pcre, just install the pcre-dev or pcre-devel system package before building.

[vinsonzou]

@agentzh system pcre-devel can't support utf-8 and jit.
The openresty is not support buildin --with-pcre=the dir ?

nginx version: openresty/1.15.8.1
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC)
built with OpenSSL 1.1.0j  20 Nov 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.1rc1 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.15 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.7 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-select_module --without-http_scgi_module --with-http_stub_status_module --with-http_gzip_static_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_sub_module --with-pcre=/tmp/pcre-8.41 --with-pcre-jit --with-pcre-conf-opt='--enable-utf --enable-unicode-properties' --with-openssl=/tmp/openssl-1.1.0j --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module

Error output

--- Error log
2019/05/22 18:30:02 [error] 11838#0: *85 lua entry thread aborted: runtime error: /usr/local/openresty//lualib/ngx/re.lua:47: missing declaration for symbol 'ngx_http_lua_ffi_exec_regex'
stack traceback:
coroutine 0:
	[C]: in function 'require'
	content_by_lua(test.conf:22):2: in main chunk, client: 127.0.0.1, server: 127.0.0.1, request: "GET /re HTTP/1.1", host: "127.0.0.1:8080"

[agentzh]

@vinsonzou It does:

$ ./configure --help | grep pcre
  --without-pcre                     disable PCRE library usage
  --with-pcre                        force PCRE library usage
  --with-pcre=DIR                    set path to PCRE library sources
  --with-pcre-opt=OPTIONS            set additional make options for PCRE
  --with-pcre-conf-opt=OPTIONS       set additional configure options for PCRE
  --with-pcre-jit                    build PCRE with JIT compilation support

You can always find it out yourself this way.

[agentzh]

BTW, you can also specify the installation tree location of your own build of PCRE, just as in our openresty rpm packaging process:

https://github.com/openresty/openresty-packaging/blob/master/rpm/SPECS/openresty.spec#L145

[thibaultcha]

Aside: There is a lack of testing for PCRE support in ngx.re. It has always been like so. 30a7266 ensured that resty.core could be loaded in binaries without PCRE support, but ngx.re is an optional module, and doesn't have such a test. Patch incoming.

That said @vinsonzou, you won't be able to use ngx.re if you don't link OpenResty to libpcre, as @agentzh instructed.

[thibaultcha]

A simple patch has been proposed at #259

[vinsonzou]

@thibaultcha @agentzh tks.

[huntkalio]

I have the same problem with building with --with-pcre

2019/06/12 11:06:06 [error] 3936#0: *126458 lua entry thread aborted: runtime error: /opt/openresty/lualib/ngx/re.lua:47: missing declaration for symbol 'ngx_http_lua_ffi_exec_regex' stack traceback:coroutine 0: [C]: in function 'require'

if we build openresty with --with-pcre (use static lib),then there will no function C.pcre_version() in nginx .So,should we not use pcre_version in regex.lua ? @agentzh @thibaultcha

[alubbe]

We actually now get the same error message when compiling openresty with --with-pcre=DIR after upgrading to the current version. Did something change?

[thibaultcha]

@alubbe @huntkalio There is indeed an issue in 1.15.8.1 when statically linking to libpcre. The workarounds are:

• Dynamically link against libpcre; e.g. by specifying --with-pcre (note: without a path)
• By statically linking against libpcre (--with-pcre=DIR), but with the following flag (at least for GNU ld): --with-ld-opt="-Wl,-u,pcre_version"

The fix is already in ngx_http_lua's master branch (see openresty/lua-nginx-module@2014dd8), but still has some issues on Darwin platforms. I just pushed openresty/lua-nginx-module#1577 yesterday evening which brings many improvements to our config script and fixes all of those issues at least on Fedora and Darwin. It should work if you drop it into any 1.15.8.x OpenResty release in-lieu of the bundled one.

[thibaultcha]

The above PR also fixes other cases such as --without-pcre or unspecified --with-pcre flags, but including modules silently requiring libpcre (e.g. ngx_http_rewrite_module). It also fixes the detection of LuaJIT with FFI (something that isn't included in 1.15.8.x, but will ship in 1.17.x.1 as we retire the old CFunction API).

[thibaultcha]

One more thing: in upcoming releases (1.17.x.1+), the ngx.re module will gracefully error out when NGINX is compiled without libpcre: 724ef60. This wasn't the case before because we lack test cases for builds without libpcre in our CI (I will add them, as well as many other edge-case compilation scenarios in the future).

[alubbe]

We're using ubuntu and are already setting --with-ld-opt='-L/usr/local/lib' - will that workaround help us and how do we add it together? Is it just this?

--with-ld-opt='-L/usr/local/lib,-Wl,-u,pcre_version'

[alubbe]

by the way, thank you so much for 724ef60 and your amazingly fast response time here!

[alubbe]

or... could we just apply this patch openresty/lua-nginx-module@2014dd8 before compiling? any reason why that might not work on ubuntu?

[thibaultcha]

This should work:

--with-ld-opt='-L/usr/local/lib -Wl,-u,pcre_version'

There is an issue with openresty/lua-nginx-module@2014dd8 (which is fixed by openresty/lua-nginx-module#1577) when specifying --without-pcre (this is one of the edge-cases the PR fixes). I'd recommend specifying the linker flag yourself for the time being, which is much simpler than having to maintain a patch on your end.

[huntkalio]

many thanks for your reply!

[alubbe]

It turned out to be easier to apply patches (724ef60 & openresty/lua-nginx-module@2014dd8) rather than changing the flags in our build setup, but I can confirm that it's working again - thank you so much!