Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use (L)MDB backend instead of HDB #8

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

sergiodj
Copy link

OpenLDAP 2.5.x has deprecated and removed support for the HDB backend,
which makes the tests fail. This commit changes the backend to MDB,
which is the recommended one now.

OpenLDAP 2.5.x has deprecated and removed support for the HDB backend,
which makes the tests fail.  This commit changes the backend to MDB,
which is the recommended one now.

Closes rbarrois#7

Signed-off-by: Sergio Durigan Junior <[email protected]>
@sergiodj sergiodj force-pushed the move-from-hdb-to-mdb-openldap-2.5 branch from 09a9754 to 42baf3b Compare June 25, 2021 22:04
@sergiodj
Copy link
Author

Closes #7

@rbarrois
Copy link
Owner

rbarrois commented Jun 28, 2021

This seems to fail on a local setup with openldap-2.4; have you been able to run the tests on your end?

Output:

% make test
python -Wdefault -m unittest discover tests
....config file testing succeeded
.config file testing succeeded
127.0.0.1 - - [28/Jun/2021 19:00:11] "GET /config/ HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:11] "POST /entry/ HTTP/1.1" 201 -
127.0.0.1 - - [28/Jun/2021 19:00:11] "GET /entry/ou=people HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:11] "POST /control/stop/ HTTP/1.1" 204 -
.config file testing succeeded
127.0.0.1 - - [28/Jun/2021 19:00:11] "GET /config/ HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:11] "GET /entry/ou=people HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:11] "POST /control/stop/ HTTP/1.1" 204 -
.config file testing succeeded
127.0.0.1 - - [28/Jun/2021 19:00:12] "GET /config/ HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:12] "POST /control/stop/ HTTP/1.1" 204 -
.config file testing succeeded
127.0.0.1 - - [28/Jun/2021 19:00:12] "GET /config/ HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:12] "POST /entry/ HTTP/1.1" 201 -
127.0.0.1 - - [28/Jun/2021 19:00:12] "GET /entry/ou=people HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 47722)
Traceback (most recent call last):
  File "/usr/lib/python3.7/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.7/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.7/socketserver.py", line 720, in __init__
    self.handle()
  File "/usr/lib/python3.7/http/server.py", line 426, in handle
    self.handle_one_request()
  File "/usr/lib/python3.7/http/server.py", line 414, in handle_one_request
    method()
  File "/home/xelnor/dev/public/volatildap/volatildap/control.py", line 64, in do_POST
    self.ldap.reset()
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 244, in reset
    self._clear()
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 379, in _clear
    raise RuntimeError("ldapdelete failed with code %d: %r" % (retcode, stderr))
RuntimeError: ldapdelete failed with code 32: b'ldap_delete: No such object (32)\n'
----------------------------------------
127.0.0.1 - - [28/Jun/2021 19:00:12] "POST /control/stop/ HTTP/1.1" 204 -
Econfig file testing succeeded
127.0.0.1 - - [28/Jun/2021 19:00:13] "GET /config/ HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2021 19:00:13] "POST /control/stop/ HTTP/1.1" 204 -
.config file testing succeeded
Error starting LDAP server: ldapdelete failed with code 32: b'ldap_delete: No such object (32)\n'
Traceback (most recent call last):
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 164, in start
    self._clear()
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 379, in _clear
    raise RuntimeError("ldapdelete failed with code %d: %r" % (retcode, stderr))
RuntimeError: ldapdelete failed with code 32: b'ldap_delete: No such object (32)\n'
Econfig file testing succeeded
.config file testing succeeded
.config file testing succeeded
.config file testing succeeded
Econfig file testing succeeded
.
======================================================================
ERROR: test_reset (test_usage.ControlTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1369, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 279, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1369, in getresponse
    response.begin()
  File "/usr/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.7/http/client.py", line 279, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xelnor/dev/public/volatildap/tests/test_usage.py", line 240, in test_reset
    self.proxy.reset()
  File "/home/xelnor/dev/public/volatildap/volatildap/control.py", line 159, in reset
    response = requests.post(self._path('control/reset/'))
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/xelnor/dev/venvs/volatildap/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

======================================================================
ERROR: test_clear_data (test_usage.ReadWriteTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xelnor/dev/public/volatildap/tests/test_usage.py", line 105, in test_clear_data
    self.server.start()  # Actually a restart
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 164, in start
    self._clear()
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 379, in _clear
    raise RuntimeError("ldapdelete failed with code %d: %r" % (retcode, stderr))
RuntimeError: ldapdelete failed with code 32: b'ldap_delete: No such object (32)\n'

======================================================================
ERROR: test_cleanup (test_usage.ResetTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xelnor/dev/public/volatildap/tests/test_usage.py", line 133, in test_cleanup
    self.server.reset()
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 244, in reset
    self._clear()
  File "/home/xelnor/dev/public/volatildap/volatildap/server.py", line 379, in _clear
    raise RuntimeError("ldapdelete failed with code %d: %r" % (retcode, stderr))
RuntimeError: ldapdelete failed with code 32: b'ldap_delete: No such object (32)\n'

----------------------------------------------------------------------
Ran 16 tests in 6.398s

FAILED (errors=3)
make: *** [Makefile:27: test] Error 1

@sergiodj
Copy link
Author

This seems to fail on a local setup with openldap-2.4; have you been
able to run the tests on your end?

Hm, indeed, I can confirm the failure if I run the test with OpenLDAP
2.4. I apologize for not catching it sooner; I've been testing things
only with OpenLDAP 2.5.

It's not immediately clear to me why this is failing and how to fix
it. I will have to investigate more, but I don't have a lot of time
right now. This may take a while :-/. I will try to talk to @rtandy
(who is the OpenLDAP maintainer on Debian) and see if he's got any
ideas.

@rbarrois
Copy link
Owner

Thanks for the quick feedback! I'll try to setup some automated CI on the package to detect that sooner — and I'll look into the lack of meaningful details here, likely related to not properly propagating the slapd logs. Quite busy on my end too, but I hope to have something up by the end of the week.

@sergiodj
Copy link
Author

Ah, awesome, thank you very much! I'll keep you posted if/when I find
something interesting.

@rtandy
Copy link

rtandy commented Jun 30, 2021

Hi. The root cause appears to be ITS#9531. It's already fixed in the current OpenLDAP releases (2.4.59 and 2.5.5).

The tests succeed for me with the 2.5.5 package from Debian experimental and @sergiodj's patch. A local build based on current Debian unstable (2.4.57) with the ITS#9531 patch applied also works for me.

Do you need me to backport the fix to Debian testing (bullseye) and/or stable (buster)? The patch applies cleanly, but I would have to justify the need for it to the release managers.

@@ -144,8 +144,8 @@ def quote(base, *args):
yield quote('TLSCertificateFile %s', self._tls_certificate_path)
yield quote('TLSCertificateKeyFile %s', self._tls_key_path)

yield quote('moduleload back_hdb')
yield quote('database hdb')
yield quote('moduleload back_mdb')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering whether mapsize should also be set when using this backend. The default is capacity is 10 MiB (source).

@sergiodj
Copy link
Author

Hi. The root cause appears to be
ITS#9531. It's
already fixed in the current OpenLDAP releases (2.4.59 and 2.5.5).

Ah, thanks a lot for checking this, Ryan!

Do you need me to backport the fix to Debian testing (bullseye) and/or
stable (buster)? The patch applies cleanly, but I would have to
justify the need for it to the release managers.

I don't know if you will be able to justify backporting this patch to
bullseye (it doesn't seem like a proper candidate for a
release-critical bug), but you can try. Let me know if you need any
help.

@rbarrois
Copy link
Owner

rbarrois commented Jul 1, 2021

Oh, thanks for the analysis, that's quite helpful!

I'll look into this on two sides:

  1. Making sure we can get a details error trace from slapd;
  2. Exploring a workaround to not have to delete the original entry

@sandre35
Copy link

sandre35 commented Aug 9, 2023

Hello,

I am having the same issue, using volatildap with OpenLdap 2.5. As of now, this PR solves issues I am facing.

Is there any additional check to be made concerning this PR ?

Thanks,

@RemiCardona
Copy link

hi all, a friendly bump :)

is there anything we can do to help?

cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants