Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to run privileged nested container, on fedora 41 #24692

Open
Yarboa opened this issue Nov 27, 2024 · 5 comments · May be fixed by #24696
Open

Fail to run privileged nested container, on fedora 41 #24692

Yarboa opened this issue Nov 27, 2024 · 5 comments · May be fixed by #24696
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Yarboa
Copy link

Yarboa commented Nov 27, 2024 •
edited
Loading

Issue Description

Describe your issue

Fail to run privileged nested container, on fedora 41

Steps to reproduce the issue

Steps to reproduce the issue

  1. Fedora41
  2. dnf copr enable -y @centos-automotive-sig/bluechi-snapshot "Fedora-41-$(arch)"
  3. dnf copr enable -y rhcontainerbot/qm "fedora-41-$(arch)"
  4. dnf install -y qm
  5. Now, need to run emulated setup script to install qm systemd and podman in QM rootfs.
    bash -x /usr/share/qm/setup
  6. Verify qm service is up
    systemctl is-acive qm 
    podman ps shows qm enviroment is running.
  1. Run nested container, rootfull
    podman exec qm podman run ubi9-minimal pwd

Describe the results you received

Describe the results you received

    Resolved "ubi9-minimal" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
    Trying to pull registry.access.redhat.com/ubi9-minimal:latest...
    Getting image source signatures
    Writing manifest to image destination
    Storing signatures
Error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI permission denied

Describe the results you expected

Describe the results you expected
Nested container should run without failures.host-podman-info.txt
qm-podman-info.txt

Nested podman container is run successfully for c9s
https://artifacts.dev.testing-farm.io/0d73bf4f-14b2-43c9-8bcf-27b07f3406fb/work-kvm-tier-05glnpgbx/plans/e2e/kvm-tier-0/execute/data/guest/default-0/tests/qm-sanity-test/check_qm_podman_quadlet_is_ok-4/output.txt

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

*podman info*

Please see attached files
host-podman-info.txt
attached file for qm
qm-podman-info.txt

*podman version*

rpm -q podman
podman-5.3.1-1.fc41.x86_64

podman exec qm rpm -q podman
podman-5.3.1-1.fc41.x86_64


*operating system* 
Fedora
*version*
41, uname -r
6.11.8-300.fc41.x86_64

architecture
x86_64

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

To understand tested enviroment please refer this README
https://github.com/containers/qm?tab=readme-ov-file#qm-is-a-containerized-environment-for-running-functional-safety-qm-quality-management-software

Additional information

more details exist here
qm#660

This one containers/qm#666 (comment)
Did not help to solve the issue
@Yarboa Yarboa added the kind/bug Categorizes issue or PR as related to a bug. label Nov 27, 2024
@Yarboa Yarboa mentioned this issue Nov 27, 2024
Open
@Luap99
Copy link
Member

Luap99 commented Nov 27, 2024

Can you provide the actual podman commands used please.

@Yarboa
Copy link
Author

Yarboa commented Nov 27, 2024 •
edited
Loading

Sure see Steps to reproduce the issue

But the command itself is step number 7

podman exec qm podman run ubi9-minimal pwd

qm is running with the following quadlet
https://github.com/containers/qm/blob/main/qm.container

The issue is nested container inside qm
When remove all PodmanArgs and use --priviliged in quadlet file nested container is forked successfully

In case you need ready env in TestingFarm, i can prepare it for you

@Luap99
Copy link
Member

Luap99 commented Nov 27, 2024

LimitNOFILE=65536

Ok I see, your outer container sets a low file limit and you do not give the container CAP_SYS_RESOURCE so it cannot increase it. Since commit
5ebba75 we try to set a static default limit. So you either need to use a higher limit for the outer container or use a different limit for the inner container, either via --ulimit or there should be a containers.conf default I think.

The alternative is for podman to not assume we can increase the limits as root (check for CAP_SYS_RESOURCE) and if not just clamp it like with the rootless case: #24508 (comment)

cc @giuseppe

giuseppe added a commit to giuseppe/libpod that referenced this issue Nov 27, 2024
@giuseppe
spec: clamp rlimits without CAP_SYS_RESOURCE
c447288 
commit 5ebba75 implemented this
behaviour for rootless users and later commit
0a69aef changed it when in a user
namespace, but the same limitation exists for root without
CAP_SYS_RESOURCE.  Change the check to use the clamp to the current
values if running without CAP_SYS_RESOURCE.

Closes: containers#24692

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe linked a pull request Nov 27, 2024 that will close this issue
Open
@giuseppe
Copy link
Member

Ok I see, your outer container sets a low file limit and you do not give the container CAP_SYS_RESOURCE so it cannot increase it. Since commit 5ebba75 we try to set a static default limit. So you either need to use a higher limit for the outer container or use a different limit for the inner container, either via --ulimit or there should be a containers.conf default I think.

The alternative is for podman to not assume we can increase the limits as root (check for CAP_SYS_RESOURCE) and if not just clamp it like with the rootless case: #24508 (comment)

yes good analysis.

I've opened a PR to fix it: #24696

giuseppe added a commit to giuseppe/libpod that referenced this issue Nov 27, 2024
@giuseppe
spec: clamp rlimits without CAP_SYS_RESOURCE
4b38031 
commit 5ebba75 implemented this
behaviour for rootless users and later commit
0a69aef changed it when in a user
namespace, but the same limitation exists for root without
CAP_SYS_RESOURCE.  Change the check to use the clamp to the current
values if running without CAP_SYS_RESOURCE.

Closes: containers#24692

Signed-off-by: Giuseppe Scrivano <[email protected]>
@Yarboa
Copy link
Author

Yarboa commented Nov 27, 2024

@Luap99 thanks

Add this to quadlet drop-in

cat /etc/containers/systemd/qm.container.d/add_resource.conf 
[Container]
DropCapability=
AddCapability=sys_resource

Did the trick

So to understand the ulimit explanation
I see that in outer container i have this
open files (-n) 1024

and in internal container
open files (-n) 524288

So do i have to change one of those?
Thanks again

giuseppe added a commit to giuseppe/libpod that referenced this issue Nov 27, 2024
@giuseppe
spec: clamp rlimits without CAP_SYS_RESOURCE
4b38294 
commit 5ebba75 implemented this
behaviour for rootless users and later commit
0a69aef changed it when in a user
namespace, but the same limitation exists for root without
CAP_SYS_RESOURCE.  Change the check to use the clamp to the current
values if running without CAP_SYS_RESOURCE.

Closes: containers#24692

Signed-off-by: Giuseppe Scrivano <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

spec: clamp rlimits without CAP_SYS_RESOURCE giuseppe/libpod
3 participants
@giuseppe @Yarboa @Luap99