Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable Redis Protected Mode #194

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

esinx
Copy link
Member

@esinx esinx commented Mar 21, 2024

Courses backend was having issues with redis connectivity, turns out that redis base image has protected mode enabled by default. Fixed redis config map to disable protected mode!

@esinx esinx requested a review from joyliu-q March 21, 2024 02:47
@joyliu-q
Copy link
Contributor

joyliu-q commented Mar 21, 2024

Is there any reason this is just recently showing up, and is courses specific? Why do we need external network exccess?

Protected mode
Unfortunately, many users fail to protect Redis instances from being accessed from external networks. Many instances are simply left exposed on the internet with public IPs. Since version 3.2.0, Redis enters a special mode called protected mode when it is executed with the default configuration (binding all the interfaces) and without any password in order to access it. In this mode, Redis only replies to queries from the loopback interfaces, and replies to clients connecting from other addresses with an error that explains the problem and how to configure Redis properly.
We expect protected mode to seriously decrease the security issues caused by unprotected Redis instances executed without proper administration. However, the system administrator can still ignore the error given by Redis and disable protected mode or manually bind all the interfaces.

@esinx
Copy link
Member Author

esinx commented Mar 22, 2024

It's not that this issue has appeared from nowhere, this issue persisted after merging in kittyhawk bump to 1.1.11.

We just rolled back to 1.1.9 whenever this happened and that's why we never saw this happening so often, but then we actually wanted to apply the new updates in our master branch and deploy them, which consequently blocked us from "rolling back to 1.1.9".

As to why this is cousres-specific, I have no clue (yet). We will find out soon. But as speculated before, it could be that:

  1. Redis is running as host mode(unlikely, but could happen) and clubs/ohq backend pods are spawned into the same instance, making the access originate from loopback interfaces.

  2. We have Redis error silenced in clubs/ohq and we never noticed(also unlikely because we were able to confirm that clubs django is caching into redis properly)

@shiva-menta
Copy link

It's not that this issue has appeared from nowhere, this issue persisted after merging in kittyhawk bump to 1.1.11.

We just rolled back to 1.1.9 whenever this happened and that's why we never saw this happening so often, but then we actually wanted to apply the new updates in our master branch and deploy them, which consequently blocked us from "rolling back to 1.1.9".

As to why this is cousres-specific, I have no clue (yet). We will find out soon. But as speculated before, it could be that:

  1. Redis is running as host mode(unlikely, but could happen) and clubs/ohq backend pods are spawned into the same instance, making the access originate from loopback interfaces.
  2. We have Redis error silenced in clubs/ohq and we never noticed(also unlikely because we were able to confirm that clubs django is caching into redis properly)

Just adding context here based on a conversation on Slack, the difference is that PCx products uses Redis Stack, when we were planning on using Redis-Search functionality. Because that's not our image most likely and Redis Stack has more features, it likely has protection mode set to on by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants