Redash to MongoDB v6 connection fails

[nzidol]

Trying to connect to my local mongdb (v6.0.3) gives this error message:

Connection Test Failed:
Unsupported OP_QUERY command: connectionStatus. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal

[justinclift]

Yeah, the version of the MongoDB driver that we use doesn't support MongoDB 6.x.

We need to update the driver, which will then allow that to work. There's some other stuff we need to do first though (being worked on), which is why it hasn't happened yet.

Check back in a few days maybe, to see where things are up to then? 😄

[justinclift]

@nzidol We have a potential update for this almost ready, but would like someone that uses MongoDB 6 to try it and make sure the data source is working ok.

So, we'd need that person to install a Redash image (should be ready shortly), then try it out with their MongoDB 6.x data source, then let us know if it does or doesn't work (etc).

Is that something you'd be ok doing?

[nzidol]

Yeah no problem

…

On Fri, 30 Jun 2023, 9:07 pm Justin Clift, ***@***.***> wrote: @nzidol <https://github.com/nzidol> We have a potential update for this almost ready, but would like someone that uses MongoDB 6 to try it and make sure the data source is working ok. So, we'd need that person to install a Redash image (should be ready shortly), then try it out with their MongoDB 6.x data source, then let us know if it does or doesn't work (etc). Is that something you'd be ok doing? — Reply to this email directly, view it on GitHub <#6131 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADH46HLAPVYKHZQMAA6WYMTXN2JN3ANCNFSM6AAAAAAZX5Y3QI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[obscure723]

I'm also facing the same issue. If you don't mind, I'd like to contribute to the testing as well.

[jesraygarciano]

@justinclift
I am experiencing the same issue on "Unsupported OP_QUERY command: connectionStatus". If possible, can we have it fixed? I can also help with testing

[justinclift]

Oops, I got distracted with other stuff. The fix for this (seems like it's pretty good) got merged into our master branch a few days ago: 24b6ef7

We've disabled the preview builds in Docker Hub for now while we're still sorting out a few other things though.

I'll look into manually generating a Docker image people can test out, though I'll likely upload it online somewhere for manual download + import (eg docker image load -i somefile.tar).

Once we re-enable the preview uploads on Docker Hub, things will be easier. 😄

[obscure723]

thank you for your reply.
I will check the operation after this. I will contact you if there are any problems.
thank you.

[justinclift]

I'll hold off on creating the docker image for testing for today, as the comment below indicates it probably isn't working yet... and I'm getting pretty tired today.

[obscure723]

I've checked several replies, and it is now working properly. Thank you.

[jesraygarciano]

@justinclift Thank you, Can I ask when can I see the preview version on Docker Hub?

[justinclift]

We're being careful, so maybe 1-2 weeks away? Probably closer to the 2 weeks than 1 though.

[jesraygarciano]

I tried to create my own docker image using the Redash master branch to check if I could try to fix "Unsupported OP_QUERY command:". Then tried to query I'm not seeing the error anymore, not sure if it is fixed but now I'm getting "Query on queue" where I get no response.

[justinclift]

Interesting. Maybe the PR we merged didn't include all the needed changes after all. Damn. 😦

It could also be something else going on too. We'll likely have a better idea in a few days time, as we go through some other things. 😄

[jesraygarciano]

Here's my docker-compose kinda look like where I get, "Query on queued" or no response.

version: "3.8"

services:

    stripe:
        image: stripe/stripe-cli:latest
        container_name: stripe
        environment:
            STRIPE_API_KEY: ${STRIPE_API_KEY} 
            STRIPE_DEVICE_NAME: ${STRIPE_DEVICE_NAME}
        command: listen --forward-to fastapi:8000/webhook
        depends_on:
            - fastapi
        networks:
            - backend

    adminer:
        image: adminer
        container_name: adminer
        restart: always
        ports:
            - 8080:8080
        networks:
            - backend

    mongo:
        image: mongo
        container_name: mongo
        restart: always
        ports:
            - 27018:27017
        volumes:
            - mongodb_data:/data/db
            - mongodb_config:/data/configdb
        networks:
            - backend
        environment:
            MONGO_INITDB_ROOT_USERNAME: wati
            MONGO_INITDB_ROOT_PASSWORD: pass

    mongo-express:
        image: mongo-express:0.54.0
        container_name: mongo-express
        restart: always
        ports:
            - 8082:8081
        networks:
            - backend
        environment:
            ME_CONFIG_MONGODB_ADMINUSERNAME: wati
            ME_CONFIG_MONGODB_ADMINPASSWORD: pass
            ME_CONFIG_MONGODB_URL: mongodb://wati:pass@mongo:27017/
        depends_on:
            - mongo

    mailhog:
        image: mailhog/mailhog:latest
        container_name: mailhog
        ports:
            - "1025:1025"
            - "8025:8025"
        networks:
            - backend
        volumes:
            - mailhog_data:/var/mail

    metabase:
        image: metabase/metabase
        container_name: metabase
        ports:
            - "3003:3000"
        volumes:
            - ./metabase-data:/metabase-data
        environment:
            MB_DB_FILE: /metabase-data/metabase.db
            JAVA_TOOL_OPTIONS: "-XX:+IgnoreUnrecognizedVMOptions"
        networks:
            - backend

    redash:
        image: redash/redash:latest
        container_name: redash
        ports:
            - "5005:5000"
        depends_on:
            - postgres
            - redis
            - mongo
        environment:
            REDASH_DATABASE_URL: postgresql://postgres:password@postgres/postgres
            REDASH_REDIS_URL: redis://redis:6379/0
            CELERYD_PREFETCH_MULTIPLIER: 1
        networks:
            - backend
        volumes:
            - redash_data:/var/lib/postgresql/data

    adhocWorker:
        image: redash/redash:10.0.0.b50363
        command: worker
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "queries,celery,schemas,default,periodic"
            WORKERS_COUNT: 2
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    worker:
        image: redash/redash:10.0.0.b50363
        command: worker
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "queries" #scheduled_queries,celery
            WORKERS_COUNT: 4
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    scheduled_worker:
        image: redash/redash:10.0.0.b50363
        command: worker
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "scheduled_queries,schemas"
            WORKERS_COUNT: 2
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    scheduler:
        image: redash/redash:10.0.0.b50363
        command: scheduler
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "celery"
            WORKERS_COUNT: 2
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    postgres:
        image: postgres:9.6-alpine
        container_name: postgres
        volumes:
            - postgres_data:/var/lib/postgresql/data
        networks:
            - backend
        environment:
            POSTGRES_PASSWORD: password

    redis:
        image: redis:5.0-alpine
        container_name: redis
        networks:
            - backend

networks:
    backend:


volumes:
    mongodb_data:
    mongodb_config:
    mailhog_data:
    metabase-data:
    redash_data:
    postgres_data:

[justinclift]

Oh, that one's definitely not quite right.

The Redash image is spread across that file in a few places, but you've only updated the one in the redash section. You'll need to update all the other mentions of the redash image too.

For example:

adhocWorker:
    image: redash/redash:10.0.0.b50363

(there are several redash ones, they need updating)

[jesraygarciano]

Okay, thank you for this. I'll update it from your suggestion. Better If I can hopefully see your preview image anytime soon😄

[nzidol]

Hi,

I build a dev instance as per https://redash.io/help/open-source/dev-guide/docker and can confirm connection to MongoDB V6 now works with version 11.0.0-dev

Note that I updated requirements_dev.txt to install watchdog=3.0.0 instead of 0.0.9 otherwise redash-server-1 would crash on startup on an issue with /usr/local/lib/python3.8/site-packages/watchdog/events.py "ImportError: cannot import name 'EVENT_TYPE_OPENED' from 'watchdog.events'"

Note that within Docker there needs to be an internal network between redash-server/Redash-worker and mongodb to allow connection.

Connection string I used is mongodb://mongodb-mongodb-1:27017/admin?authmechanism=SCRAM-SHA-256

[nzidol]

docker images used are hub.docker.com
idonot/redash-server:ido-11.0.0.dev
idonot/redash-worker:ido-11.0.0.dev
idonot/redash-scheduler:ido-11.0.0.dev

[justinclift]

@nzidol Good stuff, thanks for getting that done.

Sounds like we'd better take a look at that watchdog dependency too.

[justinclift]

@nzidol Created #6161 for that watchdog update.

[justinclift]

@nzidol This might be useful info for setting up a dev environment too:

https://github.com/getredash/redash/wiki/Local-development-setup

[jesraygarciano]

With the images you pushed, does below docker-compose yml below correctly implemented with your images.

version: "3.8"

services:

    stripe:
        image: stripe/stripe-cli:latest
        container_name: stripe
        environment:
            STRIPE_API_KEY: ${STRIPE_API_KEY} 
            STRIPE_DEVICE_NAME: ${STRIPE_DEVICE_NAME}
        command: listen --forward-to fastapi:8000/webhook
        depends_on:
            - fastapi
        networks:
            - backend

    adminer:
        image: adminer
        container_name: adminer
        restart: always
        ports:
            - 8080:8080
        networks:
            - backend

    mongo:
        image: mongo
        container_name: mongo
        restart: always
        ports:
            - 27018:27017
        volumes:
            - mongodb_data:/data/db
            - mongodb_config:/data/configdb
        networks:
            - backend
        environment:
            MONGO_INITDB_ROOT_USERNAME: wati
            MONGO_INITDB_ROOT_PASSWORD: pass

    mongo-express:
        image: mongo-express:0.54.0
        container_name: mongo-express
        restart: always
        ports:
            - 8082:8081
        networks:
            - backend
        environment:
            ME_CONFIG_MONGODB_ADMINUSERNAME: wati
            ME_CONFIG_MONGODB_ADMINPASSWORD: pass
            ME_CONFIG_MONGODB_URL: mongodb://wati:pass@mongo:27017/
        depends_on:
            - mongo

    mailhog:
        image: mailhog/mailhog:latest
        container_name: mailhog
        ports:
            - "1025:1025"
            - "8025:8025"
        networks:
            - backend
        volumes:
            - mailhog_data:/var/mail

    metabase:
        image: metabase/metabase
        container_name: metabase
        ports:
            - "3003:3000"
        volumes:
            - ./metabase-data:/metabase-data
        environment:
            MB_DB_FILE: /metabase-data/metabase.db
            JAVA_TOOL_OPTIONS: "-XX:+IgnoreUnrecognizedVMOptions"
        networks:
            - backend

    redash:
        image: idonot/redash-server:ido-11.0.0.dev
        container_name: redash
        ports:
            - "5005:5000"
        depends_on:
            - postgres
            - redis
            - mongo
        environment:
            REDASH_DATABASE_URL: postgresql://postgres:password@postgres/postgres
            REDASH_REDIS_URL: redis://redis:6379/0
            CELERYD_PREFETCH_MULTIPLIER: 1
        networks:
            - backend
        volumes:
            - redash_data:/var/lib/postgresql/data

    adhocWorker:
        image: idonot/redash-worker:ido-11.0.0.dev
        command: worker
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "queries,celery,schemas,default,periodic"
            WORKERS_COUNT: 2
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    worker:
        image: idonot/redash-worker:ido-11.0.0.dev
        command: worker
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "queries" #scheduled_queries,celery
            WORKERS_COUNT: 4
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    scheduled_worker:
        image: idonot/redash-scheduler:ido-11.0.0.dev
        command: worker
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "scheduled_queries,schemas"
            WORKERS_COUNT: 2
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    scheduler:
        image: idonot/redash-scheduler:ido-11.0.0.dev
        command: scheduler
        environment:
            PYTHONUNBUFFERED: 0
            REDASH_LOG_LEVEL: "INFO"
            REDASH_REDIS_URL: "redis://redis:6379/0"
            REDASH_DATABASE_URL: "postgresql://postgres:password@postgres/postgres"
            QUEUES: "celery"
            WORKERS_COUNT: 2
        depends_on:
            - redis
            - postgres
            - mongo
        networks:
            - backend

    postgres:
        image: postgres:9.6-alpine
        container_name: postgres
        volumes:
            - postgres_data:/var/lib/postgresql/data
        networks:
            - backend
        environment:
            POSTGRES_PASSWORD: password

    redis:
        image: redis:5.0-alpine
        container_name: redis
        networks:
            - backend

networks:
    backend:


volumes:
    mongodb_data:
    mongodb_config:
    mailhog_data:
    metabase-data:
    redash_data:
    postgres_data:

[nzidol]

Maybe, these are dev-images so normal docker use is as below plus a separate ui-server (yarn build/yarn start from source). Your mileage may vary...

# This configuration file is for the **development** setup.
# For a production example please refer to getredash/setup repository on GitHub.
version: "2.2"
x-redash-service: &redash-service
  build:
    context: .
    args:
      skip_frontend_build: "true"  # set to empty string to build
  volumes:
    - .:/app
  env_file:
    - .env
x-redash-environment: &redash-environment
  REDASH_LOG_LEVEL: "INFO"
  REDASH_REDIS_URL: "redis://redis:6379/0"
  REDASH_DATABASE_URL: "postgresql://postgres@postgres/postgres"
  REDASH_RATELIMIT_ENABLED: "false"
  REDASH_MAIL_DEFAULT_SENDER: "[email protected]"
  REDASH_MAIL_SERVER: "email"
  REDASH_ENFORCE_CSRF: "true"
  REDASH_GUNICORN_TIMEOUT: 60
  # Set secret keys in the .env file
services:
  server:
    <<: *redash-service
    command: dev_server
    depends_on:
      - postgres
      - redis
    ports:
      - "5001:5000"
      - "5678:5678"
    environment:
      <<: *redash-environment
      PYTHONUNBUFFERED: 0
  scheduler:
    <<: *redash-service
    command: dev_scheduler
    depends_on:
      - server
    environment:
      <<: *redash-environment
  worker:
    <<: *redash-service
    command: dev_worker
    depends_on:
      - server
    environment:
      <<: *redash-environment
      PYTHONUNBUFFERED: 0
  redis:
    image: redis:3-alpine
    restart: unless-stopped
  postgres:
    image: postgres:9.5-alpine
    ports:
      - "15432:5432"
    # The following turns the DB into less durable, but gains significant performance improvements for the tests run (x3
    # improvement on my personal machine). We should consider moving this into a dedicated Docker Compose configuration for
    # tests.
    command: "postgres -c fsync=off -c full_page_writes=off -c synchronous_commit=OFF"
    restart: unless-stopped
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
  email:
    image: maildev/maildev
    ports:
      - "1080:80"
    restart: unless-stopped

[justinclift]

Just reformatted that for easier cut-n-paste. 😄

[jesraygarciano]

Thank you for this, I'm looking into it.

[nsakibp]

I am getting the same error with Mongodb 8.0