Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unnecessary pyenv usage in environment setup #26088

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ install_dev_python_modules_verbose:
python scripts/install_dev_python_modules.py

install_dev_python_modules_verbose_m1:
python scripts/install_dev_python_modules.py -q --include-prebuilt-grpcio-wheel
python scripts/install_dev_python_modules.py --include-prebuilt-grpcio-wheel

graphql:
cd js_modules/dagster-ui/; make generate-graphql; make generate-perms
Expand Down
26 changes: 9 additions & 17 deletions docs/content/community/contributing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,28 +9,20 @@ We love to see our community members get involved! If you are planning to contri

## Environment setup

1. Install a version of Python that Dagster supports. <DagsterVersion />

2. Create and activate a virtualenv, using the tool of your choice. On macOS you can install `pyenv` with Homebrew:
1. [Install uv](https://docs.astral.sh/uv/getting-started/installation/). On macOS, you can install use `curl` to download the script and execute it with `sh`:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a small typo in this line: "you can install use" should be either "you can install" or "you can use"

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.


```bash
brew install pyenv pyenv-virtualenv
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Then add the following commands to your shell profile:
2. Create and activate a virtual environment using uv with a Python version that Dagster supports:

```bash
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
uv venv --python 3.12
source .venv/bin/activate
```

and finally create and activate the virtualenev:

```bash
pyenv install 3.10.14
pyenv virtualenv 3.10.14 dagster310
pyenv activate dagster310
```
<DagsterVersion />

3. Ensure that you have node installed by running `node -v`, and that you have [yarn](https://yarnpkg.com/lang/en/) installed. If you are on macOS, you can install yarn with Homebrew:

Expand All @@ -44,7 +36,7 @@ We love to see our community members get involved! If you are planning to contri
git clone [email protected]:dagster-io/dagster.git
```

5. Run `make dev_install` at the root of the repository. This sets up a full Dagster developer environment with all modules and runs tests that do not require heavy external dependencies such as docker. This will take a few minutes. Note that certain sections of the makefile (sanity_check, which is part of `rebuild_ui`) require POSIX compliant shells and will fail on CMD and powershell -- if developing on windows, using something like WSL or git-bash is recommended. Note also that if this command fails while installing python packages, the problem might be resolved by ensuring you are running an up-to-date version of `pip` (upgrade with `pip install -U pip`).
5. Run `make dev_install` at the root of the repository. This sets up a full Dagster developer environment with all modules and runs tests that do not require heavy external dependencies such as docker. This will take a few minutes. Note that certain sections of the makefile (sanity_check, which is part of `rebuild_ui`) require POSIX compliant shells and will fail on CMD and powershellif developing on windows, using something like WSL or git-bash is recommended. Note also that if this command fails while installing python packages, the problem might be resolved by ensuring you are running an up-to-date version of `pip` (upgrade with `pip install -U pip`).

```bash
make dev_install
Expand Down Expand Up @@ -132,7 +124,7 @@ make mdx-format

You can find more information about developing documentation in `docs/README.md`.

## Picking a Github Issue
## Picking a GitHub Issue

We encourage you to start with an issue labeled with the tag [`good first issue`](https://github.com/dagster-io/dagster/issues?q=is%3Aopen+is%3Aissue+label%3A%22type%3A+good+first+issue%22) on the [Github issue board](https://github.com/dagster-io/dagster/issues), to get familiar with our codebase as a first-time contributor.

Expand All @@ -144,6 +136,6 @@ To submit your code, [fork the Dagster repository](https://help.github.com/en/ar

In the PR template, please describe the change, including the motivation/context, test coverage, and any other relevant information. Please note if the PR is a breaking change or if it is related to an open GitHub issue.

A Core reviewer will review your PR in around one business day and provide feedback on any changes it requires to be approved. Once approved and all the tests (including Buildkite!) pass, the reviewer will click the Squash and merge button in Github 🥳.
A Core reviewer will review your PR in around one business day and provide feedback on any changes it requires to be approved. Once approved and all the tests (including Buildkite!) pass, the reviewer will click the Squash and merge button in GitHub 🥳.

Your PR is now merged into Dagster! We’ll shout out your contribution in the weekly release notes.
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ ENV LANG=en_US.UTF-8 \
DOCKER_COMPOSE_VERSION=1.29.1 \
KIND_VERSION=v0.14.0 \
KUBECTL_VERSION=v1.20.1 \
FOSSA_VERSION=1.1.10
FOSSA_VERSION=1.1.10 \
UV_VERSION=0.5.4

# Install Kubernetes tools: kubectl, kind, helm
RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$KUBECTL_VERSION/bin/linux/amd64/kubectl" \
Expand Down Expand Up @@ -131,8 +132,8 @@ RUN curl -fsSL https://download.docker.com/linux/debian/gpg | \
&& curl -L "https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose

# Install Python build deps
RUN pip install -U pip setuptools wheel
# Install uv
COPY --from=ghcr.io/astral-sh/uv:${UV_VERSION} /uv /bin/

# Install redis
RUN wget http://download.redis.io/redis-stable.tar.gz \
Expand All @@ -159,7 +160,7 @@ WORKDIR dagster
ENV VIRTUAL_ENV=/usr/local
ENV UV_HTTP_TIMEOUT=300
RUN python scripts/install_dev_python_modules.py --system awscli || exit 1
RUN pip freeze --exclude-editable > /snapshot-reqs.txt
RUN uv pip freeze --exclude-editable > /snapshot-reqs.txt

# Final image includes both system deps and preinstalled non-Dagster python deps
FROM system_base
Expand All @@ -168,7 +169,7 @@ COPY --from=snapshot_builder /snapshot-reqs.txt .

# Preinstall non-Dagster packages in image so that the virtual environment
# builds faster in Buildkite.
RUN pip install -r /snapshot-reqs.txt \
RUN uv pip install -r /snapshot-reqs.txt \
&& rm /snapshot-reqs.txt

# New versions of debian require allowlisting folders where `git` can run
Expand Down
6 changes: 0 additions & 6 deletions scripts/install_dev_python_modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,6 @@ def main(
"python_modules/libraries/dagster-airflow",
]

# if sys.version_info > (3, 7):
# editable_target_paths += []

install_targets += list(
itertools.chain.from_iterable(
zip(["-e"] * len(editable_target_paths), editable_target_paths)
Expand All @@ -136,9 +133,6 @@ def main(
"https://github.com/dagster-io/build-grpcio/wiki/Wheels",
]

# Ensure uv is installed which we use for faster package resolution
subprocess.run(["pip", "install", "-U", "uv"], check=True)

# NOTE: These need to be installed as one long pip install command, otherwise pip will install
# conflicting dependencies, which will break pip freeze snapshot creation during the integration
# image build!
Expand Down