Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't use workspace dependencies #2652

Merged
merged 13 commits into from
Nov 4, 2024
Merged

Don't use workspace dependencies #2652

merged 13 commits into from
Nov 4, 2024

Conversation

tokatoka
Copy link
Member

@tokatoka tokatoka commented Nov 4, 2024

No description provided.

@rmalmain
Copy link
Collaborator

rmalmain commented Nov 4, 2024

good thanks for the rev work.
should we use something like cargo-outdated to track dependencies getting out of date in the CI?
what happened to dependabot btw (cc @domenukk)

@tokatoka
Copy link
Member Author

tokatoka commented Nov 4, 2024

idk it's dead after working for several weeks 🚮

@tokatoka tokatoka merged commit e0c5304 into main Nov 4, 2024
99 checks passed
@tokatoka tokatoka deleted the revert_workspace branch November 4, 2024 16:47
@Marcondiro
Copy link
Contributor

Hey @tokatoka what is the reason behind this? thx :)

@tokatoka
Copy link
Member Author

tokatoka commented Nov 4, 2024

the problem is that workspace dependencies does not allow optional = flag.
and if you list a dependency there all the crate that uses libafl have to download and compile it even if it is a optional feature.
for example if you list windows then you'll download windows crate even if you are on linux

@Marcondiro
Copy link
Contributor

workspace dependencies does not allow optional = flag.

You can mark it as optional in the packages, the absence of optional in the workspace won't force it to be mandatory in any package

if you list a dependency there all the crate that uses libafl have to download and compile it

Oh, this is weird. I checked this by creating a new workspace with dependencies not used in the package and they are not downloaded nor compiled when I compile the package. Do you have a specific example in libafl that has been fixed by this PR by chance?
Thanks 🙂

@tokatoka
Copy link
Member Author

tokatoka commented Nov 4, 2024

git checkout 1e0b5581e52feaca7311572a3f88b8dcf5cea0c7
then go to fuzzers/fuzzbench/fuzzbench_ctx and cargo build --release.
you'll see windows crate getting downloaded

@Marcondiro
Copy link
Contributor 

git checkout 1e0b5581e52feaca7311572a3f88b8dcf5cea0c7
cargo clean
cd fuzzers/inprocess/fuzzbench_ctx
cargo clean
cargo build --release

I don't see windows here

warning: unused manifest key: profile.release-abort.abort
    Updating crates.io index
     Locking 169 packages to latest compatible versions
      Adding which v6.0.3 (available: v7.0.0)
  Downloaded unicode-width v0.2.0
  Downloaded ratatui v0.29.0
  Downloaded 2 crates (815.0 KB) in 0.31s
   Compiling proc-macro2 v1.0.89
   Compiling unicode-ident v1.0.13
   Compiling libc v0.2.161
   Compiling autocfg v1.4.0
   Compiling cfg-if v1.0.0
   Compiling rustversion v1.0.18
   Compiling serde v1.0.214
   Compiling shlex v1.3.0
   Compiling parking_lot_core v0.9.10
   Compiling either v1.13.0
   Compiling memchr v2.7.4
   Compiling bitflags v2.6.0
   Compiling scopeguard v1.2.0
   Compiling log v0.4.22
   Compiling glob v0.3.1
   Compiling smallvec v1.13.2
   Compiling version_check v0.9.5
   Compiling prettyplease v0.2.25
   Compiling signal-hook v0.3.17
   Compiling allocator-api2 v0.2.18
   Compiling minimal-lexical v0.2.1
   Compiling lock_api v0.4.12
   Compiling memoffset v0.9.1
   Compiling rustix v0.38.39
   Compiling ahash v0.8.11
   Compiling clang-sys v1.8.1
   Compiling regex-syntax v0.8.5
   Compiling syn v1.0.109
   Compiling typeid v1.0.2
   Compiling quote v1.0.37
   Compiling cfg_aliases v0.2.1
   Compiling once_cell v1.20.2
   Compiling jobserver v0.1.32
   Compiling libloading v0.8.5
   Compiling nix v0.29.0
   Compiling syn v2.0.87
   Compiling num-traits v0.2.19
   Compiling bindgen v0.70.1
   Compiling paste v1.0.15
   Compiling cc v1.1.34
   Compiling zerocopy v0.7.35
   Compiling gimli v0.31.1
   Compiling linux-raw-sys v0.4.14
   Compiling foldhash v0.1.3
   Compiling ryu v1.0.18
   Compiling itoa v1.0.11
   Compiling nom v7.1.3
   Compiling equivalent v1.0.1
   Compiling adler2 v2.0.0
   Compiling heck v0.5.0
   Compiling sdd v3.0.4
   Compiling static_assertions v1.1.0
   Compiling itertools v0.13.0
   Compiling miniz_oxide v0.8.0
   Compiling hashbrown v0.15.1
   Compiling signal-hook-registry v1.4.2
   Compiling parking_lot v0.12.3
   Compiling regex-automata v0.4.8
   Compiling mio v1.0.2
   Compiling scc v2.2.4
   Compiling getrandom v0.2.15
   Compiling cexpr v0.6.0
   Compiling signal-hook-mio v0.2.4
   Compiling libafl_bolts v0.13.2 (/home/marco/code/LibAFL_upstream_main/libafl_bolts)
   Compiling castaway v0.2.3
   Compiling object v0.36.5
   Compiling aho-corasick v1.1.3
   Compiling home v0.5.9
   Compiling rustc-hash v1.1.0
   Compiling cobs v0.2.3
   Compiling addr2line v0.24.2
   Compiling utf8parse v0.2.2
   Compiling libm v0.2.11
   Compiling unicode-width v0.1.14
   Compiling typewit_proc_macros v1.8.1
   Compiling regex v1.11.1
   Compiling rustc-demangle v0.1.24
   Compiling serde_json v1.0.132
   Compiling unicode-xid v0.2.6
   Compiling unicode-segmentation v1.12.0
   Compiling const_format_proc_macros v0.2.33
   Compiling typewit v1.9.0
   Compiling anstyle-parse v0.2.6
   Compiling unicode-truncate v1.1.0
   Compiling which v6.0.3
   Compiling compact_str v0.8.0
   Compiling crossterm v0.28.1
   Compiling libmimalloc-sys v0.1.39
   Compiling serde_derive v1.0.214
   Compiling serial_test_derive v3.1.1
   Compiling num_enum_derive v0.7.3
   Compiling strum_macros v0.26.4
   Compiling serial_test v3.1.1
   Compiling c2rust-bitfields-derive v0.19.0
   Compiling num_enum v0.7.3
   Compiling libafl_derive v0.13.2 (/home/marco/code/LibAFL_upstream_main/libafl_derive)
   Compiling backtrace v0.3.74
   Compiling typed-builder-macro v0.20.0
   Compiling instability v0.3.2
   Compiling ctor v0.2.8
   Compiling lru v0.12.5
   Compiling uds v0.4.2
   Compiling hostname v0.4.0
   Compiling libafl v0.13.2 (/home/marco/code/LibAFL_upstream_main/libafl)
   Compiling xxhash-rust v0.8.12
   Compiling anstyle v1.0.10
   Compiling rand_core v0.6.4
   Compiling tuple_list v0.1.3
   Compiling strum v0.26.3
   Compiling cassowary v0.3.0
   Compiling colorchoice v1.0.3
   Compiling unicode-width v0.2.0
   Compiling is_terminal_polyfill v1.70.1
   Compiling anstyle-query v1.1.2
   Compiling indoc v2.0.5
   Compiling anstream v0.6.18
   Compiling c2rust-bitfields v0.19.0
   Compiling const_format v0.2.33
   Compiling typed-builder v0.20.0
   Compiling libafl_cc v0.13.2 (/home/marco/code/LibAFL_upstream_main/libafl_cc)
   Compiling ratatui v0.29.0
   Compiling const_panic v0.2.10
   Compiling wait-timeout v0.2.0
   Compiling strsim v0.11.1
   Compiling clap_lex v0.7.2
   Compiling rangemap v1.5.1
   Compiling clap_builder v4.5.20
   Compiling libafl_targets v0.13.2 (/home/marco/code/LibAFL_upstream_main/libafl_targets)
warning: [email protected]: src/forkserver.c: In function ‘__afl_start_forkserver’:
warning: [email protected]: src/forkserver.c:275:7: warning: ignoring return value of ‘write’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
warning: [email protected]:   275 |       write(2, "Error: could not send autotokens len\n",
warning: [email protected]:       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: [email protected]:   276 |             strlen("Error: could not send autotokens len\n"));
warning: [email protected]:       |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Compiling clap v4.5.20
   Compiling uuid v1.11.0
   Compiling hashbrown v0.14.5
   Compiling postcard v1.0.10
   Compiling erased-serde v0.4.5
   Compiling bincode v1.3.3
   Compiling meminterval v0.4.1
   Compiling mimalloc v0.1.43
warning: [email protected]: Skipping non-required compiler pass src/dump-cfg-pass.cc - Reason: Exit status exit status: 1. You can ignore this error unless you want this compiler pass.
warning: [email protected]: Skipping non-required compiler pass src/profiling-pass.cc - Reason: Exit status exit status: 1. You can ignore this error unless you want this compiler pass.
   Compiling fuzzbench_ctx v0.13.2 (/home/marco/code/LibAFL_upstream_main/fuzzers/inprocess/fuzzbench_ctx)
    Finished `release` profile [optimized + debuginfo] target(s) in 1m 11s

@Marcondiro
Copy link
Contributor

Also after rm -r ~/.cargo/registry/cache it doesn't download windows

warning: unused manifest key: profile.release-abort.abort
  Downloaded adler2 v2.0.0
  Downloaded anstyle-parse v0.2.6
  Downloaded anstream v0.6.18
  Downloaded ahash v0.8.11
  Downloaded addr2line v0.24.2
  Downloaded equivalent v1.0.1
  Downloaded mimalloc v0.1.43
  Downloaded c2rust-bitfields v0.19.0
  Downloaded anstyle v1.0.10
  Downloaded cfg_aliases v0.2.1
  Downloaded meminterval v0.4.1
  Downloaded c2rust-bitfields-derive v0.19.0
  Downloaded either v1.13.0
  Downloaded cfg-if v1.0.0
  Downloaded castaway v0.2.3
  Downloaded anstyle-query v1.1.2
  Downloaded foldhash v0.1.3
  Downloaded hostname v0.4.0
  Downloaded itoa v1.0.11
  Downloaded clap_lex v0.7.2
  Downloaded home v0.5.9
  Downloaded heck v0.5.0
  Downloaded ctor v0.2.8
  Downloaded cobs v0.2.3
  Downloaded embedded-io v0.6.1
  Downloaded indoc v2.0.5
  Downloaded glob v0.3.1
  Downloaded errno v0.3.9
  Downloaded colorchoice v1.0.3
  Downloaded embedded-io v0.4.0
  Downloaded autocfg v1.4.0
  Downloaded scopeguard v1.2.0
  Downloaded rand_core v0.6.4
  Downloaded cexpr v0.6.0
  Downloaded parking_lot_core v0.9.10
  Downloaded lru v0.12.5
  Downloaded bincode v1.3.3
  Downloaded rustc-hash v1.1.0
  Downloaded num_enum_derive v0.7.3
  Downloaded num_enum v0.7.3
  Downloaded signal-hook-registry v1.4.2
  Downloaded serial_test v3.1.1
  Downloaded rustversion v1.0.18
  Downloaded paste v1.0.15
  Downloaded cassowary v0.3.0
  Downloaded memoffset v0.9.1
  Downloaded lock_api v0.4.12
  Downloaded sdd v3.0.4
  Downloaded instability v0.3.2
  Downloaded quote v1.0.37
  Downloaded is_terminal_polyfill v1.70.1
  Downloaded libloading v0.8.5
  Downloaded once_cell v1.20.2
  Downloaded getrandom v0.2.15
  Downloaded const_format_proc_macros v0.2.33
  Downloaded erased-serde v0.4.5
  Downloaded rustc-demangle v0.1.24
  Downloaded num-traits v0.2.19
  Downloaded clang-sys v1.8.1
  Downloaded const_panic v0.2.10
  Downloaded allocator-api2 v0.2.18
  Downloaded jobserver v0.1.32
  Downloaded parking_lot v0.12.3
  Downloaded clap v4.5.20
  Downloaded ryu v1.0.18
  Downloaded log v0.4.22
  Downloaded bitflags v2.6.0
  Downloaded serde_derive v1.0.214
  Downloaded postcard v1.0.10
  Downloaded compact_str v0.8.0
  Downloaded miniz_oxide v0.8.0
  Downloaded const_format v0.2.33
  Downloaded serial_test_derive v3.1.1
  Downloaded rangemap v1.5.1
  Downloaded proc-macro2 v1.0.89
  Downloaded prettyplease v0.2.25
  Downloaded strum v0.26.3
  Downloaded backtrace v0.3.74
  Downloaded typewit_proc_macros v1.8.1
  Downloaded typed-builder-macro v0.20.0
  Downloaded tuple_list v0.1.3
  Downloaded typed-builder v0.20.0
  Downloaded memchr v2.7.4
  Downloaded minimal-lexical v0.2.1
  Downloaded shlex v1.3.0
  Downloaded strsim v0.11.1
  Downloaded version_check v0.9.5
  Downloaded unicode-truncate v1.1.0
  Downloaded typeid v1.0.2
  Downloaded serde v1.0.214
  Downloaded cc v1.1.34
  Downloaded mio v1.0.2
  Downloaded hashbrown v0.15.1
  Downloaded nom v7.1.3
  Downloaded libm v0.2.11
  Downloaded scc v2.2.4
  Downloaded crossterm v0.28.1
  Downloaded itertools v0.13.0
  Downloaded aho-corasick v1.1.3
  Downloaded hashbrown v0.14.5
  Downloaded clap_builder v4.5.20
  Downloaded libmimalloc-sys v0.1.39
  Downloaded bindgen v0.70.1
  Downloaded unicode-xid v0.2.6
  Downloaded gimli v0.31.1
  Downloaded utf8parse v0.2.2
  Downloaded regex v1.11.1
  Downloaded regex-syntax v0.8.5
  Downloaded object v0.36.5
  Downloaded nix v0.29.0
  Downloaded rustix v0.38.39
  Downloaded signal-hook-mio v0.2.4
  Downloaded which v6.0.3
  Downloaded wait-timeout v0.2.0
  Downloaded static_assertions v1.1.0
  Downloaded strum_macros v0.26.4
  Downloaded xxhash-rust v0.8.12
  Downloaded ratatui v0.29.0
  Downloaded smallvec v1.13.2
  Downloaded regex-automata v0.4.8
  Downloaded unicode-ident v1.0.13
  Downloaded uds v0.4.2
  Downloaded uuid v1.11.0
  Downloaded libc v0.2.161
  Downloaded signal-hook v0.3.17
  Downloaded typewit v1.9.0
  Downloaded serde_json v1.0.132
  Downloaded linux-raw-sys v0.4.14
  Downloaded unicode-segmentation v1.12.0
  Downloaded zerocopy v0.7.35
  Downloaded syn v1.0.109
  Downloaded unicode-width v0.2.0
  Downloaded unicode-width v0.1.14
  Downloaded syn v2.0.87
  Downloaded 134 crates (11.8 MB) in 0.43s (largest was `linux-raw-sys` at 1.8 MB)

@tokatoka
Copy link
Member Author

tokatoka commented Nov 5, 2024

can you reset --hard?
you are still on the latest commit it seems

@Marcondiro
Copy link
Contributor

still no win* crate downloaded nor compiled

@tokatoka
Copy link
Member Author

tokatoka commented Nov 5, 2024

image
did you remove cargo.lock?
it does for me

@Marcondiro
Copy link
Contributor 

cargo clean
rm Cargo.lock
cargo build --release

still nothing
my toolchain:

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.84.0-nightly (fbab78289 2024-11-04)

@tokatoka
Copy link
Member Author

tokatoka commented Nov 5, 2024

now i understand that this change didn't really fix the problem. the real cultprit was the tui_monitor feature.

@rmalmain
Copy link
Collaborator

rmalmain commented Nov 5, 2024

so workspace dependencies work as expected?

@tokatoka
Copy link
Member Author

tokatoka commented Nov 5, 2024

no man, but there was apparently that pyo3 issue that could never happen

@tokatoka
Copy link
Member Author

tokatoka commented Nov 5, 2024 •
edited
Loading

At least before this other PR, #2635 every fuzzers has to download pyo3 and caused linker error on ci. for that he removed it from workspace deps then it's gone.
so i think my windows stuff is just a different issue

@Marcondiro
Copy link
Contributor

Marcondiro commented Nov 5, 2024 •
edited
Loading

I think it stopped being downloaded not because of the workspace dep but because nautilus was removed from libafl default (pyo3 is required by nautilus feature)

@domenukk
Copy link
Member

domenukk commented Nov 5, 2024

Sounds like we want workspace deps back/undo this PR then?

@tokatoka
Copy link
Member Author

tokatoka commented Nov 5, 2024

but because nautilus was removed from libafl default

yeah i think you are right

@Marcondiro Marcondiro mentioned this pull request Nov 6, 2024
Merged
@domenukk
Copy link
Member

domenukk commented Nov 6, 2024

So, are we gonna revert this @tokatoka ?

@tokatoka
Copy link
Member Author

tokatoka commented Nov 6, 2024

yeah

@domenukk
Copy link
Member

domenukk commented Nov 7, 2024

Will you do it?

This was referenced Nov 9, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tokatoka @rmalmain @Marcondiro @domenukk