Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the num_inference_steps to the AI API for text-to-image endpoint #80

Merged
merged 2 commits into from
May 13, 2024
Merged

Add the num_inference_steps to the AI API for text-to-image endpoint #80

merged 2 commits into from
May 13, 2024

Conversation

mikezupper
Copy link
Contributor

This change allows higher quality image generation to the AI text-to-image endpoint. The the num_inference_steps parameter will be set at a sensible default of 25 (if no value is provided).

This change will require the Gateway and Orchestrator nodes to be updated.

@mikezupper mikezupper requested a review from rickstaa as a code owner May 11, 2024 12:09
rickstaa
rickstaa approved these changes May 12, 2024
View reviewed changes
Copy link
Member

@rickstaa rickstaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mikezupper, thanks a lot for this pull request. I reviewed it, and it works perfectly 🚀. I did, however, add one commit to handle an edge case that could crash the pipeline. Additionally, I changed the default num_inference_steps from 25 to 50 to match the default set by the diffusers library. Once this issue gets resolved, we can remove the default value from our codebase altogether.

@rickstaa rickstaa force-pushed the task/add-missing-ai-params branch 2 times, most recently from 2e89e06 to 0e97245 Compare May 12, 2024 11:01
mikezupper and others added 2 commits May 12, 2024 13:03
@mikezupper @rickstaa
add num_inference_steps to text-to-image endpoints
eeaa620 
added the generated runner code

final changes for num_inference_steps

removed debug logging

fixed spacing

fixed spacing
@rickstaa
refactor(runner): handle 'num_inference_steps' edge cases
8b38b4c 
Ensure the pipeline does not crash when the `num_inference_steps`
argument is set to a value lower than 1. Update the default
`num_inference_steps` to align with diffusion model defaults.
@rickstaa rickstaa force-pushed the task/add-missing-ai-params branch from 0e97245 to 8b38b4c Compare May 12, 2024 11:03
@rickstaa rickstaa merged commit fa2f0ab into livepeer:main May 13, 2024
1 check passed
rickstaa added a commit to rickstaa/docs that referenced this pull request May 13, 2024
@rickstaa
chore(ai): update AI subnet OpenAPI spec
cf1c0c1 
This commit updates the AI subnet OpenApi spec so that the reference
documentation correctly represents the new `num_inference_steps`
parameter introduced in livepeer/ai-worker#80.
rickstaa added a commit to rickstaa/docs that referenced this pull request May 13, 2024
@rickstaa
chore(ai): update AI subnet OpenAPI spec
c652980 
This commit updates the AI subnet OpenApi spec so that the reference
documentation correctly represents the new `num_inference_steps`
parameter introduced in livepeer/ai-worker#80.
@mikezupper mikezupper deleted the task/add-missing-ai-params branch May 18, 2024 10:29
rickstaa added a commit to livepeer/docs that referenced this pull request May 21, 2024
@rickstaa @mikezupper
@AuthorityNull @ChuckChain @eliteprox @dob
docs(ai): add ai-subnet documentation (#549)
61d59fc 
* docs(ai): add on-chain discovery documentation

This commit adds documentation on how Orchestrators can broadcast
their AI subnet service URIs to AI subnet Gateways, enabling their
discovery on the subnet for job execution.

* docs(ai): improve subsection titles and discovery text

This commit improves the subsection titles and Discovery text.

* docs(ai): improve introduction text

* ci(ai): add AI pipelines document structure

* docs(ai): add artibtrum explorer discovery steps

This commit adds a guide on how AI nodes can advertise their AI URI
using the Arbitrum explorer.

* docs(ai): add AI API reference

This commit adds an initial draft for the AI Subnet API reference. It is
not yet linked to an upstream API endpoint so the playground is not yet
working.

* chore(ai): cleanup redundant OpenAPI entries

* ci(ai): add AI docs codeowners

* docs(ai): add pipelines documentation

This commit adds the initial draft for the documentation that explains
the pipelines available on the network.

* docs(ai): fix broken images

This commit ensures that images are working when deployed on the remote.

* feat(ai): temporarily use external paths for pipeline images

This commit replaces the relative image paths used in the pipelines
documentation with external paths. We can change this back if we found a
way to use relative paths inside mermaid graphs.

* feat(ai): add contribution pages scaffold

This commit adds a general scaffold for the contribution pages. It is a
quick draft and should not yet be published to the upstream docs.

* feat(ai): add contribution pages scaffold (#553)

This commit adds a general scaffold for the contribution pages. It is a
quick draft and should not yet be published to the upstream docs.

* Livepeer.Cloud SPE Milestone #1 Deliverables: Add Gateway Docs

* docs(ai): improve AI subnet introduction

This commit improves the AI subnet introduction page.

* fix(ai): fix kickstart collab card link

This commit ensures that the collaboration card is linked correctly.

* refactor(ai): improve pipeline mermaid diagrams

This commit improves the styling used in the pipeline mermaid
diagrams.

* refactor(ai): improve pipeline documentation

This commit updates the pipelines documentation to increase
readability.

* fix(ai): implement temporary solution for mermaid image issue

This commit implements a temporary solution to address the issue of
images not displaying correctly when embedded in a mermaid graph on the
deployed site. Can be replaced with relative path if mintlify fixes the
issue (see
https://mintlify-users.slack.com/archives/C05D00NS55F/p1714485030542069).

* feat(ai): add builders page

This commit adds the builders page and a showcase to show dApps already
build on the AI subnet.

* refactor(ai): add cloud API disclaimer

This commit adds a small disclaimer to the API documentation that states
that the livepeer.cloud API is not production ready.

* feat(ai): add orchestrator setup guide

This commit adds the orchestrator setup guide.

* refactor(ai): improve showcase descriptions

This commit improves the descriptions of the projects in the showcase.

* refactor(ai): highlight showcase keywords

This commit highlights some keywords in the showcase descriptions.

* feat(ai): add AI gateway docs

This commit adds the documentation for setting up a AI gateway on the AI
subnet. It also improves the project structure.

* refactor(ai): adopt impersonal writing style (#556)

This commit updates the text to reflect an impersonal writing style,
consistent with the ecosystem's voice. It eliminates personal pronouns
such as "our" and "we".

* fix(ai): fix broken AI gateway links

This commit fixes several broken internal links in the AI gateway docs.

* refactor(ai): improve AI subnet introduction section

This commit improves the readability of the AI subnet introduction
section.

* refactor(ai): hide contribution/SDK pages and cleanup pipelines

This commit hides the contribution and SDK pages and cleansup the
pipeline pages.

* refactor(ai): improve text formatting

This commit applies some last text formatting improvements.

* addressed review comments from Rick

* refactor(ai): add model parameter callout to API docs

This commit adds a model parameter callout to the API docs to explain
that not all parameters are used by all models.

* chore(ai): fix small typo

This commit fixes some small typos.

* refactor(ai): fix spelling

This commit fixes some small spelling errors.

* made Prerequisites - more concise

* chore(ai): add Analytics token

This commit adds the Google Analytics GA4 token to the documentation
config.

* docs(gateway): apply prettier mdx formatting

This commit applies the prettier formatting suggestions on the Gateway
docs (see https://prettier.io/blog/2021/11/25/2.5.0.html).

* refactor(ai): gateway former naming comment

This commit makes it more clear that Gateways were formaly named
Broadcasters.

* refactor(ai): simplify binary install

This commit simplifies the binary install of the AI subnet by giving
users one simple download url they can use to downloade the latest
`ai-video` branch binary. This was made possible in
livepeer/go-livepeer@fb9764b.

* docs(ai): apply community feedback

This commit applies the feedback I received from the community about
things that could be improved.

Co-authored-by: Authority_Null <[email protected]>
Co-authored-by: Chuck Chain <[email protected]>

* docs(ai): add RealViz warm comment

This commit adds a comment letting users know that several orchestrators
have the https://huggingface.co/SG161222/RealVisXL_V4.0 model warm on
their GPU.

* docs(ai): fix incorrect RealViLightning model link

This commit ensures that the RealVizLightning model link is pointing to the
right url.

* chore(ai): update AI subnet OpenAPI spec

This commit updates the AI subnet OpenApi spec so that the reference
documentation correctly represents the new `num_inference_steps`
parameter introduced in livepeer/ai-worker#80.

* docs: replace -broadcaster flag with -gateway in AI subnet documentation (#564)

This commit replaces the `broadcaster` flag with the new `gateway` flag. See livepeer/go-livepeer#3048 for more information.

* docs(ai): remove pricePerUnit requirement for AI docs (#565)

This commit removes the dependency on the `pricePerUnit` flag in the AI orchestrator documentation now that livepeer/go-livepeer#3047 has been merged.

* docs(ai): add RealVisXL to I2I pipeline

This commit adds the
[SG161222/RealVisXL_V4.0_Lightning](https://huggingface.co/SG161222/RealVisXL_V4.0_Lightning)
model to the supported models on the I2I pipeline documentation page.

* docs: add Gateway network page

This commit adds a Gateway page under the core-concepts category, explaining the
role and functions of a Gateway.

* docs: improve Gateway configuration wording

* chore: apply prettier to Gateway node page

* Update gateways.mdx

Clarified gateways as the type of node, and talked about hosted gateways as something that individual entities may run.

* docs: fix Gateway node page links

This commit fixes some broken links in the Gateway node page.

* docs(ai): remove double subnet requirements text

This commit removes duplicate text in the AI subnet requirements
section.

* docs(ai): address community feedback

This commit clarifies points that the community identified as unclear.

* docs(ai): enhance introduction

This commit strengthens the AI Subnet introduction, providing clearer
and more detailed information about its capabilities and benefits.

* docs(ai): add DeepCache optimization method

This commit adds the DeepCache optimization method to the AI models
configuration page. This method can be used to speedup the pipelines to
up to 50%.

* docs(ai): replace dApp with application

This commit replaces the parts that mention dApps with applications
since the AI subnet can serve both web2 and web3 applications.

* docs(ai): ensure GPU binaries are listed

This commit ensures the GPU binaries are listed in the Orchestrator
binary setup guide.

* docs(ai): fix incorrect binary format

This commit fixes a small syntax error in the AI subnet GPU binary
download url.

* docs(ai): add HuggingFace token read permission comment

This commit ensures that people are aware that they only need to create
a HuggingFace token with read permissions.

* docs(ai): improve introduction

This commit improves the introduction to make clear that orchestrators
can have multiple workers. It also improves the diagram.

* docs(ai): change dApp to app

This commit ensures that all references to dApp are replaced by app or
application.

* docs(ai): apply some small text improvements

This commit applies some small text improvements to the AI
documentation.

* docs(ai): update AI API reference

This commit updates the AI API reference to include the new `nsfw`
feature (see
livepeer/ai-worker@272ac74).

* refactor(ai) improve introduction wording (#570)

* Update verbiage in introduction

* docs(ai): mention AI Video subnet

---------

Co-authored-by: Elite Encoder <[email protected]>

---------

Co-authored-by: Mike Zupper <[email protected]>
Co-authored-by: Authority_Null <[email protected]>
Co-authored-by: Chuck Chain <[email protected]>
Co-authored-by: Elite Encoder <[email protected]>
Co-authored-by: Mike Zupper <[email protected]>
Co-authored-by: Doug Petkanics <[email protected]>
eliteprox pushed a commit to eliteprox/ai-worker that referenced this pull request Jun 9, 2024
@mikezupper @rickstaa @eliteprox
feat: add the 'num_inference_steps' to T2I endpoint (livepeer#80)
e57d52b 
* add num_inference_steps to text-to-image endpoints

added the generated runner code

final changes for num_inference_steps

removed debug logging

fixed spacing

fixed spacing

* refactor(runner): handle 'num_inference_steps' edge cases

Ensure the pipeline does not crash when the `num_inference_steps`
argument is set to a value lower than 1. Update the default
`num_inference_steps` to align with diffusion model defaults.

---------

Co-authored-by: Rick Staa <[email protected]>
@rickstaa rickstaa mentioned this pull request Jun 20, 2024
Closed
eliteprox pushed a commit to eliteprox/ai-worker that referenced this pull request Jul 26, 2024
@mikezupper @rickstaa @eliteprox
feat: add the 'num_inference_steps' to T2I endpoint (livepeer#80)
41cb844 
* add num_inference_steps to text-to-image endpoints

added the generated runner code

final changes for num_inference_steps

removed debug logging

fixed spacing

fixed spacing

* refactor(runner): handle 'num_inference_steps' edge cases

Ensure the pipeline does not crash when the `num_inference_steps`
argument is set to a value lower than 1. Update the default
`num_inference_steps` to align with diffusion model defaults.

---------

Co-authored-by: Rick Staa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rickstaa rickstaa rickstaa approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mikezupper @rickstaa