Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Light SDK without Anchor #1349

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat: Light SDK without Anchor #1349

wants to merge 1 commit into from

Conversation

vadorovsky
Copy link
Member

Provide a possibility to use the Rust Light SDK for programs which are not using Anchor.

Related changes:

  • Rename the former sdk-test program to sdk-anchor-test.
  • Add the new sdk-test program, which uses only solana-program.
  • Replace all re-imports from anchor_lang with direct imports from solana_program.
  • Introduce anchor feature flag in light_sdk.
  • Make the whole light_sdk::verify module independent from Anchor.

Copy link
Contributor

@ananas-block ananas-block left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the pr! Couple comments.

@@ -11,13 +11,14 @@ crate-type = ["cdylib", "lib"]
name = "light_sdk"

[features]
anchor = []
no-entrypoint = []
no-idl = []
no-log-ix-name = []
cpi = ["no-entrypoint"]
custom-heap = ["light-heap"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do we need light-heap for in light-sdk?
can we remove light-heap and the custom heap feature from this cargo toml?
(one feedback from squads was that they use their own heap implementation and encountered conflicts.)

#[cfg(feature = "anchor")]
use anchor_lang::{AnchorDeserialize, AnchorSerialize};
#[cfg(not(feature = "anchor"))]
use borsh::{BorshDeserialize as AnchorDeserialize, BorshSerialize as AnchorSerialize};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice fix

Comment on lines +286 to +359
pub fn new_address_params(&self) -> Option<PackedNewAddressParams> {
unimplemented!()
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the purpose of this placeholder?

sdk/src/error.rs Outdated
impl From<LightSdkError> for u32 {
fn from(e: LightSdkError) -> Self {
match e {
LightSdkError::ConstraintViolation => 13001,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already use 13000 + for errors of the light-verifier crate, can we use 14000 instead?

};
use thiserror::Error;

entrypoint!(process_instruction);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file would be easier to read if we put the implementation of process_instruction up here as well.

.data
.ok_or(LightSdkError::ExpectedData)?;
let mut data = data.borrow_mut();
let mut cpda = MyCompressedAccount::deserialize(&mut data.as_slice())?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are using borsh here, can we do a test which uses LightAccountdirectly (just duplicate this one but use LightAccount without anchor?
(It definitely makes sense to have a test without LightAccount as this one to make sure that the sdk works with any deserialization.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you think about renaming LightAccount to LightBorshAccount ?
(This would fit nicely with future LightZeroCopyAccount.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea.

The reason why I went for LightAccount before is sticking with Anchor's naming. But I can see the appeal of implementing more wrappers (bytemuck, bincode etc.).

@vadorovsky vadorovsky force-pushed the dns-without-anchor branch 5 times, most recently from e914db7 to 5fa6ba1 Compare November 21, 2024 17:59
Copy link
Contributor

@ananas-block ananas-block left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice accounts look a lot cleaner now! Comments are mainly about account checks and some naming here a summary:

  1. when initing system accounts, imo we should treat fee_payer and authority separately.
  2. For system account checks we need to check that fee_payer and authority are signers, invoking program is the correct program and system program is the correct program.
  3. instruction accounts and system accounts are the same we should unify naming see comment.

}

pub fn fee_payer(&self) -> &'c AccountInfo<'info> {
// PANICS: We are sure about the bounds of the slice.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does split_at panic if requested len is out of range?

Comment on lines 44 to 55
pub fn new(accounts: &'c [I]) -> Self {
// `split_at` doesn't make any copies.

// Take `CPI_ACCOUNTS_LEN` elements.
let (accounts, _) = accounts.split_at(SYSTEM_ACCOUNTS_LEN);
Self {
accounts,
_marker: PhantomData,
}
}

pub fn new_with_start_index(accounts: &'c [I], start_index: usize) -> Self {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if used with anchor it doesn't feel intuitive to me to assume thatfee_payer and authority are passed with the other accounts in correct order with remaining accounts since you will want to do a signer check for authority.
Can we pass fee_payer and authority as separate account infos and check that these are signers?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

additionally we need to check that light_system_program is the actual light system program and that it is executable. We need to do the same for invoking_program that it's id is the program id. The program id should be passed as an argument or const generic. We do a check like this here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub fn new(accounts: &'c [I]) -> Self {
// `split_at` doesn't make any copies.
// Take `CPI_ACCOUNTS_LEN` elements.
let (accounts, _) = accounts.split_at(SYSTEM_ACCOUNTS_LEN);
Self {
accounts,
_marker: PhantomData,
}
}
pub fn new_with_start_index(accounts: &'c [I], start_index: usize) -> Self {
pub fn new( fee_payer: AccountInfo<'info>, authority:AccountInfo<'info>, program_id: &Pubkey, accounts: &'c [I])) -> Self {
Self::new_with_start_index(accounts, 0, fee_payer, authority, program_id)
}
pub fn new_with_start_index(fee_payer: AccountInfo<'info>, authority:AccountInfo<'info>, program_id: &Pubkey, accounts: &'c [I], start_index: usize) -> Self {
// check invoking_program_id vs program id here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we pass fee_payer and authority separately maybe it makes sense to store these references in separate fields to avoid to create a vector in case that's necessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the main reason I wanted to avoid that is that it screws up my nice trick with just assigning the slice to the struct. 😅

With your approach, I will have to create a vector. I can try to do that and make a vector of references, so we don't make any copies, but we will still do a wasteful allocation of 11 * 8 bytes for storing the pointers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you add two fields fee_payer and authority you could use your trick for the remaining accounts without creating a vector right?

pub struct CreateRecord<'info> {
#[account(mut)]
#[fee_payer]
pub signer: Signer<'info>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels intuitive to me to keep the signer here so that it can be used for anchor checks see comment for LightSystemAccounts

Comment on lines +19 to +27
registered_program_pda: &Pubkey,
account_compression_authority: &Pubkey,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these accounts are always the same, you can hardcode these.
I think you are missing the cpi signer of the invoking program which is also deterministic you can derive it inside of new from the program id.

program_id: &Pubkey,
accounts: &[AccountInfo],
instruction_data: &[u8],
) -> ProgramResult {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

signer checks are missing

use solana_program::{account_info::AccountInfo, instruction::AccountMeta};

#[repr(usize)]
pub enum LightSystemAccountIndex {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are probably aware just a reminder for final review, there is a duplicate enum in instruction_accounts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good point. I kept this one and removed the other one.

};

/// Collection of Light Protocol system accounts which are sent to the program.
pub struct LightInstructionAccounts {
Copy link
Contributor

@ananas-block ananas-block Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could rename to LightInstructionAccountsBuilder and LightSystemAccounts to LightInstructionAccounts or the other way around to keep naming consistent and intuitive for devs on and offchain.
wdyt @vadorovsky ?
cc @SwenSchaeferjohann .

@vadorovsky vadorovsky force-pushed the dns-without-anchor branch 2 times, most recently from 05ed541 to 1be0fcb Compare November 22, 2024 09:15
pub struct LightCpiAccounts<'c, 'info, I>
where
// The reason behind using `AsRef` is to handle the inconsistency between
// solana-program and Anchor:
//
// - solana-program passes all accounts as `&[&AccountInfo]` (slice of
// references to `AccountInfo`).
// - Anchor stores `remaining_accounts` as `&[AccountInfo]` (slices of
// owned `AccountInfo` instances).
//
// Taking `AsRef<AccountInfo<'info>>` is the only way which makes this
// wrapper work with both owned and referenced `AccountInfo`s.
I: AsRef<AccountInfo<'info>>,
{
accounts: &'c [I],
_marker: PhantomData<&'info ()>,
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub struct LightCpiAccounts<'c, 'info, I>
where
// The reason behind using `AsRef` is to handle the inconsistency between
// solana-program and Anchor:
//
// - solana-program passes all accounts as `&[&AccountInfo]` (slice of
// references to `AccountInfo`).
// - Anchor stores `remaining_accounts` as `&[AccountInfo]` (slices of
// owned `AccountInfo` instances).
//
// Taking `AsRef<AccountInfo<'info>>` is the only way which makes this
// wrapper work with both owned and referenced `AccountInfo`s.
I: AsRef<AccountInfo<'info>>,
{
accounts: &'c [I],
_marker: PhantomData<&'info ()>,
}
pub struct LightCpiAccounts<'c, 'info, I>
where
// The reason behind using `AsRef` is to handle the inconsistency between
// solana-program and Anchor:
//
// - solana-program passes all accounts as `&[&AccountInfo]` (slice of
// references to `AccountInfo`).
// - Anchor stores `remaining_accounts` as `&[AccountInfo]` (slices of
// owned `AccountInfo` instances).
//
// Taking `AsRef<AccountInfo<'info>>` is the only way which makes this
// wrapper work with both owned and referenced `AccountInfo`s.
I: AsRef<AccountInfo<'info>>,
{
fee_payer: &'c I,
authority: &'c I,
accounts: &'c [I],
_marker: PhantomData<&'info ()>,
}

@vadorovsky vadorovsky force-pushed the dns-without-anchor branch 2 times, most recently from be5ca6c to 3870020 Compare November 22, 2024 13:52
Provide a possibility to use the Rust Light SDK for programs which are
not using Anchor.

Related changes:

- Rename the former `sdk-test` program to `sdk-anchor-test`.
- Add the new `sdk-test` program, which uses only solana-program.
- Replace all re-imports from `anchor_lang` with direct imports from
  `solana_program`.
- Introduce `anchor` feature flag in `light_sdk`.
- Make the whole `light_sdk::verify` module independent from Anchor.
@vadorovsky
Copy link
Member Author

vadorovsky commented Nov 22, 2024

Unfortunately, there are still errors when running the tests for name-service-without-macros program. I'm building and running them with the following commands:

cd examples/name-service
anchor build --p program-name name_service_without_macros
cargo test-sbf  -p name-service-without-macros -- --nocapture

The latest error is:

[2024-11-22T15:37:06.804098854Z DEBUG solana_runtime::message_processor::stable_log] Program log: Instruction: CreateRecord
[2024-11-22T15:37:06.804265514Z DEBUG solana_runtime::message_processor::stable_log] Program log: ACCOUNT METAS (len: 14):
[2024-11-22T15:37:06.804299084Z DEBUG solana_runtime::message_processor::stable_log] Program log: 0: AccountMeta { pubkey: BZa8GUz3kraqvnFpt76f5JC2PEr1WViKeaFBJHoy5Azz, is_signer: true, is_writable: true }
[2024-11-22T15:37:06.804328484Z DEBUG solana_runtime::message_processor::stable_log] Program log: 1: AccountMeta { pubkey: Bs4aBjKGMFsSn5q27NtDBQEHhHMRvNc5HkF18N9jsLSK, is_signer: true, is_writable: false }
[2024-11-22T15:37:06.804357114Z DEBUG solana_runtime::message_processor::stable_log] Program log: 2: AccountMeta { pubkey: 35hkDgaAKwMCaxRz2ocSZ6NaUrtKkyNqU6c4RV3tYJRh, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804384954Z DEBUG solana_runtime::message_processor::stable_log] Program log: 3: AccountMeta { pubkey: noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804419254Z DEBUG solana_runtime::message_processor::stable_log] Program log: 4: AccountMeta { pubkey: HwXnGK3tPkkVY6P439H2p68AxpeuWXd5PcrAxFpbmfbA, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804447594Z DEBUG solana_runtime::message_processor::stable_log] Program log: 5: AccountMeta { pubkey: compr6CUsB5m2jS4Y3831ztGSTnDpnKJTKS95d64XVq, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804480844Z DEBUG solana_runtime::message_processor::stable_log] Program log: 6: AccountMeta { pubkey: 7yucc7fL3JGbyMwg4neUaenNSdySS39hbAk89Ao3t1Hz, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804509924Z DEBUG solana_runtime::message_processor::stable_log] Program log: 7: AccountMeta { pubkey: SySTEM1eSU2p4BGQfQpimFEWWSC1XDFeun3Nqzz3rT7, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804541044Z DEBUG solana_runtime::message_processor::stable_log] Program log: 8: AccountMeta { pubkey: SySTEM1eSU2p4BGQfQpimFEWWSC1XDFeun3Nqzz3rT7, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804554484Z DEBUG solana_runtime::message_processor::stable_log] Program log: 9: AccountMeta { pubkey: 11111111111111111111111111111111, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804586914Z DEBUG solana_runtime::message_processor::stable_log] Program log: 10: AccountMeta { pubkey: SySTEM1eSU2p4BGQfQpimFEWWSC1XDFeun3Nqzz3rT7, is_signer: false, is_writable: false }
[2024-11-22T15:37:06.804620694Z DEBUG solana_runtime::message_processor::stable_log] Program log: 11: AccountMeta { pubkey: 5bdFnXU47QjzGpzHfXnxcEi5WXyxzEAZzd1vrE39bf1W, is_signer: false, is_writable: true }
[2024-11-22T15:37:06.804654324Z DEBUG solana_runtime::message_processor::stable_log] Program log: 12: AccountMeta { pubkey: C83cpRN6oaafjNgMQJvaYgAz592EP5wunKvbokeTKPLn, is_signer: false, is_writable: true }
[2024-11-22T15:37:06.804687594Z DEBUG solana_runtime::message_processor::stable_log] Program log: 13: AccountMeta { pubkey: HNjtNrjt6irUPYEgxhx2Vcs42koK9fxzm3aFLHVaaRWz, is_signer: false, is_writable: true }
[2024-11-22T15:37:06.804720304Z DEBUG solana_runtime::message_processor::stable_log] Program SySTEM1eSU2p4BGQfQpimFEWWSC1XDFeun3Nqzz3rT7 invoke [2]
[2024-11-22T15:37:06.804936714Z DEBUG solana_runtime::message_processor::stable_log] Program log: Instruction: InvokeCpi
[2024-11-22T15:37:06.805085865Z DEBUG solana_runtime::message_processor::stable_log] Program log: panicked at programs/system/src/invoke_cpi/verify_signer.rs:168:83:
    called `Result::unwrap()` on an `Err` value: AnchorError(AnchorError { error_name: "AccountDiscriminatorMismatch", error_code_number: 3002, error_msg: "8 byte discriminator did not match what was expected", error_origin: None, compared_values: None })
[2024-11-22T15:37:06.805092825Z DEBUG solana_runtime::message_processor::stable_log] Program SySTEM1eSU2p4BGQfQpimFEWWSC1XDFeun3Nqzz3rT7 consumed 38052 of 1175817 compute units
[2024-11-22T15:37:06.805100295Z DEBUG solana_runtime::message_processor::stable_log] Program SySTEM1eSU2p4BGQfQpimFEWWSC1XDFeun3Nqzz3rT7 failed: SBF program panicked
[2024-11-22T15:37:06.805106255Z DEBUG solana_runtime::message_processor::stable_log] Program 7yucc7fL3JGbyMwg4neUaenNSdySS39hbAk89Ao3t1Hz consumed 262235 of 1400000 compute units
[2024-11-22T15:37:06.805111595Z DEBUG solana_runtime::message_processor::stable_log] Program 7yucc7fL3JGbyMwg4neUaenNSdySS39hbAk89Ao3t1Hz failed: Program failed to complete
thread 'test_name_service' panicked at examples/name-service/programs/name-service-without-macros/tests/test.rs:94:6:
called `Result::unwrap()` on an `Err` value: TransactionError(InstructionError(0, ProgramFailedToComplete))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test test_name_service ... FAILED

This means that there is something wrong going on with the Merkle tree incides.

To give you some context here - there are 3 places in which we deal with account packing:

  1. LightInstructionAccounts- which is a client-side wrapper over a hash map, where all the account pubkeys are accumilated. The system accounts are added first:

pub fn new(
registered_program_pda: &Pubkey,
account_compression_authority: &Pubkey,
program_id: &Pubkey,
sol_pool_pda: Option<&Pubkey>,
decompression_recipient: Option<&Pubkey>,
) -> Self {
let mut accounts = Self {
// We reserve the first two incides for `sol_pool_pd`
next_index: 2,
map: HashMap::new(),
sol_pool_pda: sol_pool_pda.cloned(),
decompression_recipient: decompression_recipient.cloned(),
};
accounts.insert_or_get(*registered_program_pda);
accounts.insert_or_get(PROGRAM_ID_NOOP);
accounts.insert_or_get(*account_compression_authority);
accounts.insert_or_get(PROGRAM_ID_ACCOUNT_COMPRESSION);
accounts.insert_or_get(*program_id);
accounts.insert_or_get(PROGRAM_ID_SYSTEM);
accounts.insert_or_get(PROGRAM_ID_LIGHT_SYSTEM);
accounts
}

Then the Merkle tree accounts are being added through the insert_or_get method during the Merkle context packing:

let output_merkle_tree_index = remaining_accounts.insert_or_get(*output_merkle_tree);

let merkle_tree_pubkey_index = remaining_accounts.insert_or_get(*merkle_tree_pubkey);
let nullifier_queue_pubkey_index = remaining_accounts.insert_or_get(*nullifier_queue_pubkey);

  1. LightCpiAccounts - which is the on-chain wrapper. The way it works is that it uses the remaining accounts and adds fee_payer and authority at the start:

pub fn new<I>(
fee_payer: &'c AccountInfo<'info>,
authority: &'c AccountInfo<'info>,
accounts: &'c [I],
) -> Self
where
I: AsRef<AccountInfo<'info>>,
{
let mut cpi_accounts = Vec::with_capacity(accounts.len() + 2);
cpi_accounts.push(fee_payer.as_ref());
cpi_accounts.push(authority.as_ref());
cpi_accounts.extend(accounts.into_iter().map(|acc| acc.as_ref()));
Self {
accounts: cpi_accounts,
}
}

let light_cpi_accounts = LightCpiAccounts::new(
ctx.accounts.signer.as_ref(),
ctx.accounts.cpi_signer.as_ref(),
ctx.remaining_accounts,
);

In Anchor-less programs, this wrapper can be used just with AccountInfos. Unfortunately, I didn't get yet to adjusting of the Anchor-less test. 😞

  1. Merkle tree accounts

Before making a CPI, we subtract the Merkle tree account indices by the number of "system accounts" passed in the wrapper:

address_queue_account_index: address_merkle_context
.address_queue_pubkey_index
// Merkle tree accounts are at the end of "remaining accounts".
// "Remaining acocunts" passed to programs contain also system
// accounts.
// However, in light-system-program, system accounts are
// specified as regular accounts and don't end up as remaining
// anymore.
// Therefore, we need to update the Merkle tree accout indices.
.saturating_sub(SYSTEM_ACCOUNTS_LEN as u8),

Before doing so, the error was even worse, because system-program would crash with "out of bounds error".

The reason why it's done is that the most of what we considered as "remaining accounts" in the example program, is being recognized by Anchor in light-system-program instructions. Merkle trees are being the only "remaining accounts".

But still, even after adjusting the indices, we're hitting the error.

I think the bug resides in this exact mechanism and subtraction. Maybe to avoid that entirely, we should construct the LightInstructionAccounts wrapper in a way that it has the Merkle tree accounts first. Unfortunately, I ran out of time to try that solution.

Summary - remaining work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants