Temp error backendError - occurring with 80GB restore that previously was working

[mcopley08]

Full steps to reproduce the issue:

1. I successfully started an 80GB mbox restore to a gmail account that got to ~12% completed. I stopped the run and re-started it multiple times and it worked, picking back up where it was before.
2. Suddenly it does not make any more progress, every time I run the command I get the Temp error backendError. that others have been seeing. I've tried doing this on a different computer, and making a new Google Cloud project, using different internet connections - but none of it resolves the issue

What else can be done to continue with the inbox restore?

Please confirm the following:

• [CONFIRMED] I have upgraded to the latest GYB release from https://github.com/jay0lee/got-your-back/releases and I still have this issue.
• [CONFIRMED] I am typing the command as described in the GAM Wiki at https://github.com/jay0lee/got-your-back/wiki

[urbaned121]

Hmm I have alike problem with GYB recently (begins about 10 days ago(?)).
Im backing up mailbox what goes smoothly but when restoring it to another (both on the same google workspace) it constantly throws an errors:

500: Temporary System Problem. Try again later. - backendError

• Giving up.
  restoring 15 messages (45/25178)
  Temp error backendError. Backing off 16 seconds...attempt 5/10

Temp error backendError. Backing off 32 seconds...attempt 6/10

Temp error backendError. Backing off 60 seconds...attempt 7/10

Temp error backendError. Backing off 60 seconds...attempt 8/10

Temp error backendError. Backing off 60 seconds...attempt 9/10

command to backup is:
gyb --email [email protected] --action backup --service-account
and to restore I'm doing:
gyb --email [email protected] --action restore --local-folder [email protected] --label-restored "BackupLabel" --service-account

I'm using GYB and GAM latest versions on MacOS M1 which doesn't really mean anything because initially (about a month ago) nothing like this appeared.

I suspect, but haven't confirmed it yet.
I administer two Google Workspaces, both of which I deal with large mailboxes, and so far the problem is annoying in one of the Google Workspace instances.

EDIT: adding --batch-size 5 to restore command does not change anything

[mcopley08]

Thanks @urbaned121 , and can confirm changing the batch size does not change anything for me either

[wazimshizm]

Also having this issue. Even on a tiny inbox (62 messages). Large inbox as well.

[urbaned121]

@mcopley08 @wazimshizm
Guys try with --batch-size 1
no errors so far and I managed to send 5k messages since the morning 5k/23k
but the strange thing is that recently there was no need to add batch size

[wazimshizm]

@mcopley08 @wazimshizm Guys try with --batch-size 1 no errors so far and I managed to send 5k messages since the morning 5k/23k but the strange thing is that recently there was no need to add batch size

Can confirm batch-size 1 works, slower but it works. Thanks @urbaned121

[imoseyon]

I can confirm this issue. batch-size 2 also works for me.

[mpcusack]

I have also found that --batch-size 2 is the largest I can use and have a restore work reliably. Taking a few days to restore a 210k inbox, but better than nothing.

[jay0lee]

I just released GYB 1.81 that should fix this issue by reducing batch sizes. Please test and lmk if you are still seeing issues.

https://github.com/GAM-team/got-your-back/releases

Jay

[mpcusack]

@jay0lee I see you reduced restore batch size to 1. In my testing at least 2 also works (and at 3 I start seeing errors).

Any idea what changed to cause these errors on batches larger than 2? Just an invisible change on google's side?

[wazimshizm]

This is working with batch size 2 at a time, but on a 160,000 messages it's going at a snails pace. Do we know any more about the situation?