[Discussion] Training 1.5B model on Google Cloud

[EibrielInv]

Hi, I'm looking to train GPT-2 in Spanish. And I would like to know exactly what hardware is needed.

On the readme you state that can be trained using TPU pod.

• Are you referring to the following TPU POD?

TPU (v3) type	TPU v3 cores	TPU total memory	Available zones
v3-256	256	4 TiB	europe-west4-a

• Could you tell me how many TPU time was needed to setup and run the training?

• And do you remember what other services were needed? How much storage? And for how long?

Thank you!

[irfanahme]

if you want to train than contact with me on github and email, [email protected]