Self hosting: replication not working/unreliable when only two buckets are configured

[smncd]

Hello there!
I've been playing around with self hosting Ente and love it so far.

I've tried to set up replication between the primary b2-eu-cen bucket and scw-eu-fr-v3, using Backblaze and Scaleway respectively.

However, it doesn't seem to quite do it. It seems like something is going on, as the scaleway bucket does get somewhat populated with files, however looking at the logs of the museum docker container, there are lots of errors related to replication. I'll try to grab the logs and post here.
As soon as I configure a third bucket, wasabi-eu-central-2-v3, the replication starts working without issues.

So my question is, is this expected behaviour, and if so, are there plans to make replication work for a two bucket setup (hot and glacier)?

Cheers!

[mnvr]

So my question is, is this expected behaviour,

Yes. Currently replication only works in the exact 3 (1 hot, 1 cold, 1 glacier) setup.

and if so, are there plans to make replication work for a two bucket setup (hot and glacier)?

Yes! No timelines yet, but we are working on the sides to make this more configurable, and things like arbitrary bucket names (instead of the current hardcoded ones), arbitrary replication setups (e.g. 1 hot 1 glacier) will be possible.

[smncd]

Got it! Sounds great, I will keep my eyes out for that. Thank you :)

[gymnae]

I ran into the same issue. For now, I have helped myself by enabling the integrated replication functionality of minio. For that, I had to enable versioning on the main hot storage in minio and set up a second minio install at a different location.

[fredix]

Hi all,
my Ente server self-hosted crash with 2 S3 servers : primary: b2-eu-cen and secondary: wasabi-eu-central-2-v3
The primary is is a scaleway S3 and secondary is self-hosted. So the replication doesn't works with 2 S3 server ?

`INFO[0000]main.go:95 main Booting up local server with commit #b0674cd5f37cebda0f446db7ce1e87a3e6d8b9fc
INFO[0000]main.go:791 setupDatabase Setting up db
INFO[0000]main.go:798 setupDatabase Connected to DB
INFO[0000]main.go:803 setupDatabase Pinged DB
INFO[0000]main.go:812 setupDatabase Loaded migration scripts
INFO[0000]main.go:821 setupDatabase Database was configured successfully.
INFO[0000]lock.go:65 ReleaseHostLock Released 0 locks held by 6bb434ef78d1
INFO[0000]s3config.go:114 initialize Hot storage: b2-eu-cen (secondary: wasabi-eu-central-2-v3)
ERRO[0000]offer.go:49 NewOfferController Error reading config fileopen : no such file or directory
ERRO[0000]offer.go:53 NewOfferController Error un-marshalling JSONunexpected end of JSON input
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory
INFO[0000]replication3.go:96 StartReplication replication.worker-url was not defined, files will downloaded directly during replication
INFO[0000]replication3.go:146 createTemporaryStorage Temporary storage for replication v3 is: /replication/data
INFO[0000]replicate.go:22 StartReplication replication.worker-url was not defined, file data will downloaded directly during replication
INFO[0000]object_cleanup.go:107 StartRemovingUnreportedObjects Starting 1 workers to remove-unreported-objects
INFO[0000]replicate.go:36 startWorkers Starting 6 workers for replication v3
INFO[0000]replication3.go:120 startWorkers Starting 6 workers for replication v3
INFO[0000]delete.go:22 startDeleteWorkers Starting 1 delete workers for fileData
INFO[0000]object_cleanup.go:132 removeUnreportedObjects Removing unreported objects task=remove-unreported-objects
INFO[0000]discord.go:56 sendMessage Skipping sending Discord message: 6bb434ef78d1 has taken off 🚀
INFO[0000]main.go:734 main We have lift-off.
INFO[0000]object_cleanup.go:152 removeUnreportedObjects Removed 0 objects task=remove-unreported-objects
INFO[0000]repository.go:79 DeleteOldKeys Deleted old keys less than 1731649098611137 old
INFO[0000]replication3.go:261 tryReplicate Replication attempt start object_key=1580559962386438/51053210-54c9-409b-aee8-86039d162d3d task=replication
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe74d2]

goroutine 35 [running]:
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).downloadObject(0xc000c4ae10, {0x16b4fd0, 0xc00083c0e0}, {0xc000f100f0, 0x2a}, {0xc000fce9d0, 0xc})
/etc/ente/pkg/controller/filedata/s3.go:61 +0x1f2
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicateRowData(0xc000c4ae10, {0x16b4fd0, 0xc00083c0e0}, {0x99707a, 0x59d82cb1ab006, {0xc000fce9c0, 0x6}, 0x1a8f, {0xc000fce9d0, 0xc}, ...})
/etc/ente/pkg/controller/filedata/replicate.go:97 +0x238
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).tryReplicate(0xc000c4ae10)
/etc/ente/pkg/controller/filedata/replicate.go:70 +0x230
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicate(0xc0003caf50?, 0x0)
/etc/ente/pkg/controller/filedata/replicate.go:50 +0x25
created by github.com/ente-io/museum/pkg/controller/filedata.(*Controller).startWorkers in goroutine 63
/etc/ente/pkg/controller/filedata/replicate.go:39 +0x77
INFO[0000]main.go:95 main Booting up local server with commit #b0674cd5f37cebda0f446db7ce1e87a3e6d8b9fc
INFO[0000]main.go:791 setupDatabase Setting up db
INFO[0000]main.go:798 setupDatabase Connected to DB
INFO[0000]main.go:803 setupDatabase Pinged DB
INFO[0000]main.go:812 setupDatabase Loaded migration scripts
INFO[0000]main.go:821 setupDatabase Database was configured successfully.
INFO[0000]lock.go:65 ReleaseHostLock Released 0 locks held by 6bb434ef78d1
INFO[0000]s3config.go:114 initialize Hot storage: b2-eu-cen (secondary: wasabi-eu-central-2-v3)
ERRO[0000]offer.go:49 NewOfferController Error reading config fileopen : no such file or directory
ERRO[0000]offer.go:53 NewOfferController Error un-marshalling JSONunexpected end of JSON input
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory
INFO[0000]replication3.go:96 StartReplication replication.worker-url was not defined, files will downloaded directly during replication
INFO[0000]replication3.go:146 createTemporaryStorage Temporary storage for replication v3 is: /replication/data
INFO[0000]replicate.go:22 StartReplication replication.worker-url was not defined, file data will downloaded directly during replication
INFO[0000]replication3.go:120 startWorkers Starting 6 workers for replication v3
INFO[0000]delete.go:22 startDeleteWorkers Starting 1 delete workers for fileData
INFO[0000]replicate.go:36 startWorkers Starting 6 workers for replication v3
INFO[0000]object_cleanup.go:107 StartRemovingUnreportedObjects Starting 1 workers to remove-unreported-objects
INFO[0000]object_cleanup.go:132 removeUnreportedObjects Removing unreported objects task=remove-unreported-objects
INFO[0000]discord.go:56 sendMessage Skipping sending Discord message: 6bb434ef78d1 has taken off 🚀
INFO[0000]main.go:734 main We have lift-off.
INFO[0000]repository.go:79 DeleteOldKeys Deleted old keys less than 1731649116947697 old
INFO[0000]object_cleanup.go:152 removeUnreportedObjects Removed 0 objects task=remove-unreported-objects
INFO[0000]replication3.go:261 tryReplicate Replication attempt start object_key=1580559962386438/c34f6d4c-f97e-4cbe-af85-a4099f3c63d4 task=replication
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe74d2]

goroutine 51 [running]:
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).downloadObject(0xc000b44820, {0x16b4fd0, 0xc000e920e0}, {0xc00189c150, 0x2a}, {0xc000d80190, 0xc})
/etc/ente/pkg/controller/filedata/s3.go:61 +0x1f2
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicateRowData(0xc000b44820, {0x16b4fd0, 0xc000e920e0}, {0x996ba3, 0x59d82cb1ab006, {0xc000d80180, 0x6}, 0x24f7, {0xc000d80190, 0xc}, ...})
/etc/ente/pkg/controller/filedata/replicate.go:97 +0x238
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).tryReplicate(0xc000b44820)
/etc/ente/pkg/controller/filedata/replicate.go:70 +0x230
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicate(0xc000b44820?, 0x0)
/etc/ente/pkg/controller/filedata/replicate.go:50 +0x25
created by github.com/ente-io/museum/pkg/controller/filedata.(*Controller).startWorkers in goroutine 66
/etc/ente/pkg/controller/filedata/replicate.go:39 +0x77
INFO[0000]main.go:95 main Booting up local server with commit #b0674cd5f37cebda0f446db7ce1e87a3e6d8b9fc
INFO[0000]main.go:791 setupDatabase Setting up db
INFO[0000]main.go:798 setupDatabase Connected to DB
INFO[0000]main.go:803 setupDatabase Pinged DB
INFO[0000]main.go:812 setupDatabase Loaded migration scripts
INFO[0000]main.go:821 setupDatabase Database was configured successfully.
INFO[0000]lock.go:65 ReleaseHostLock Released 0 locks held by 6bb434ef78d1
INFO[0000]s3config.go:114 initialize Hot storage: b2-eu-cen (secondary: wasabi-eu-central-2-v3)
ERRO[0000]offer.go:49 NewOfferController Error reading config fileopen : no such file or directory
ERRO[0000]offer.go:53 NewOfferController Error un-marshalling JSONunexpected end of JSON input
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory
ERRO[0000]billing.go:91 parsePricingFile Error reading file : open : no such file or directory
INFO[0000]replication3.go:96 StartReplication replication.worker-url was not defined, files will downloaded directly during replication
INFO[0000]replication3.go:146 createTemporaryStorage Temporary storage for replication v3 is: /replication/data
INFO[0000]replicate.go:22 StartReplication replication.worker-url was not defined, file data will downloaded directly during replication
INFO[0000]object_cleanup.go:107 StartRemovingUnreportedObjects Starting 1 workers to remove-unreported-objects
INFO[0000]replication3.go:120 startWorkers Starting 6 workers for replication v3
INFO[0000]replicate.go:36 startWorkers Starting 6 workers for replication v3
INFO[0000]delete.go:22 startDeleteWorkers Starting 1 delete workers for fileData
INFO[0000]object_cleanup.go:132 removeUnreportedObjects Removing unreported objects task=remove-unreported-objects
INFO[0000]discord.go:56 sendMessage Skipping sending Discord message: 6bb434ef78d1 has taken off 🚀
INFO[0000]main.go:734 main We have lift-off.
INFO[0000]repository.go:79 DeleteOldKeys Deleted old keys less than 1731649135457333 old
INFO[0000]object_cleanup.go:152 removeUnreportedObjects Removed 0 objects task=remove-unreported-objects
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe74d2]

goroutine 51 [running]:
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).downloadObject(0xc0009caff0, {0x16b4fd0, 0xc0010b2070}, {0xc001129230, 0x2a}, {0xc000ec0e90, 0xc})
/etc/ente/pkg/controller/filedata/s3.go:61 +0x1f2
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicateRowData(0xc0009caff0, {0x16b4fd0, 0xc0010b2070}, {0x996c3e, 0x59d82cb1ab006, {0xc000ec0e88, 0x6}, 0x1a8f, {0xc000ec0e90, 0xc}, ...})
/etc/ente/pkg/controller/filedata/replicate.go:97 +0x238
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).tryReplicate(0xc0009caff0)
/etc/ente/pkg/controller/filedata/replicate.go:70 +0x230
github.com/ente-io/museum/pkg/controller/filedata.(*Controller).replicate(0x0?, 0x0)
/etc/ente/pkg/controller/filedata/replicate.go:50 +0x25
created by github.com/ente-io/museum/pkg/controller/filedata.(*Controller).startWorkers in goroutine 43
/etc/ente/pkg/controller/filedata/replicate.go:39 +0x77`

[mnvr]

Yes. Currently replication only works in the exact 3 (1 hot, 1 cold, 1 glacier) setup.

#3167 (comment)

[mngshm]

This is a wild guess:

But you've set primary s3 to cold storage (like you mentioned) and the second (is perhaps hot?). That could be another reason why it is failing. Also, like @mnvr mentioned in the Answer for this discussion, s3 will only work in the exact 3 (1 hot, 1 cold, 1 glacier) setup.

[fredix]

Hi @mngshm ,
it seems the b2-eu-cen bucket is not the cold but the hot storage. My setup is :

s3:
    # Override the primary and secondary hot storage. The commented out values
    # are the defaults.
    #
    hot_storage:
        primary: b2-eu-cen
        secondary: wasabi-eu-central-2-v3
    wasabi-eu-central-2-v3:
        key: KEY
        secret: SECRET
        endpoint: http://LOCAL_IP:3900
        region: garage
        bucket: ente
    b2-eu-cen:
        key: KEY
        secret: SECRET
        endpoint: https://s3.fr-par.scw.cloud
        region: fr-par
        bucket: bucket
    are_local_buckets: false

[smncd]

@fredix replication might be unreliable with only 2 buckets set up but in my experience that shouldn't crash the app... Perhaps try Scaleway only and see what it does?

[fredix]

With only one S3 bucket at Scaleway it works but it's not the point. I want a replication from Scaleway to my S3 bucket at home. I will test to build a third S3 at home to match the requested architecture.