Be confident when client sync do not sync all files

[Kwadz]

Hello,

Sometimes Pydio does not sync properly server to clients. To be more confident with the client, what do you with about implementing a comparison between the number of files on the server and locally?
This would allow the user to know if some files are not synchronized.

Kwadz

[7omate]

Yes.
Many small tricks could be used to increase confidence that everything is going fine, this is one that we're seriously considering.
Currently this is not done because it would require a new API call. Soon we'll tackle a redesign of major parts of the server and client but until them, we do our best with what we have.

When are some files not synchronized?

[Kwadz]

When the client try to sync new files from the server (shared hosting). I have already seen the pydio folder icon green and some files was missing.
I noticed a problem with the letter E with accent (É), it seems it's not supported.
Also we can see folder icons blue (synchronizing) and the pydio state in the top bar on synchronized state:

[7omate]

Hi, I just checked, from a Mac to a Linux server with a folder called Éllà, everything worked fine. There might be something wrong in your server configuration.
However I'd like to warn you, files manipulated from the web interface by a Mac can result in encoding issues to PydioSync.
Since the pdf file is marked as Uploading and you're using shared hosting (insufficient resources) it could be that the transfer failed, try to resync clicking on the resync button from the settings of the sync task.
A slow reprocessor for failed changes is ready, it will be out soon.

[Kwadz]

Thank you for this information. Is there an open issue about your warn on files manipulated from the web interface by a Mac?
Glad to know about a new reprocessor. In his current state, due to some synchronization problems, we can't really trust in Pydio. As we said this can be corrected by implementing a checking (for example a recursive checking of number of files per folder). That said, if the number of files is wrong how to make able the app to resync the related folder automatically? Is your current reprocessor able to do it?
I think with the md5 checking + number of files checking + auto reprocess on failed or bad number, we will be 100% confident :-)

[7omate]

There is no public open issue, but we're working on it.
You can trust PydioSync if you have reasonable hardware and a good connection. Otherwise the wrong icons are a sign that you need to resync...

[Kwadz]

Sorry 7omate, I am not agree with your last post.

First, some files can be not synchronized from the server while the root icon sign is green. I had the issue. If there's missing files due to an indexation issue, how can the user know it without any notification?

Secondly, I'm not talking about the hardware or the Internet connection quality. If I remember correctly, there's already a MD5 checking, right? So, we can trust in the transport, the connection quality doesn't matter. However, how can we trust in the files indexation management?

Even if it's already a good tool, it only lacks one feature to make the sync process 100% under control and make the app totally reliable. In my opinion, the level of confidence make the difference between a reliable tool and a gadget. The data integrity should not be taken lightly.

As there is no checking beyond the transport, how can we trust in Pydio in his current state?

[7omate]

There are options to check:

• monitor pydio.log for ERRORS
• use the local API http://localhost:5556/check_sync/sync_id

[Kwadz]

Thank you for the workaround. As I can't spend time every day to check Pydio logs, I'll wait the integration of the automated checking feature.

[7omate]

Small update here, the missing É is probably linked to a miss-configured server. See sysvano last answers (pydio/pydio-core#1272)

[7omate]

What's the status of this ?

Wait for the sync between each file, also if you notice problems, please have a look at pydio.log (About > Open Logs from PydioSync interface).
Then visit https://YOURSERVER/pydio/api/ws-slug/changes/0 (Or send me credentials to reproduce)

Please do some more precise tests:

• create a folder on the server F_Éllò
• create a file on the server Éllàw.txt
• create a file on the client ÉllàClient.txt
• create a folder on the client F_ÉllàClient

[Kwadz]

I just made a fresh install of the server and the client (with last version for both of them). There are still issues about synchronization and accent support.

I don't think the icon on the root folder should be green while the subfolders are not.

Please find attached screenshots below. I just also made a new pydio server for you:
http://protonik.org/pydio-core-7.0.4/

Here are the credentials ( are same for the db and pydio):

database: protonik_pydio_test
user: protonik_pydio_test
password: pydio_test

---

---

[7omate]

Indeed the icons should be the same. As in: the top most icon status' is defined to be the "worst" status of the children of the folders. (Maybe this isn't true for the root for some reason?)

Please check on http://demo.pyd.io

For me creating files / folders called Électronique works fine both from the client and from the web.

I tried connecting to your server but it doesn't work, either you or someone else changed the credentials.

You should look into your server's encoding

[7omate]

[Kwadz]

I created a new one, with the following credentials:

db name: protonik_pydiotest
user: protonik_pydiotest
pass: REDACTED

[7omate]

I added files, please have a look and try to sync.

[Kwadz]

Did you change something? It seems to synchronize the files properly. However I realize that the status is not displayed in real time, for example it's "Idle (Synchronized)" while it's still uploading, same about the icons (all are green while it's still uploading).

I also noticed that some system files are not synchronized:

If it's normal, the icon should be different than the blue one. No icon at all would be better. That could be a new issue.

For your information, the resources consumption are pretty high:

[7omate]

The thing I changed and you probably missed was to configure an encoding for Pydio EN_US.UTF-8 instead of just UTF-8. It's strange, because the fallback value since v7 should be EN_US.UTF-8 but on your server it displayed UTF-8 .

.DS_Store -> I don't even know how to display hidden files in Finder, they are in the "exclude" list it can be your choice to synchronize those files but I doubt there would be any use.

We are doing concurrent HTTP connections and transferring files and serving a web UI... The resource should be very low when the agent is idle ("Synchronized"). You can set the number of concurrent connections in the task settings.

The "magic" green instead of "still uploading" could be that the UI is actually letting you believe some transfers are still happening but they were already done. If you want to understand what's going on look at pydio.log

[Kwadz]

I'm going to test changing encoding.

About .DS_Store if the file is excluded, no icon (neither green or blue) should be displayed. Icons are also supposed to show if a file is followed or not by Pydio.

The "magic" green instead of "still uploading" could be that the UI is actually letting you believe some transfers are still happening but they were already done. If you want to understand what's going on look at pydio.log

User should not have to have a look at the logs after each sync to check if everything's ok. That should be done by the icons. If the icons show wrong information, they are useless. So, the convention for the icons should be:

• green: synchronized (really)
• blue: still downloading/uploading
• red: error, not synchronized
• no icon: not tracked

The parent directory icon should show the worst case in its children directories icon.
Looking at the log should only be required to understand why there is a red icon.

And, of course, the Pydio UI should match with the icons:

I think this is the foundation of a reliable icon management. What do you think?

[Kwadz]

Where did you change the encoding?

[Kwadz]

@7omate Where have you been able to change the encoding please?

[7omate]

In the file conf/bootstrap_conf.php

define("AJXP_LOCALE", "en_US.UTF-8");

You'll find details in the https://pydio.com/en/docs/v8/quick-start "Advanced > Encoding"

[Kwadz]

@7omate Are you sure that you modified conf/bootstrap_conf.php?
Because I just checked it:

<?php
/*
 * Copyright 2007-2013 Charles du Jeu <contact (at) cdujeu.me>
 * This file is part of Pydio.
 *
 * Pydio is free software: you can redistribute it and/or modify
 * it under the terms of the GNU Affero General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * Pydio is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU Affero General Public License for more details.
 *
 * You should have received a copy of the GNU Affero General Public License
 * along with Pydio.  If not, see <http://www.gnu.org/licenses/>.
 *
 * The latest code can be found at <https://pydio.com>.
 *
 * These configuration must be set at the very root loading of the framework
 */

/*
 * If you have a charset warning, or problems displaying filenames with accented characters,
 * check your system locale and set it in the form lang_country.charset
 * Example : fr_FR.UTF-8, fr_FR.ISO-8859-1, fr_FR.CP1252 (windows), en_EN.UTF-8, etc.
 *
 * Windows users may define an empty string
 * define("AJXP_LOCALE", "");
 */
//define("AJXP_LOCALE", "en_EN.UTF-8");
//define("AJXP_LOCALE", "");


/*
 * If you encounter problems writing to the standard php tmp directory, you can
 * define your own tmp dir here. Suggested value is ajxp_path/data/tmp/
 * AJXP_DATA_PATH, AJXP_INSTALL_PATH are replaced automatically.
 *
 * See php.ini settings below for the session.save_path value as well.
 */
//define("AJXP_TMP_DIR", AJXP_DATA_PATH."/tmp");


/*
 * Additionnal php.ini settings
 * > Problems with tmp dir : set your own session tmp dir (create it and make it writeable!)
 * > Concurrent versions of AjaXplorer : use session.cookie_path to differentiate them.
 */
$AJXP_INISET = array();
//$AJXP_INISET["session.save_path"] = AJXP_DATA_PATH."/tmp/sessions";
//$AJXP_INISET["session.cookie_path"] = "/ajaxplorer";

/*
 * If you want to force the https, uncomment the line below. This will automatically
 * redirect all calls to ajaxplorer via http to the same URL with https
 */
//define("AJXP_FORCE_SSL_REDIRECT", true);

[Kwadz]

Just in case I add interesting lines from pydio.log:

2017-08-02 17:29:37 ERROR   4480036864 Thread-18 logging : 1601 | error | Invalid JSON Response, line was {"seq"","repository_identifier":"1-kwad"}}d98f00b204e9800998ecf8427e","mtime":1486424197,"node_path":"\/Icon
2017-08-02 17:29:37 ERROR   4480036864 Thread-18 logging : 1601 | error | Invalid JSON value received while getting remote changes
Traceback (most recent call last):
  File "/Users/charles/Desktop/TeamCity/buildAgent/work/a00136d723583ad3/build/pydio-agent/out01-PYZ.pyz/pydio.job.continous_merger", line 464, in run
  File "/Users/charles/Desktop/TeamCity/buildAgent/work/a00136d723583ad3/build/pydio-agent/out01-PYZ.pyz/pydio.job.continous_merger", line 786, in load_remote_changes_in_store
  File "/Users/charles/Desktop/TeamCity/buildAgent/work/a00136d723583ad3/build/pydio-agent/out01-PYZ.pyz/pydio.sdkremote.remote", line 438, in changes_stream
Exception: Invalid JSON value received while getting remote changes

I wonder how comes File "/Users/charles/Desktop/TeamCity/buildAgent/work can be there...

[Kwadz]

@7omate, what do you think about my comment regarding the icons management and its consistency?