Dropbox halts on certain unicode characters

[Sethur]

Hi,

there is again a problem in OMERO v5.1.0 with using certain non standard unicode characters in filenames. Our Dropbox just halted because someone used the German umlaut "ä" in a file name (at least I think that this caused it). The MonitorServer.log tails as follows:

Code: Select all

2015-05-13 11:56:41,236 INFO  [        fsclient.fsNotificationScheduler] (Thread-5  ) Notification queue size = 1
2015-05-13 11:56:41,236 INFO  [        fsclient.fsNotificationScheduler] (Thread-5  ) Notification queue 1 items removed.
2015-05-13 11:56:41,237 INFO  [        fsclient.fsNotificationScheduler] (Thread-5  ) Notification queue size = 0
2015-05-13 11:56:41,237 INFO  [                fsserver.fsMonitorServer] (Thread-5  ) Event notification on monitor id= e34e826a-f7c1-11e4-8ac4-001e6752dcf8
2015-05-13 12:08:00,555 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) New directory event of type IN_CREATE|IN_ISDIR at: /srv/omero_data/DropBox/jmertins/New folder
2015-05-13 12:08:00,555 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) Not propagated.
2015-05-13 12:08:04,476 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) Deleted directory event of type IN_MOVED_FROM|IN_ISDIR at: /srv/omero_data/DropBox/jmertins/New folder
2015-05-13 12:08:04,476 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) Not propagated.
2015-05-13 12:08:04,477 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) New directory event of type IN_MOVED_TO|IN_ISDIR at: /srv/omero_data/DropBox/jmertins/MM Färbung
2015-05-13 12:08:04,477 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) Not propagated.
2015-05-13 12:08:04,542 WARNI [                                  stderr] (Thread-4  ) Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/share/omero/OMERO.server-5.1.0-ice35-b40/lib/python/omero_ext/pyinotify.py", line 1411, in run
    self.loop()
  File "/usr/local/share/omero/OMERO.server-5.1.0-ice35-b40/lib/python/omero_ext/pyinotify.py", line 1397, in loop
    self.process_events()
  File "/usr/local/share/omero/OMERO.server-5.1.0-ice35-b40/lib/python/omero_ext/pyinotify.py", line 1198, in process_events
    self._default_proc_fun(revent)
  File "/usr/local/share/omero/OMERO.server-5.1.0-ice35-b40/lib/python/omero_ext/pyinotify.py", line 841, in __call__
    return _ProcessEvent.__call__(self, event)
  File "/usr/local/share/omero/OMERO.server-5.1.0-ice35-b40/lib/python/omero_ext/pyinotify.py", line 578, in __call__
    return self.process_default(event)
  File "/usr/local/share/omero/OMERO.server-5.1.0-ice35-b40/lib/python/fsPyinotifyMonitor.py", line 260, in process_default
    pathModule.path(name).parent].getAutoAdd():
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 37: ordinal not in range(128)


Any ideas how to fix this?

Cheers,

Tristan

[cblackburn]

Hi Tristan,

Thanks for raising this issue. The underlying Python library that handles file paths was modified on the 5.1 branch and that seems to have caused this regression. We are currently investigating this and will be looking to push a fix as soon as possible.

Cheers,

Colin

[Sethur]

Hi Colin,

I updated to v5.1.1 last week and the problem is even more severe now. The DropBox now crashes on directly after startup of the OMERO.server with the same error, i.e.

Code: Select all

    ignoreSysFiles, ignoreDirEvents, platformCheck, proxy, monitorId)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsMonitor.py", line 73, in __init__
    ignoreSysFiles, ignoreDirEvents, self)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsPyinotifyMonitor.py", line 93, in __init__
    auto_add=follow)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/    ignoreSysFiles, ignoreDirEvents, platformCheck, proxy, monitorId)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsMonitor.py", line 73, in __init__
    ignoreSysFiles, ignoreDirEvents, self)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsPyinotifyMonitor.py", line 93, in __init__
    auto_add=follow)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsPyinotifyMonitor.py", line 153, in addBaseWatch
    self.addWatch(str(d), mask)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsPyinotifyMonitor.py", line 165, in addWatch
    self.addWatch(str(d), mask)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 37: ordinal not in range(128)", line 153, in addBaseWatch
    self.addWatch(str(d), mask)
  File "/usr/local/share/omero/OMERO.server-5.1.1-ice35-b43/lib/python/fsPyinotifyMonitor.py", line 165, in addWatch
    self.addWatch(str(d), mask)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 37: ordinal not in range(128)


This is probably because some watches on already existing files with problematic unicode characters (mainly German umlauts) are added after startup. Possibly between v5.1.0 and v5.1.1 there was a change in the code that addressed the way the DropBox starts up.

In any case, this is bad news for us since we cannot use the DropBox at all now until a fix is released. Could you maybe point me at the proper code line of the python code so that I can work around this until a patch gets released?

Cheers,

Tristan

[Sethur]

PS: Apparently this error is the very first example in the Python Unicode HOWTO and comes from using str() instead of encode()...

EDIT: Fixed it by removing the unnecessary str(...) in line 165 and 153 in OMERO.server-5.1.1-ice35-b43/lib/python/fsPyinotifyMonitor.py. Apparently the variable d was already in unicode. I hope this does not break anything else.

[cblackburn]

Hi Tristan,

Thanks for the replies. We are looking at the same parts of the code ourselves here. It may be that your patch isn't quite extensive enough for all cases, but we are looking into it and will be pushing a patch shortly.

Cheers,

Colin

[Sethur]

Hi,

today, our DropBox crashed again because somebody created a directory with Unicode characters. After the crash, the DropBox did not recover, but halted until a restart of the resprective OMERO subsystems. The error log is as follows:

Code: Select all

2015-06-23 11:51:03,556 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) New directory event of type IN_MOVED_TO|IN_ISDIR at: /srv/omero_data/DropBox/someuser/150623_Bilder für SomeInstitute
2015-06-23 11:51:03,557 INFO  [             fsserver.fsPyinotifyMonitor] (Thread-4  ) Not propagated.
2015-06-23 11:51:03,629 WARNI [                                  stderr] (Thread-4  ) Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/share/omero/OMERO.server-5.1.2-ice35-b45/lib/python/omero_ext/pyinotify.py", line 1411, in run
    self.loop()
  File "/usr/local/share/omero/OMERO.server-5.1.2-ice35-b45/lib/python/omero_ext/pyinotify.py", line 1397, in loop
    self.process_events()
  File "/usr/local/share/omero/OMERO.server-5.1.2-ice35-b45/lib/python/omero_ext/pyinotify.py", line 1198, in process_events
    self._default_proc_fun(revent)
  File "/usr/local/share/omero/OMERO.server-5.1.2-ice35-b45/lib/python/omero_ext/pyinotify.py", line 841, in __call__
    return _ProcessEvent.__call__(self, event)
  File "/usr/local/share/omero/OMERO.server-5.1.2-ice35-b45/lib/python/omero_ext/pyinotify.py", line 578, in __call__
    return self.process_default(event)
  File "/usr/local/share/omero/OMERO.server-5.1.2-ice35-b45/lib/python/fsPyinotifyMonitor.py", line 279, in process_default
    pathModule.path(name).parent].getAutoAdd():
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)


I'd be happy if somebody points out what I need to change in the python code.

Cheers,

Tristan

[jmoore]

Hi,

Hi Tristan,

today, our DropBox crashed again because somebody created a directory with Unicode characters. After the crash, the DropBox did not recover, but halted until a restart of the resprective OMERO subsystems. The error log is as follows:
...
I'd be happy if somebody points out what I need to change in the python code.

There was a longish investigation after your last problem. (Much of which was captured under the dropbox-and-unicode-characters card but there were various attempts like my path-unicode branch) Ultimately, the problem is that different methods in path.py have different behavior for unicode characters. We found no small code change which would work across the board. For example, without adding in normalization, etc. "path1 == path2" fails as does "path1 in some_collection". In other words, it's going to take an extensive review and possibly partial rewrite to make DropBox behave correctly.

That being said, there were workarounds which we found if you think they'll be sufficiently safe in your Python environment.

See https://fedoraproject.org/wiki/Features/PythonEncodingUsesSystemLocale (and a number of other posts like http://www.ianbicking.org/illusive-setdefaultencoding.html). They all come down to forcing the use of UTF-8 across the board, for example by putting

Code: Select all

import sys
reload(sys)
sys.setdefaultencoding('utf-8')


either in a sitecustomize or usercustomize file:

Code: Select all

$ cat > sitecustomize.py << EOF
> import sys
> reload(sys)
> sys.setdefaultencoding('utf-8')
> EOF
$ python -c "import sys; print sys.getdefaultencoding()"
ascii
$ PYTHONPATH=. python -c "import sys; print sys.getdefaultencoding()"
utf-8


which could be done only for the DropBox process itself.

Cheers,
Tristan

Cheers,
~Josh.

[Sethur]

Hi Josh,

thanks for the quick response and the suggested workaround. I will try to add this to our server asap.

I cannot access to linked Trello page, I think you need to add me first, my username there is also "Sethur".

Cheers,

Tristan

PS: Currently, I'm always restarting the whole server when the DropBox has crashed, but somebody mentioned a more efficient way to restart only this subsystem. Could you point that out again?

[jmoore]

Hi Josh,

Hi Tristan,

thanks for the quick response and the suggested workaround. I will try to add this to our server asap.

+1. Keep us posted.

I cannot access to linked Trello page, I think you need to add me first, my username there is also "Sethur".

Ah, sorry about that. That board was still set to "organization only". It's now public and you've been added to the org, the board, and the card.

PS: Currently, I'm always restarting the whole server when the DropBox has crashed, but somebody mentioned a more efficient way to restart only this subsystem. Could you point that out again?

Code: Select all

bin/omero admin ice server stop DropBox


which will stop and restart the process if it's in a stalled state. If IceGrid has already tried to re-start the process several times, it will be in a disabled state. In that case, use:

Code: Select all

bin/omero admin ice server enable DropBox


You can also check beforehand which state it is in with:

Code: Select all

omero admin ice server state DropBox


Cheers,
~Josh.

[Sethur]

Ah thanks, this comes in handy, especially to quickly check if the DropBox is even running.

PS: Is there a way to find out which group is currently targeted for a particular user who wants to use the DropBox? I know it's supposed to be the last one that user was active in, but this is a quite vague definition.