Dropbox crashes, java 64 & colors in ImageJ

[ppouchin]

Hi,

I've set up the DropBox to automatically import files from the users' folders, but periodically, files stop being imported so I need to restart the server and refresh the files which were not imported, but not the others as to avoid duplicates...
I have made a small script that checks if each file from the server has been imported into OMERO and outputs commands (bash) that I then run as root to refresh the files, but if I were able to directly start the imports from the script (run by the user), it would be nice. I have not found such a function in the API though...
Could the crashes be caused by too much data being imported at the same time ?

I also noticed that master.err is filled with:

Code: Select all

2012-04-16 11:03:46,033 4416       [      main] WARN   enums.handlers.MicroscopeTypeEnumHandler  - Unknown MicroscopeType value 'Unknown' will be stored as "Other"

So the Leica SP5 stores its type as "unknown"... ? Isn't it more an "INFO" than a "WARN" ?

Moreover, I saw this warning in the logs:

Code: Select all

loci.formats.MissingLibraryException: QuickTime for Java is not supported with a 64-bit JVM. Please invoke the 32-bit JVM (-d32) to utilize QTJava functionality.

What should I do ? Install a 32-bit JVM ? Change something ?

Finally, when I import images in ImageJ or Fiji through the plugin, the images lose their colors. Would there be a way for users to get their channel colors... ?

[cblackburn]

Hi,

I've set up the DropBox to automatically import files from the users' folders, but periodically, files stop being imported so I need to restart the server and refresh the files which were not imported, but not the others as to avoid duplicates...

Could the crashes be caused by too much data being imported at the same time ?

It is possible that too many files being copied into the directories at once could be overwhelming the import process. Could you send the logs for a session where such a restart was needed by zipping up the whole log directory?

Colin

[cxallan]

Hi,
...
Moreover, I saw this warning in the logs:

Code: Select all

loci.formats.MissingLibraryException: QuickTime for Java is not supported with a 64-bit JVM. Please invoke the 32-bit JVM (-d32) to utilize QTJava functionality.

What should I do ? Install a 32-bit JVM ? Change something ?
...

There's really very little you can do about this, particularly on Linux. It's just a warning that extended QuickTime functionality is not available without QuickTime for Java installed and without running a 32-bit JVM. As long as you don't have the intention of importing "exotic" QuickTime movies into OMERO you can safely ignore the warning.

[ppouchin]

Hi,
...
Moreover, I saw this warning in the logs:

Code: Select all

loci.formats.MissingLibraryException: QuickTime for Java is not supported with a 64-bit JVM. Please invoke the 32-bit JVM (-d32) to utilize QTJava functionality.

What should I do ? Install a 32-bit JVM ? Change something ?
...

There's really very little you can do about this, particularly on Linux. It's just a warning that extended QuickTime functionality is not available without QuickTime for Java installed and without running a 32-bit JVM. As long as you don't have the intention of importing "exotic" QuickTime movies into OMERO you can safely ignore the warning.

Ok !

Hi,

I've set up the DropBox to automatically import files from the users' folders, but periodically, files stop being imported so I need to restart the server and refresh the files which were not imported, but not the others as to avoid duplicates...

Could the crashes be caused by too much data being imported at the same time ?

It is possible that too many files being copied into the directories at once could be overwhelming the import process. Could you send the logs for a session where such a restart was needed by zipping up the whole log directory?

Colin

I zipped it and tried to upload it, but it doesn't work.
However, I think I found why it crashed several times in the past months.
One of my users like to make some access restrictions on his data (770). I thought I had solved it by adding the user "omero" (that runs the server) to the group "users". But I still have something like that in MonitorServer.log :

Code: Select all

2012-05-14 11:29:57,331 ERROR [                               pyinotify] (Thread-4  ) add_watch: cannot watch /data/OMERO/DropBox/username/some_folder WD=-1 Errno=Permission denied (EACCES)
2012-05-14 11:29:57,396 WARNI [                                  stderr] (Thread-4  ) Exception in thread Thread-4:

Which means, he must be setting them on 700 sometimes...

I also have 1 weird line :

Code: Select all

2012-02-24 19:09:36,256 WARNI [                                  stderr] (Dummy-6   ) 2012-02-24 19:09:36.256649 MonitorServer: warning: dispatch exception: ObjectAdapterI.cpp:993: Ice::ObjectAdapterDeactivatedException:
object adapter `omerofs.MonitorServer' deactivated
identity: MonitorServer
facet:
operation: stopMonitor

[cblackburn]

Hi,

it looks like having inaccessible files or directories in the DropBox folders is causing problems for the file monitoring system. The monitoring system should be more robust and so I have raised to a ticket to investigate and fix this:

http://trac.openmicroscopy.org.uk/ome/ticket/9156

If you'd like to be cc'ed on updates to this ticket let me know.

We would recommend, though, that directories and files in the DropBox area are accessible so that changes can be monitored and files imported successfully.

Cheers,

Colin

[ppouchin]

That's what I thought...

It seems I should tell my users not to hide their files...

Thank you for your help !

[ppouchin]

Apparently, the program responsible for this crash is rsync.
Since I can not modify the way it behaves by default, I had a look at the DropBox error and I might have a fix.

In the script lib/python/path.py, I replaced the following line:

Code: Select all

names = os.listdir(self)

by this:

Code: Select all

try:
  names = os.listdir(self)
except OSError:
  # ignore unreadable directories
  names = []


I tried to create an unreadable folder, and the MonitorServer didn't crash...

[cblackburn]

Hi,

thanks for letting us know how you fixed this locally. path.py is a third-party package that is no longer maintained and so we have already made some modifications ourselves. I'll add details of your fix to the ticket and then look at incorporating this.

Cheers,

Colin

[ppouchin]

By the way, what settings could I use to silence some of the events in the MonitorServer.log ?
I can change the global python server log level from INFO to WARN, but it does not just affect the MonitorServer log file...

[cblackburn]

Hi,

you can change the Python logging in:

Code: Select all

$OMERO_HOME\etc\grid\templates.xml

To change the logging for all Python processes change omero.logging.level property under:

Code: Select all

    <properties id="PythonServer">
      <property name="Ice.ImplicitContext" value="Shared"/>
      <!-- Default logging settings for Python servers. -->
      <property name="omero.logging.timedlog" value="False"/>
      <property name="omero.logging.logsize" value="5000000"/>
      <property name="omero.logging.lognum" value="9"/>
      <property name="omero.logging.level" value="20"/>
      <target name="debug">
        <property name="omero.logging.level" value="10"/>
      </target>
      <!-- From the python documentation
            CRITICAL  50
            ERROR     40
            WARNING   30
            INFO      20
            DEBUG     10
            NOTSET    0
      -->
    </properties>


To override this setting for just the MonitorServer add an omero.logging.level property to:

Code: Select all

    <server-template id="MonitorServerTemplate">
        ...
        <properties>
          ...
          <property name="omero.logging.level" value="30"/>
        </properties>
      </server>
    </server-template>


Cheers,

Colin