Problems batch/macro exporting tiffs

[rowellw]

Hello,
I'd really appreciate a little help on this. Even an angry shove in the right direction would be nice.
I'm trying to set up a very basic batch macro to import a tiff (or bigtiff) using b-f importer, run a plugin on the image in place, and export the new tiff (or bigtiff) using b-f exporter. Performing these steps in normal/interactive mode works fine, but running it as a macro or as a batch macro gives me this error during the export step:
Output & Error Message

Code: Select all

Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
java.lang.IllegalArgumentException: Invalid Z size: 0
        at loci.formats.FormatTools.getZCTCoords(FormatTools.java:403)
        at loci.plugins.out.Exporter.run(Exporter.java:545)
        at loci.plugins.LociExporter.run(LociExporter.java:77)
        at ij.plugin.filter.PlugInFilterRunner.processOneImage(PlugInFilterRunner.java:256)
        at ij.plugin.filter.PlugInFilterRunner.<init>(PlugInFilterRunner.java:105)
        at ij.IJ.runUserPlugIn(IJ.java:187)
        at ij.IJ.runPlugIn(IJ.java:152)
        at ij.Executer.runCommand(Executer.java:127)
        at ij.Executer.run(Executer.java:64)
        at ij.IJ.run(IJ.java:251)
        at ij.macro.Functions.doRun(Functions.java:579)
        at ij.macro.Functions.doFunction(Functions.java:82)
        at ij.macro.Interpreter.doStatement(Interpreter.java:216)
        at ij.macro.Interpreter.doStatements(Interpreter.java:204)
        at ij.macro.Interpreter.run(Interpreter.java:103)
        at ij.macro.Interpreter.run(Interpreter.java:73)
        at ij.macro.Interpreter.run(Interpreter.java:84)
        at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:105)
        at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:90)
        at ij.IJ.runMacroFile(IJ.java:119)
        at ij.ImageJ.main(ImageJ.java:670)


I've found this occurs even if the intermediate step (image stabilization using the Image Stabilizer plugin http://www.cs.cmu.edu/~kangli/code/Imag ... lizer.html) is left out, as in the macro below:
Macro stabilize.ijm

Code: Select all

args = getArgument;
args_array = split(args, ":");
input_name = args_array[0];
output_name = args_array[1];

run("Bio-Formats Importer", "open=[" + input_name + "] autoscale color_mode=Default view=Hyperstack stack_order=XYCZT");
//this is where my Image Stabilization step would run
run("Bio-Formats Exporter", "save=[" + output_name + "] compression=Uncompressed");


This occurs with both a fresh installation of Fiji or a fresh installation of ImageJ for 64 bit Linux.

Thanks for any help,

William

[mlinkert]

Hi William,

Is the version of Bio-Formats that is installed completely up to date? In plain ImageJ, "Help > About Plugins > LOCI Plugins" will tell you the installed version; 4.4.6 is the most recent, though you may also want to try the latest development ("trunk") build if that doesn't work:

http://www.openmicroscopy.org/site/prod ... /downloads

In Fiji, "Help > Update Fiji" should give you the most recent version in Fiji's updater.

-Melissa

[rowellw]

Hi Melissa,

Thanks for the reply.

1) When I was posting this error yesterday, I was using standard ImageJ with the versions below:
ImageJ version 1.46r with Java 1.6.0_24 (64-bit)
LOCI Plugins for Imagej, revision 3f142f7, built 7 February 2013, Release 4.4.6

2) I actually tried again just now using FIJI with the following versions and it worked as -macro but not as -batch.
ImageJ version 1.47n5 with Java 1.6.0_24 (64-bit)
LOCI Plugins for Imagej, revision 0b6f4e3, built 18 March 2013, Release 4.5-DEV

3) And, following your suggestion, I downloaded the newest trunk dev build of LOCI Plugins into my standard ImageJ installation:
LOCI Plugins for ImageJ, revision e36c059, built 27 March 2013, Release 4.5-DEV
This works in -macro and -batch modes without a problem.

So it looks like updating to 4.5 fixed the problem. Thank you Melissa.

William

[rowellw]

Hi Melissa,

Unfortunately, it got more complicated. I need to be able to run this in headless mode.

When I run my gutted script (only importer and exporter plugins, in my original post) using Fiji + LOCI 4.5 in -macro mode, it works.

Code: Select all

~/bin/Fiji.app/ImageJ-linux64 --no-splash -macro stabilize.ijm test_in.tif:test_out.tif

When I run it in -batch mode it seems to work.

Code: Select all

~/bin/Fiji.app/ImageJ-linux64 --no-splash -batch stabilize.ijm test_in.tif:test_out.tif

When I run it in --headless mode, I get the following error:

Code: Select all

~/bin/Fiji.app/ImageJ-linux64 --no-splash --headless stabilize.ijm test_in.tif:test_out.tif

Error:

Code: Select all

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at ij.Command.runPlugIn(Command.java:148)
        at ij.Command.runCommand(Command.java:97)
        at ij.Executer.run(Executer.java:64)
        at ij.IJ.run(IJ.java:259)
        at ij.macro.Functions.doRun(Functions.java:586)
        at ij.macro.Functions.doFunction(Functions.java:89)
        at ij.macro.Interpreter.doStatement(Interpreter.java:219)
        at ij.macro.Interpreter.doStatements(Interpreter.java:207)
        at ij.macro.Interpreter.run(Interpreter.java:104)
        at ij.macro.Interpreter.run(Interpreter.java:74)
        at ij.macro.Interpreter.run(Interpreter.java:85)
        at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:120)
        at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:104)
        at ij.IJ.runMacroFile(IJ.java:127)
        at ij.ImageJ.main(ImageJ.java:678)
        at fiji.Main.main(Main.java:130)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at imagej.ClassLauncher.launch(ClassLauncher.java:224)
        at imagej.ClassLauncher.run(ClassLauncher.java:161)
        at imagej.ClassLauncher.main(ClassLauncher.java:72)
Caused by: java.lang.VerifyError: (class: loci/plugins/in/MainDialog, method: rebuildDialog signature: (Lij/gui/GenericDialog;)V) Incompatible argument to function
        at loci.plugins.in.ImporterPrompter.promptMain(ImporterPrompter.java:128)
        at loci.plugins.in.ImporterPrompter.statusUpdated(ImporterPrompter.java:81)
        at loci.plugins.in.ImportProcess.notifyListeners(ImportProcess.java:463)
        at loci.plugins.in.ImportProcess.step(ImportProcess.java:756)
        at loci.plugins.in.ImportProcess.execute(ImportProcess.java:144)
        at loci.plugins.in.Importer.showDialogs(Importer.java:141)
        at loci.plugins.in.Importer.run(Importer.java:79)
        at loci.plugins.LociImporter.run(LociImporter.java:81)
        at ij.IJ.runUserPlugIn(IJ.java:193)
        at ij.IJ.runPlugIn(IJ.java:160)
        ... 27 more
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at ij.Command.runPlugIn(Command.java:148)
        at ij.Command.runCommand(Command.java:97)
        at ij.Executer.run(Executer.java:64)
        at ij.IJ.run(IJ.java:259)
        at ij.macro.Functions.doRun(Functions.java:586)
        at ij.macro.Functions.doFunction(Functions.java:89)
        at ij.macro.Interpreter.doStatement(Interpreter.java:219)
        at ij.macro.Interpreter.doStatements(Interpreter.java:207)
        at ij.macro.Interpreter.run(Interpreter.java:104)
        at ij.macro.Interpreter.run(Interpreter.java:74)
        at ij.macro.Interpreter.run(Interpreter.java:85)
        at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:120)
        at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:104)
        at ij.IJ.runMacroFile(IJ.java:127)
        at ij.ImageJ.main(ImageJ.java:678)
        at fiji.Main.main(Main.java:130)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at imagej.ClassLauncher.launch(ClassLauncher.java:224)
        at imagej.ClassLauncher.run(ClassLauncher.java:161)
        at imagej.ClassLauncher.main(ClassLauncher.java:72)
Caused by: java.lang.VerifyError: (class: loci/plugins/in/MainDialog, method: rebuildDialog signature: (Lij/gui/GenericDialog;)V) Incompatible argument to function
        at loci.plugins.in.ImporterPrompter.promptMain(ImporterPrompter.java:128)
        at loci.plugins.in.ImporterPrompter.statusUpdated(ImporterPrompter.java:81)
        at loci.plugins.in.ImportProcess.notifyListeners(ImportProcess.java:463)
        at loci.plugins.in.ImportProcess.step(ImportProcess.java:756)
        at loci.plugins.in.ImportProcess.execute(ImportProcess.java:144)
        at loci.plugins.in.Importer.showDialogs(Importer.java:141)
        at loci.plugins.in.Importer.run(Importer.java:79)
        at loci.plugins.LociImporter.run(LociImporter.java:81)
        at ij.IJ.runUserPlugIn(IJ.java:193)
        at ij.IJ.runPlugIn(IJ.java:160)
        ... 27 more


I also have an unrelated error, but maybe someone has experience with it. When I try to both use --headless mode and pass arguments to java, similar to the last command above (--headless mode), as in:

Code: Select all

~/bin/Fiji.app/ImageJ-linux64 -Xms8g -Xmx8g --no-splash --headless stabilize.ijm test_in.tif:test_out.tif


I get the following error:

Code: Select all

--headless without a parameter?


So, it can obviously see the parameters for --headless, because it does so when I don't pass the java arguments -Xms8g -Xmx8g. Is there some tricky syntax that I'm missing?

Thanks for your time.

William

[rowellw]

I found a workaround for my first question (Ext.openImagePlus(input_name)) and eventually found a workaround for my second question (--heap instead of -Xmx). I'm sorry if any of those questions/problems wasted your time.

I am running into one last roadblock and I can't find any documentation that answers this question specifically. Is it possible to force Bio-Format Exporter to write a bigtiff (somehow calling setBigTiff(true)) in a macro, and if so, what is the syntax?

Thanks,

William

[mlinkert]

It's not possible at the moment to call setBigTiff(true) from a macro; BigTIFF files will only be written if the amount of pixel data to be written is greater than 2 GB.

We don't have plans to support the complete Bio-Formats API in ImageJ macros, so if you need access to anything beyond what is currently in the macro extensions I would suggest writing a plugin instead.

-Melissa

[rowellw]

Thanks for your help Melissa.

I'm testing my macro on BigTIFF files with over 4 GB of pixel data and I get the following exception:

Code: Select all

loci.formats.FormatException: File is too large; call setBigTiff(true)
   at loci.formats.out.TiffWriter.prepareToWriteImage(TiffWriter.java:294)
   at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:184)
   at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:378)
   at loci.formats.FormatWriter.saveBytes(FormatWriter.java:136)
   at loci.plugins.out.Exporter.run(Exporter.java:571)
   at loci.plugins.LociExporter.run(LociExporter.java:77)
   etc,etc,etc


If I understand you correctly, since there is over 2 GB of pixel data, it should automatically be writing a BigTIFF file, but instead it's throwing an exception and writing a regular TIFF file that isn't read correctly by other programs.

Thanks for the suggestion about switching to a plugin. I will start focusing some energy on that in case I can't sort this bigtiff problem out.

William

[rowellw]

It looks like automatically switching to BigTIFF with large TIFF files is still an open task on Trac.

http://trac.openmicroscopy.org/ome/ticket/6589

[mlinkert]

Sorry, that's my mistake - you are correct that automatically switching to BigTIFF is not supported yet, but we do plan to make that happen in the near future. For now, you would still need to call the 'setBigTiff' method on the TiffWriter, or write to multiple smaller files (the "Write each ... to a separate file" checkboxes in the exporter plugin allow you to do that).

[rowellw]

No worries Melissa. I'm just happy to see that I wasn't doing anything incorrectly. I look forward to that change, but in the meantime, I'll just write each frame individually and cat them using tiffcp afterwards.

Thanks for your time and help, and thanks for your work on bio-formats!

William