There’s no way to avoid the document opening and closing, it just needs to happen for Retrobatch to run the workflows.
You can however use Retrobatch from a shell like so:
Retrobatch.app/Contents/MacOS/Retrobatch --workflow path/to/workflow.retrobatch --output ~/Desktop/foo/ /path/to/files.jpeg /or/folders/