Copying and retrieving files
Aside from passing data and variables as function arguments to a batch task, it is possible to manually upload files from the local disk to a batch task. This avoids having to read a file into Julia before passing it as a function argument or broadcasting it. Similarily, it is possible to copy files from the batch worker disk back to the local disk.
Upload files for batch task
To upload a file from the local disk to your batch tasks, use the fileinclude
function:
@batchdef fileinclude("my_local_file.dat")
This expression first copies the file my_local_file.dat
to the Azure blob store and then includes it as a resource file with any future batch task.
Retrieve file from batch task
If your remotely executed Julia function saves a file to the local batch worker disk, you can use the filereturn()
function to move that file from the disk of the batch worker to blob storage:
filereturn("my_remote_file.dat")
Note that the filereturn
statement must be included in the function that will executed remotely as a batch task. See the following function as an example:
@batchdef function create_file()
# Do some work
data = ...
# Write output to local file
iostream = open("my_remote_file.dat", "w")
write(iostream, data)
# Copy remote file to blob store
filereturn("my_remote_file.dat")
end
Similar to return arguments, files from filereturn
will be included as a Future
in the batch controller. Once your function has executed successfully, you can retrieve the file via:
# Remotely run function via azure batch
bctrl = @batchexec create_file()
# Wait for task to finish
wait_for_tasks_to_complete(bctrl)
# Copy "my_remote_file.dat" to your local disk
fetch(bctrl.files; path=pwd())