This project has moved and is read-only. For the latest updates, please go here.

Retrieving files in a batch from sftp server

Nov 10, 2011 at 11:07 AM

Hi all,

I was testing a particular scenario where there are multiple very large files and some small files being retrieved by the polling blogical sftp adapter. I noticed from the sftp server logs that the downloading of the files are done sequentially, only after 1st file is finished downloading then the 2nd file will start. In the blogical config file, we can actually specify the max no. of connections per biztalk server that can allow to connect and also in the receive location config, we can configure how many files per batch to download.

Up to this point, is all the above understanding correct? Can anyone confirm?

I am actually expecting the files to be downloaded concurrently instead of one by one as this is too slow and what's the point if we allow a max no. of connections of 10 but end up using a single connection to retrieve the file one by one? Is there any settings where we can configure to enable the adapter (per receive location) to download files concurrently in a batch.

Also another thing to clarify is, is it only after the whole batch of files are downloaded then the delete command will be issued (if we configure as such) and only then it will be send to the pipelines for processing? What if out of 10 files, 5 have been downloaded and somehow the connection reset or server simply drops connection, will the successful 5 get into pipeline processing? What I noticed is the files wont be deleted and once connection is restored, all 10 files will be downloaded again.