This project has moved. For the latest updates, please go here.

SFTP Adapter in Clustered Enviroment

Mar 11, 2009 at 11:29 AM
Hi All,
I'm planning to use bLobical SFTP Adapter in BizTalk Clustered Environment configured in Active-Passive mode (fail over). I understand that adapter can handle receive locations which are behind a load balancer. I've following queries for the adapter behaiviour
1- What will happen if a BizTalk active server carshes while transferring (download/upload) the file. Will the passive server will take over the file being transferred or orchestration have to be restarted
2-Is FTP issue mentioned in http://msdn.microsoft.com/en-us/library/cc296808.aspx applicable to bLogical Adapter
3- Any other consideration while using bLogical in clustered environment?
Regards,
FM
Coordinator
Mar 11, 2009 at 10:51 PM
Edited Mar 11, 2009 at 10:52 PM
1. Files that are picked up from the adapter will not be deleted until they are successfully submitted to BizTalk. If the host crashes while reading the file, the file will remain at its location and can be picked up from the other node. On the transmit side, everything works as normal, the message will be submitted and resent

2. Yes and no... The SFTP adapter can run in two "modes". There is a UseLoadbalancing flag on the receive adapter which ,if enabled, will "check out" the file, preventing other hosts from picking up the same file (have a look in the documentation for more details). There is a small performance overhead, but it might prevent you from having to cluster the host. If you're not using this feature, you'll have the same issue as with the FTP adapter.
3. Not that I know of.

HTH

//Mikael
Mar 12, 2009 at 3:07 AM
Hi Mikael,
Thanks for clarifying. I've a related query concerning 2.
UseLoadbalancing flag solves half of my problem i.e.
a- No Issue: Clustered BizTalk will not try to pick up the file if one host is already downloading the file.
b- Issue: My understanding is that UseLoadbalancing flag will keep the status in DB regarding which host is downloading to prevent multiple download of the same file. Assume one host in clustered BizTalk is donwloading the file using UseLoadbalancing flag and it crashes. The status in DB will still show that file download is in progress preventing other hosts to download and process the file unless DB status is cleared.
Is my understanding correct?
I'd be glad if it is automatic and no manual intervention required for recovery.
Regards,
FM
Coordinator
Mar 12, 2009 at 7:45 AM
First of all, if you'd use the UseLoadbalancing flag, there is no real reason to cluster is there?
Secondly, there is a timeout which, if I remember correctly, is 10 min. If the file is not "check-in" before, then the other node will try to get it.
//Mikael
Oct 7, 2009 at 9:33 AM
Edited Oct 8, 2009 at 3:15 AM

Hi Mikael,

Would like to check with you on this timeout value of 10minutes, is it configurable? If so, how can we configure it?

Also, just to clarify my understanding, when we are using load balancing for the SFTP adapter, the files being transferred as well as the node handling will be logged into the SftpWorkingProcess table, am I correct?

One more question would be when the files are being transferred and suddenly one of the host instances goes down during transfer, the files handled by the host instance that went down would still be "checked-in" the SftpWorkingProcess table, correct? So, would the release of this "checked-in" status of these files be done automatically after the 10 minute timeout value or would I have to manually remove these file transfer data from the table.

Hope you can clarify if this process is automatically done, or is some kind of manual intervention needed to remove the file entry from the SftpWorkingProcess table?

So having done this file entry removal from the SftpWorkingProcess table, would the file transfer be then be automatically picked up by the other node which is still active?

Best regards,

HW

Coordinator
Oct 8, 2009 at 6:13 AM

Q: Would like to check with you on this timeout value of 10minutes, is it configurable? If so, how can we configure it?

A: No it's not configurable (for now). It's defined in a query (DatabaseHelper.cs). When the file is checked out. It checks if the file is already checkedout or is older than 10 minutes:

if(
(select count(*) 
from [dbo].[SftpWorkingProcess] 
where [URI] = '{0}' 
and [FileName] ='{2}'
and datediff(minute, [Timestamp], getdate()) <10) = 0
)
begin
INSERT INTO [dbo].[SftpWorkingProcess]
           ([URI]
           ,[Node]
           ,[FileName])
     VALUES
           ('{0}','{1}','{2}') 

select 1 as WorkInProcess
end
else
select 0 as WorkInProcess

Q: Also, just to clarify my understanding, when we are using load balancing for the SFTP adapter, the files being transferred as well as the node handling will be logged into the SftpWorkingProcess table, am I correct?

A: Yes.

Q: One more question would be when the files are being transferred and suddenly one of the host instances goes down during transfer, the files handled by the host instance that went down would still be "checked-in" the SftpWorkingProcess table, correct? So, would the release of this "checked-in" status of these files be done automatically after the 10 minute timeout value or would I have to manually remove these file transfer data from the table.

A: Yes,  see the first answer.

Q: Hope you can clarify if this process is automatically done, or is some kind of manual intervention needed to remove the file entry from the SftpWorkingProcess table?

A: When checked in, all entries are removed:

delete from [dbo].[SftpWorkingProcess] 
where [URI] = '{0}' 
and [FileName] ='{1}' 

Q: So having done this file entry removal from the SftpWorkingProcess table, would the file transfer be then be automatically picked up by the other node which is still active?

A: It is not removed until any of the nodes picks it up and checks it in again.

HTH

//Mikael

Oct 9, 2009 at 6:40 AM

Hi Mikael,

Thanks for the very thorough clarification. Sure does help with my understanding of how the blogical SFTP adapter works in clustered environment. Will try testing this out....

Appreciate your quick reply.

Regards,

HW

Apr 4, 2012 at 8:42 PM

What if I do not want to use the UseLoadbalancing flag? Can you run the receive adapter under a clustered Biztalk host (active/passive)? The OoTB FTP adapter and the nSoftware SFTP adapter supports this.