This project has moved and is read-only. For the latest updates, please go here.

NullReferenceException on multiple receive locations

Dec 11, 2008 at 9:42 AM
Hi guys.

I'm doing some testing of the 1.3.2 release and have stumbled upon an issue. When using multiple receive locations for pooling a SFTP server, the adapter tends to throw NullReferenceExceptions now and then. The exceptions has the form:

----
Method: Blogical.Shared.Adapters.Sftp.SftpReceiverEndpoint.EndpointTask
Error: Uri:SFTP://**.***:***:***:22/opt/edi/xxxxxxxx/xxxxx/*.edi

------------------------------
Information:
Type: System.Exception
Target:
Message: Uri:SFTP://**.***:***:***:22/opt/edi/xxxxxxxx/xxxxx/*.edi
Stacktrace:

------------------------------
Type: System.NullReferenceException
Target: Boolean PickupFilesAndSubmit()
Message: Object reference not set to an instance of an object.
Stacktrace:
   at Blogical.Shared.Adapters.Sftp.SftpReceiverEndpoint.PickupFilesAndSubmit()
   at Blogical.Shared.Adapters.Sftp.SftpReceiverEndpoint.EndpointTask()

----

From what I can figure, the exception is initially thrown in Blogical.Shared.Adapters.Sftp.SharpSsh.Sftp.dir(string fileMask, string uri, int maxNumberOfFiles, ArrayList filesInProcess, bool trace), but I'm not sure exactly where. I've tried to reproduce the error using a custom class running the same method, but no luck so far. The receive locations run on a timely schedule every 60 seconds and the exceptions are thrown on random locations every 10-15 minutes.

I can also add that this seems to work fine on Win2003 SP2 32bit/BTS 2006R2 32bit, but exceptions are thrown on Win2003 SP2 64bit/BTS 2006R2 64 bit. Please note that the adapter runs in a 32 bit host process on both environments.

Any ideas?
Dec 11, 2008 at 9:05 PM
Is it possible for you to run with the trace flag turned on? And catch it with either a TraceListener or Dbgview.

If that doesn't take us closer to the solution, perhaps we need to make the logging more verbose.

I haven't seen this before, but then again I don't currently run this on a 64-bit environment.
Dec 17, 2008 at 9:20 AM
Hi,

This is an example of the trace printed when exceptions are caught:

[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] Exception #2
[1880] [SftpReceiverEndpoint] ControlledTermination.Leave()
[1880] [TimeSchedule]Executing GetNextActivationTime
[1880] [SftpReceiverEndpoint] Dir(/opt/edi/xxxx/*.edi)
[1880] [SftpReceiverEndpoint] Dir(/opt/edi/xxxx/*.edi)
[1880] [SftpReceiverEndpoint] Found 0 files.
[1880] [SftpReceiverEndpoint] ControlledTermination.Leave()
[1880] [TimeSchedule]Executing GetNextActivationTime
[1880] [SftpReceiverEndpoint] Found 0 files.
[1880] [SftpReceiverEndpoint] ControlledTermination.Leave()
[1880] [TimeSchedule]Executing GetNextActivationTime
[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] Dir(/opt/edi/xxxx/*.edi)
[1880] [SftpReceiverEndpoint] Found 0 files.
[1880] [SftpReceiverEndpoint] ControlledTermination.Leave()
[1880] [TimeSchedule]Executing GetNextActivationTime
[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] Dir(/opt/edi/xxxx/*.edi)
[1880] [SftpReceiverEndpoint] Found 0 files.
[1880] [SftpReceiverEndpoint] ControlledTermination.Leave()
[1880] [TimeSchedule]Executing GetNextActivationTime
[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] ControlledTermination.Enter()
[1880] [SftpReceiverEndpoint] PickupFilesAndSubmit called [SFTP://xx.xxx.x.xx:22/opt/edi/xxxx/*.edi]
[1880] [SftpReceiverEndpoint] Exception #1
[1880] [SftpReceiverEndpoint] ControlledTermination.Leave()
[1880] [TimeSchedule]Executing GetNextActivationTime

I'm currently trying a modified version of 1.3.2 with a lot more tracing, but cant seem to find a reliable pattern for these exceptions. The strange thing is, they seem to disappear over time. The exceptions occur approximately once per hour at start up, but after a few days there's only a few left per 24 hours.

Strange behavior indeed.
Dec 17, 2008 at 9:22 AM
I forgot to mention that the trace referred to above describes the execution of 4 receive locations at the same time.
Apr 13, 2009 at 6:35 PM
I have the same issue which 'janode' has, i.e. of NullReferenceExceptions with multiple receive locations.

I have two receive loactions with SFTP adapter but I receive error only for one receive location. Both receive location have the same configurations

Thanks.
Apr 22, 2009 at 9:32 AM
Hi,

I am facing the same problem with multiple receive locations but the strange thing is it used to work fine until now. Nothing appears to have been changed on the server. When I start a host instance it throws Null reference exceptions and I can see three errors in event log straightaway, I have got four receive locations in my project. Any help would be highly appreciated. Just to confirm my server is 32 bit and not 64 bit.

Many thanks,
Apr 22, 2009 at 12:57 PM
Hi, guys.

Thought I should provide some feedback, being the one to start this thread. We are currently running in a 64 bit clustered environment with 32 bit hosts and still experiencing these problems. As far as I have been able to debug, there are no issues linked directly with these error messages. We have therefore chosen to ignore them until further notice.

We consider the 1.3.2 release stable and have had next to no problems using this adapter. Thanks guys! 
Apr 23, 2009 at 2:03 PM
Hi Janode,

Thank you very much for your reply. But in my case the adapter stops working and does not pick up any file from the remote location. When I restart the host instance, it gives series of erros in the event log as belows.

Type: System.NullReferenceException

Target: Boolean PickupFilesAndSubmit()

Message: Object reference not set to an instance of an object.

Stacktrace:

And then another series of errors as below

type: System.Exception

Target: Tamir.SharpSsh.jsch.HostKey get_HostKey()

Message: SSH session is not connected.

Stacktrace:

at Tamir.SharpSsh.SshBase.get_HostKey()


I think at this point the adapter stops working

Any help would be highly appreciated as this is the issue in the production. I tried reinstalling the adapter but no luck.

Thank you.
Amit

 

 

Apr 23, 2009 at 2:46 PM
Hi Amit,

When I debug the sftp adapter code step by step, I come up with the following conclusions, (Below are my personal experience and opinions)

1) Whenever the host instance is restarted / started the SFTP adapter starts pulling files (may be) irrespective of the time schedule.
2) If we have multiple locations within one receive port, and when the host instance is restarted or if one location is disabled or enabled the other enabled locations starts pulling files.
3) The error occurs because all the enabled sftp locations try to access a same config file which leads to deadlock (Multiple instance trying to access the same file)

4) I found the solution for this BUT the solution restricts me to schedule the adapter only for daily schedule basis.
5) In daily schedule basis you can also schedule for weekly basis.

Let me know if this works for you so I can provide you the solution.

Thanks,
Piyush
Apr 23, 2009 at 2:52 PM
Edited Apr 23, 2009 at 4:08 PM
Hi,

Thanks for your response. In my application I have got four different port with four different receive locations, all polling the same remote directory but for different files (i.e. the file mask is different for each receive location). One more thing, I am using timely schedule as I have to pull the files as soon as possible so I have set it as every 5 seconds.

Let me have your thoughts please.

Guys, I ran a debug view and logged the trace. It appears that after 5 - 7 minutes of Host Instance restart, only ONE receive location is working out of FOUR. Although all the four locations appear to be enabled, the trace shows polling only ONE servived location.

Kind regards,
Amit
Apr 23, 2009 at 8:08 PM
Edited Apr 23, 2009 at 8:13 PM
Sorry guys that we havn't been active in this discussion yet.
I don't have a good testing environment at the moment, but I'll try to help out.

-piyushthakkar;
Your conclusions does not make sense to me. But since you are able to debug, could you find out where the "Object reference not set to an instance of an object" exception is origining from. It semse to be in the PickupFilesAndSubmit method, but a line number would help.

If you need help, you can contact me through messenger wmmihaa at hot mail dot com
Apr 23, 2009 at 8:10 PM
-amitmca; Are you running load balancing? Are all Receive Locations running on the same host?
//Mikael
Apr 24, 2009 at 10:04 AM
Hi Mikael

I am not running load balancing. Yes all the receive locations are running on the same host. All four receive location polling the same remote directory with different file masks. I really apprecite your help.

Regards,
Amit
May 11, 2009 at 10:43 AM

Hi Mikael,

Sorry to chase you up but I do not seem to have received any response regarding an issue with multiple receive locations polling the same remote directory with different file masks does not work at all. Out of four such receive locations, only one survives and other three dies after 5 - 10 mins of starting of host instance.

I am under pressure to get it resolved and will be really grateful to you.

Kind regards,

Amit

May 11, 2009 at 10:26 PM
Edited May 11, 2009 at 10:28 PM

What puzzles me is the "Tamir.SharpSsh.jsch.HostKey get_HostKey()" exception which normally indicates that the server has changed, which seams unlikely. Could there be a server setting that limits the connections?

What value have you set to ErrorThreshold? (it won't solve the problem, but I'm curious for why the port stops polling for messages).

Does it make a difference if there are any messages to pick up? 

Is the server accessable from the internet? If it is I might be able to debug from here, given that you can limit the access rights for me.

 //Mikael

May 12, 2009 at 10:48 AM

Hi Mikael,

I do not know much about AIX server, so wouldn't know if there is any connection limits set up there.

The ErrorThreshold is 0 which is a default value.

I dont think so it makes any difference if any messages to be picked up because as soon as I start the host instance (all four receive locations run in same host instance), it starts logging errors in event log.

I work at a client site in a constained environment so the server will not be accessible from the internet.

I would be useful to perform the same test in Windows environment to see if the adapter still throws exceptions.

Regards,

Amit

Jul 13, 2009 at 5:20 AM

Hi All,

I debug the source code and found a easy fix to solve the issue above.

 

Root Casue:

Two or more receive locations try to access same configuration file.

Solution:

Put a lock to stop multipe thread to access the configuration file at same time,

 

const string settingsFileName = "SftpHostFiles.config";
const string objlock = "lock";

public static ArrayList Load()
        {
            IsolatedStorageFile isoStore = IsolatedStorageFile.GetStore(IsolatedStorageScope.User | IsolatedStorageScope.Assembly, null, null);
            ArrayList ApplicationStorage = new ArrayList();
            if (isoStore.GetFileNames(settingsFileName).Length == 0)
            {
                return ApplicationStorage;
            }

            // Read the stream from Isolated Storage.
            lock (objlock)
            {
                Stream stream = new IsolatedStorageFileStream(settingsFileName, FileMode.OpenOrCreate, isoStore);
                if (stream != null)
                {
                    try
                    {
                        XmlSerializer ser = new XmlSerializer(typeof(ApplicationStorage[]));
                        TextReader reader = new StreamReader(stream);
                        ApplicationStorage[] arr = (ApplicationStorage[])ser.Deserialize(reader);
                        ApplicationStorage = ArrayList.Synchronized(new ArrayList(arr));
                        reader.Close();
                    }
                    finally
                    {
                        // We are done with it.
                        stream.Close();
                    }
                }
            }
            return ApplicationStorage;
        }

Jul 13, 2009 at 9:24 AM

Many thanks.

Jul 13, 2009 at 8:17 PM

Great, I'm planing to do some fixes while on vaccation. I'll add this fix.

Thanks dingwanyou!

 

Sep 13, 2009 at 7:52 PM

Hello,

 

I am also facing the same problem described above.I have 4 receive locations and 4 receive ports corrresponding to each location.ALl the receive location point to the same SFTP location with the difference of one folder: e.g.

SFTP://XXX:22/XX/XX/XX/AAA/Received/*.xml

SFTP://XXX:22/XX/XX/XX/AAA/Received/*.pdf

SFTP://XXX:22/XX/XX/XX/BBB/Received/*.xml

SFTP://XXX:22/XX/XX/XX/BBB/Received/*.pdf

and are scheduled timely for 60 sec

I am using version 1.3 of the SFTP adapter .I get the 'Null reference' exception every few minutes(Not consistently).Also, once one receive location downloads the files, the others do not(although all are enabled).Once I restart the application the other 3 receive location also downloads the files.

I am assuming that the fix mentioned above,will also fix my issue.Please let me know if it is ver 1.3.2 or ver 1.3.3.

 

Sep 14, 2009 at 9:04 PM

The fix is done in the 1.3.3 version.

//Mikael