Outbound and inbound FTP stopped working

[kcpoole]

I have been using PfSense for about 2 years with great success until last week
Once of my users advised that they cannot upload files to an FTP site.
the run a batch job every week to upoload abotu 20 files and this now does not work.

there have been no changes to the environment at all, but my server had a crash the wek before and last weekend woud have been the first time since the crash the process was attempted.

using Command line FTP client on Windows ( multiple versions) and Linx client all fail to different FTP servers. ( i have tested several to check not a server issue).

The FTP client connects and can log in, but try to do an ls or put results in an error.
ftp> ls
200 PORT command successful
425 Unable to build data connection: Connection timed out

I have tried to set PASV and still not working.

I have tried multiple hosts and OS versions behind the firewall to get out and they are the same.
Turned off FTP proxy and get port errors as described elsewhere.

Using Automatic outbound NAT, and all other services and connection work fine.

I host an FTP server inside the network and have port forwarding to it which has worked for years. This alos now exhibits the same issue from outside my network.

How can I find out why PfSense is blocking FTP now?

Thanks
Ken

[johnpoz]

Well the first thing in troubleshooting ftp is understanding what your actually using and what your clients are actually using - and are they behind nats, double nats? etc..

Active and passive  determines which side make the data connection.  Making a control connection is normally quite easy - where it becomes a pain is data side.  And who is initiating the connection.  What is your server set for passive - what ports does it use?  ftp help is going to be on unless your actually forwarding the passive ports.  Did you ftp server change IPs when you said it crashed?

Here is great write up on active vs passive.  http://slacksite.com/other/ftp.html

What version of pfsense are you using - you say its been without issue for years, are you running a 2 year old version?

C:>ftp ftp.microsoft.com
Connected to ftp.microsoft.akadns.net.
220 Microsoft FTP Service
User (ftp.microsoft.akadns.net:(none)): anonymous
331 Anonymous access allowed, send identity (e-mail name) as password.
Password:
230-Welcome to FTP.MICROSOFT.COM. Also visit http://www.microsoft.com/downloads.
230 User logged in.
ftp> ls
200 PORT command successful.
125 Data connection already open; Transfer starting.
bussys
deskapps
developr
KBHelp
MISC

switch to pasv and still works, even though there was an error at first

ftp> quote pasv
227 Entering Passive Mode (134,170,188,232,169,136).
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection.
425 Cannot open data connection.
ftp> ls
200 PORT command successful.
125 Data connection already open; Transfer starting.
bussys
deskapps
developr
KBHelp
MISC
MISC1

Are you behind a double nat?  You clearly should be able to ftp outbound using the ftp helper in pfsense without any issue if current version 2.1.5

[kcpoole]

The version i was using last week was 2.1.4. Updated last night to 2.1.5 and no different
I have been using Pfsense for 2 years and it has been updated continually as patches and updates are realesed

no I am not behind Double natting and know quite initimately the FTP process and communications.
The host I am trying to connect is not natted and does not have an issue.

[kcpoole]

Posted before I could finish typing :-(
the comments about pasv is to let anyone who sees this that I have tried different solutions to test where the issue might be.

The remote host has no problem and my Pfsense wan side is on a public ip.
outbound nat is auto.

inbound NAT and firewalls rule to my own Ftp server is also affected in the same way.

You clearly should be able to ftp outbound using the ftp helper in pfsense without any issue if current version 2.1.5

I have been using the FTP helper since initially installed and still am. it was workign up until last week.

Ken

[johnpoz]

Well validate that it still is - simple enough to do a sniff on wan, connect to ftp server and validate that it changes your IP to your public so it can connect with active

So you can see in the active connection, sniffing on the wan, my public IP is sent in the port command..  Even though client sends private.

Then in the passive connection I get told what IP to connect to, and the syn is sent from my side - helper has nothing to do with this sort of connection.  When your the client behind pfsense, only if the server was behind pfsense and it sent private would helper have to change the port and open it for the syn.

If you are so clear on the active/passive ftp connectivity I would assume you would of already looked at simple sniff to see what is not working, what info is being sent or not sent, etc.

I can tell you I have 2.1.5 running - and don't have any problem connecting to public ftp servers from behind pfsense.  And if I turn on a ftp server behind pfsense.. outside can connect to my ftp server active or passive.  All that is needed is simple forward to 21 of the server private ip.

You can see in bottom 3 attachments, on the wan of pfsense passive is given as public IP, but server clearly sent its private..  So that is the helper at work changing that.  And the first attachments are client connecting to public IP outside pfsense where the helper changes port command for the active connection.

[kcpoole]

@johnpoz:

Well validate that it still is - simple enough to do a sniff on wan, connect to ftp server and validate that it changes your IP to your public so it can connect with active

So you can see in the active connection, sniffing on the wan, my public IP is sent in the port command..  Even though client sends private.

Then in the passive connection I get told what IP to connect to, and the syn is sent from my side - helper has nothing to do with this sort of connection.  When your the client behind pfsense, only if the server was behind pfsense and it sent private would helper have to change the port and open it for the syn.

If you are so clear on the active/passive ftp connectivity I would assume you would of already looked at simple sniff to see what is not working, what info is being sent or not sent, etc.

I can tell you I have 2.1.5 running - and don't have any problem connecting to public ftp servers from behind pfsense.  And if I turn on a ftp server behind pfsense.. outside can connect to my ftp server active or passive.  All that is needed is simple forward to 21 of the server private ip.

You can see in bottom 3 attachments, on the wan of pfsense passive is given as public IP, but server clearly sent its private..  So that is the helper at work changing that.  And the first attachments are client connecting to public IP outside pfsense where the helper changes port command for the active connection.

Ok thanks. will try that and see what i can see. I do not have another host on the wan side of the firewall, (It is in a remote data centre),  so will have to set one up and sniff.
That will also give me host outside the firewall to confirm that it is actually Pfsense causing the issue.

Ken

[johnpoz]

pfsense under diag, packet capture can do all the packet capture you need to do..

If you need a host outside to confirm – just PM the info, happy to to do a test connection to it and tell you what I see active and passive what is being sent to the client, etc.

[kcpoole]

@johnpoz:

pfsense under diag, packet capture can do all the packet capture you need to do..

If you need a host outside to confirm – just PM the info, happy to to do a test connection to it and tell you what I see active and passive what is being sent to the client, etc.

Ok thanks
Will let you know

Ken

[kcpoole]

@johnpoz:

pfsense under diag, packet capture can do all the packet capture you need to do..

If you need a host outside to confirm – just PM the info, happy to to do a test connection to it and tell you what I see active and passive what is being sent to the client, etc.

I Did some testing and the IP is being rewritten as it is supposed to.
Some of the Connection attempts i have made work for a moment or two but most just fail at the data connection and times out

ken

[kejianshi]

I'm confused.  FTP is dirt simple.
So my first thought is what might you have changed that is breaking it?

[kcpoole]

@kejianshi:

I'm confused.  FTP is dirt simple.
So my first thought is what might you have changed that is breaking it?

As far as I can tell, Nothing has changed at my servers, The remote server, or the firewall. The Firewall was rebooted after power failure and then next time the users could not connect sucessfully and upload files:-(

just trying an inbound connection to my own FTP server while capture the traffic and this is the capture once already connected.

the 114 IP is my firewall wan address and 60.240 is my client here that I am trying to connect from.

it does not matter it seems whether I am trying to connect into my ftp server or out to an external one, the ftp connection is setup correctly but once data is to be tranferred it fails.

Ken

[kejianshi]

Are you using a static public IP or dynamic one?  Do you use a dynamic dns updater?

I'm wondering if its as simple as your IP changed.

Also thinking if its not, wipe the pfsense and reinstall then restore your config.

[johnpoz]

I don't see any data attempts in your post..  And where are you sniffing, see send fin,ack and then retrans it twice - I assume that is your wan..  So what does pfsense have to do with not seeing a a response and sending retrans?  Not like the answer is seen at the wan at pfsense?

In the first looks like see the syn from 182 to 114, and then 114 sending back syn,ack - and then sending it 2 more times.. because no answer?  That is on your wan is in not - so what does pfsense have to do with no answer from 182?

can you post up the sniff, so we can look at the details?  To me looks like you have problem outside pfsense.  If these sniffs are taken on the pfsense wan?

[kcpoole]

@johnpoz:

I don't see any data attempts in your post..  And where are you sniffing, see send fin,ack and then retrans it twice - I assume that is your wan..  So what does pfsense have to do with not seeing a a response and sending retrans?  Not like the answer is seen at the wan at pfsense?

In the first looks like see the syn from 182 to 114, and then 114 sending back syn,ack - and then sending it 2 more times.. because no answer?  That is on your wan is in not - so what does pfsense have to do with no answer from 182?

can you post up the sniff, so we can look at the details?  To me looks like you have problem outside pfsense.  If these sniffs are taken on the pfsense wan?

Yes the sniffs are on the outside interface using pfsense to take them.
the last one posted was limited to a single IP and port 21 as there  is much other traffic to that host.

Happy to post the full sniff but how can i remove the password from it? i do nt really want to post the ftp details on the web

Ken

[kcpoole]

@johnpoz:

To me looks like you have problem outside pfsense.

I am wondering that myself. how can I prove or test that thought?
I am setting up another host on my network that is outside the pfsense and hope I can find the issue there.

Ken

[kcpoole]

Does anyone have an ftp server I can test uploading to?
The one my client is trying to send to does not work for me, I have my own that i use at my home and that works fine :-(

TIA
Ken

[johnpoz]

Valid point with the passwords - which is why there are anon tools for sniffs ;)

You could use http://www.tracewrangler.com/ to remove the passwords..

So see my first sniff where there billy password sent to ftp.microsoft – You add an anon task, set everything to passthrough except the text part.  You put in the original and what you want to replace it with.  See 1st attachment

You run the task, and then the new pcap it creates doesn't have your password in there ;)  But all the other info is in tact for looking at what could be wrong..

You can anon other stuff as well like IPs, etc.  Don't go all crazy on it, it can make it difficult to spot issues if too much manipulation is done.

[kcpoole]

Thanks John for the link to the anonymizer

I have rerun the capture and attached it below
Any help to work out why it is failing will be great.

note: I have appended .txt to the end of the file so i can upload.

Thanks
Ken

[Capture - FailFTP_anon.pcapng.txt](/public/imported_attachments/1/Capture - FailFTP_anon.pcapng.txt)

[johnpoz]

well clearly this is wrong.

So see the port command to IP 114 port 52030

Why is it trying to go to IP 170?

And where is this address coming from - see second image.

[kcpoole]

Ok i did not see them at all.
the Address 114 is my own server and the server on 182.50.153.244 is the host I am trying to get to
I have no idea what the other IP addresses are and what they are doing in the converstaion. thanks I will have to investigate where they fit inot it.

Ken

[johnpoz]

well that is not the capture you gave then..

This is 170 talking to a 88, that seems to report its IP address is 182 in the banner.

The port command yes tells this server 88 hey come talk to 114 on port 52030 for the data connection, but the SYN from 20 source port to 52030 is going to to 170.  Which yeah never answers..  You see the retrans, and you see it tell it hey come talk to more ports as well where the port commands are 114, but traffic is to 170.x.x.x

You have dual wan connection on this pfsense?

[kcpoole]

Nope only single wan network connected

is there any possibility that somehow the port 20 traffic redirected or hacked?

What does the handling of the traffic on port 20 ?

Ken

[kcpoole]

@johnpoz:

well that is not the capture you gave then..

When i compare the non anoymised capture it show the correct IP addresses wher the one I posted has different IPs in it !

Sorry but the process must have changed them all. I will repost the other.

Ken

[kcpoole]

Okay sorry but In the original file i uploaded i anonymized the IP addresses too by mistake
Redid the process and here is the correct file

Ken

[Capture - FailFTP_anon.pcapng.txt](/public/imported_attachments/1/Capture - FailFTP_anon.pcapng.txt)

[kcpoole]

I note that it is the traffic returning on Port 20 that fails.
As I am using outbound NAT, how does the firewall know that the returning traffic is part of the original outbound conversation on port 21?

I assume the ftp proxy on pfsense does that magic, how do i test whether that is working properly?

Ken

[johnpoz]

ok 114 is the client, and 182 is the server..

your 114 box tells 182 hey come talk to me on Port X with the port command.. You see the connections from 182 to the port, but never see an answer back.  Yes if this on sniffed on wan of pfsense, and 114 is behind pfsense we can see the traffic get to wan of pfsense.

Can you see the traffic at the lan interface get sent to 114?

Do you see the states get opened?  What if you try a passive connection vs active.  Where the client will make the connection to the server, ie so 114 will make a connection to 182 on the port 182 says to connect too.

As to the IPs being changed - remember in my post where I said to set everything else in the task to passthru ;) "You add an anon task, set everything to passthrough except the text part. " ;)

Why would you be doing an outbound nat, other than automatic?  Do you have some strange outbound net setup?  I would assume all traffic from lan of pfsense going to wan network would get natted to the pfsense wan IP.  Ie the 114 address.

Normally in an active connection the ftp helper would change the lan IP of the server to the public, and no that hey that server is going to be coming in from source IP 20 to port X in the port command, so will send that on to the lan IP of the ftp client.

You can run into problems if that port is already in use, sure.  Try a passive connection, set on 114 when you talk to the 182 server.  What does the sniff show then?

Just to clarify - your the client in this conversation right, your 114 is behind pfsense.  and 182 is some public ftp server?  Or is 182 behind pfsense?

[kcpoole]

@johnpoz:

ok 114 is the client, and 182 is the server..

your 114 box tells 182 hey come talk to me on Port X with the port command.. You see the connections from 182 to the port, but never see an answer back.  Yes if this on sniffed on wan of pfsense, and 114 is behind pfsense we can see the traffic get to wan of pfsense.

Can you see the traffic at the lan interface get sent to 114?
Yes, See the new sniff on the LAN Side. I see the same traffic there too

Do you see the states get opened?  What if you try a passive connection vs active.  Where the client will make the connection to the server, ie so 114 will make a connection to 182 on the port 182 says to connect too.
this trace was PASv and failed as wwell.
yes the states are opened ( at least I think they are correct , See the attached shot.

As to the IPs being changed - remember in my post where I said to set everything else in the task to passthru ;) "You add an anon task, set everything to passthrough except the text part. " ;)
I am little tired these day or is it just dumb! :-)

Why would you be doing an outbound nat, other than automatic?  Do you have some strange outbound net setup?  I would assume all traffic from lan of pfsense going to wan network would get natted to the pfsense wan IP.  Ie the 114 address.

Outbound NAT is automatic. no rules have been changed for many months and has only just stopped working :-(

You can run into problems if that port is already in use, sure.  Try a passive connection, set on 114 when you talk to the 182 server.  What does the sniff show then?
See attached Sniff from inside using PASV, This connection failed too

Just to clarify - your the client in this conversation right, your 114 is behind pfsense.  and 182 is some public ftp server?  Or is 182 behind pfsense?
yes that is correct, Teh remote site is not behind an firewall ( as far as I know, I d not have anythig to do with that host)

[Capture LAN- FailFTP_anon.pcapng.txt](/public/imported_attachments/1/Capture LAN- FailFTP_anon.pcapng.txt)

[johnpoz]

ok there is not passive traffic in that sniff.  I see the passive command but then you see port command and the server coming from 182

source 20 to port 56488..  But seems your client on 192.168.53.26 doesn't answer.

Do you have some firewall running on this 192.168 box?  Clearly pfsense sent it the traffic – that sniff is clearly on the lan side.  But there is no answer back.

I see the passive command where the server on 182 said hey connect to me on port 33133, see the command pasv 129x256 + 109 but there is not traffic from your 192.168 to that IP and port.

so what I see from that sniff, your client never sends traffic on the pasv port that server gave.  But then again sends port command and the server sends traffic to that port, that pfsense clearly sent on changing the IP to the client 192.168 IP -- but the client never answers.

edit: does this client have more than 1 interface, or routes where it would of sent that passive traffic to something else than pfsense, or try and answer that syn traffic from 20 somewhere else?

again clearly pfsense put on the wire to that 192.168.53.26 the data traffic it got from the server, but there is no answer from the client in that sniff.  So yeah data would fail.

[kcpoole]

@johnpoz:

edit: does this client have more than 1 interface, or routes where it would of sent that passive traffic to something else than pfsense, or try and answer that syn traffic from 20 somewhere else?

again clearly pfsense put on the wire to that 192.168.53.26 the data traffic it got from the server, but there is no answer from the client in that sniff.  So yeah data would fail.

Ok thanks
No the Client does not have a firewall on it, and there is only 1 NIC on each client,

Any thoughts on Why the client would not respond? not forgetting I have tried 3 different clients behind the firewall, Win 2003, 2008, and my debian server. None of which work. The debian server is my own FTP host which I also cannot now connect to.

Ken

[johnpoz]

can we draw up your network, and work through it

From what I can see with the passive command your client didn't try and connect to the IP and port given, and instead sent port command again, that the ftp server tried to connect too and pfsense sent on the lan to the 192.168 address.  But that ftp server never responded.

None of this points to a pfsense problem.  It seems the helper is changing the private ips to the public ones, and creating the states to allow the active connection from 20 to come in and sends it on to the ftp server.

You sniffed at both the wan and the lan interfaces and don't see where pfsense is doing anything wrong or not doing anything.  It changes the IPs and forwards on the traffic it gets in answer to the port command..

Here the thing, from a client talking pasv to a ftp server on the pubic internet pfsense ftp helper doesn't really do anything.  Your client makes a connection to the servers port 21, this is no different than a client going to a website.  If the client then does a passive connection - the server says hey connect to me on port X, the client is then suppose to go connect to that port.  Again pfsense ftp helper is not involved, your just a client making a connection to some port just like a web site.

Only if your server is behind pfsense and client come from the public does the passive does ftp helper have to do anything it has to open the port the server told the client to connect to and forward it to the server.  And possible change the IP from a private to public if the servers pasv command gave a private.

if the client is on the public internet and does a active to your server behind pfsense.  Pfsense does nothing different than if your ftp server was going to some website.

If the client is behind pfsense talking to a ftp server on public and does active connection, then ftp helper has to change the IP in the port command and to the public one and allow and forward the port that the sever on the public internet is going to talk to from port 20.

Since pfsense is not really doing anything in a client behind pfsense going to a pubic ftp server using passive, lets troubleshoot that problem.  From a box on your 192 network, try and connect to say ftp.microsoft.com using passive!!  Make sure the client support passive and easy to change and watch the connection.  For example filezilla shows you what is happening and can easy be changed from active to passive mode on the client.  If you sniff on pfsense wan, and client we can validate that pfsense is passing traffic and that client from the filezilla log and sniff that it got the command for what IP and port to connect to and that it actually tries to connect, and sniff on wan of pfsense will show us that connection went out to the internet.

[kejianshi]

You get A+ for patience.  (-:

[johnpoz]

It is wearing thin ;)  But would be easier if had access to a box inside his network, and pfsense..

While ftp can be a PITA, its not a complicated protocol.  I really don't understand why anyone still uses it any more.  Use sftp - its 1 port, its encrypted.  You don't have these pasv vs active on a different session to deal with, etc..

I really don't see why anyone still uses ftp, other than maybe anonymous serving up files?  If your going to serve them up anonymous - why not just do it over http ;)

[kcpoole]

@johnpoz:

It is wearing thin ;)  But would be easier if had access to a box inside his network, and pfsense..

While ftp can be a PITA, its not a complicated protocol.  I really don't understand why anyone still uses it any more.  Use sftp - its 1 port, its encrypted.  You don't have these pasv vs active on a different session to deal with, etc..

I really don't see why anyone still uses ftp, other than maybe anonymous serving up files?  If your going to serve them up anonymous - why not just do it over http ;)

FTP in use because that is why the clients website provider uses :-(
I have asked is they support sftp or SCP to no avail..

Re network diagram is super simple so did not think necessary to doc
192.168.53.x –--> 192.168.53.1 (Pfsense) 114.111.141.50 ------> whlac.org.au (ftp server)

for inbound to my own ftp server ( used for testing)
192.168.53.5 (my internal FTP server) ---- 192.168.53.1 (Pfsense Inbound NAT) ) 114.111.141.50 <------ Any external client

Several internal host have the identical issue

I have just tried using Filezilla on the client and the connection / upload worked fine ! . It still fails using the command line though.
note I am uploading as part of a script and thus the command line is what I am needing to use and thus have not ried anything else

Ken

[kcpoole]

@kejianshi:

You get A+ for patience.  (-:

Hey boys Dont worry OK if that is the attitude. I Dunno why it might be wearing thin 'cos this is a really weird one that has got me stumped. As you say FTP is brain dead simple and shold "Just work" but this does not for some reason.

Thanks for your help John and Sorry to take up your time
Ken

[johnpoz]

So filezilla works in what test you uploading to whlac.org.au in passive mode?

So 182.50.153.244 is the webhost ftp server you need to send stuff too.  And your doing this from something on your 192.168 network..  Why can you not just use passive?  What ftp client are scripting to?  Windows build int ftp?  You can script filezilla easy..  Or  you could use winscp again scripts easy.

If on linux what ftp client are you using?

So when you used active connection pfsense changed the IP for your port command, and sent on the syn part of the connection from the server as you saw that on your sniff.  But your client did not respond..

So you have something wrong on your client.

When you test from filezilla did you test both active and passive - did that work?  Or just the passive worked?

As to webhost not supporting sftp, that is easy enough - change webhosts ;)  Run your own vps, etc. Its not like you can not host a website at a billion different places.  If one doesn't provide the services you want - move to one that does ;)

[kcpoole]

@johnpoz:

So filezilla works in what test you uploading to whlac.org.au in passive mode? When I tried just now yep

So 182.50.153.244 is the webhost ftp server you need to send stuff too.  And your doing this from something on your 192.168 network..  Why can you not just use passive?  What ftp client are scripting to?  Windows build int ftp?
YEP, batch file that does a billion other things too and has been running for about 4 years now, behing pfsense for 2 years no issue

You can script filezilla easy..  Or  you could use winscp again scripts easy.
Yep and I then need to rewrite other apps to go and call a windows app to fix what went wrong.

If on linux what ftp client are you using? ftp command line. it is a server so no X server

So when you used active connection pfsense changed the IP for your port command, and sent on the syn part of the connection from the server as you saw that on your sniff.  But your client did not respond..
So traces tell us, When I set pasv in the FTP comand it still no work.

So you have something wrong on your client.  Possibly but why would 4 different clients ( 2 win2003, 1 x win 2008, and Debian ( server) all break at the same time?

When you test from filezilla did you test both active and passive - did that work?  Or just the passive worked? Passive worked ( the default) so did not test further, I might later if get time

As to webhost not supporting sftp, that is easy enough - change webhosts ;)  Run your own vps, etc. Its not like you can not host a website at a billion different places.  If one doesn't provide the services you want - move to one that does ;)
It not my Webhost or my Script. I host the application on my server for a client. I have no control over their webhost or application.

[johnpoz]

So if it was pfsense - why would filezilla work?  Test the active connection using filezilla.

As I stated from the beginning understanding ftp is key to troubleshooting this.  Why would you not test the active???  Just to have the info, be it you use it or not.  And pretty sure that is what your script it using.

Your going to have to follow the path..  Sniff traffic on both sides of pfsense, are the ports right for the command given.  Clearly taffic was sent to your client from pfsense.  So why did it not respond??

[kcpoole]

@johnpoz:

Clearly taffic was sent to your client from pfsense.  So why did it not respond??

Absolutely no idea which is why this is swo frustrating.

The only common thing between any of this is PFsense, Differnet FTP site works, Different client works, Seems that different FTP application works too, but I need to investigate that more it seems.

Ken

[johnpoz]

So different ftp sties work, different clients works.  You have validated that pfsense is doing what it is suppose to do - yet you still think pfsense is the issue :rolleyes:

You have a sniff of pfsense sending your ftp machine traffic, and that machine not answering - but hey its pfsense, dude really??

[kcpoole]

@johnpoz:

So different ftp sties work, different clients works.  You have validated that pfsense is doing what it is suppose to do - yet you still think pfsense is the issue :rolleyes:

You have a sniff of pfsense sending your ftp machine traffic, and that machine not answering - but hey its pfsense, dude really??

Rep Really!

Finally had a chance to dedicate some time to this again and schedule downtime to try some things.
First one was to install a new firewall with no rules aside from the default outbound nat and an inbound rule to my protected server i am using to upload from.

Happy to say the result is that ftp upload works perfectly thru the newly installed  firewall.

Reinstalled all of my other firewall rules and services to my hosted network and all is good.

I have no idea what has caused pfSense to break, but it most definitely did. From now on I will ensure that i will always have a second device to failover to, or a method to easily roll back to known good configuration at all times.

Thanks for your assistance

ps, if anyone is interested I can make available a vmware backup of the installation to facilitate further analysis.

Ken