Two (active FTP + SACK) sourceware.org FTP download problems

Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

Hi,

Cannot download GDB from FTP, 86.214640 seconds to login.

If I ever login in default/active mode data transfer never starts:
ftp> pass
Passive mode off.
ftp> cd /pub/gdb/snapshots/branch/
ftp> get gdb-7.2.90.20110703.tar.bz2
local: gdb-7.2.90.20110703.tar.bz2 remote: gdb-7.2.90.20110703.tar.bz2
421 Service not available, remote server has closed connection

IP host1.jankratochvil.net.59123 > server1.sourceware.org.ftp: Flags [P.], seq 95:122, ack 615, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27
IP server1.sourceware.org.ftp > host1.jankratochvil.net.59123: Flags [R.], seq 615:642, ack 122, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27


In passive mode it starts to transfer but gets stuck later, apparently that
ICMP error appears on first SelectiveACK:

IP host1.jankratochvil.net.42232 > server1.sourceware.org.netiq-endpt: Flags [.], ack 3376741, win 1643, options [nop,nop,TS val 566893884 ecr 181994339,nop,nop,sack 2 {3375393:3376741}{3378089:3477841}], length 0
IP server1.sourceware.org > host1.jankratochvil.net: ICMP host server1.sourceware.org unreachable - admin prohibited, length 80

Using just one NAT done by my GNU/Linux box; tcpdumps are from the host behind
that NAT, I could re-run them from the server - doing the NAT.

Unfortunately sourceware.org still does not support IPv6 so there is some NAT
pain going on.

I was always getting these stuck transfers before, I was always reget-ting it
many times.  Finally I found the reason - SACK.
`echo 0 >/proc/sys/net/ipv4/tcp_sack' really makes the full file download by
single `get'.


Thanks,
Jan

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Christopher Faylor]

On Sun, Jul 03, 2011 at 09:32:06AM +0200, Jan Kratochvil wrote:
>Hi,
>
>Cannot download GDB from FTP, 86.214640 seconds to login.
>
>If I ever login in default/active mode data transfer never starts:
>ftp> pass
>Passive mode off.
>ftp> cd /pub/gdb/snapshots/branch/
>ftp> get gdb-7.2.90.20110703.tar.bz2
>local: gdb-7.2.90.20110703.tar.bz2 remote: gdb-7.2.90.20110703.tar.bz2
>421 Service not available, remote server has closed connection
>
>IP host1.jankratochvil.net.59123 > server1.sourceware.org.ftp: Flags [P.], seq 95:122, ack 615, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27
>IP server1.sourceware.org.ftp > host1.jankratochvil.net.59123: Flags [R.], seq 615:642, ack 122, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27
>
>
>In passive mode it starts to transfer but gets stuck later, apparently that
>ICMP error appears on first SelectiveACK:
>
>IP host1.jankratochvil.net.42232 > server1.sourceware.org.netiq-endpt: Flags [.], ack 3376741, win 1643, options [nop,nop,TS val 566893884 ecr 181994339,nop,nop,sack 2 {3375393:3376741}{3378089:3477841}], length 0
>IP server1.sourceware.org > host1.jankratochvil.net: ICMP host server1.sourceware.org unreachable - admin prohibited, length 80
>
>Using just one NAT done by my GNU/Linux box; tcpdumps are from the host behind
>that NAT, I could re-run them from the server - doing the NAT.
>
>Unfortunately sourceware.org still does not support IPv6 so there is some NAT
>pain going on.

"More NAT pain?" No idea what that means.  The box is obviously behind a
firewall but it does have its own IPv4 address.  It is using ipfilter to
block some things but I am not aware of any problems caused by that.

>I was always getting these stuck transfers before, I was always reget-ting it
>many times.  Finally I found the reason - SACK.
>`echo 0 >/proc/sys/net/ipv4/tcp_sack' really makes the full file download by
>single `get'.

I can download this file without problem.  It sure sounds like you have
a problem on your end.  If it was a general problem I would expect a lot
of complaints.

cgf

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

On Sun, 03 Jul 2011 18:13:30 +0200, Christopher Faylor wrote:
> On Sun, Jul 03, 2011 at 09:32:06AM +0200, Jan Kratochvil wrote:
> > Cannot download GDB from FTP, 86.214640 seconds to login.

You did not reply why the FTP access is so slow.


> > If I ever login in default/active mode data transfer never starts:
> > ftp> pass
> > Passive mode off.
> > ftp> cd /pub/gdb/snapshots/branch/
> > ftp> get gdb-7.2.90.20110703.tar.bz2
> > local: gdb-7.2.90.20110703.tar.bz2 remote: gdb-7.2.90.20110703.tar.bz2
> > 421 Service not available, remote server has closed connection
> > 
> > IP host1.jankratochvil.net.59123 > server1.sourceware.org.ftp: Flags [P.], seq 95:122, ack 615, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27
> > IP server1.sourceware.org.ftp > host1.jankratochvil.net.59123: Flags [R.], seq 615:642, ack 122, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27

You did not reply why the default active FTP data transfers do not work.


> "More NAT pain?" No idea what that means.

That I have to use NAT myself to access sourceware.org.  This is because
sourceware.org does not support IPv6.  I do not see any direct problem from
it, it was just a general feedback the network situation of sourceware.org is
needlessly complicated.


> I can download this file without problem.  It sure sounds like you have
> a problem on your end.  If it was a general problem I would expect a lot
> of complaints.

It is because you are in U.S. with better connectivity to sourceware.org,
therefore you will most probably never lose a single packet.

Also it may be possible you are using machine/OS not capable of generating the
SACK TCP extension.

I find the tcpdump output as a good enough proof the firewall on
sourceware.org is broken - what is your explanation of the ICMP packet
otherwise?


Thanks,
Jan

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Christopher Faylor]

On Sun, Jul 03, 2011 at 06:21:22PM +0200, Jan Kratochvil wrote:
>On Sun, 03 Jul 2011 18:13:30 +0200, Christopher Faylor wrote:
>> On Sun, Jul 03, 2011 at 09:32:06AM +0200, Jan Kratochvil wrote:
>> > Cannot download GDB from FTP, 86.214640 seconds to login.
>
>You did not reply why the FTP access is so slow.

I guess I wasn't clear that "can download this file without problem"
means that it downloaded speedily without any delay in logging in.

>> > If I ever login in default/active mode data transfer never starts:
>> > ftp> pass
>> > Passive mode off.
>> > ftp> cd /pub/gdb/snapshots/branch/
>> > ftp> get gdb-7.2.90.20110703.tar.bz2
>> > local: gdb-7.2.90.20110703.tar.bz2 remote: gdb-7.2.90.20110703.tar.bz2
>> > 421 Service not available, remote server has closed connection
>> > 
>> > IP host1.jankratochvil.net.59123 > server1.sourceware.org.ftp: Flags [P.], seq 95:122, ack 615, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27
>> > IP server1.sourceware.org.ftp > host1.jankratochvil.net.59123: Flags [R.], seq 615:642, ack 122, win 54, options [nop,nop,TS val 568265800 ecr 183361626], length 27
>
>You did not reply why the default active FTP data transfers do not work.
>
>
>> "More NAT pain?" No idea what that means.
>
>That I have to use NAT myself to access sourceware.org.  This is because
>sourceware.org does not support IPv6.  I do not see any direct problem from
>it, it was just a general feedback the network situation of sourceware.org is
>needlessly complicated.

Since you're familiar enough with sourceware's firewall to feel
comfortable calling it "needlessly complicated" you really should
provide specific details on what we need to do to fix your problem -
other than implementing IPv6, of course.  That is outside of our
control, as far as I know.  The firewall and IP connectivity is under
the control of the organization who provides the machine and bandwidth,
as I'm sure you know, given your intimate knowledge.

>> I can download this file without problem.  It sure sounds like you have
>> a problem on your end.  If it was a general problem I would expect a lot
>> of complaints.
>
>It is because you are in U.S. with better connectivity to sourceware.org,
>therefore you will most probably never lose a single packet.
>
>Also it may be possible you are using machine/OS not capable of generating the
>SACK TCP extension.
>
>I find the tcpdump output as a good enough proof the firewall on
>sourceware.org is broken - what is your explanation of the ICMP packet
>otherwise?

I detect an unacceptable amount of latent hostility here so, since I am
a volunteer, and it is a holiday weekend in the US, I am going to let
someone else deal with you.

Good luck.

cgf

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

On Sun, 03 Jul 2011 18:33:58 +0200, Christopher Faylor wrote:
> I guess I wasn't clear that "can download this file without problem"
> means that it downloaded speedily without any delay in logging in.

Initially I could not download it.  After some fiddling I found workarounds
how to download the GDB snapshot I needed for the new Fedora release.

That is first enter the passive mode as it otherwise does not even start the
transfer.

And second disable /proc/sys/net/ipv4/tcp_sack . This SACK problem I was
always workarounding by several `reget' commands before.

I do not find such workarounds as acceptable for downloads for users.


> other than implementing IPv6, of course.

Why not IPv6?  Many websites have IPv6 access already for some years.
We can sure discuss about the pros and cons of default AAAA availability but
for technical sites I find it as an advantage.


> That is outside of our control, as far as I know.

It is not, if your connectivity provider still does not support native IPv6
you can use IPv6 tunnel provider for free.  I was using http://sixxs.net
(before my server ISP started to support IPv6 natively (*)), there are many
other free IPv6 tunnel provides.

(*) But my home ISP still does not support IPv6 so I tunnel myself now.


> The firewall and IP connectivity is under
> the control of the organization who provides the machine and bandwidth,

When the firewall is outside of your control going to open a ticket with Red
Hat IT.  It is true there is ext-gw1-phx2.redhat.com one hop before.


> as I'm sure you know, given your intimate knowledge.

I really only have sparse information about the machine.  And primarily
I expected there is rather some bug in iptables rules on that host itself.


> I detect an unacceptable amount of latent hostility here so, since I am
> a volunteer, and it is a holiday weekend in the US, I am going to let
> someone else deal with you.

Thanks; I am also a volunteer (using redhat.com domain in my free time).


Thanks,
Jan

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Ian Lance Taylor]

Jan Kratochvil <jan.kratochvil@redhat.com> writes:

> On Sun, 03 Jul 2011 18:33:58 +0200, Christopher Faylor wrote:
>> I guess I wasn't clear that "can download this file without problem"
>> means that it downloaded speedily without any delay in logging in.
>
> Initially I could not download it.  After some fiddling I found workarounds
> how to download the GDB snapshot I needed for the new Fedora release.
>
> That is first enter the passive mode as it otherwise does not even start the
> transfer.
>
> And second disable /proc/sys/net/ipv4/tcp_sack . This SACK problem I was
> always workarounding by several `reget' commands before.
>
> I do not find such workarounds as acceptable for downloads for users.

What specifically do you suggest that we change on sourceware.org?  We
are not networking experts.


>> The firewall and IP connectivity is under
>> the control of the organization who provides the machine and bandwidth,
>
> When the firewall is outside of your control going to open a ticket with Red
> Hat IT.  It is true there is ext-gw1-phx2.redhat.com one hop before.

I'm not sure if you are saying that you are going to open a ticket with
Red Hat IT, or if you are suggesting that we do so.  In all honesty I
suspect they are more likely to respond to your ticket.


> I really only have sparse information about the machine.  And primarily
> I expected there is rather some bug in iptables rules on that host itself.

The iptables rules on sourceware.org itself are fairly simple.  There
are a bunch of rules blocking specific IP addresses which have attacked
us in various ways.  Those rules all use --dport 80 or --dport 3690.
Other than that, the rules are appended (from /etc/sysconfig/iptables).

I strongly suspect that Red Hat is applying other rules before packets
get to sourceware.org.  In particular they seem to filter out SYN
packets when some IP address exceeds some unknown threshold; this is
painful for us at Google, since many people share a single outgoing NAT
IP address.

Ian

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -m udp -p udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -m udp -p udp --sport 24441 -s 82.94.255.100 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 123 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 209 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 174 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 873 -j ACCEPT
# -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 1529 -j ACCE
PT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 2401 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 3690 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 9418 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5999 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

On Sun, 03 Jul 2011 19:05:28 +0200, Ian Lance Taylor wrote:
> Jan Kratochvil <jan.kratochvil@redhat.com> writes:
> > I do not find such workarounds as acceptable for downloads for users.
> 
> What specifically do you suggest that we change on sourceware.org?

After you have shown iptables there I no longer think there is any problem.


> I'm not sure if you are saying that you are going to open a ticket with
> Red Hat IT, or if you are suggesting that we do so.

I have opened RH internal ticket INC000000306811 (all Cced Frank Eigler) for
the SACK problem.


The FTP active data transfer problem was a test mistake on my side, it got
firewalled by _my_ NAT server (this seems to be a RHEL bug but that is OT
here).  For IPv6 or a tunnel security permission I have opened RH internal
ticket INC000000306811.


> In particular they seem to filter out SYN packets when some IP address
> exceeds some unknown threshold; this is painful for us at Google, since many
> people share a single outgoing NAT IP address.

I also see this problem, opened RH internal ticket INC000000306813 for it.


Thanks,
Jan

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Ian Lance Taylor]

Jan Kratochvil <jan.kratochvil@redhat.com> writes:

> On Sun, 03 Jul 2011 19:05:28 +0200, Ian Lance Taylor wrote:
>> Jan Kratochvil <jan.kratochvil@redhat.com> writes:
>> > I do not find such workarounds as acceptable for downloads for users.
>> 
>> What specifically do you suggest that we change on sourceware.org?
>
> After you have shown iptables there I no longer think there is any problem.
>
>
>> I'm not sure if you are saying that you are going to open a ticket with
>> Red Hat IT, or if you are suggesting that we do so.
>
> I have opened RH internal ticket INC000000306811 (all Cced Frank Eigler) for
> the SACK problem.
>
>
> The FTP active data transfer problem was a test mistake on my side, it got
> firewalled by _my_ NAT server (this seems to be a RHEL bug but that is OT
> here).  For IPv6 or a tunnel security permission I have opened RH internal
> ticket INC000000306811.
>
>
>> In particular they seem to filter out SYN packets when some IP address
>> exceeds some unknown threshold; this is painful for us at Google, since many
>> people share a single outgoing NAT IP address.
>
> I also see this problem, opened RH internal ticket INC000000306813 for it.


Great, thanks!

Ian

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

On Sun, 03 Jul 2011 19:05:28 +0200, Ian Lance Taylor wrote:
> What specifically do you suggest that we change on sourceware.org?  We
> are not networking experts.

Frank Eigler has tcpdump-ed the problem now and it seems to me the problem is
really on sourceware.org itself (and not in the RH network infrastructure):

14:41:28.248127 IP 46.28.109.124.42872 > 209.132.180.131.bctp-server: Flags [.], ack 3271597, win 1580, options [nop,nop,TS val 1593072627 ecr 2362128182,nop,nop,sack 2
{1855640264:1855641612}{1855655092:1855735972}], length 0
14:41:28.248157 IP 209.132.180.131 > 46.28.109.124: ICMP host 209.132.180.131 unreachable - admin prohibited, length 80
 - this is on the 209.132.180.131 = sourceware.org host itself

I do not know why this happens, my guess is that kernel there has netfilter
bug and fails to match that SACK packet by that `--state ESTABLISHED,RELATED'.
There were some such bugs in the netfilter code, maybe it is not backported:
	https://lists.netfilter.org/pipermail/netfilter/2005-June/061101.html

Sorry for the delay, mostly due to me.


Thanks,
Jan

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

On Thu, 28 Jul 2011 15:16:00 +0200, Jan Kratochvil wrote:
> I do not know why this happens, my guess is that kernel there has netfilter
> bug and fails to match that SACK packet by that `--state ESTABLISHED,RELATED'.

not sure if/where is the real bug but Frank Eigler has now (persistently):
	echo 0 >/proc/sys/net/ipv4/tcp_sack

on sourceware.org which should be enough unidirectionally:
RFC2018
	If the data receiver has received a SACK-Permitted option on the SYN
	for this connection, the data receiver MAY elect to generate SACK
	options as described below. 

as the problem is only in receiving SACKs, not in their possibly transmission.
The problem is no longer reproducible for me.

17:40:22.527441 IP 46.28.109.124.34131 > 209.132.180.131.ftp: Flags [S], seq 4057857005, win 5840, options [mss 1361,sackOK,TS val 1690206962 ecr 0,nop,wscale 7], length 0
17:40:22.694011 IP 209.132.180.131.ftp > 46.28.109.124.34131: Flags [S.], seq 3206468532, ack 4057857006, win 5792, options [mss 1360,nop,nop,TS val 2459263683 ecr 1690206962,nop,wscale 4], length 0
17:40:22.714454 IP 46.28.109.124.34131 > 209.132.180.131.ftp: Flags [.], ack 1, win 46, options [nop,nop,TS val 1690207193 ecr 2459263683], length 0

(no sackOK from sourceware.org)


Thanks,
Jan

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Ian Lance Taylor]

Jan Kratochvil <jan.kratochvil@redhat.com> writes:

> On Thu, 28 Jul 2011 15:16:00 +0200, Jan Kratochvil wrote:
>> I do not know why this happens, my guess is that kernel there has netfilter
>> bug and fails to match that SACK packet by that `--state ESTABLISHED,RELATED'.
>
> not sure if/where is the real bug but Frank Eigler has now (persistently):
> 	echo 0 >/proc/sys/net/ipv4/tcp_sack

Cool.

By the way, a couple of weeks ago I set net.ipv4.tcp_tw_recycle to 0 on
the advice of some Google networking people.  This fixes some problems
when using sourceware with large numbers of systems behind a NAT.

Ian

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Hans-Peter Nilsson]

On Sun, 3 Jul 2011, Christopher Faylor wrote:
> On Sun, Jul 03, 2011 at 09:32:06AM +0200, Jan Kratochvil wrote:
> >In passive mode it starts to transfer but gets stuck later, apparently that
> >ICMP error appears on first SelectiveACK:
> >
> >IP host1.jankratochvil.net.42232 > server1.sourceware.org.netiq-endpt: Flags [.], ack 3376741, win 1643, options [nop,nop,TS val 566893884 ecr 181994339,nop,nop,sack 2 {3375393:3376741}{3378089:3477841}], length 0
> >IP server1.sourceware.org > host1.jankratochvil.net: ICMP host server1.sourceware.org unreachable - admin prohibited, length 80

Yup, stuck after ~700kB.

> >I was always getting these stuck transfers before, I was always reget-ting it
> >many times.  Finally I found the reason - SACK.
> >`echo 0 >/proc/sys/net/ipv4/tcp_sack' really makes the full file download by
> >single `get'.

Yup.

> I can download this file without problem.  It sure sounds like you have
> a problem on your end.  If it was a general problem I would expect a lot
> of complaints.

I'm not saying the problem is not at my end too (like, flawed
network gear), but at least I can reproduce it and I see bounced
ICMP messages.  N.B. I'm not sure how they're supposed to be
handled, maybe bouncing them at the firewall is ok.  I do not
recalling having problem with other sites.  (Mentioning IPv6 is
a bit of a red herring.)

brgds, H-P

Re: Two (active FTP + SACK) sourceware.org FTP download problems

[Jan Kratochvil]

On Sun, 03 Jul 2011 09:32:06 +0200, Jan Kratochvil wrote:
> Cannot download GDB from FTP, 86.214640 seconds to login.

This problem I have no longer reproducible.

Which seems to be due to the sourceware.org host high load sometimes being
discussed, a longterm known hosting problem.


Thanks,
Jan