any work arounds for bug 2176

[Brett Lymn]

Hi,

I think I am being bitten hard by bugid 2176
(http://bugs.squid-cache.org/show_bug.cgi?id=2176) which is preventing
people from uploading documents over about 50k in size.

I can upload fine bypassing squid but when using squid the browser
keeps popping up an authentication window and the upload fails.  If
the file is under about 50kbytes then the upload actually works but
files over 50kbytes will fail with the authentication pop ups.

I have checked request_header_max_size (made this a large number),
request_body_max_size is at the default 0 and reply_body_max_size is
at the default too but the problem persists.

I see the following in the cache.log:

2009/12/02 11:34:24| httpAppendBody: Request not yet fully sent "POST http://..."
2009/12/02 11:34:25| httpAppendBody: Request not yet fully sent "POST http://..."

The site is using IIS, they probably are using NTLM and I am not sure
if they will change to basic authentication.

Does anyone have any suggestions for how to work around this problem
(apart from getting the other end to change the auth scheme) or how to
track down the bug in the source?

I am running Squid Version 2.7.STABLE6.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Amos Jeffries-2]

Brett Lymn wrote:

> Hi,
>
> I think I am being bitten hard by bugid 2176
> (http://bugs.squid-cache.org/show_bug.cgi?id=2176) which is preventing
> people from uploading documents over about 50k in size.
>
> I can upload fine bypassing squid but when using squid the browser
> keeps popping up an authentication window and the upload fails.  If
> the file is under about 50kbytes then the upload actually works but
> files over 50kbytes will fail with the authentication pop ups.
>
> I have checked request_header_max_size (made this a large number),
> request_body_max_size is at the default 0 and reply_body_max_size is
> at the default too but the problem persists.
>
> I see the following in the cache.log:
>
> 2009/12/02 11:34:24| httpAppendBody: Request not yet fully sent "POST http://..."
> 2009/12/02 11:34:25| httpAppendBody: Request not yet fully sent "POST http://..."
>
> The site is using IIS, they probably are using NTLM and I am not sure
> if they will change to basic authentication.
>
> Does anyone have any suggestions for how to work around this problem
> (apart from getting the other end to change the auth scheme) or how to
> track down the bug in the source?
>
> I am running Squid Version 2.7.STABLE6.
>

Workaround: see comment 6 of the bug report.
"
Simply enabling plaintext authention and disabling NTLM
on IIS was enough for making everything work, even passing through squid.
"

I'm attaching a test patch to the bug as well to see if it works.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

[Brett Lymn]

On Wed, Dec 02, 2009 at 06:43:43PM +1300, Amos Jeffries wrote:
> >
> Workaround: see comment 6 of the bug report.
> "
> Simply enabling plaintext authention and disabling NTLM
> on IIS was enough for making everything work, even passing through squid.
> "
>

Yes I saw that but I did say:

"  and I am not sure
if they will change to basic authentication."

 :)

I will ask anyway but I expect the answer will be no.

> I'm attaching a test patch to the bug as well to see if it works.
>

did it get stripped?

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Amos Jeffries-2]

Brett Lymn wrote:

> On Wed, Dec 02, 2009 at 06:43:43PM +1300, Amos Jeffries wrote:
>> Workaround: see comment 6 of the bug report.
>> "
>> Simply enabling plaintext authention and disabling NTLM
>> on IIS was enough for making everything work, even passing through squid.
>> "
>>
>
> Yes I saw that but I did say:
>
> "  and I am not sure
> if they will change to basic authentication."
>
>  :)
>
> I will ask anyway but I expect the answer will be no.
>
>> I'm attaching a test patch to the bug as well to see if it works.
>>
>
> did it get stripped?
>

Sorry. I attached it to the bug report.
Yes this list strips all attachments.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

[Brett Lymn]

On Wed, Dec 02, 2009 at 07:22:57PM +1300, Amos Jeffries wrote:
>
> Sorry. I attached it to the bug report.
>

I manually applied the patch - I couldn't be bothered with patch for a
simple #if removal.  The symptoms have changed.  We no longer get an
auth pop up but at the end of the upload the browser displays:

Bad Request (Invalid Verb)

and the document is not shown in the list.  So, the patch made a
difference but something else is amiss still.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[bill.allison]

As the reporter of this bug, apologies Amos for not responding promptly and thanks Brett for doing so - my excuse is pressure of work. It's particularly on my conscience because we so badly need this fix. Today I have a test instance I can use, and I guess the next step, and my contribution, is apply the patch, tcpdump the result, and report back (unless Brett gets there first ;-) )

Bill A.

-----Original Message-----
From: Brett Lymn [mailto:[hidden email]]
Sent: 07 December 2009 01:39
To: Amos Jeffries
Cc: [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

On Wed, Dec 02, 2009 at 07:22:57PM +1300, Amos Jeffries wrote:
>
> Sorry. I attached it to the bug report.
>

I manually applied the patch - I couldn't be bothered with patch for a
simple #if removal.  The symptoms have changed.  We no longer get an
auth pop up but at the end of the upload the browser displays:

Bad Request (Invalid Verb)

and the document is not shown in the list.  So, the patch made a
difference but something else is amiss still.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Amos Jeffries-2]

Brett Lymn wrote:

> On Wed, Dec 02, 2009 at 07:22:57PM +1300, Amos Jeffries wrote:
>> Sorry. I attached it to the bug report.
>>
>
> I manually applied the patch - I couldn't be bothered with patch for a
> simple #if removal.  The symptoms have changed.  We no longer get an
> auth pop up but at the end of the upload the browser displays:
>
> Bad Request (Invalid Verb)
>
> and the document is not shown in the list.  So, the patch made a
> difference but something else is amiss still.
>

Strange. I've never see that one before.

I think another trace of the request-reply sequence is needed to see if
there is anything different now and what.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

[Brett Lymn]

On Mon, Dec 07, 2009 at 10:36:52PM +1300, Amos Jeffries wrote:
>
> I think another trace of the request-reply sequence is needed to see if
> there is anything different now and what.
>

I do have a trace from snoop.  I don't want to post it to the list
due to it containing details of the site we are trying to upload to.
Can I mail it to you off list?

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Amos Jeffries-2]

On Tue, 8 Dec 2009 08:49:00 +1030, Brett Lymn <[hidden email]>
wrote:
> On Mon, Dec 07, 2009 at 10:36:52PM +1300, Amos Jeffries wrote:
>>
>> I think another trace of the request-reply sequence is needed to see if

>> there is anything different now and what.
>>
>
> I do have a trace from snoop.  I don't want to post it to the list
> due to it containing details of the site we are trying to upload to.
> Can I mail it to you off list?

Sure.

Amos

[bill.allison]

Amos

I've done some more testing / tracing with this - one finding and one question to help me do more.

My finding is that the response to a large POST varies - put simply, a small (< 5Kb) POST succeeds, larger POST pops up UID/PWD request, large (> 80Kb) gives the INVALID VERB error that Brett reported. There is no exact borderline size. Sometimes a 6Kb upload will succeed, sometimes the max is around 4Kb. This is on an otherwise idle test server. So far I've only tcpdumped the first two cases and can see that in the middle case, the proxy issues FIN packets to server and client just after receiving and passing on the second 401 response from the server. A feature, if not a factor, common to failures, at least in traces taken so far, is that transfer of the upload to the server begins before receipt of the upload from the client has completed. Comment please?

My question - I'd now like to marry up tcpdump traces with squid debug output. Having read up on debug_options, I've used 5,6 17,6 33,6 41,6 48,6 58,6 73,6 85,6 87,6 88,6. What would be a better set?

For the avoidance of doubt - I'm a rank amateur (as if you haven't already guessed :-) ) but really need to find a fix or workaround, despite knowing that Microsoft state that IIS NTLM authentication can not work through proxy servers. Any pointers gratefully received.

Kind regards
Bill A.


-----Original Message-----
From: Bill Allison
Sent: 10 December 2009 12:53
To: 'Brett Lymn'; Amos Jeffries
Cc: [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

Hi

I finally found time to test the patch - with different results from Brett. In my case it made no apparent difference - I still get the UID/PWD popup. Attached are two wireshark traces, for the same POST attempt before and after patching. The traces were taken on the squid box and show client 192.0.1.145 and webserver 192.0.1.105 traffic - both are on our LAN. Also attached is my squid.conf. We're still on 2.6-17 - sorry.

If there is anything you want me to try, I have this test instance available for a few days before it has to go live.

Thanks
Bill A.

-----Original Message-----
From: Brett Lymn [mailto:[hidden email]]
Sent: 07 December 2009 22:19
To: Amos Jeffries
Cc: [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

On Mon, Dec 07, 2009 at 10:36:52PM +1300, Amos Jeffries wrote:
>
> I think another trace of the request-reply sequence is needed to see
> if there is anything different now and what.
>

I do have a trace from snoop.  I don't want to post it to the list due to it containing details of the site we are trying to upload to.
Can I mail it to you off list?

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is confidential to BAE Systems Australia. If you are not the intended recipient, any use, disclosure or copying of this email or any attachments is expressly prohibited.  If you have received this email in error, please notify us immediately. VIRUS: Every care has been taken to ensure this email and its attachments are virus free, however, any loss or damage incurred in using this email is not the sender's responsibility.  It is your responsibility to ensure virus checks are completed before installing any data sent in this email to your computer."



--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Amos Jeffries-2]

Bill Allison wrote:

> Amos
>
> I've done some more testing / tracing with this - one finding and one question to help me do more.
>
> My finding is that the response to a large POST varies - put simply, a small (< 5Kb) POST succeeds, larger POST pops up UID/PWD request, large (> 80Kb) gives the INVALID VERB error that Brett reported. There is no exact borderline size. Sometimes a 6Kb upload will succeed, sometimes the max is around 4Kb. This is on an otherwise idle test server. So far I've only tcpdumped the first two cases and can see that in the middle case, the proxy issues FIN packets to server and client just after receiving and passing on the second 401 response from the server. A feature, if not a factor, common to failures, at least in traces taken so far, is that transfer of the upload to the server begins before receipt of the upload from the client has completed. Comment please?
>
> My question - I'd now like to marry up tcpdump traces with squid debug output. Having read up on debug_options, I've used 5,6 17,6 33,6 41,6 48,6 58,6 73,6 85,6 87,6 88,6. What would be a better set?
>
> For the avoidance of doubt - I'm a rank amateur (as if you haven't already guessed :-) ) but really need to find a fix or workaround, despite knowing that Microsoft state that IIS NTLM authentication can not work through proxy servers. Any pointers gratefully received.
>
> Kind regards
> Bill A.
>

Hmm, I wonder...

Do you get the same result as Brett with
persistent_connection_after_error set to ON?

The default in all Squid is OFF which forces the connection closed on
all 4xx replies regardless of the patched Squid now seeing it as a
viable connection.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

[bill.allison]

"Do you get the same result as Brett with persistent_connection_after_error set to ON?"

Yes - I've had it set on throughout my testing.

-----Original Message-----
From: Amos Jeffries [mailto:[hidden email]]
Sent: 16 December 2009 10:46
To: [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

Bill Allison wrote:

> Amos
>
> I've done some more testing / tracing with this - one finding and one question to help me do more.
>
> My finding is that the response to a large POST varies - put simply, a small (< 5Kb) POST succeeds, larger POST pops up UID/PWD request, large (> 80Kb) gives the INVALID VERB error that Brett reported. There is no exact borderline size. Sometimes a 6Kb upload will succeed, sometimes the max is around 4Kb. This is on an otherwise idle test server. So far I've only tcpdumped the first two cases and can see that in the middle case, the proxy issues FIN packets to server and client just after receiving and passing on the second 401 response from the server. A feature, if not a factor, common to failures, at least in traces taken so far, is that transfer of the upload to the server begins before receipt of the upload from the client has completed. Comment please?
>
> My question - I'd now like to marry up tcpdump traces with squid debug output. Having read up on debug_options, I've used 5,6 17,6 33,6 41,6 48,6 58,6 73,6 85,6 87,6 88,6. What would be a better set?
>
> For the avoidance of doubt - I'm a rank amateur (as if you haven't already guessed :-) ) but really need to find a fix or workaround, despite knowing that Microsoft state that IIS NTLM authentication can not work through proxy servers. Any pointers gratefully received.
>
> Kind regards
> Bill A.
>

Hmm, I wonder...

Do you get the same result as Brett with
persistent_connection_after_error set to ON?

The default in all Squid is OFF which forces the connection closed on
all 4xx replies regardless of the patched Squid now seeing it as a
viable connection.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[bill.allison]

Sorry - that was misleading. I've had persistent_connection_after_error set on throughout my testing. I  get the same error as Brett only when the body of the post is much greater than that which causes the post to fail.

I'd like to correlate network traces with debug output and would appreciate suggestions as to which debug_options would include all possibly relevant info


-----Original Message-----
From: Bill Allison [mailto:[hidden email]]
Sent: 16 December 2009 12:07
To: Amos Jeffries; [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

"Do you get the same result as Brett with persistent_connection_after_error set to ON?"

Yes - I've had it set on throughout my testing.

-----Original Message-----
From: Amos Jeffries [mailto:[hidden email]]
Sent: 16 December 2009 10:46
To: [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

Bill Allison wrote:

> Amos
>
> I've done some more testing / tracing with this - one finding and one question to help me do more.
>
> My finding is that the response to a large POST varies - put simply, a small (< 5Kb) POST succeeds, larger POST pops up UID/PWD request, large (> 80Kb) gives the INVALID VERB error that Brett reported. There is no exact borderline size. Sometimes a 6Kb upload will succeed, sometimes the max is around 4Kb. This is on an otherwise idle test server. So far I've only tcpdumped the first two cases and can see that in the middle case, the proxy issues FIN packets to server and client just after receiving and passing on the second 401 response from the server. A feature, if not a factor, common to failures, at least in traces taken so far, is that transfer of the upload to the server begins before receipt of the upload from the client has completed. Comment please?
>
> My question - I'd now like to marry up tcpdump traces with squid debug output. Having read up on debug_options, I've used 5,6 17,6 33,6 41,6 48,6 58,6 73,6 85,6 87,6 88,6. What would be a better set?
>
> For the avoidance of doubt - I'm a rank amateur (as if you haven't already guessed :-) ) but really need to find a fix or workaround, despite knowing that Microsoft state that IIS NTLM authentication can not work through proxy servers. Any pointers gratefully received.
>
> Kind regards
> Bill A.
>

Hmm, I wonder...

Do you get the same result as Brett with
persistent_connection_after_error set to ON?

The default in all Squid is OFF which forces the connection closed on
all 4xx replies regardless of the patched Squid now seeing it as a
viable connection.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Brett Lymn]

On Wed, Dec 16, 2009 at 07:57:21AM -0600, Bill Allison wrote:
> Sorry - that was misleading. I've had
> persistent_connection_after_error set on throughout my testing.

I don't have that in my config file at all so I would guess it is at
the default.

> I  get the same error as Brett only when the body of the post is much greater than that which causes the post to fail.
>

I only tried a large-ish document.  We did observe the same strange
limit that Bill has seen when we tested without the patch applied,
under a certain "magic" threshold the document would upload - the
threshold seemed to be around the 50k mark, over that threshold we
would just get popups.

> I'd like to correlate network traces with debug output and would appreciate suggestions as to which debug_options would include all possibly relevant info
>

I am a C coder and may have some time to do some debugging on this
between christmas and new year so, Amos, if you have any thoughts or
hints as to where to go looking I can certainly have a stab at it.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Amos Jeffries-2]

Brett Lymn wrote:
> On Wed, Dec 16, 2009 at 07:57:21AM -0600, Bill Allison wrote:
>> Sorry - that was misleading. I've had
>> persistent_connection_after_error set on throughout my testing.
>
> I don't have that in my config file at all so I would guess it is at
> the default.
>

Which is off. Now I'm confused.

>> I  get the same error as Brett only when the body of the post is much greater than that which causes the post to fail.
>>
>
> I only tried a large-ish document.  We did observe the same strange
> limit that Bill has seen when we tested without the patch applied,
> under a certain "magic" threshold the document would upload - the
> threshold seemed to be around the 50k mark, over that threshold we
> would just get popups.
>
>> I'd like to correlate network traces with debug output and would appreciate suggestions as to which debug_options would include all possibly relevant info
>>
>
> I am a C coder and may have some time to do some debugging on this
> between christmas and new year so, Amos, if you have any thoughts or
> hints as to where to go looking I can certainly have a stab at it.
>

Thank you. Any help at all would be great.

I *think* the relevant code is off src/client_side_reply.cc, but what to
look for is where I'm currently stuck. The keep_alive values resolved
things for you Brett but not Bill.


The variable nature of the threshold looks like some timing between
actions triggering the bug vs the rate at which Squid is sucking the
request in.

AFAIK popups only occur when the client gets sent two re-auth
challenges. Which in the un-patched Squid was caused by the first
half-authenticated link being closed by Squid before auth could
complete. Then the second link being challenged for more auth would
cause popup.

I think the next step is to find out what the difference between your
two setups is exactly:
  * squid.conf
  * headers between Squid and the POSTing app.
  * headers between Squid and the web server.

Particularly in what reply headers are going back.  That should give us
a little more of an idea what areas to look at.

If as you say the patch solved the issue but you saw the same thing
earlier. Then I suspects it's probably a squid.conf detail being overlooked.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

[bill.allison]

Reposted for info to the list, without the attachments that cause the list to bounce the message

-----Original Message-----
From: Bill Allison
Sent: 18 December 2009 09:43
To: 'Amos Jeffries'; Brett Lymn
Cc: [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

"I  get the same error as Brett only when the body of the post is much greater than that which causes the post to fail."

Correction after further testing...

I  get the same error as Brett only when the body of the post is much greater than that which causes the post to fail, and even then only sometimes, in repeated tests with the same file being uploaded.

Other times the browser reports "The connection was reset" and tcpdump shows that the proxy sent a FIN to the server then to the client in response to the second 401 from the server. THe server closes the connection but the client continues sending a POST and the proxy then sends the client a string of RSTs.

For info "Invalid Verb" is issued by http.sys in IIS 6.0, in response to receiving a header that is not strictly rfc-compliant (including truncated).

Attached as requested is my squid.conf and tcpdumps of the Invalid Verb and RST failure cases.

Unlike Brett I'm very much a novice C coder but I'm perfectly happy to patch, compile and test if it helps generate a solution.

Regards
Bill A.

-----Original Message-----
From: Amos Jeffries [mailto:[hidden email]]
Sent: 17 December 2009 09:10
To: Brett Lymn
Cc: Bill Allison; [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

Brett Lymn wrote:
> On Wed, Dec 16, 2009 at 07:57:21AM -0600, Bill Allison wrote:
>> Sorry - that was misleading. I've had
>> persistent_connection_after_error set on throughout my testing.
>
> I don't have that in my config file at all so I would guess it is at
> the default.
>

Which is off. Now I'm confused.

>> I  get the same error as Brett only when the body of the post is much greater than that which causes the post to fail.
>>
>
> I only tried a large-ish document.  We did observe the same strange
> limit that Bill has seen when we tested without the patch applied,
> under a certain "magic" threshold the document would upload - the
> threshold seemed to be around the 50k mark, over that threshold we
> would just get popups.
>
>> I'd like to correlate network traces with debug output and would
>> appreciate suggestions as to which debug_options would include all
>> possibly relevant info
>>
>
> I am a C coder and may have some time to do some debugging on this
> between christmas and new year so, Amos, if you have any thoughts or
> hints as to where to go looking I can certainly have a stab at it.
>

Thank you. Any help at all would be great.

I *think* the relevant code is off src/client_side_reply.cc, but what to look for is where I'm currently stuck. The keep_alive values resolved things for you Brett but not Bill.


The variable nature of the threshold looks like some timing between actions triggering the bug vs the rate at which Squid is sucking the request in.

AFAIK popups only occur when the client gets sent two re-auth challenges. Which in the un-patched Squid was caused by the first half-authenticated link being closed by Squid before auth could complete. Then the second link being challenged for more auth would cause popup.

I think the next step is to find out what the difference between your two setups is exactly:
  * squid.conf
  * headers between Squid and the POSTing app.
  * headers between Squid and the web server.

Particularly in what reply headers are going back.  That should give us a little more of an idea what areas to look at.

If as you say the patch solved the issue but you saw the same thing earlier. Then I suspects it's probably a squid.conf detail being overlooked.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

[vincent.blondel]

 
Hello all,

Just to inform you I exactly get the same problem. Firstly I thought it
was a problem with WWW-Authenticate but it is not ONLY ....

next is the reference of my first post ...
http://www.squid-cache.org/mail-archive/squid-users/200912/0029.html

I also get this same message ( httpReadReply: Request not yet fully sent
) when sending some POST requests bigger than x bytes to an  IIS server
...

I applied the patch from the bugzilla (2176) on a 2.7.4. The user does
not receive the traditional 'Page cannot be displayed' from Internet
Explorer any more but the browser freeze instead :(-

below the current config ...


client_persistent_connections on
server_persistent_connections on
acl protime url_regex -i ^http://services.group.intranet/rec
acl protime_src src all
cache_peer 1.2.3.4 parent 80 0 forceddomain=services.group.intranet
originserver proxy-only no-query no-digest connection-auth=on login=PASS
cache_peer_access 1.2.3.4 allow protime


I am certainly interested with a definitive solution so if I can be part
of the tests, just say it ...


many thks
Vincent.

-----Original Message-----
From: Bill Allison [mailto:[hidden email]]
Sent: Friday, December 18, 2009 10:47 AM
Cc: [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

Reposted for info to the list, without the attachments that cause the
list to bounce the message

-----Original Message-----
From: Bill Allison
Sent: 18 December 2009 09:43
To: 'Amos Jeffries'; Brett Lymn
Cc: [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

"I  get the same error as Brett only when the body of the post is much
greater than that which causes the post to fail."

Correction after further testing...

I  get the same error as Brett only when the body of the post is much
greater than that which causes the post to fail, and even then only
sometimes, in repeated tests with the same file being uploaded.

Other times the browser reports "The connection was reset" and tcpdump
shows that the proxy sent a FIN to the server then to the client in
response to the second 401 from the server. THe server closes the
connection but the client continues sending a POST and the proxy then
sends the client a string of RSTs.

For info "Invalid Verb" is issued by http.sys in IIS 6.0, in response to
receiving a header that is not strictly rfc-compliant (including
truncated).

Attached as requested is my squid.conf and tcpdumps of the Invalid Verb
and RST failure cases.

Unlike Brett I'm very much a novice C coder but I'm perfectly happy to
patch, compile and test if it helps generate a solution.

Regards
Bill A.

-----Original Message-----
From: Amos Jeffries [mailto:[hidden email]]
Sent: 17 December 2009 09:10
To: Brett Lymn
Cc: Bill Allison; [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

Brett Lymn wrote:
> On Wed, Dec 16, 2009 at 07:57:21AM -0600, Bill Allison wrote:
>> Sorry - that was misleading. I've had
>> persistent_connection_after_error set on throughout my testing.
>
> I don't have that in my config file at all so I would guess it is at
> the default.
>

Which is off. Now I'm confused.

>> I  get the same error as Brett only when the body of the post is much
greater than that which causes the post to fail.

>>
>
> I only tried a large-ish document.  We did observe the same strange
> limit that Bill has seen when we tested without the patch applied,
> under a certain "magic" threshold the document would upload - the
> threshold seemed to be around the 50k mark, over that threshold we
> would just get popups.
>
>> I'd like to correlate network traces with debug output and would
>> appreciate suggestions as to which debug_options would include all
>> possibly relevant info
>>
>
> I am a C coder and may have some time to do some debugging on this
> between christmas and new year so, Amos, if you have any thoughts or
> hints as to where to go looking I can certainly have a stab at it.
>

Thank you. Any help at all would be great.

I *think* the relevant code is off src/client_side_reply.cc, but what to
look for is where I'm currently stuck. The keep_alive values resolved
things for you Brett but not Bill.


The variable nature of the threshold looks like some timing between
actions triggering the bug vs the rate at which Squid is sucking the
request in.

AFAIK popups only occur when the client gets sent two re-auth
challenges. Which in the un-patched Squid was caused by the first
half-authenticated link being closed by Squid before auth could
complete. Then the second link being challenged for more auth would
cause popup.

I think the next step is to find out what the difference between your
two setups is exactly:
  * squid.conf
  * headers between Squid and the POSTing app.
  * headers between Squid and the web server.

Particularly in what reply headers are going back.  That should give us
a little more of an idea what areas to look at.

If as you say the patch solved the issue but you saw the same thing
earlier. Then I suspects it's probably a squid.conf detail being
overlooked.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

-----------------------------------------------------------------
ATTENTION:
The information in this electronic mail message is private and confidential, and only intended for the addressee. Should you receive this message by mistake, you are hereby notified that any disclosure, reproduction, distribution or use of this message is strictly prohibited. Please inform the sender by reply transmission and delete the message without copying or opening it.

Messages and attachments are scanned for all viruses known.
If this message contains password-protected attachments, the files have NOT been scanned for viruses by the ING mail domain.
Always scan attachments before opening them.
-----------------------------------------------------------------
ING Belgium SA/nv -  Bank/Lender - Avenue Marnix 24, B-1000 Brussels, Belgium -  Brussels RPM/RPR - vat BE 0403.200.393 - BIC (SWIFT) : BBRUBEBB - Account: 310-9156027-89 (IBAN BE 45310-9156027-89).
An insurance broker, registered with the Banking, Finance and Insurance Commission under the code number 12381A.

ING Belgique SA - Banque/Preteur, Avenue Marnix 24, B-1000 Bruxelles - RPM Bruxelles - tva BE 0403 200 393 - BIC (SWIFT) : BBRUBEBB - Compte: 310-9156027-89 (IBAN: BE 45310-9156027-89).
Courtier d'assurances inscrit a la CBFA sous le no 12381A

ING Belgie nv - Bank/Kredietgever - Marnixlaan 24, B-1000 Brussel - RPR Brussel - btw BE 0403.200.393 - BIC (SWIFT) : BBRUBEBB - Rekening: 310-9156027-89 (IBAN: BE45 3109 1560 2789).
Verzekeringsmakelaar ingeschreven bij de CBFA onder het nr. 12381A.
-----------------------------------------------------------------

[bill.allison]

Amos (or anyone)

I'm trying to marry up debug output with tcpdump traces taken on the squid box but I want greater precision. Is there a debug_options setting which will cause entries in cache.log to be timestamped in micro-seconds? If not, is there something I can change (in debug.c ?) to make that happen (bearing in mind I'm an absolute novice C coder)?

Bill A.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: 22 December 2009 07:36
To: [hidden email]; [hidden email]
Cc: Bill Allison; [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

 
Hello all,

Just to inform you I exactly get the same problem. Firstly I thought it
was a problem with WWW-Authenticate but it is not ONLY ....

next is the reference of my first post ...
http://www.squid-cache.org/mail-archive/squid-users/200912/0029.html

I also get this same message ( httpReadReply: Request not yet fully sent
) when sending some POST requests bigger than x bytes to an  IIS server
...

I applied the patch from the bugzilla (2176) on a 2.7.4. The user does
not receive the traditional 'Page cannot be displayed' from Internet
Explorer any more but the browser freeze instead :(-

below the current config ...


client_persistent_connections on
server_persistent_connections on
acl protime url_regex -i ^http://services.group.intranet/rec
acl protime_src src all
cache_peer 1.2.3.4 parent 80 0 forceddomain=services.group.intranet
originserver proxy-only no-query no-digest connection-auth=on login=PASS
cache_peer_access 1.2.3.4 allow protime


I am certainly interested with a definitive solution so if I can be part
of the tests, just say it ...


many thks
Vincent.

-----Original Message-----
From: Bill Allison [mailto:[hidden email]]
Sent: Friday, December 18, 2009 10:47 AM
Cc: [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

Reposted for info to the list, without the attachments that cause the
list to bounce the message

-----Original Message-----
From: Bill Allison
Sent: 18 December 2009 09:43
To: 'Amos Jeffries'; Brett Lymn
Cc: [hidden email]
Subject: RE: [squid-users] any work arounds for bug 2176

"I  get the same error as Brett only when the body of the post is much
greater than that which causes the post to fail."

Correction after further testing...

I  get the same error as Brett only when the body of the post is much
greater than that which causes the post to fail, and even then only
sometimes, in repeated tests with the same file being uploaded.

Other times the browser reports "The connection was reset" and tcpdump
shows that the proxy sent a FIN to the server then to the client in
response to the second 401 from the server. THe server closes the
connection but the client continues sending a POST and the proxy then
sends the client a string of RSTs.

For info "Invalid Verb" is issued by http.sys in IIS 6.0, in response to
receiving a header that is not strictly rfc-compliant (including
truncated).

Attached as requested is my squid.conf and tcpdumps of the Invalid Verb
and RST failure cases.

Unlike Brett I'm very much a novice C coder but I'm perfectly happy to
patch, compile and test if it helps generate a solution.

Regards
Bill A.

-----Original Message-----
From: Amos Jeffries [mailto:[hidden email]]
Sent: 17 December 2009 09:10
To: Brett Lymn
Cc: Bill Allison; [hidden email]
Subject: Re: [squid-users] any work arounds for bug 2176

Brett Lymn wrote:
> On Wed, Dec 16, 2009 at 07:57:21AM -0600, Bill Allison wrote:
>> Sorry - that was misleading. I've had
>> persistent_connection_after_error set on throughout my testing.
>
> I don't have that in my config file at all so I would guess it is at
> the default.
>

Which is off. Now I'm confused.

>> I  get the same error as Brett only when the body of the post is much
greater than that which causes the post to fail.

>>
>
> I only tried a large-ish document.  We did observe the same strange
> limit that Bill has seen when we tested without the patch applied,
> under a certain "magic" threshold the document would upload - the
> threshold seemed to be around the 50k mark, over that threshold we
> would just get popups.
>
>> I'd like to correlate network traces with debug output and would
>> appreciate suggestions as to which debug_options would include all
>> possibly relevant info
>>
>
> I am a C coder and may have some time to do some debugging on this
> between christmas and new year so, Amos, if you have any thoughts or
> hints as to where to go looking I can certainly have a stab at it.
>

Thank you. Any help at all would be great.

I *think* the relevant code is off src/client_side_reply.cc, but what to
look for is where I'm currently stuck. The keep_alive values resolved
things for you Brett but not Bill.


The variable nature of the threshold looks like some timing between
actions triggering the bug vs the rate at which Squid is sucking the
request in.

AFAIK popups only occur when the client gets sent two re-auth
challenges. Which in the un-patched Squid was caused by the first
half-authenticated link being closed by Squid before auth could
complete. Then the second link being challenged for more auth would
cause popup.

I think the next step is to find out what the difference between your
two setups is exactly:
  * squid.conf
  * headers between Squid and the POSTing app.
  * headers between Squid and the web server.

Particularly in what reply headers are going back.  That should give us
a little more of an idea what areas to look at.

If as you say the patch solved the issue but you saw the same thing
earlier. Then I suspects it's probably a squid.conf detail being
overlooked.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

-----------------------------------------------------------------
ATTENTION:
The information in this electronic mail message is private and confidential, and only intended for the addressee. Should you receive this message by mistake, you are hereby notified that any disclosure, reproduction, distribution or use of this message is strictly prohibited. Please inform the sender by reply transmission and delete the message without copying or opening it.

Messages and attachments are scanned for all viruses known.
If this message contains password-protected attachments, the files have NOT been scanned for viruses by the ING mail domain.
Always scan attachments before opening them.
-----------------------------------------------------------------
ING Belgium SA/nv -  Bank/Lender - Avenue Marnix 24, B-1000 Brussels, Belgium -  Brussels RPM/RPR - vat BE 0403.200.393 - BIC (SWIFT) : BBRUBEBB - Account: 310-9156027-89 (IBAN BE 45310-9156027-89).
An insurance broker, registered with the Banking, Finance and Insurance Commission under the code number 12381A.

ING Belgique SA - Banque/Preteur, Avenue Marnix 24, B-1000 Bruxelles - RPM Bruxelles - tva BE 0403 200 393 - BIC (SWIFT) : BBRUBEBB - Compte: 310-9156027-89 (IBAN: BE 45310-9156027-89).
Courtier d'assurances inscrit a la CBFA sous le no 12381A

ING Belgie nv - Bank/Kredietgever - Marnixlaan 24, B-1000 Brussel - RPR Brussel - btw BE 0403.200.393 - BIC (SWIFT) : BBRUBEBB - Rekening: 310-9156027-89 (IBAN: BE45 3109 1560 2789).
Verzekeringsmakelaar ingeschreven bij de CBFA onder het nr. 12381A.
-----------------------------------------------------------------


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Brett Lymn]

On Thu, Dec 17, 2009 at 10:10:12PM +1300, Amos Jeffries wrote:
>
> Which is off. Now I'm confused.
>

I know the feeling well :)

> >
> >I am a C coder and may have some time to do some debugging on this
> >between christmas and new year so, Amos, if you have any thoughts or
> >hints as to where to go looking I can certainly have a stab at it.
> >
>
> Thank you. Any help at all would be great.
>
> I *think* the relevant code is off src/client_side_reply.cc, but what to
> look for is where I'm currently stuck. The keep_alive values resolved
> things for you Brett but not Bill.
>

src/client_side.c?  I think you are referring to a squid 3 file there
at a guess since it is c++

>
> The variable nature of the threshold looks like some timing between
> actions triggering the bug vs the rate at which Squid is sucking the
> request in.
>

I have done some traces and it looks like the entire file is not being
sent to the remote server... "something" happens between squid and the
remote server that stops the sending short.  The client sends the
entire file to squid.

> AFAIK popups only occur when the client gets sent two re-auth
> challenges. Which in the un-patched Squid was caused by the first
> half-authenticated link being closed by Squid before auth could
> complete. Then the second link being challenged for more auth would
> cause popup.
>

Yes, that is what we were seeing unpatched.

> I think the next step is to find out what the difference between your
> two setups is exactly:
>  * squid.conf


Here is a lightly edited squid.conf - just removed some acls that
should not affect the upload:

http_port 3128
cache_mem  32 MB
maximum_object_size 16000 KB
cache_dir aufs /cache 15000 16 256
cache_dir aufs /cache2 15000 16 256
cache_access_log /cache/logs/access.log
cache_log /cache/logs/cache.log
cache_store_log none
pid_filename /var/run/squid.pid
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
request_header_max_size 10000 KB
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 80 443 444 563
acl Safe_ports port 80 81 82 21 443 444 563 70 1025-65535
acl CONNECT method CONNECT
acl threat dstdomain "/opt/local/squid/etc/block_list.txt"
acl Safe_ports port 86
acl Safe_ports port 554
icp_access deny all
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny threat
http_access allow all
http_access deny all
http_reply_access allow all
icp_access allow all
miss_access allow all
always_direct allow all
never_direct deny all
snmp_access deny all
coredump_dir /cache/logs

>  * headers between Squid and the POSTing app.
>  * headers between Squid and the web server.
>

I have these and will send them off list.  As I mentioned before it
seems like the entire file is not being sent to the server for some
reason.  I don't understand enough to tell if this is because the
server is terminating the connection early or squid is sending
something it does not like.

>
> If as you say the patch solved the issue but you saw the same thing
> earlier. Then I suspects it's probably a squid.conf detail being overlooked.
>

If I understand Bill correctly I think we are both getting the same
thing.  I have not tested smaller files again since the patch but the
response to large file uploads is consistent with what I am seeing.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Amos Jeffries-2]

Brett Lymn wrote:

> On Thu, Dec 17, 2009 at 10:10:12PM +1300, Amos Jeffries wrote:
>> Which is off. Now I'm confused.
>>
>
> I know the feeling well :)
>
>>> I am a C coder and may have some time to do some debugging on this
>>> between christmas and new year so, Amos, if you have any thoughts or
>>> hints as to where to go looking I can certainly have a stab at it.
>>>
>> Thank you. Any help at all would be great.
>>
>> I *think* the relevant code is off src/client_side_reply.cc, but what to
>> look for is where I'm currently stuck. The keep_alive values resolved
>> things for you Brett but not Bill.
>>
>
> src/client_side.c?  I think you are referring to a squid 3 file there
> at a guess since it is c++
>
>> The variable nature of the threshold looks like some timing between
>> actions triggering the bug vs the rate at which Squid is sucking the
>> request in.
>>
>
> I have done some traces and it looks like the entire file is not being
> sent to the remote server... "something" happens between squid and the
> remote server that stops the sending short.  The client sends the
> entire file to squid.
>
>> AFAIK popups only occur when the client gets sent two re-auth
>> challenges. Which in the un-patched Squid was caused by the first
>> half-authenticated link being closed by Squid before auth could
>> complete. Then the second link being challenged for more auth would
>> cause popup.
>>
>
> Yes, that is what we were seeing unpatched.
>
>> I think the next step is to find out what the difference between your
>> two setups is exactly:
>>  * squid.conf
>
>
> Here is a lightly edited squid.conf - just removed some acls that
> should not affect the upload:
>
> http_port 3128
> cache_mem  32 MB
> maximum_object_size 16000 KB
> cache_dir aufs /cache 15000 16 256
> cache_dir aufs /cache2 15000 16 256
> cache_access_log /cache/logs/access.log
> cache_log /cache/logs/cache.log
> cache_store_log none
> pid_filename /var/run/squid.pid
> auth_param basic children 5
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours
> request_header_max_size 10000 KB
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern . 0 20% 4320
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 80 443 444 563
> acl Safe_ports port 80 81 82 21 443 444 563 70 1025-65535
> acl CONNECT method CONNECT
> acl threat dstdomain "/opt/local/squid/etc/block_list.txt"
> acl Safe_ports port 86
> acl Safe_ports port 554
> icp_access deny all
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny threat
> http_access allow all
> http_access deny all
> http_reply_access allow all
> icp_access allow all
> miss_access allow all
> always_direct allow all
> never_direct deny all
> snmp_access deny all
> coredump_dir /cache/logs
>
>>  * headers between Squid and the POSTing app.
>>  * headers between Squid and the web server.
>>
>
> I have these and will send them off list.  As I mentioned before it
> seems like the entire file is not being sent to the server for some
> reason.  I don't understand enough to tell if this is because the
> server is terminating the connection early or squid is sending
> something it does not like.
>
>> If as you say the patch solved the issue but you saw the same thing
>> earlier. Then I suspects it's probably a squid.conf detail being overlooked.
>>
>
> If I understand Bill correctly I think we are both getting the same
> thing.  I have not tested smaller files again since the patch but the
> response to large file uploads is consistent with what I am seeing.
>

I've taken a good look at the trace files on this. It's clear that the
client is in fact not sending the whole initial POST.

What I see happening is that the server early response gets relayed by
Squid and if the connection is not aborted Squid receives a small
further portion of data from the client before it abruptly stops and
starts sending the re-send POST with auth details.

Since the client has indicated a certain length X of data then only
sends N bytes the start of second request is lost and the server
complains that some random bytes mid-way down the repeat POST are an
invalid request method "verb".

To get this going we are going to have to add to the patch a bit to make
Squid delay the relayed reply until the initial POST is fully received.

PS: This has pushed Squid very, very close to the wanted behavior for
Expect-100 HTTP/1.1 requests/replies. Thanks guys.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.15