Updating RFC 2617 (HTTP Digest) to use UTF-8

Many thanks to Bjoern for the detailed checking and report.

My summary of the situation would be as follows:
There is currently widely varying practice. Current
implementations
are anyways broken and non-interoperable. The main reason
for this
is most probably that there is no clear specification. This
means
that an update of RFC 2617 is desirable. The new
specification
should probably go for UTF-8, while noting that there is
still
some varying practice.

Regards,   Martin.


At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>* Alexey Melnikov wrote:
>>Does anybody know if updating RFC 2617 to say that
username/passwords 
>>are UTF-8 would break any major implementation? For
example, does 
>>anybody know if a major HTTP client/server
implementation assume ISO 8859-1?
>
>It appears that for Basic authentication the german
version of Internet
>Explorer 6 running on the german version of Windows 2003
as well as the
>latest english Internet Explorer 7 release candidate
running on the
>german version of Windows XP will use something like
ISO-8859-1 for both
>manual as well as XMLHttpRequest requests. Trying to use
U+20AC as user
>name and password they got encoded as 0x80
(Windows-1252) for manual re-
>quests, and to '?' for XHR. Characters not included in
Windows-1252 come
>out as '?' regardless of the method used. For XHR my
test cases include
>documents encoded as ISO-8859-1 and UTF-8; there did not
appear to be
>any difference.
>
>The latest en-us version of Firefox uses UTF-8 for XHR
and the lower
>byte of the character when encoded using UTF-16BE (so
for U+20AC you
>get 0xAC) when using manual input. For manually entered
http://u:p...
>URLs Firefox uses Windows-1252 if possible, UTF-8
otherwise. When XHR
>is used with such a URL, it uses UTF-8. The latest en-us
version of
>Opera9 always uses UTF-8, as far as I can tell based on
my limited
>testing. Results might well be different on with
different default code
>pages, language settings, and so on. Note that the
illegal http://u:p..
>addressing scheme allows to use arbitrary octet
sequences using %hh
>escape sequences, with some browser-specific
limitations.
>-- 
>Bj$B‹S(Bn H$B‹I(Brmann $B%-(B mailto:bjoernhoehrmann.de $B%-(B http://bjoern.hoehrmann.de

>Weinh. Str. 22 $B%-(B Telefon: +49(0)621/4309674
$B%-(B http://www.bjoernsworld.de

>68309 Mannheim $B%-(B PGP Pub. KeyID: 0xA4357E78
$B%-(B http://www.websitedev.de/ 
>_______________________________________________
>Ietf-http-auth mailing list
>Ietf-http-authosafoundation.org
>http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama
Gakuin University
#-#-#  http://www.sw.it.aoyama
.ac.jp       mailto:duerstit.aoyama.ac.jp     

_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

I agree it would be good to update RFC2617.  Is anybody
going to take  
a stab at it?

Lisa

On Sep 22, 2006, at 7:55 PM, Martin Duerst wrote:

> Many thanks to Bjoern for the detailed checking and
report.
>
> My summary of the situation would be as follows:
> There is currently widely varying practice. Current
implementations
> are anyways broken and non-interoperable. The main
reason for this
> is most probably that there is no clear specification.
This means
> that an update of RFC 2617 is desirable. The new
specification
> should probably go for UTF-8, while noting that there
is still
> some varying practice.
>
> Regards,   Martin.
>
>
> At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>> * Alexey Melnikov wrote:
>>> Does anybody know if updating RFC 2617 to say
that username/ 
>>> passwords
>>> are UTF-8 would break any major implementation?
For example, does
>>> anybody know if a major HTTP client/server
implementation assume  
>>> ISO 8859-1?
>>
>> It appears that for Basic authentication the german
version of  
>> Internet
>> Explorer 6 running on the german version of Windows
2003 as well  
>> as the
>> latest english Internet Explorer 7 release
candidate running on the
>> german version of Windows XP will use something
like ISO-8859-1  
>> for both
>> manual as well as XMLHttpRequest requests. Trying
to use U+20AC as  
>> user
>> name and password they got encoded as 0x80
(Windows-1252) for  
>> manual re-
>> quests, and to '?' for XHR. Characters not
included in  
>> Windows-1252 come
>> out as '?' regardless of the method used. For XHR
my test cases  
>> include
>> documents encoded as ISO-8859-1 and UTF-8; there
did not appear to be
>> any difference.
>>
>> The latest en-us version of Firefox uses UTF-8 for
XHR and the lower
>> byte of the character when encoded using UTF-16BE
(so for U+20AC you
>> get 0xAC) when using manual input. For manually
entered http:// 
>> u:p...
>> URLs Firefox uses Windows-1252 if possible, UTF-8
otherwise. When XHR
>> is used with such a URL, it uses UTF-8. The latest
en-us version of
>> Opera9 always uses UTF-8, as far as I can tell
based on my limited
>> testing. Results might well be different on with
different default  
>> code
>> pages, language settings, and so on. Note that the
illegal http:// 
>> u:p..
>> addressing scheme allows to use arbitrary octet
sequences using %hh
>> escape sequences, with some browser-specific
limitations.
>> -- 
>> Bj�n H�rmann キ mailto:bjoernhoehrmann.de キ http:// 
>> bjoern.hoehrmann.de
>> Weinh. Str. 22 ã‚­ Telefon: +49(0)621/4309674 ã‚­
http:// 
>> www.bjoernsworld.de
>> 68309 Mannheim ã‚­ PGP Pub. KeyID: 0xA4357E78 ã‚­
http:// 
>> www.websitedev.de/
>> _______________________________________________
>> Ietf-http-auth mailing list
>> Ietf-http-authosafoundation.org
>> http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/ietf-http- 
>> auth
>
>
> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama
Gakuin University
> #-#-#  http://www.sw.it.aoyama
.ac.jp        
> mailto:duerstit.aoyama.ac.jp
>
> _______________________________________________
> Ietf-http-auth mailing list
> Ietf-http-authosafoundation.org
> http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

Hi,

Do not the HTTP headers from the browser already say what
character
encoding has been used, and do not the HTTP headers and HTML
from the
server beforehand dictate this stuff too?  Except for the
case where
someone is trying to type foreign characters on a PC which
can't
understand them (eg: holidaymakers at net cafes where the
net cafe has
not installed the language they want to use) - everything
should
already be working just fine, yes?

Kind Regards,
Chris Drake


Tuesday, September 26, 2006, 4:31:13 AM, you wrote:

LD> I agree it would be good to update RFC2617.  Is
anybody going to take
LD> a stab at it?

LD> Lisa

LD> On Sep 22, 2006, at 7:55 PM, Martin Duerst wrote:

>> Many thanks to Bjoern for the detailed checking and
report.
>>
>> My summary of the situation would be as follows:
>> There is currently widely varying practice. Current
implementations
>> are anyways broken and non-interoperable. The main
reason for this
>> is most probably that there is no clear
specification. This means
>> that an update of RFC 2617 is desirable. The new
specification
>> should probably go for UTF-8, while noting that
there is still
>> some varying practice.
>>
>> Regards,   Martin.
>>
>>
>> At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>>> * Alexey Melnikov wrote:
>>>> Does anybody know if updating RFC 2617 to
say that username/ 
>>>> passwords
>>>> are UTF-8 would break any major
implementation? For example, does
>>>> anybody know if a major HTTP client/server
implementation assume
>>>> ISO 8859-1?
>>>
>>> It appears that for Basic authentication the
german version of  
>>> Internet
>>> Explorer 6 running on the german version of
Windows 2003 as well  
>>> as the
>>> latest english Internet Explorer 7 release
candidate running on the
>>> german version of Windows XP will use something
like ISO-8859-1  
>>> for both
>>> manual as well as XMLHttpRequest requests.
Trying to use U+20AC as
>>> user
>>> name and password they got encoded as 0x80
(Windows-1252) for  
>>> manual re-
>>> quests, and to '?' for XHR. Characters not
included in  
>>> Windows-1252 come
>>> out as '?' regardless of the method used. For
XHR my test cases  
>>> include
>>> documents encoded as ISO-8859-1 and UTF-8;
there did not appear to be
>>> any difference.
>>>
>>> The latest en-us version of Firefox uses UTF-8
for XHR and the lower
>>> byte of the character when encoded using
UTF-16BE (so for U+20AC you
>>> get 0xAC) when using manual input. For manually
entered http:// 
>>> u:p...
>>> URLs Firefox uses Windows-1252 if possible,
UTF-8 otherwise. When XHR
>>> is used with such a URL, it uses UTF-8. The
latest en-us version of
>>> Opera9 always uses UTF-8, as far as I can tell
based on my limited
>>> testing. Results might well be different on
with different default
>>> code
>>> pages, language settings, and so on. Note that
the illegal http://
>>> u:p..
>>> addressing scheme allows to use arbitrary octet
sequences using %hh
>>> escape sequences, with some browser-specific
limitations.
>>> -- 
>>> Bj?n H?rmann ? mailto:bjoernhoehrmann.de ? http:// 
>>> bjoern.hoehrmann.de
>>> Weinh. Str. 22 ? Telefon: +49(0)621/4309674 ?
http:// 
>>> www.bjoernsworld.de
>>> 68309 Mannheim ? PGP Pub. KeyID: 0xA4357E78 ?
http:// 
>>> www.websitedev.de/
>>> _______________________________________________
>>> Ietf-http-auth mailing list
>>> Ietf-http-authosafoundation.org
>>> http://lists.osafoundation.org/cgi-bin/mailman
/listinfo/ietf-http-
>>> auth
>>
>>
>> #-#-#  Martin J. Du"rst, Assoc. Professor,
Aoyama Gakuin University
>> #-#-#  http://www.sw.it.aoyama
.ac.jp        
>> mailto:duerstit.aoyama.ac.jp
>>
>> _______________________________________________
>> Ietf-http-auth mailing list
>> Ietf-http-authosafoundation.org
>> http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

LD> _______________________________________________
LD> Ietf-http-auth mailing list
LD> Ietf-http-authosafoundation.org
LD> http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth
 



_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

Chris Drake schrieb:
> Hi,
> 
> Do not the HTTP headers from the browser already say
what character
> encoding has been used, and do not the HTTP headers and
HTML from the

That's for the request body, not the headers.

> server beforehand dictate this stuff too?  Except for
the case where
> someone is trying to type foreign characters on a PC
which can't
> understand them (eg: holidaymakers at net cafes where
the net cafe has
> not installed the language they want to use) -
everything should
> already be working just fine, yes?

I don't think so.

Speaking of which, where does HTML come into play here?
We're talking 
about HTTP authentication à la RFC2617, not HTML forms based
login.

Best regards, Julian
_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

* Julian Reschke wrote:
>Speaking of which, where does HTML come into play here?
We're talking 
>about HTTP authentication à la RFC2617, not HTML forms
based login.

For HTML form submissions, unless the author indicated
something else,
web browsers tend to use the character encoding of the
document that in-
cludes the form to encode the characters; they could apply
the same
logic to the submission of credentials when there is such a
document
(e.g., the user clicked a link to a HTTP Auth protected
page, the page
with the link could then be used to determine some
encoding). Based on
my limited testing, I found this to be not the case.
-- 
Björn Höhrmann · mailto:bjoernhoehrmann.de · http://bjoern.hoehrmann.de

Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de

68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 
_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

On Tue, 26 Sep 2006 00:08:11 +0200, Bjoern Hoehrmann
<derhoermigmx.net>  
wrote:

> * Julian Reschke wrote:
>> Speaking of which, where does HTML come into play
here? We're talking
>> about HTTP authentication à la RFC2617, not HTML
forms based login.
>
> For HTML form submissions, unless the author indicated
something else,
> web browsers tend to use the character encoding of the
document that in-
> cludes the form to encode the characters; they could
apply the same
> logic to the submission of credentials when there is
such a document
> (e.g., the user clicked a link to a HTTP Auth protected
page, the page
> with the link could then be used to determine some
encoding). Based on
> my limited testing, I found this to be not the case.

In the HTML case the information is directly available, as
part of the  
form's own environment, which it would not be when the
authentication  
credentials are being processed.

Which characterset/encoding should the application choose
when that  
information is not available?

E.g. The information would not be available if the user goes
directly to  
the site, by entering the URL directly. Similar situation
could arise when  
the URL is loaded in another tab than the originating
document, and  
obtaining the information would also not be entirely
straight forward even  
if it is opened in the same tab. Or what about proxy
authentication?

Also: What if the original page is using a different
characterset/encoding  
than used by the server requesting authentication? E.g. What
if a Russian  
language page directs you to an authenticated Japanese site?
And AFAIK  
several languages actually have multiple encodings.

Also, characterset/encoding information may not be available
in the HTTP  
header either.

It is not possible to define a heuristic that will fit all
scenarios. The  
best approach is to define a common characterset/encoding
that will be  
used by all compliant servers.

As RFC 2617 was not able to assist, the only guidance I had
when I chose  
the I18N policy for Opera's RFC 2617 support, was RFC
2277/BCP 18, which I  
interprete to say that protocols should use UTF-8 unless
they specify  
otherwise either in the specification (i.e. the RFC) or in a
specific  
field of the protocol (in this case, that would mean an
attribute in the  
WWW-Authenticate header). The problem is that work on the
RFC 2617  
protocol probably started before RFC 2217 was finished.

Given that the current system is broken anyway, since client
and server  
have to agree out-of-band on which characterset/encoding to
use, it is in  
my opinion best to define a proper solution, which IMO means
UTF-8,  
instead of trying to patch up the broken system . (And
remember: Even a  
patch of the current system would have to be deployed in new
clients and  
servers).


-- 
Sincerely,
Yngve N. Pettersen
 
************************************************************
********
Senior Developer                     Email: yngveopera.com
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
************************************************************
********
_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

I can help with the pieces related to i18n/utf-8, but I'm
not at all an expert in HTTP auth, so I won't try, sorry.

Regards,     Martin.

At 03:31 06/09/26, Lisa Dusseault wrote:
>I agree it would be good to update RFC2617.  Is anybody
going to take  
>a stab at it?
>
>Lisa


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama
Gakuin University
#-#-#  http://www.sw.it.aoyama
.ac.jp       mailto:duerstit.aoyama.ac.jp     

_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

Martin Duerst wrote:

>I can help with the pieces related to i18n/utf-8, but
I'm
>not at all an expert in HTTP auth, so I won't try,
sorry.
>  
>
I can do 2617 update, but I am not an HTTP expert either.
So I would need a co-editor, who is.

>Regards,     Martin.
>
>At 03:31 06/09/26, Lisa Dusseault wrote:
>  
>
>>I agree it would be good to update RFC2617.  Is
anybody going to take  
>>a stab at it?
>>
>>Lisa
>>    
>>

_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth

On 9/25/06, Lisa Dusseault <lisaosafoundation.org>
wrote:
> I agree it would be good to update RFC2617.  Is anybody
going to take
> a stab at it?
>

As a data point, Mozilla always uses a lossy UTF16-to-ASCII
conversion
for Basic [1] and always uses UTF-8 for Digest [2]. Both
implementations date from 2001. We have no bugs filed on
Digest
authentication charsets, and one bug on Basic that's been
filed
several times.[3]

I don't think a 2617 update is a good use of time, but I
will update
our networking code in the event an update is completed.
MD5-sess is
essentially unused. An update should drop it.

1.) http://lxr.mozilla.org/
seamonkey/source/netwerk/protocol/http/src/nsHttpBasicAuth.c
pp#108
2.) http://lxr.mozilla.org
/seamonkey/source/netwerk/protocol/http/src/nsHttpDigestAuth
.cpp#330
3.) ht
tps://bugzilla.mozilla.org/show_bug.cgi?id=41489

-- 

Robert Sayre
_______________________________________________
Ietf-http-auth mailing list
Ietf-http-authosafoundation.org
http://lists.osafoundation.org/cgi-bin/mai
lman/listinfo/ietf-http-auth