This is version 2 of my xmlIO HTTPS handler using libcurl.

It fixes the following issues:

1) now installs curlhttps.h in include/libxml2/
2) mimics redirect handling of nanohttp.c

There is also now a configuration API for enabling
certification 
verification via libcurl. Basically:

typedef enum {
     XML_HTTPS_VERIFYHOST	= 1<<0,	
     /* verify that the server cert is for the server it is
known as */
     XML_HTTPS_VERIFYPEER	= 1<<1	
     /* verify the authenticity of the peer's certificate
*/
} xmlHttpsOption;

XMLPUBFUN int XMLCALL xmlIOHTTPSSetup (int options);

jr


_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

On Wed, Mar 21, 2007 at 12:37:26PM -0400, joel reed wrote:
> This is version 2 of my xmlIO HTTPS handler using
libcurl.

  Hi Joel,

> It fixes the following issues:
> 
> 1) now installs curlhttps.h in include/libxml2/
> 2) mimics redirect handling of nanohttp.c
> 
> There is also now a configuration API for enabling
certification 
> verification via libcurl. Basically:
> 
> typedef enum {
>     XML_HTTPS_VERIFYHOST	= 1<<0,	
>     /* verify that the server cert is for the server it
is known as */
>     XML_HTTPS_VERIFYPEER	= 1<<1	
>     /* verify the authenticity of the peer's
certificate */
> } xmlHttpsOption;
> 
> XMLPUBFUN int XMLCALL xmlIOHTTPSSetup (int options);

  Thanks for the patch, though I'm still wondering abit
about it:
    - I have said in the past that I didn't want to add too
much to nano*
    - adding a dependancy to libxml2 always annoys me
on the other hand
    - the patch is nice
    - other people may use this
    - the code is not compiled and the dependancy does not
exist by default

I'm still undecided about this, more code means that this
gets factored,
which is potentally good for a few users, but grows the
maintainance a bit.

Also it seems to me the patch avoids the core of the issue
which is 
authentication, that was one of my main discussion w.r.t.
extending
nanohttp is that people wanted auth, and IMHO that was
outside the scope
of libxml2, and I suggested to plug curl in and deal with it
there.
So how is https really useful if you don't have
authentication (or maybe I
missed it and it's done by the curl layer), but I would at
least expect
some kind of callback to handle 401 auth requests, and I
don't see 
any in the patch, so I'm wondering what's the limitation
here 

Daniel

-- 
Red Hat Virtualization group http://redhat.com/v
irtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillardredhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ |
Rpmfind RPM search engine  http://rpmfind.net/
_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

Daniel Veillard wrote:

<snip>
>   Thanks for the patch, though I'm still wondering abit
about it:
>     - I have said in the past that I didn't want to add
too much to nano*
>     - adding a dependancy to libxml2 always annoys me
> on the other hand
>     - the patch is nice
>     - other people may use this
>     - the code is not compiled and the dependancy does
not exist by default
>
> I'm still undecided about this, more code means that
this gets factored,
> which is potentally good for a few users, but grows the
maintainance a bit.
>   
> Also it seems to me the patch avoids the core of the
issue which is 
> authentication, that was one of my main discussion
w.r.t. extending
> nanohttp is that people wanted auth, and IMHO that was
outside the scope
> of libxml2, and I suggested to plug curl in and deal
with it there.
> So how is https really useful if you don't have
authentication (or maybe I
> missed it and it's done by the curl layer), but I would
at least expect
> some kind of callback to handle 401 auth requests, and
I don't see 
> any in the patch, so I'm wondering what's the
limitation here 
>
> Daniel
>
>   
Yes I agree another dependency is annoying. We are doing
forms based 
authentication over SSL here, so the need to handle http
basic/digest 
type authentication hasn't come up but I can see the
usefulness for others.

One thing I had considered is making the curl patch handle
http, https, 
ftp, sftps, and whatever else (http://curl.ha
xx.se/docs/features.html) 
libcurl can handle that would be of interest for xmlIO. As
part of that 
I could add an authentication callback.

Could we say that if you enable curl support,  nano http/ftp
handlers 
are disabled at compile time or would you like to approach
it some other 
way? (if the above sounds interesting) If the dependency is
really too 
annoying, no problem, patch can sit on mailing list for
others if they 
need it.

jr
_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

On Wed, Mar 21, 2007 at 01:38:43PM -0400, joel reed wrote:
> Yes I agree another dependency is annoying. We are
doing forms based 
> authentication over SSL here, so the need to handle
http basic/digest 
> type authentication hasn't come up but I can see the
usefulness for others.

  I see,

> One thing I had considered is making the curl patch
handle http, https, 
> ftp, sftps, and whatever else (http://curl.ha
xx.se/docs/features.html) 
> libcurl can handle that would be of interest for xmlIO.
As part of that 
> I could add an authentication callback.
> 
> Could we say that if you enable curl support,  nano
http/ftp handlers 
> are disabled at compile time or would you like to
approach it some other 
> way? (if the above sounds interesting) If the
dependency is really too 
> annoying, no problem, patch can sit on mailing list for
others if they 
> need it.

  I tend to think that if we integrate curl support then
let's do it
fully, and avoid the nano side of things. Then the configure
option 
would really be useful to a larger audience.
  At this point my take is that an optional feature plugging
curl fully
replacing the nano modules and adding authentication would
makes a
lot more sense than a limited plug, and that should be added
to the 
default code base because a lot more users are likely to
use/debug/fix
potential problems.

   Makes sense ?

Daniel

-- 
Red Hat Virtualization group http://redhat.com/v
irtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillardredhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ |
Rpmfind RPM search engine  http://rpmfind.net/
_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

Daniel Veillard wrote:
> On Wed, Mar 21, 2007 at 01:38:43PM -0400, joel reed
wrote:
>   
>> Yes I agree another dependency is annoying. We are
doing forms based 
>> authentication over SSL here, so the need to handle
http basic/digest 
>> type authentication hasn't come up but I can see
the usefulness for others.
>>     
>
>   I see,
>
>   
>> One thing I had considered is making the curl patch
handle http, https, 
>> ftp, sftps, and whatever else (http://curl.ha
xx.se/docs/features.html) 
>> libcurl can handle that would be of interest for
xmlIO. As part of that 
>> I could add an authentication callback.
>>
>> Could we say that if you enable curl support,  nano
http/ftp handlers 
>> are disabled at compile time or would you like to
approach it some other 
>> way? (if the above sounds interesting) If the
dependency is really too 
>> annoying, no problem, patch can sit on mailing list
for others if they 
>> need it.
>>     
>
>   I tend to think that if we integrate curl support
then let's do it
> fully, and avoid the nano side of things. Then the
configure option 
> would really be useful to a larger audience.
>   At this point my take is that an optional feature
plugging curl fully
> replacing the nano modules and adding authentication
would makes a
> lot more sense than a limited plug, and that should be
added to the 
> default code base because a lot more users are likely
to use/debug/fix
> potential problems.
>
>    Makes sense ?
>
> Daniel
>
>   
This does make sense Daniel, thanks for thinking through the
right 
approach to take.
I'll take a crack at a v3 patch - give me some time.

jr

_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

Daniel Veillard wrote:
> On Wed, Mar 21, 2007 at 01:38:43PM -0400, joel reed
wrote:
>   
>> Yes I agree another dependency is annoying. We are
doing forms based 
>> authentication over SSL here, so the need to handle
http basic/digest 
>> type authentication hasn't come up but I can see
the usefulness for others.
>>     
>
>   I see,
>
>   
>> One thing I had considered is making the curl patch
handle http, https, 
>> ftp, sftps, and whatever else (http://curl.ha
xx.se/docs/features.html) 
>> libcurl can handle that would be of interest for
xmlIO. As part of that 
>> I could add an authentication callback.
>>
>> Could we say that if you enable curl support,  nano
http/ftp handlers 
>> are disabled at compile time or would you like to
approach it some other 
>> way? (if the above sounds interesting) If the
dependency is really too 
>> annoying, no problem, patch can sit on mailing list
for others if they 
>> need it.
>>     
>
>   I tend to think that if we integrate curl support
then let's do it
> fully, and avoid the nano side of things. Then the
configure option 
> would really be useful to a larger audience.
>   At this point my take is that an optional feature
plugging curl fully
> replacing the nano modules and adding authentication
would makes a
> lot more sense than a limited plug, and that should be
added to the 
> default code base because a lot more users are likely
to use/debug/fix
> potential problems.
>
>    Makes sense ?
>
> Daniel
>
>   
I think I'll rename this thing curlIO.{h,c}. Any thoughts on

configuration API? I think I need to change xmlIO to call a

configuration callback function or some such thing, as my
current 
xmlIOHTTPSSetup approach is library wide, but what we really
want is a 
context specific configuration capability. Also, I'm not
sure it makes 
sense to wrap every possible curl configuration option 
(h
ttp://curl.haxx.se/libcurl/c/curl_easy_setopt.html). I'm
thinking if 
there was a callback that got the context pointer, maybe all
we need is 
a function which returns the CURL*, then the application
could use 
whatever CURL config options were available. There are a ton
of curl 
config options. Does this sound OK?

jr


_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

On Wed, Mar 21, 2007 at 02:22:17PM -0400, joel reed wrote:
> Daniel Veillard wrote:
> >  I tend to think that if we integrate curl support
then let's do it
> >fully, and avoid the nano side of things. Then the
configure option 
> >would really be useful to a larger audience.
> >  At this point my take is that an optional feature
plugging curl fully
> >replacing the nano modules and adding
authentication would makes a
> >lot more sense than a limited plug, and that should
be added to the 
> >default code base because a lot more users are
likely to use/debug/fix
> >potential problems.
> >  
> This does make sense Daniel, thanks for thinking
through the right 
> approach to take.
> I'll take a crack at a v3 patch - give me some time.

  Cool, no hurry 

Daniel

-- 
Red Hat Virtualization group http://redhat.com/v
irtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillardredhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ |
Rpmfind RPM search engine  http://rpmfind.net/
_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

On Wed, Mar 21, 2007 at 02:45:36PM -0400, joel reed wrote:
> I think I'll rename this thing curlIO.{h,c}. Any
thoughts on 

  makes sense.

> configuration API? I think I need to change xmlIO to
call a 
> configuration callback function or some such thing, as
my current 
> xmlIOHTTPSSetup approach is library wide, but what we
really want is a 
> context specific configuration capability. Also, I'm
not sure it makes 

  should we need to compile in nano* if curl support is
configured in ?
I would probably change xmlRegisterDefaultInputCallbacks()
and
xmlRegisterDefaultOutputCallbacks() undefining
LIBXML_HTTP_ENABLED
and LIBXML_FTP_ENABLED and using a special register entry
point for
curl. Some of the things which may not be trivial are the
write
function for FTP and HTTP, at least testing they work okay.

> sense to wrap every possible curl configuration option

> (h
ttp://curl.haxx.se/libcurl/c/curl_easy_setopt.html). I'm
thinking if 
> there was a callback that got the context pointer,
maybe all we need is 
> a function which returns the CURL*, then the
application could use 
> whatever CURL config options were available. There are
a ton of curl 
> config options. Does this sound OK?

  You mean the (void *) context used as first arg for
read/write/close,
then yes sure, that is curl specific and I would assume it's
a 
CURL *handle, but it could be libxml2 specific structure
with the handle
and provide an accessor.
  I'm sure other people have used curl in a libxml2 context,
their feedback
would be welcome.

Daniel

-- 
Red Hat Virtualization group http://redhat.com/v
irtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillardredhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ |
Rpmfind RPM search engine  http://rpmfind.net/
_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

Hi Daniel,

>   I'm sure other people have used curl in a libxml2
context, their feedback
> would be welcome.

We use curl with libxml2 in Prince by specifying our own
entity loader. 
One downside of this approach is that we cannot fully
support the 
"referer" HTTP header, as the entity loader is
just given a URL to load 
but is not given the URL of the referring document. I guess
that using 
curl as an IO handler directly within libxml2 would avoid
this issue?

Best regards,

Michael

-- 
Print XML with Prince!
http://www.princexml.com

_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml

Michael Day wrote:
> Hi Daniel,
>
>>   I'm sure other people have used curl in a libxml2
context, their 
>> feedback
>> would be welcome.
>
> We use curl with libxml2 in Prince by specifying our
own entity 
> loader. One downside of this approach is that we cannot
fully support 
> the "referer" HTTP header, as the entity
loader is just given a URL to 
> load but is not given the URL of the referring
document. I guess that 
> using curl as an IO handler directly within libxml2
would avoid this 
> issue?
>
> Best regards,
>
> Michael
>
yes, i believe this would avoid this issue. Acc'd to 
h
ttp://curl.haxx.se/libcurl/c/curl_easy_setopt.html,
you would use CURLOPT_REFERER

jr
_______________________________________________
xml mailing list, project page  http://xmlsoft.org/
xmlgnome.org
http://mai
l.gnome.org/mailman/listinfo/xml