new scholarly content in (vanilla) Google?

Some google searches this morning consistently brought back
results
from JSTOR, which is a very new wrinkle 9at least to me).

If I recall correctly, Google spiders were not allowed to
crawl sites
they did not have full access to, and thus Google Scholar
was born to
have different rules and allow the spidering of journal home
pages and
other password blocked sites. However, this morning's
results were in
plain old vanilla google, and also failed to include the
"Full text "
links we see from Schoogle results.

Has anyone any solid information about this? Have Google's
rules
changed? What else is now being crawled by regular google
spiders?
Will scholar be eliminated? Bueller?

-- 
Rudy Leon
Instruction & Collection Development Librarian
College Libraries            SUNY Potsdam
http://www2.potsdam.ed
u/leonre
http://www.deepeni
ng.wordpress.com
(315) 267-3309
AIM: leonre3309
Meebo: rudyleon
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

Can you give us an example of what search returned JSTOR
results?  I'd like
to see it!

jf

On Mon, Feb 25, 2008 at 11:23 AM, Rudy Leon
<rudy.leongmail.com> wrote:

> Some google searches this morning consistently brought
back results
> from JSTOR, which is a very new wrinkle 9at least to
me).
>
> If I recall correctly, Google spiders were not allowed
to crawl sites
> they did not have full access to, and thus Google
Scholar was born to
> have different rules and allow the spidering of journal
home pages and
> other password blocked sites. However, this morning's
results were in
> plain old vanilla google, and also failed to include
the "Full text "
> links we see from Schoogle results.
>
> Has anyone any solid information about this? Have
Google's rules
> changed? What else is now being crawled by regular
google spiders?
> Will scholar be eliminated? Bueller?
>
> --
> Rudy Leon
> Instruction & Collection Development Librarian
> College Libraries            SUNY Potsdam
> http://www2.potsdam.ed
u/leonre
> http://www.deepeni
ng.wordpress.com
> (315) 267-3309
> AIM: leonre3309
> Meebo: rudyleon
> _______________________________________________
> Web4lib mailing list
> Web4libwebjunction.org
> http://lists.we
bjunction.org/web4lib/
>



-- 
http://libgrunt.blogspot
.com -- library culture and technology.
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

Can you give us an example of what search returned JSTOR
results?  I'd like
to see it!

jf

On Mon, Feb 25, 2008 at 11:23 AM, Rudy Leon
<rudy.leongmail.com> wrote:

> Some google searches this morning consistently brought
back results
> from JSTOR, which is a very new wrinkle 9at least to
me).
>
> If I recall correctly, Google spiders were not allowed
to crawl sites
> they did not have full access to, and thus Google
Scholar was born to
> have different rules and allow the spidering of journal
home pages and
> other password blocked sites. However, this morning's
results were in
> plain old vanilla google, and also failed to include
the "Full text "
> links we see from Schoogle results.
>
> Has anyone any solid information about this? Have
Google's rules
> changed? What else is now being crawled by regular
google spiders?
> Will scholar be eliminated? Bueller?
>
> --
> Rudy Leon
> Instruction & Collection Development Librarian
> College Libraries            SUNY Potsdam
> http://www2.potsdam.ed
u/leonre
> http://www.deepeni
ng.wordpress.com
> (315) 267-3309
> AIM: leonre3309
> Meebo: rudyleon
> _______________________________________________
> Web4lib mailing list
> Web4libwebjunction.org
> http://lists.we
bjunction.org/web4lib/
>



-- 
http://libgrunt.blogspot
.com -- library culture and technology.
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

I was searching for a number of books, trying primarily to
get to
publisher home pages (JSTOR reviews were a useful surprise,
but I'd
like to know more....)

One example: slavery, colonialism and economic growth

I just looked over the official Google blog, and the google
librarian
page, and found nothing about this change

On Mon, Feb 25, 2008 at 11:28 AM, John Fink
<john.finkgmail.com> wrote:
> Can you give us an example of what search returned
JSTOR results?  I'd like
> to see it!
>
> jf
>
>
>
> On Mon, Feb 25, 2008 at 11:23 AM, Rudy Leon
<rudy.leongmail.com> wrote:
>
> >
> >
> >
> > Some google searches this morning consistently
brought back results
> > from JSTOR, which is a very new wrinkle 9at least
to me).
> >
> > If I recall correctly, Google spiders were not
allowed to crawl sites
> > they did not have full access to, and thus Google
Scholar was born to
> > have different rules and allow the spidering of
journal home pages and
> > other password blocked sites. However, this
morning's results were in
> > plain old vanilla google, and also failed to
include the "Full text "
> > links we see from Schoogle results.
> >
> > Has anyone any solid information about this? Have
Google's rules
> > changed? What else is now being crawled by regular
google spiders?
> > Will scholar be eliminated? Bueller?
> >
> > --
> > Rudy Leon
> > Instruction & Collection Development
Librarian
> > College Libraries            SUNY Potsdam
> > http://www2.potsdam.ed
u/leonre
> > http://www.deepeni
ng.wordpress.com
> > (315) 267-3309
> > AIM: leonre3309
> > Meebo: rudyleon
> > _______________________________________________
> > Web4lib mailing list
> > Web4libwebjunction.org
> > http://lists.we
bjunction.org/web4lib/
> >
>
>
>
> --
> http://libgrunt.blogspot
.com -- library culture and technology.



-- 
Rudy Leon
Instruction & Collection Development Librarian
College Libraries            SUNY Potsdam
http://www2.potsdam.ed
u/leonre
http://www.deepeni
ng.wordpress.com
(315) 267-3309
AIM: leonre3309
Meebo: rudyleon
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

I was searching for a number of books, trying primarily to
get to
publisher home pages (JSTOR reviews were a useful surprise,
but I'd
like to know more....)

One example: slavery, colonialism and economic growth

I just looked over the official Google blog, and the google
librarian
page, and found nothing about this change

On Mon, Feb 25, 2008 at 11:28 AM, John Fink
<john.finkgmail.com> wrote:
> Can you give us an example of what search returned
JSTOR results?  I'd like
> to see it!
>
> jf
>
>
>
> On Mon, Feb 25, 2008 at 11:23 AM, Rudy Leon
<rudy.leongmail.com> wrote:
>
> >
> >
> >
> > Some google searches this morning consistently
brought back results
> > from JSTOR, which is a very new wrinkle 9at least
to me).
> >
> > If I recall correctly, Google spiders were not
allowed to crawl sites
> > they did not have full access to, and thus Google
Scholar was born to
> > have different rules and allow the spidering of
journal home pages and
> > other password blocked sites. However, this
morning's results were in
> > plain old vanilla google, and also failed to
include the "Full text "
> > links we see from Schoogle results.
> >
> > Has anyone any solid information about this? Have
Google's rules
> > changed? What else is now being crawled by regular
google spiders?
> > Will scholar be eliminated? Bueller?
> >
> > --
> > Rudy Leon
> > Instruction & Collection Development
Librarian
> > College Libraries            SUNY Potsdam
> > http://www2.potsdam.ed
u/leonre
> > http://www.deepeni
ng.wordpress.com
> > (315) 267-3309
> > AIM: leonre3309
> > Meebo: rudyleon
> > _______________________________________________
> > Web4lib mailing list
> > Web4libwebjunction.org
> > http://lists.we
bjunction.org/web4lib/
> >
>
>
>
> --
> http://libgrunt.blogspot
.com -- library culture and technology.



-- 
Rudy Leon
Instruction & Collection Development Librarian
College Libraries            SUNY Potsdam
http://www2.potsdam.ed
u/leonre
http://www.deepeni
ng.wordpress.com
(315) 267-3309
AIM: leonre3309
Meebo: rudyleon
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

> Can you give us an example of what search returned
JSTOR results?  I'd like
> to see it!

As it happens, a few days ago I had a patron searching for

mashpee wampanoag census 1832

and it returned results from both JSTOR and Project MUSE.

-- 
Bob Sullivan
Schenectady Digital History Archive
<http://www.sch
enectadyhistory.org/>
Schenectady County (NY) Public Library
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

> Can you give us an example of what search returned
JSTOR results?  I'd like
> to see it!

As it happens, a few days ago I had a patron searching for

mashpee wampanoag census 1832

and it returned results from both JSTOR and Project MUSE.

-- 
Bob Sullivan
Schenectady Digital History Archive
<http://www.sch
enectadyhistory.org/>
Schenectady County (NY) Public Library
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

>> On Mon, Feb 25, 2008 at 11:23 AM, Rudy Leon
<rudy.leongmail.com> wrote:

> Has anyone any solid information about this? Have
Google's rules
> changed? What else is now being crawled by regular
google spiders?
> Will scholar be eliminated? Bueller?

Google has been crawling both JSTOR and Project MUSE for a
couple of years
now, and, as far as I know, our articles have always shown
up in the
regular Google interface.  (In fact, I have often gotten
better results
looking for humanities material in vanilla Google than
Scholar, which seems
to have an STM bias as a side-effect of its ranking
algorithm).

I'm pretty sure that Scholar's here to stay.

             -- Brian


--
Brian Harrington
Project MUSE
The Johns Hopkins University Press
brianjhu.edu
_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

On Feb 25, 2008, at 11:23 AM, Rudy Leon wrote:

> If I recall correctly, Google spiders were not allowed
to crawl sites
> they did not have full access to, and thus Google
Scholar was born to
> have different rules and allow the spidering of journal
home pages and
> other password blocked sites. However, this morning's
results were in
> plain old vanilla google, and also failed to include
the "Full text "
> links we see from Schoogle results.


JSTOR and Muse, as well as a number of other sites, both
specifically  
allow the Google crawlers to index their site. You can see
part of  
their instructions to Google and other crawlers in the
robots.txt:

http://www.jstor.org/
robots.txt

They might also be checking if the request comes from a
crawler and  
changing the content they return based on that. A good
example is  
NPR.org, which lets Google index their full text
transcripts, but  
requires users to pay for them.

But your example is a good one: when information is opened
up to  
Google, it's usually findable there.


Casey Bisson
__________________________________________

Information Architect
Plymouth State University
Plymouth, New Hampshire
http://MaisonBisson.com/

ph: 603-535-2256

_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/

On Feb 25, 2008, at 11:23 AM, Rudy Leon wrote:

> If I recall correctly, Google spiders were not allowed
to crawl sites
> they did not have full access to, and thus Google
Scholar was born to
> have different rules and allow the spidering of journal
home pages and
> other password blocked sites. However, this morning's
results were in
> plain old vanilla google, and also failed to include
the "Full text "
> links we see from Schoogle results.


JSTOR and Muse, as well as a number of other sites, both
specifically  
allow the Google crawlers to index their site. You can see
part of  
their instructions to Google and other crawlers in the
robots.txt:

http://www.jstor.org/
robots.txt

They might also be checking if the request comes from a
crawler and  
changing the content they return based on that. A good
example is  
NPR.org, which lets Google index their full text
transcripts, but  
requires users to pay for them.

But your example is a good one: when information is opened
up to  
Google, it's usually findable there.


Casey Bisson
__________________________________________

Information Architect
Plymouth State University
Plymouth, New Hampshire
http://MaisonBisson.com/

ph: 603-535-2256

_______________________________________________
Web4lib mailing list
Web4libwebjunction.org
http://lists.we
bjunction.org/web4lib/