BSDStats v4.0: Attempt to address some major issues ...

As painful as it was to do, I backed up the old data tonight
and wiped out 
the stats ... for one major reason: the stats lost their
accuracy.

As several have noticed, the OpenBSD numbers have been
increasing quite 
steadily, but looking at the RAW urls coming in, 99% of
those #s were from 
a couple of IPs, and they were submitting the full range of
releases and 
architectures ...

Unfortunately, since we store neither IP or hostname, trying
to 
'surgically remove' the false entries proved to be very
difficult ...

I've have just committed BSDstats 4.0 to CVS, and there is a
copy of it 
available at http
://www.bsdstats.org/downloads/300.statistics ...
download 
and replace the one you are currently running, as it will
not work anymore 
... the new version even knows to remove your
/var/db/bsdstats file, since 
its format is no longer valid either ...

I've increased the size of the IDTOKEN to 32 from 16, since
I've been 
noticing alot of duplicates when two hosts submit at close
to the same 
time ...

I've also added an 'enable/disable' record before/after
submitting the 
reports ...

And, when reporting, both your TOKEN *and* KEY are sent,
instead of just 
the KEY ... that way we not only check what we returned to
you the first 
time, but also what you sent to us ...

This version also has a 'Network Connectivity' check, in the
form of 
checking a TXT record in DNS, so that if the Network is
down, it doesn't 
bother going through the whole procedure ...

And, finally, I've made 'checkin_server' configurable, so
that you don't 
have to modify the script itself to change that value ...
default being, 
of course, bsdstats.org ... Antony is working on docs for
how to use 
Apache as a 'quick-n-dirty proxy server', which he'll put up
onto the web 
site as soon as completed ...

As I said, you just need to download the new version and run
it, you don't 
have to wait for the port to go through, assuming you have
already 
installed from the port and /etc/periodic.conf is setup ...

Make sure you run it right after downloading though ...

If anyone out there can see a flaw in the script ... or
something that I 
may have overlooked as far as a 'loophole' that could be
used to screw 
around with the data, please let me know ... I know its not
possible, 
minus registration, to get rid of all holes, but, hopefully
I've now 
gotten rid of the ones that a truck could (and did) drive
though ...



----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . scrappyhub.org                              MSN .
scrappyhub.org
Yahoo . yscrappy               Skype: hub.org        ICQ .
7615664
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

Marc G. Fournier wrote:

> I've increased the size of the IDTOKEN to 32 from 16,
since I've been
> noticing alot of duplicates when two hosts submit at
close to the same
> time ...

Ummm... that's actually really bad.  That means that the RNG
used by OpenSSL
(hence SSH and others) is not actually producing anything
like a proper
random sequence for a lot of people.  Hence all sorts of
crypto handled by
those machines is potentially vulnerable to attack.  If this
is the case,
going from 16 to 32 bytes of random token won't actually
help at all.

On the other hand, the duplicates could be the result of
people deliberately
trying to frig the statistics or just innocently running the
300.statistics
script manually several times.  In either case, entries with
duplicate tokens
should be discarded -- I guess you'ld always want to keep
just the last entry
for any token.

	Cheers,

	Matthew

-- 
Dr Matthew J Seaman MA, D.Phil.                       7
Priory Courtyard
                                                      Flat 3
PGP: http://www.i
nfracaninophile.co.uk/pgpkey         Ramsgate
                                                      Kent,
CT11 9PW

On 9/29/06, Marc G. Fournier <freebsdhub.org> wrote:
>
> As painful as it was to do, I backed up the old data
tonight and wiped out
> the stats ... for one major reason: the stats lost
their accuracy.
>
> As I said, you just need to download the new version
and run it, you don't
> have to wait for the port to go through, assuming you
have already
> installed from the port and /etc/periodic.conf is setup
...
>
> Make sure you run it right after downloading though ...
>
> If anyone out there can see a flaw in the script ... or
something that I
> may have overlooked as far as a 'loophole' that could
be used to screw
> around with the data, please let me know ... I know its
not possible,
> minus registration, to get rid of all holes, but,
hopefully I've now
> gotten rid of the ones that a truck could (and did)
drive though ...
>

I just updated the script and it ran fine 
I'm the only guy yet from Portugal and the only sparc cpu


On another subject, with the addition of the other BSDs the
releases
stats for example are pretty much nonsense. Do you plan to
work on
that?

-- 
Joao Barros
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

On 29/09/2006 1:11 AM, Joao Barros wrote:
> On another subject, with the addition of the other BSDs
the releases
> stats for example are pretty much nonsense. Do you plan
to work on
> that?

Yep, each individual *BSD is getting its own detailed stats
summary 
section... they're not finished yet, so at the moment I've
left the 
links to the old (nonsensical) pages, but it's a long
weekend here this 
weekend so I'm hoping to try and finalise them 

See here for the FreeBSD page:

     http://www.bsdstats.
org/freebsd/

Thus far I have Releases and Countries done, so it's just a
matter of 
some further formatting and then the Platforms + Devices
pages...

Cheers
Antony

_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

On 9/29/06, Antony Mawer <fbsd-questionsmawer.org> wrote:
> On 29/09/2006 1:11 AM, Joao Barros wrote:
> > On another subject, with the addition of the other
BSDs the releases
> > stats for example are pretty much nonsense. Do you
plan to work on
> > that?
>
> Yep, each individual *BSD is getting its own detailed
stats summary
> section... they're not finished yet, so at the moment
I've left the
> links to the old (nonsensical) pages, but it's a long
weekend here this
> weekend so I'm hoping to try and finalise them 
>
> See here for the FreeBSD page:
>
>      http://www.bsdstats.
org/freebsd/
>
> Thus far I have Releases and Countries done, so it's
just a matter of
> some further formatting and then the Platforms +
Devices pages...
>
> Cheers
> Antony
>
>

It looks very nice indeed, good work! 

-- 
Joao Barros
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

Matthew Seaman wrote:

> On the other hand, the duplicates could be the result
of people deliberately
> trying to frig the statistics or just innocently
running the 300.statistics
> script manually several times.  In either case, entries
with duplicate tokens
> should be discarded -- I guess you'ld always want to
keep just the last entry
> for any token.

How is the country determined? by whois lookup? I am just
surprised that 
after the wipe and required update of the stats-script,
Panama has 75% 
of the hosts, 10 times the US.

Cheers, Erik
-- 
Ph: +34.666334818                      web: http://www.locolomo.org
X.509 Certificate: http
://www.locolomo.org/crt/8D03551FFCE04F0C.crt
Key ID:
69:79:B8:2C:E3:8F:E7:BE:5D:C3:C3:B1:74:62:B8:3F:9F:1F:69:B9
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

On 29/09/2006 2:01 AM, Erik Norgaard wrote:
> Matthew Seaman wrote:
> 
>> On the other hand, the duplicates could be the
result of people 
>> deliberately
>> trying to frig the statistics or just innocently
running the 
>> 300.statistics
>> script manually several times.  In either case,
entries with duplicate 
>> tokens
>> should be discarded -- I guess you'ld always want
to keep just the 
>> last entry
>> for any token.
> 
> How is the country determined? by whois lookup? I am
just surprised that 
> after the wipe and required update of the stats-script,
Panama has 75% 
> of the hosts, 10 times the US.

Via the GeoIP module. Marc's servers are mostly/all located
in Panama 
(hub.org), hence why they're in there quickly after the
stats wipe 

--Antony

_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

Antony is working on operating system sub-pages that will be
linked from the 
operating system summary page ... check out what he has so
far by going to:

         http://www.bsdstats.o
rg/freebsd


--On Friday, September 29, 2006 12:11:51 +0100 Joao Barros 
<joao.barrosgmail.com> wrote:

> On 9/29/06, Marc G. Fournier <freebsdhub.org> wrote:
>>
>> As painful as it was to do, I backed up the old
data tonight and wiped out
>> the stats ... for one major reason: the stats lost
their accuracy.
>>
>> As I said, you just need to download the new
version and run it, you don't
>> have to wait for the port to go through, assuming
you have already
>> installed from the port and /etc/periodic.conf is
setup ...
>>
>> Make sure you run it right after downloading though
...
>>
>> If anyone out there can see a flaw in the script
... or something that I
>> may have overlooked as far as a 'loophole' that
could be used to screw
>> around with the data, please let me know ... I know
its not possible,
>> minus registration, to get rid of all holes, but,
hopefully I've now
>> gotten rid of the ones that a truck could (and did)
drive though ...
>>
>
> I just updated the script and it ran fine 
> I'm the only guy yet from Portugal and the only sparc
cpu 
>
> On another subject, with the addition of the other BSDs
the releases
> stats for example are pretty much nonsense. Do you plan
to work on
> that?
>
> --
> Joao Barros



----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . scrappyhub.org                              MSN .
scrappyhub.org
Yahoo . yscrappy               Skype: hub.org        ICQ .
7615664

_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"