Re: flash with DBIC session storage

On Fri, Jul 27, 2007 at 11:57:01AM -0700, Mesdaq, Ali
wrote:
> Are you sure that InnoDB would solve this issue? Even
if just a row was
> locked and you have 2 inserts at the exact same time
how would that
> resolve the issue?

One transaction would succeed and the other would fail.  If
you want
different behavior, you'll have to change the isolation
level (or
actualy, in this case, rethink your app).

It's more of an issue of this:

User visits page A.
User visits page B in another tab.
User submits form A.
User submits form B.
A updates the flash.
B updates the flash.
Page A2 loads with whatever flash won.
Page B2 loads with whatever flash won.

The point is that both transaction A and transaction B can't
go
through.  Web apps aren't designed to be hit concurrently by
the same
user; it just doesn't make sense (or work) when you're
keeping state.
One state has to win; this is what the DB error is telling
you.

You could setup locks over [submit, update flash, display
flash]:

User submits form A.
Form A gets the lock.
User submits form B.
B submission blocks waiting for the lock.
A updates the flash.
Page A2 loads.
A unlocks.
B gets the lock.
B updates the flash.
...

This adds a lot of complexity, though.  Is this really a
problem that
your users experience, or is it an artifact of poor load
testing
practices?

Regards,
Jonathan Rockway

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

I think in the case of the person who initially emailed the
group the
problem is poor load testing. But it does bring up the point
of better
handling of exception statements by DBIx. The
DBIx::Class::ResultSet::find_or_create(): call should handle
this
exception better because it can happen in tons of different
places and
its not always an application or benchmarking issue. In high
load
situations you WILL see this happen. 

To illustrate here is an example:
Your DB has a first_name table to store user first names
You get a flood of new people registering and you normalize
their first
names to get id's
2 Users have the first name of Tom which does not exist yet
You call DBIx::Class::ResultSet::find_or_create():
Both calls detect the name does not exist
Both calls insert with one getting this error

Now there is no real way to handle this in your app unless
you do a lot
of concurrency checking or queuing of inserts. 

Actually I should write a bug for this so the DBIx guys can
check it
out.

Thanks,
------------------------------------------
Ali Mesdaq
Security Researcher II
Websense Security Labs
http://www.Websen
seSecurityLabs.com
------------------------------------------

-----Original Message-----
From: Jonathan T. Rockway [mailto:jonjrock.us] 
Sent: Friday, July 27, 2007 5:27 PM
To: The elegant MVC web framework
Subject: [BULK] - Re: [BULK] - Re: [Catalyst] flash with
DBIC session
storage

On Fri, Jul 27, 2007 at 11:57:01AM -0700, Mesdaq, Ali
wrote:
> Are you sure that InnoDB would solve this issue? Even
if just a row 
> was locked and you have 2 inserts at the exact same
time how would 
> that resolve the issue?

One transaction would succeed and the other would fail.  If
you want
different behavior, you'll have to change the isolation
level (or
actualy, in this case, rethink your app).

It's more of an issue of this:

User visits page A.
User visits page B in another tab.
User submits form A.
User submits form B.
A updates the flash.
B updates the flash.
Page A2 loads with whatever flash won.
Page B2 loads with whatever flash won.

The point is that both transaction A and transaction B can't
go through.
Web apps aren't designed to be hit concurrently by the same
user; it
just doesn't make sense (or work) when you're keeping
state.
One state has to win; this is what the DB error is telling
you.

You could setup locks over [submit, update flash, display
flash]:

User submits form A.
Form A gets the lock.
User submits form B.
B submission blocks waiting for the lock.
A updates the flash.
Page A2 loads.
A unlocks.
B gets the lock.
B updates the flash.
...

This adds a lot of complexity, though.  Is this really a
problem that
your users experience, or is it an artifact of poor load
testing
practices?

Regards,
Jonathan Rockway

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive:
http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

On Fri, Jul 27, 2007 at 06:11:03PM -0700, Mesdaq, Ali
wrote:
> I think in the case of the person who initially emailed
the group the
> problem is poor load testing. But it does bring up the
point of better
> handling of exception statements by DBIx. 

DBIx:: is a generic namespace for DBI extensions. 
DBIx::Class is
abbreviated as DBIC.  (This comes up a lot, I should add it
to a FAQ
or something.)

> The DBIx::Class::ResultSet::find_or_create(): call
should handle
> this exception better because it can happen in tons of
different
> places and its not always an application or
benchmarking issue.

What do you think it should do?  create failed.  Only your
app knows
what to do in that situation.

Maybe you are suggesting that errors be hidden so your app
silently
loses data?  I am not in favor of that.

> In high load situations you WILL see this happen.

I have a feeling that your app is broken then.  I often do
dumbass
load testing on my apps and don't have any problems with
transactions
failing unchecked.

I really think you should read about transactional isolation
and how
deadlocks/conflicts are handled by your RDBMS.

The Berkeley DB has really good documentation on both of
these things,
describing both theory and how to avoid common problems with
the
library itself.  SQLite's docs are also good.


> To illustrate here is an example: Your DB has a
first_name table to
> store user first names You get a flood of new people
registering and
> you normalize their first names to get id's 2 Users
have the first
> name of Tom which does not exist yet You call
> DBIx::Class::ResultSet::find_or_create(): Both calls
detect the name
> does not exist Both calls insert with one getting this
error
> 
> Now there is no real way to handle this in your app
unless you do a lot
> of concurrency checking or queuing of inserts. 

Rollback and try again.  Your DBMS' docs will tell you about
the best
strategy for dealing with this case.  I'm sure there's a
good-enough
generic solution... if someone knows it, I would like to
hear.

Anyway, you should probably continue this thread on the DBIC
mailing
list instead.  DBIC has *nothing* to do with Catalyst.

Regards,
Jonathan Rockway

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

On 7/27/07, Jonathan T. Rockway <jonjrock.us> wrote:
> On Fri, Jul 27, 2007 at 06:11:03PM -0700, Mesdaq, Ali
wrote:
> > I think in the case of the person who initially
emailed the group the
> > problem is poor load testing. But it does bring up
the point of better
> > handling of exception statements by DBIx.
>
> DBIx:: is a generic namespace for DBI extensions. 
DBIx::Class is
> abbreviated as DBIC.  (This comes up a lot, I should
add it to a FAQ
> or something.)
>
> > The DBIx::Class::ResultSet::find_or_create(): call
should handle
> > this exception better because it can happen in
tons of different
> > places and its not always an application or
benchmarking issue.
>
> What do you think it should do?  create failed.  Only
your app knows
> what to do in that situation.
>
> Maybe you are suggesting that errors be hidden so your
app silently
> loses data?  I am not in favor of that.
>
> > In high load situations you WILL see this happen.
>
> I have a feeling that your app is broken then.  I often
do dumbass
> load testing on my apps and don't have any problems
with transactions
> failing unchecked.
>
> I really think you should read about transactional
isolation and how
> deadlocks/conflicts are handled by your RDBMS.
>
> The Berkeley DB has really good documentation on both
of these things,
> describing both theory and how to avoid common problems
with the
> library itself.  SQLite's docs are also good.
>
>
> > To illustrate here is an example: Your DB has a
first_name table to
> > store user first names You get a flood of new
people registering and
> > you normalize their first names to get id's 2
Users have the first
> > name of Tom which does not exist yet You call
> > DBIx::Class::ResultSet::find_or_create(): Both
calls detect the name
> > does not exist Both calls insert with one getting
this error
> >
> > Now there is no real way to handle this in your
app unless you do a lot
> > of concurrency checking or queuing of inserts.
>
> Rollback and try again.  Your DBMS' docs will tell you
about the best
> strategy for dealing with this case.  I'm sure there's
a good-enough
> generic solution... if someone knows it, I would like
to hear.
>
> Anyway, you should probably continue this thread on the
DBIC mailing
> list instead.  DBIC has *nothing* to do with Catalyst.
>
> Regards,
> Jonathan Rockway
>

Agreed on all counts, doubly so on moving to the DBIC
mailing list.

If you believe that concurrency isn't handled properly in
DBIC, I'm
sure several people would love to hear it.  I'm sure
they'll
immediately shoot you down, though for the same reasons
Jonathan
posted.

Here's a good test script that you can even port to DBIC
that
demonstrates a concurrency bug in MySQL:
htt
p://bugs.mysql.com/file.php?id=6906&text=1

-Jay

-- 
J. Shirley :: jshirleygmail.com :: Killing two stones with one
bird...
http://www.toeat.com

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

Am 28.07.2007 um 02:26 schrieb Jonathan T. Rockway:

> This adds a lot of complexity, though.  Is this really
a problem that
> your users experience, or is it an artifact of poor
load testing
> practices?

I think you're right: The error is due to my poor-man's
approach of  
load-testing.
I believe that it's very unlikely to happen in a production 

environment with real
users hitting the site.

--Tobias


_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/

* Jonathan T. Rockway <jonjrock.us> [2007-07-28
02:35]:
> On Fri, Jul 27, 2007 at 11:57:01AM -0700, Mesdaq, Ali
wrote:
> > Are you sure that InnoDB would solve this issue?
Even if just
> > a row was locked and you have 2 inserts at the
exact same
> > time how would that resolve the issue?
> 
> One transaction would succeed and the other would fail.
If you
> want different behavior, you'll have to change the
isolation
> level (or actualy, in this case, rethink your app).
> 
> It's more of an issue of this:
> 
> User visits page A.
> User visits page B in another tab.
> User submits form A.
> User submits form B.
> A updates the flash.
> B updates the flash.
> Page A2 loads with whatever flash won.
> Page B2 loads with whatever flash won.

Keeping implicit state on the server blows. News at 11.

(The right way: don’t use flash. Put the stuff somewhere
where you can give it a URI and send the user to that
address.)

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/&g
t;

_______________________________________________
List: Catalystlists.rawmode.org
Listinfo: ht
tp://lists.rawmode.org/mailman/listinfo/catalyst
Searchable archive: http://www.mail-
archive.com/catalystlists.rawmode.org/
Dev site: http://dev.catalyst.per
l.org/