debugging and serving web pages at the same time in lisp (specifically clisp)

On 10/5/06, Yin-So Chen <yinso.chengmail.com> wrote:
> Hi lisp gurus-
>
> sorry if this (newbie) question has been asked before -
I tried to search
> for the information on the web but can't (probably
because it's hard to tell
> what the keyword is).  I am wondering if it is possible
to debug and serve
> webpage at the same time in lisp (specifically clisp)
and verify my
> underestanding.
>
> The reason I am wondering is that I've read Paul
Graham's note on his
> experience in viaweb he was able to diagnose a problem
while the customer is
> on the phone.  His statement implies that he was
running a separate instance
> of the website (so not the same one as the customer's
looking at), but he
> attempt to reproduce the problem and can fix it at the
same time (all
> appearing without having to shutdown the production
image).
>
> You can find the excerpt here:
> http://lib.store.yahoo.net/lib/paulgraham/bbnexcerpts.t
xt

IIRC, Paul Graham was running CLISP as CGI script.  That is,
separate
process for each request.  This was OK, because only the
store
creation/management application was written in Lisp, so you
don't have
that much hits.  The stores themselves were C(?) programs.

>
> After playing a bit with araneida's example - it seem
that with clisp this
> is not possible - because (host-serve-events) blocks
repl.  I didn't try
> this on SBCL, but assuming it has multi-threads I can
see it being a
> possibility.
>
> Another way I can think of is that he built an
abstraction layer that allow
> him to work from repl to reproduce a problem discovered
through a web server
> request, so he can work in session to fix the problem,
but I can't see how
> he can update the running clisp image without having to
kill the current web
> server session (assuming he has something that works
similar to araneida)
> and reload code.

Again, from what I read, the Lisp application was separated
in several
layers, and they were making changes mainly to the UI-level
layer.
For the others they had to be much more careful in what they
touch and
how.

>
> Please let me know if I am missing something obvious. 
While I don't know if
> I agree with the practice of directly changing
production code in real-time,
> the concept is intriguing, and certainly would be great
to know how it's
> done.

I'm sure some googling in comp.lang.lisp will reveal a lot
of examples.

>
> Oh - I am using clisp now as I am doing work on
windows.  I might move to
> sbcl on linux but certainly would love to understand
how it's done.

There is a still experimental version of SBCL for Windows. 
You can
try it and report any problems to the developers.

Cheers,
Ivan


P.S.  I'd like to clarify, that I've only read about all
this Viaweb
stuff.  I hadn't been there.  But I'm not sure I didn't
imply it using
improper tenses in my not-so-good English.

>
> Thanks for any comments,
> yinso
>
>
> _______________________________________________
> lispweb mailing list
> lispwebred-bean.com
> http
://www.red-bean.com/mailman/listinfo/lispweb
>
>
>


-- 
"...Please don't assume Lisp is only useful for
Animation and Graphics,
AI, Bioinformatics, B2B and E-Commerce, Data Mining,
EDA/Semiconductor
applications, Expert Systems, Finance, Intelligent Agents,
Knowledge
Management, Mechanical CAD, Modeling and Simulation, Natural
Language,
Optimization, Research, Risk Analysis, Scheduling, Telecom,
and Web
Authoring just because these are the only things they
happened to
list."
  -- Kent Pitman

_______________________________________________
lispweb mailing list
lispwebred-bean.com
http
://www.red-bean.com/mailman/listinfo/lispweb

> After playing a bit with araneida's example - it seem
that with  
> clisp this is not possible - because
(host-serve-events) blocks  
> repl.  I didn't try this on SBCL, but assuming it has
multi-threads  
> I can see it being a possibility.

You should still be able to (trace) your functions and
Araneida's  
(e.g., handle-request-response). You could also wrap h-r-r,
or modify  
your methods, to preserve or print the request object, which
will  
allow you to replay a request.

> Another way I can think of is that he built an
abstraction layer  
> that allow him to work from repl to reproduce a problem
discovered  
> through a web server request, so he can work in session
to fix the  
> problem, but I can't see how he can update the running
clisp image  
> without having to kill the current web server session
(assuming he  
> has something that works similar to araneida) and
reload code.
>
> Please let me know if I am missing something obvious. 
While I  
> don't know if I agree with the practice of directly
changing  
> production code in real-time, the concept is
intriguing, and  
> certainly would be great to know how it's done.

I've done it — just play around with your methods and
functions while  
serving pages. If you have appropriate conditionals (e.g.,
you only  
generate the in-development page for your username) you can
even do  
this while serving pages to users. Risky, sure, but it
works.

-R



_______________________________________________
lispweb mailing list
lispwebred-bean.com
http
://www.red-bean.com/mailman/listinfo/lispweb

Yinso,
      I run PortableAserve on top of Lispworks and I fix hot
live 
production
bugs all the time.  It's the fastest way to fix bugs;  I
inspect the 
actual
production data in the Listener to diagnose and then
redefine actual 
production
functions and data in the Listener to fix the bugs.  My web
pages are
all served from a set of structures rooted in global
variables which
define multiple concurrent session states,
so individual sessions state is fully accessible from any
Lisp thread,
such as the Listener.   The individual web pages are
computed
from these structures for each http-get,  so on the client
web browser,
just flushing cache and reloading the page to displays the
fixed 
behavior.

     This avoid having to replicate bugs before fixing them,
and avoids time consuming turnover actions.  I fix actual
problems
directly,  instead of approximations of those problems. 
Most bugs
are just typos,  which are great to fix this way.  Of course
if the bug
required a serious rewrite,  I'd do the rewrite on a test
server.
However,  the data I can pull from a production Listener is
really
valuable in setting up the test cases on the test server. 
Since Lisp
structure print out as loadable versions of their contents, 
I can
just cut and paste the full session state to a test server
to replicate
specific problems there.  Of course it's important to drive 
PortableAserve's
http handlers with functions that take a single structure
argument,
(whose structure fully describe session state)  otherwise
that cut and 
paste
would have multiple pieces which would have to be loaded
together
by hand.

    Paul Graham is right.  I have found Lisp to be about 3x
more elegant 
than
any other web serving technology.  By this I mean that one
person
can easily do the work of three.

Lawrence Au
Q-Phrase LLC



On Oct 5, 2006, at 1:41 AM, Yin-So Chen wrote:

> Hi lisp gurus-
>
> sorry if this (newbie) question has been asked before -
I tried to 
> search for the information on the web but can't
(probably because it's 
> hard to tell what the keyword is).  I am wondering if
it is possible 
> to debug and serve webpage at the same time in lisp
(specifically 
> clisp) and verify my underestanding.
>
> The reason I am wondering is that I've read Paul
Graham's note on his 
> experience in viaweb he was able to diagnose a problem
while the 
> customer is on the phone.  His statement implies that
he was running a 
> separate instance of the website (so not the same one
as the 
> customer's looking at), but he attempt to reproduce the
problem and 
> can fix it at the same time (all appearing without
having to shutdown 
> the production image). 
>
> You can find the excerpt here: 
> http://lib.store.yahoo.net/lib/paulgraham/bbnexcerpts.t
xt
>
> After playing a bit with araneida's example - it seem
that with clisp 
> this is not possible - because (host-serve-events)
blocks repl.  I 
> didn't try this on SBCL, but assuming it has
multi-threads I can see 
> it being a possibility.
>
> Another way I can think of is that he built an
abstraction layer that 
> allow him to work from repl to reproduce a problem
discovered through 
> a web server request, so he can work in session to fix
the problem, 
> but I can't see how he can update the running clisp
image without 
> having to kill the current web server session (assuming
he has 
> something that works similar to araneida) and reload
code.
>
> Please let me know if I am missing something obvious. 
While I don't 
> know if I agree with the practice of directly changing
production code 
> in real-time, the concept is intriguing, and certainly
would be great 
> to know how it's done.
>
> Oh - I am using clisp now as I am doing work on
windows.  I might move 
> to sbcl on linux but certainly would love to understand
how it's done.
>
> Thanks for any comments,
> yinso
>
> _______________________________________________
> lispweb mailing list
> lispwebred-bean.com
> http
://www.red-bean.com/mailman/listinfo/lispweb


_______________________________________________
lispweb mailing list
lispwebred-bean.com
http
://www.red-bean.com/mailman/listinfo/lispweb

Yinso,
    The single function Save-application worked great in MCL
(macintosh 
common lisp).
It produced a double-clickable executable with the saved
state of the 
entire Lisp environment.
Unfortunately I've yet to fully master the more complex
equivalent in 
Lispworks.
My systems are always in source code form,  because I only
run code in 
the Listener
after copying and pasting it from my source code files.  So
if I 
rebuild,
I just recompile from my latest source files.  Typically my
source code 
files
consist of defvar global variables,  defstruct structure
definitions 
and defun function definitions,
each followed by specific,  commented-out sets of test cases
for unit 
testing.

    Yes,  Lisp supports a completely different and more
unified 
methodology than
other environments.  There is essentially a single runtime
phase and 
all states
and abstractions are directly accessible from that phase. 
Syntactic 
and semantic bugs
are caught while compiling functions and structure
definitions but 
these compilations
are performed directly within a runtime context.

    As far as rolling back and verifying changes,  Lisp has
unique 
advantages
which make source control seem like a giant afterthought. 
Since the 
nested parentheses
and optional type declarations of Lisp supports a pure
tree-structured 
syntax,
re-factoring of function calling trees is far simpler than
other 
languages.
Consequently I can make deep changes without the immense
re-factoring cost which causes systems to be run
half-modified,  
half-modifications
which require source control to keep track of.   This all
holds true as 
long as I
keep passing most of my data through the function arguments.
In Lisp it's unusually easy to pass data through function
arguments.

    Consequently making deep changes to a system is much
easier,
especially with complex systems.  That said,  I still prefer
to make 
deep changes
to a test system,  simply because I make typos when I code, 
and I don't
always think of all the test cases needed to root them out. 
But I've 
never had
to resort to a source control system to rollback changes in
Lisp,  
because as long
as I pass most of my data through function arguments and
results,  the 
test cases needed to
verify code changes are very simple and I've never had to
rollback to 
more
than a previous version of a function.

    Don't rush to grok all this at once.  I used to program
in Fortran, 
C, C++,  Java and SQL
before learning Lisp properly from Paul Graham's book.  I
really had to 
read his book
carefully to free myself of encumbering Fortran, C, C++, 
Java and SQL 
notions.
For instance,  I used to pass data through global variables
when they 
should have been
passed through function arguments.  This made my code harder
to test,  
harder to re-factor
and much more error-prone,  like a typical Fortran, C, C++, 
Java or 
SQL program.

    The parentheses of Lisp make re-factoring of function
arguments in Lisp so much easier than Fortran, C, C++,  Java
and SQL 
that almost all
function inputs can come via calling arguments,  and almost
all 
function outputs can return in the
function results,  without causing the kind of syntactic
overhead that 
makes
global variables attractive.

Lawrence Au
Q-Phrase LLC


On Oct 5, 2006, at 6:11 PM, Yin-So Chen wrote:

> Thanks Ivan, Richard, and Lawrence for responding -
this is all very 
> intriguing.  Sounds like CGI is the only sure way to go
with Clisp if 
> I am looking to run multiple concurrent sessions?
>
> Also - Lawrence, I am assuming you have a way to
capture your changes 
> in life system back into source code form easily?  I've
heard the 
> save-image capability, but would think you still need
to have source 
> code so you can rebuild image again if you can, is this
correct?
>
> Richard - you meant you can (trace) h-r-r in SBCL,
right?  In CLISP, 
> once (host-serve-events) is called, it blocks REPL, so
I can't do 
> trace just-in-time without killing off the lisp session
(I can't tell 
> whether there is a way to accept a control-character
within araneida).
>
> BTW Ivan - I did find a blog article by BillClementson
discussing the 
> same topic at http://bc.tech.
coop/blog/040224.html.
>
> Based on all the descriptions so far, I am wondering if
the 
> development "methodology" is completely
different with lisp.  I so far 
> lived in the compile-link-install-test world, where
most of syntactic 
> bugs and some semantic bugs are caught during the
compile phase. And 
> that leaves the bugs that are harder to find in live
systems.  Of 
> course, examining trace files is horrible compared to
live traces, but 
> I've found that a good majority of the bugs found in
production 
> requires non-trivial changes (even with decent
abstraction layers) and 
> hence my experience is that without going through
another test cycle, 
> changing is extremely risky.
>
> Unless... there is a way to quickly verify and rollback
changes easily 
> in lisp?
>
> It would be great if someone with experiences in both
worlds (I am 
> sure many of you do ;p) can share some light on whether
lisp's 
> development cycle is completely different than
c/c++/java/c# world.  
> I've started to write lisp code, but my limited
experience does not 
> allow me to grok this yet.
>
> Thanks to all for help,
> yinso
>
> On 10/5/06, Lawrence Au <laauerols.com> wrote:
>>       I run PortableAserve on top of Lispworks and
I fix hot live
>> production
>> bugs all the time.  It's the fastest way to fix
bugs;  I inspect the
>> actual
>> production data in the Listener to diagnose and
then redefine actual
>> production
>> functions and data in the Listener to fix the
bugs.  My web pages are
>> all served from a set of structures rooted in
global variables which
>> define multiple concurrent session states,
>> so individual sessions state is fully accessible
from any Lisp thread,
>> such as the Listener.   The individual web pages
are computed
>> from these structures for each http-get,  so on the
client web 
>> browser,
>> just flushing cache and reloading the page to
displays the fixed
>> behavior.
>>
>>      This avoid having to replicate bugs before
fixing them,
>> and avoids time consuming turnover actions.  I fix
actual problems
>> directly,  instead of approximations of those
problems.  Most bugs
>> are just typos,  which are great to fix this
way.  Of course if the 
>> bug
>> required a serious rewrite,  I'd do the rewrite on
a test server.
>> However,  the data I can pull from a production
Listener is really
>> valuable in setting up the test cases on the test
server.  Since Lisp
>> structure print out as loadable versions of their
contents,  I can
>> just cut and paste the full session state to a test
server to 
>> replicate
>> specific problems there.  Of course it's important
to drive
>> PortableAserve's
>> http handlers with functions that take a single
structure argument,
>> (whose structure fully describe session
state)  otherwise that cut and
>> paste
>> would have multiple pieces which would have to be
loaded together
>> by hand.
>>
>>     Paul Graham is right.  I have found Lisp to be
about 3x more 
>> elegant
>> than
>> any other web serving technology.  By this I mean
that one person
>> can easily do the work of three.
>>
>> Lawrence Au
>> Q-Phrase LLC
>>
>>
>>
>> On Oct 5, 2006, at 1:41 AM, Yin-So Chen wrote:
>>
>> > Hi lisp gurus-
>> >
>> > sorry if this (newbie) question has been asked
before - I tried to
>> > search for the information on the web but
can't (probably because 
>> it's
>> > hard to tell what the keyword is). I am
wondering if it is possible
>> > to debug and serve webpage at the same time in
lisp (specifically
>> > clisp) and verify my underestanding.
>> >
>> > The reason I am wondering is that I've read
Paul Graham's note on 
>> his
>> > experience in viaweb he was able to diagnose a
problem while the
>> > customer is on the phone. His statement
implies that he was running 
>> a
>> > separate instance of the website (so not the
same one as the
>> > customer's looking at), but he attempt to
reproduce the problem and
>> > can fix it at the same time (all appearing
without having to 
>> shutdown
>> > the production image).
>> >
>> > You can find the excerpt here:
>> > http://lib.store.yahoo.net/lib/paulgraham/bbnexcerpts.t
xt
>> >
>> > After playing a bit with araneida's example -
it seem that with 
>> clisp
>> > this is not possible - because
(host-serve-events) blocks repl. I
>> > didn't try this on SBCL, but assuming it has
multi-threads I can see
>> > it being a possibility.
>> >
>> > Another way I can think of is that he built an
abstraction layer 
>> that
>> > allow him to work from repl to reproduce a
problem discovered 
>> through
>> > a web server request, so he can work in
session to fix the problem,
>> > but I can't see how he can update the running
clisp image without
>> > having to kill the current web server session
(assuming he has
>> > something that works similar to araneida) and
reload code.
>> >
>> > Please let me know if I am missing something
obvious. While I don't
>> > know if I agree with the practice of directly
changing production 
>> code
>> > in real-time, the concept is intriguing, and
certainly would be 
>> great
>> > to know how it's done.
>> >
>> > Oh - I am using clisp now as I am doing work
on windows. I might 
>> move
>> > to sbcl on linux but certainly would love to
understand how it's 
>> done.
>> >
>> > Thanks for any comments,
>> > yinso
>> >
>> >
_______________________________________________
>> > lispweb mailing list
>> > lispwebred-bean.com
>>  > http
://www.red-bean.com/mailman/listinfo/lispweb
>>
_______________________________________________
lispweb mailing list
lispwebred-bean.com
http
://www.red-bean.com/mailman/listinfo/lispweb