RFC: Structure of background parser and language support

Presently, each language support has its own background
parser, which
has its own thread weaver, rather than there being only one
background
parser that serves all languages.  What we end up with is x
* y number
of background parsing threads (where x is the number of
languages that
are loaded and y is the specified number of threads per
language) rather
than just y number of parsing threads.  This seems
non-intuitive to me.
I'd like to make one background parser instance on Core that
will use
the appropriate language(s) for creating parse jobs instead
of having
each language determing if it should parse a file and adding
it to its
own background parser.

Ultimately, when a project is loaded, we would iterate
through the files
in the project and do the following:

1. Check if there are any languages loaded that support the
files MIME
type.  If not, load the appropriate language support
plugins.
2. Check if a duchain exists for the file in our database
(if we decide
to go this route; hasn't been designed yet).  If not parse
it, otherwise
check the timestamp and last modified date on disk to
determine if the
file needs to be re-parsed.
3. If the file needs to be parsed, add it to the background
parser,
which will then create a parse job for each language that
supports the
file's MIME type.

Thoughts on this?  Anything I might not have considered?

Kris Wong

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On Monday 09 July 2007, Kris Wong wrote:
> Presently, each language support has its own background
parser, which
> has its own thread weaver, rather than there being only
one background
> parser that serves all languages.  What we end up with
is x * y number

Wow.  Things must have really changed.  I designed the
background parser from 
the very beginning to serve all language parts and it
operated just that way.  
Who changed it?

Adam

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On Jul 11, 2007, at 12:20 AM, Adam Treat wrote:

> On Monday 09 July 2007, Kris Wong wrote:
>> Presently, each language support has its own
background parser, which
>> has its own thread weaver, rather than there being
only one  
>> background
>> parser that serves all languages.  What we end up
with is x * y  
>> number
>
> Wow.  Things must have really changed.  I designed the
background  
> parser from
> the very beginning to serve all language parts and it
operated just  
> that way.
> Who changed it?
>
> Adam
>

I'm not sure that anybody changed it. I don't think anybody
knew that  
it was meant to serve all the language parts at once.
--
Matt



_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On Jul 9, 2007, at 11:08 AM, Kris Wong wrote:

> Presently, each language support has its own background
parser, which
> has its own thread weaver, rather than there being only
one background
> parser that serves all languages.  What we end up with
is x * y number
> of background parsing threads (where x is the number of
languages that
> are loaded and y is the specified number of threads per
language)  
> rather
> than just y number of parsing threads.  This seems
non-intuitive to  
> me.
> I'd like to make one background parser instance on Core
that will use
> the appropriate language(s) for creating parse jobs
instead of having
> each language determing if it should parse a file and
adding it to its
> own background parser.
>
> Ultimately, when a project is loaded, we would iterate
through the  
> files
> in the project and do the following:
>
> 1. Check if there are any languages loaded that support
the files MIME
> type.  If not, load the appropriate language support
plugins.
> 2. Check if a duchain exists for the file in our
database (if we  
> decide
> to go this route; hasn't been designed yet).  If not
parse it,  
> otherwise
> check the timestamp and last modified date on disk to
determine if the
> file needs to be re-parsed.
> 3. If the file needs to be parsed, add it to the
background parser,
> which will then create a parse job for each language
that supports the
> file's MIME type.
>
> Thoughts on this?  Anything I might not have
considered?
>
> Kris Wong
>

Please make this change.
--
Matt



_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On Monday 09 July 2007, Kris Wong wrote:
> Presently, each language support has its own background
parser, which
> has its own thread weaver, rather than there being only
one background
> parser that serves all languages.  What we end up with
is x * y number
> of background parsing threads (where x is the number of
languages that
> are loaded and y is the specified number of threads per
language) rather
> than just y number of parsing threads.  This seems
non-intuitive to me.
> I'd like to make one background parser instance on Core
that will use
> the appropriate language(s) for creating parse jobs
instead of having
> each language determing if it should parse a file and
adding it to its
> own background parser.
>
> Ultimately, when a project is loaded, we would iterate
through the files
> in the project and do the following:
>
> 1. Check if there are any languages loaded that support
the files MIME
> type.  If not, load the appropriate language support
plugins.
> 2. Check if a duchain exists for the file in our
database (if we decide
> to go this route; hasn't been designed yet).  If not
parse it, otherwise
> check the timestamp and last modified date on disk to
determine if the
> file needs to be re-parsed.
> 3. If the file needs to be parsed, add it to the
background parser,
> which will then create a parse job for each language
that supports the
> file's MIME type.
>
> Thoughts on this?  Anything I might not have
considered?

The only thing that comes to my mind is that you should not
assume that a 
language plugin has exactly one parser to offer. In the case
of Quanta we 
might implement different parsers for different mimetypes we
want to support. 

Jens

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

> The only thing that comes to my mind is that you should
not assume
that
> a language plugin has exactly one parser to offer. In
the case of
Quanta
> we might implement different parsers for different
mimetypes we want
to
> support.

You want a one to many relationship between a language and
its
parser(s)?  I can't see a benefit of trying to parse the
same language
in different ways.  Also, the platform does not currently
support this.
You can, however, have multiple languages loaded for the
same file
(i.e., HTML/PHP/CSS/JS).  Would this accomplish what it is
you need to
support?

Kris Wong

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

>
> You want a one to many relationship between a language
and its
> parser(s)?  I can't see a benefit of trying to parse
the same language
> in different ways.  Also, the platform does not
currently support this.
> You can, however, have multiple languages loaded for
the same file
> (i.e., HTML/PHP/CSS/JS).  Would this accomplish what it
is you need to
> support?

My understanding is that one languagesupport can support
more than one 
mime-type. So what I want is that you are asking the
languagesupport for a 
parser for the file/mimetype so that the languagesupport can
decide 
internally which parser to choose. You would get back the
parsejob and can do 
what you want with it.

Jens

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

> My understanding is that one languagesupport can
support more than one

> mime-type. So what I want is that you are asking the
languagesupport
for a 
> parser for the file/mimetype so that the
languagesupport can decide 
> internally which parser to choose. You would get back
the parsejob and
can do 
> what you want with it.

I don't think this is really what you want.  A language is
much more
than just a parser.  It is also the duchain and code
completion
specializations, supports code navigation, etc...  A
language support
supports more than one MIME type in the sense that a
javascript language
support could be loaded for:

text/html files
text/javascript files
text/php files

etc...  In the same sense a text/php file could load a php
language
support, html language support, javascript langauge support,
etc...

Kris Wong

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

> I don't think this is really what you want.  A language
is much more
> than just a parser.  It is also the duchain and code
completion
> specializations, supports code navigation, etc...  A
language support
> supports more than one MIME type in the sense that a
javascript language
> support could be loaded for:
>
> text/html files
> text/javascript files
> text/php files
>
> etc...  In the same sense a text/php file could load a
php language
> support, html language support, javascript langauge
support, etc...

Do you mean that for a text/html file we would load
javascript, php, python, 
html, xml (could be xhtml) ... language support parts? I
mean we actually 
don't know what is inside of the file and all different
language support 
parts would have to parse it, right?
I don't think this would work very well. Actually Quanta
might want to parse 
the same file with different parsers. First an HTML parser
which detects 
embedded scripts and than with the suitable script parser. 

I think for web development the one parser fits all approach
does not work.

Jens

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

> I don't think this would work very well. Actually
Quanta might want to
> parse the same file with different parsers. First an
HTML parser which
> detects embedded scripts and than with the suitable
script parser. 

> I think for web development the one parser fits all
approach does not
> work.

I will put some more thought into this once I start moving
things around
and see if I can come up with a design that will work well
for both
kdevelop & quanta.

Kris Wong


_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel