Design issues (opengl, hdr, channels, pigment(?))

I'm going to try an summarize a couple of design issues that
have been 
discussed here and on irc for the past months. I hope we can
get to a 
resolution and then, hopefully, even an implementation.

* OpenGL

Adrian Page already started thinking about using opengl for
more than just 
fast zooming. Michael recently posted his experiments with
an opengl based 
image manipulation program. I would really like to make good
use of opengl 
(if available). 

These are the places I would like to use opengl:

- display (fast zooming, rotated canvas, hdr exposure,
wetness, canvas, paint 
thickness visualisation. Optionally also fast but inaccurate
projection 
composition?).
- filters (convolution, for instance)
- cursor visualization (perhaps with opengl overlays, so we
can have 3-d 
stylus?)
- brushop implementations (which are a kind of filters)
- paint device composition

For display we need to make the projection and the current
masks (selection, 
height, wetness) available as textures & allow shaders
to work on them. 

For cursor visualisation we simply need to check whether the
opengl overlay 
extension is available and use it on the widget.

For composition, filters and brushops we need to make the
paint device data 
available as textures. That's the same, in a sense, as for
display, except 
that for display we need the projection data converted to
rgb (the masks can 
stay single-channel).

I can see two approaches:

- store the tiles themselves as textures if opengl is
available and make the 
tiles available through the datamanager (eek!)

- add a getTexture(), writeTexture() to KisPaintDevice (and
the tile backend). 
There is no reason why glsl shaders need rgb data. You can
put anything in a 
texture and work on it as if it were just another
multi-dimensioned array. 
The glsl operations should come from the colorspaces.
Cyrille: you need to 
remind me of how we were going to do glsl, mmx and software
operations in 
pigment.

I think we should go the getTexture() and writeTexture()
route. This can 
always be optimized later with caching the textures; making
the tile backend 
have textures as another level of caching (texture, memory,
compressed 
memory, swapfile) sounds too complicated for me.

* OpenEXR multi-layer files

I'm not sure whether this is just an import/export filter
thing or whether we 
need to review our projection composition or layer stack
design. But it's an 
interesting area our most-favoured user (Basse) is very
interested in.

* Channels

We discussed this briefly yesterday. Basically, the question
is: is there a 
need to view in-band pixel data as separate bands? Or a set
of separate 
bands? Or is the math toolbox solution sufficient? 

Possible uses of subset-of-channel views on paint devices
are:

- filtering on one or a subset of the channels (like we do
for color 
adjustment, but without separate settings for each channel,
so you can do 
things like sharpen the L channel of a Lab image).

- compositing one or a subset of channels of two paint
devices.

- drawing on one or a subset of channels (same as above, but
for the user it 
feels different)

- copying one or a subset of channels (ditto)

The implementation in trunk I started for this (but didn't
finish) was to add 
a const QBitArray & channelFlags parameter to
KisFilter::process() and 
KoColorSpace::bitBlt() that has a 1/0 flag for all channels
in the pixels of 
the current colorspace and force the implementation to check
that before 
working with the given channels. 

That's a bit fragile. If we can come up with either a
KisChannel class that 
returns iterators that point to the chosen channel instead
of the beginning 
of the pixel and have a pixelSize the size of the chosen
channel, then we 
don't need to do much work on the composite ops or the
filters: they should 
work out of the box. to/fromQImage and to/fromTexture would
return a 
grayscale QImage or a texture with only the right pixel.

But there's a performance hit: I cannot image how we could
work on a subset of 
the channels except by iterating over all the selected
channels individually 
and calling the filter or composite op for all the selected
channels 
individually. 

* Pigment

As I said, I've lost track of pigment a little, what's done,
what needs to be 
done and what the design is... I wouldn't mind a little
summary .


-- 
Boudewijn Rempt 
http://www.va
ldyas.org/fading/index.cgi

_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

On Friday 26 January 2007, Boudewijn Rempt wrote:

> - filters (convolution, for instance)

I think that this would make the best testbed for using
opengl; convolution is 
so slow that converting to and from a texture isn't going to
be that 
important.

-- 
Boudewijn Rempt 
http://www.va
ldyas.org/fading/index.cgi

_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

> - add a getTexture(), writeTexture() to KisPaintDevice
(and the tile
> backend). There is no reason why glsl shaders need rgb
data. You can put
> anything in a texture and work on it as if it were just
another
> multi-dimensioned array. The glsl operations should
come from the
> colorspaces. Cyrille: you need to remind me of how we
were going to do
> glsl, mmx and software operations in pigment.

Write an implementation of KoColorTransformation (or the
relevant 
Transformation like KoConvolutionTransformation for
convolution). And then we 
need to create a mechanism to determine which transformation
to select.

> I think we should go the getTexture() and
writeTexture() route. 
what is this route ?

> * OpenEXR multi-layer files
>
> I'm not sure whether this is just an import/export
filter thing or whether
> we need to review our projection composition or layer
stack design. But
> it's an interesting area our most-favoured user (Basse)
is very interested
> in.
>From the screenshot, one of the blender guys gave us, it
looks like it needs a 
layer stack design changes, but not sure if it won't require
something more 
complicated than a stack.

> * Channels
>
> We discussed this briefly yesterday. Basically, the
question is: is there a
> need to view in-band pixel data as separate bands? Or a
set of separate
> bands? Or is the math toolbox solution sufficient?

I am affraid that you are confusing two different problems:
 - apply filter on only one channel
 - signal based processing for filters (which is for what
KisMathToolbox is 
for), it only works for colorspace with continuous data (so
not for the index 
colorspace) and it doesn't work for colorspace with an
"angle" as channel, 
but one would need to provide a KisMathToolbox
implementation for those.

> The implementation in trunk I started for this (but
didn't finish) was to
> add a const QBitArray & channelFlags parameter to
KisFilter::process() and
> KoColorSpace::bitBlt() that has a 1/0 flag for all
channels in the pixels
> of the current colorspace and force the implementation
to check that before
> working with the given channels.
>
> That's a bit fragile. If we can come up with either a
KisChannel class that
> returns iterators that point to the chosen channel
instead of the beginning
> of the pixel and have a pixelSize the size of the
chosen channel, then we
> don't need to do much work on the composite ops or the
filters: they should
> work out of the box. to/fromQImage and to/fromTexture
would return a
> grayscale QImage or a texture with only the right
pixel.
>
> But there's a performance hit: I cannot image how we
could work on a subset
> of the channels except by iterating over all the
selected channels
> individually and calling the filter or composite op for
all the selected
> channels individually.

see below (at the end of my comment on pigment), I think the
QBitArray 
solution, after all, we didn't even choose the solution to
enforce 
KisSelection...

> * Pigment
>
> As I said, I've lost track of pigment a little, what's
done, what needs to
> be done and what the design is... I wouldn't mind a
little summary .

Not much has change you know ;) It's just that instead of
calling a virtual 
function of the colorspace, you call a virtual function of a
transformation 
object that you did created with a call to a function of the
colorspace. This 
allow us to dynamically creates operations and choose the
best one for the 
CPU or the task at hand (I am thinking that this might be a
possibility to 
keep fast implementation for the previous point, I need to
think a little bit 
more about it).

Now, we need to think if we need or if we want to have
extensibility of those 
operations.

-- 
Cyrille Berger
_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

(My answer is a bit late, I was too sick this weekend to do
much, and even now 
I'm not really up and running.)

On Saturday 27 January 2007, Cyrille Berger wrote:

> Write an implementation of KoColorTransformation (or
the relevant
> Transformation like KoConvolutionTransformation for
convolution). And then
> we need to create a mechanism to determine which
transformation to select.

That mechanism is another issue. For asm routines, we should
use kcpuinfo.h to 
switch, I think, but for opengl we may need either a
priority scheme or a 
configuration option. Or let the calling code determine
whether it wants to 
use opengl.

> > I think we should go the getTexture() and
writeTexture() route.
>
> what is this route ?

For getting the paint device data in a form that shader
programs can use. It's 
not much use passing only up to a maximum 4096 pixels worth
of data to a 
shader, that way the gpu hardly gets to flex its muscles. So
I was thinking 
of having a toTexture/fromTexture pair similar to
to/fromQImage and 
read/writeBytes that would copy image data into a texture
object and from a 
texture object. I

NVidia's GPUFilter framework for Photoshop filters is
interesting, too. I'm 
reading through that at the moment.

I'm trying to come up with an experimental viewplugin that
does this (initally 
through the read/writeBytes methods) to test whether the
cost is prohibitive. 
(Just for fun & kicks, and because I want to do
something new for a change, 
but maybe this plugin will be our equivalent of the nvidia
filter framework.)

> I am affraid that you are confusing two different
problems:
>  - apply filter on only one channel
>  - signal based processing for filters (which is for
what KisMathToolbox is
> for), it only works for colorspace with continuous data
(so not for the
> index colorspace) and it doesn't work for colorspace
with an "angle" as
> channel, but one would need to provide a KisMathToolbox
implementation for
> those.

Okay, I think I understand now. In any case, I think it's
useful to have a 
general method to extract one channel into a single-channel
colorspace paint 
device and to put it back. KisPainter::bitBlt should be able
to do that, but 
that interferes with colorspace conversions.

> see below (at the end of my comment on pigment), I
think the QBitArray
> solution, after all, we didn't even choose the solution
to enforce
> KisSelection...

I think there's something missing in this sentence . But if I
fill in the 
gaps correctly, you prefer the bit array solution? Do we
need to make it 
automatic to have a filter configuration per channel for all
filters?

> Not much has change you know ;) It's just that instead
of calling a virtual
> function of the colorspace, you call a virtual function
of a transformation
> object that you did created with a call to a function
of the colorspace.
> This allow us to dynamically creates operations and
choose the best one for
> the CPU or the task at hand (I am thinking that this
might be a possibility
> to keep fast implementation for the previous point, I
need to think a
> little bit more about it).

We could have the factory function make the choice using
kcpuinfo and a 
configuration parameter for opengl (because opengl is fast
but inaccurate we 
cannot mandate it); that way the object returned always is
the fastest 
version.

> Now, we need to think if we need or if we want to have
extensibility of
> those operations.

You mean make it possible to add functionality to
colorspaces using plugins? 
Then we'd need to have a kind of per-colorspace registry of
ops that the 
colorspace can query to get the asm, plain-software or
opengl variant of an 
op.

-- 
Boudewijn Rempt 
http://www.va
ldyas.org/fading/index.cgi

_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

> > > I think we should go the getTexture() and
writeTexture() route.
> >
> > what is this route ?
>
> (blahblah)
ah ok, it's for display only ?

> > see below (at the end of my comment on pigment), I
think the QBitArray
> > solution, after all, we didn't even choose the
solution to enforce
> > KisSelection...
>
> I think there's something missing in this sentence . But if I
fill in
> the gaps correctly, you prefer the bit array solution?
Do we need to make
> it automatic to have a filter configuration per channel
for all filters?
hum, it's my turn to be unsure about the question :/ Yes I
prefer the bit 
array solution. I don't think enforcing it is a good idea. I
don't see the 
use for a per channel configuration for a red eye removal
filter, for 
instance.

> > Not much has change you know ;) It's just that
instead of calling a
> > virtual function of the colorspace, you call a
virtual function of a
> > transformation object that you did created with a
call to a function of
> > the colorspace. This allow us to dynamically
creates operations and
> > choose the best one for the CPU or the task at
hand (I am thinking that
> > this might be a possibility to keep fast
implementation for the previous
> > point, I need to think a little bit more about
it).
>
> We could have the factory function make the choice
using kcpuinfo and a
> configuration parameter for opengl (because opengl is
fast but inaccurate
> we cannot mandate it); that way the object returned
always is the fastest
> version.

I also saw some library that where doing some timing of the

various "algorithms" at startup to determine which
algorithm was fastest for 
the specific hardware.

> > Now, we need to think if we need or if we want to
have extensibility of
> > those operations.
>
> You mean make it possible to add functionality to
colorspaces using
> plugins?
Yes something like that.

> Then we'd need to have a kind of per-colorspace
registry of ops 
> that the colorspace can query to get the asm,
plain-software or opengl
> variant of an op.
Hum yes.

Note: I don't favor seeing more asm in krita, I don't think
the speed 
improvement is worth the few drawbacks: decrease of code
readability, not 
portable and not that maintanable. I think C/C++ is fast
enought (and we can 
use MMX/SSE/SSE2 instructions in C/C++, I need to resume my
work in that 
area, but my last attempt was with an algorithm that can't
be accellerated 
with them :/ ).
-- 
Cyrille Berger
_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

On Tuesday 30 January 2007, Cyrille Berger wrote:
> > > > I think we should go the getTexture()
and writeTexture() route.
> > >
> > > what is this route ?
> >
> > (blahblah)
>
> ah ok, it's for display only ?

No! It may have sounded like blabla, but that's quite the
wrong conclusion 
from what I wrote. I'm experimenting with using shaders in a
Krita plugin 
right now but I'm already sure that our iterator based
approach to image data 
does not work for anything that uses opengl.

I mean: we use iterators to get as large as possible chunks
of bytes that we 
pass to the colorspaces who segment it into pixels and do
stuff with those 
pixels. That's fine for C++ and for mmx and similar. These
are fairly small 
1d arrays.

OpenGL needs to load the data into 2d textures (for
preference) that are still 
often constrained to power-of-2. And bigger and fewer
textures are better 
from a performance point of view. So, if we want to convolve
with opengl, 
filter with opengl or do anything else with opengl (and I
want to, it's 
fascinating me), then we need to be able get the paint
device data as 
efficiently as possible into textures.

My provisional conclusion from the experiment I'm working is
that the opengl 
operations cannot have the same api as the c++ or mmx
operations and that we 
need a separate architecture to expose paint device data to
opengl. I'm 
taking nvidia's photoshop opengl sdk as an exampole for what
I think will be 
needed.


> hum, it's my turn to be unsure about the question :/
Yes I prefer the bit
> array solution. I don't think enforcing it is a good
idea. I don't see the
> use for a per channel configuration for a red eye
removal filter, for
> instance.

Okay. I think I need to put down all the issues and
variables for this thing 
to see whether we can have a good design. I'll do that in
answer to Ben's 
mail.

> I also saw some library that where doing some timing of
the
> various "algorithms" at startup to determine
which algorithm was fastest
> for the specific hardware.

That would be cool!

> Note: I don't favor seeing more asm in krita, I don't
think the speed
> improvement is worth the few drawbacks: decrease of
code readability, not
> portable and not that maintanable. I think C/C++ is
fast enought (and we
> can use MMX/SSE/SSE2 instructions in C/C++, I need to
resume my work in
> that area, but my last attempt was with an algorithm
that can't be
> accellerated with them :/ ).

mmx and similar (used in C++, that would be better indeed)
are especially 
worthhwile for the composite ops. Those are very
parallizable, simple and 
used a lot. We might nick the Gimp's code for composition in
mmx etc. 
(Although now I think of it -- those are maybe plain asm
files again. I'll 
check.)

-- 
Boudewijn Rempt 
http://www.va
ldyas.org/fading/index.cgi

_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

On Thursday 01 February 2007, Boudewijn Rempt wrote:
> On Tuesday 30 January 2007, Cyrille Berger wrote:
> > > > > I think we should go the
getTexture() and writeTexture() route.
> > > >
> > > > what is this route ?
> > >
> > > (blahblah)
> >
> > ah ok, it's for display only ?
>
> No! It may have sounded like blabla,
Sorry, I didn't mean to offense you !

> but that's quite the wrong conclusion 
> from what I wrote. I'm experimenting with using shaders
in a Krita plugin
> right now but I'm already sure that our iterator based
approach to image
> data does not work for anything that uses opengl.
>
> (...)
>
> My provisional conclusion from the experiment I'm
working is that the
> opengl operations cannot have the same api as the c++
or mmx operations and
> that we need a separate architecture to expose paint
device data to opengl.
> I'm taking nvidia's photoshop opengl sdk as an exampole
for what I think
> will be needed.

Which means, that we would need to have two plugins
implementation, one for 
opengl and one for C++/mmx ?

> > I also saw some library that where doing some
timing of the
> > various "algorithms" at startup to
determine which algorithm was fastest
> > for the specific hardware.
>
> That would be cool!
>

But quiet impossible if opengl and C++/mmx doesn't share the
smae kind of  API 
than Ko{Color|Convolution|Mix|Whatever}Transformation API.

> > Note: I don't favor seeing more asm in krita, I
don't think the speed
> > improvement is worth the few drawbacks: decrease
of code readability, not
> > portable and not that maintanable. I think C/C++
is fast enought (and we
> > can use MMX/SSE/SSE2 instructions in C/C++, I need
to resume my work in
> > that area, but my last attempt was with an
algorithm that can't be
> > accellerated with them :/ ).
>
> mmx and similar (used in C++, that would be better
indeed) are especially
> worthhwile for the composite ops. Those are very
parallizable, simple and
> used a lot. We might nick the Gimp's code for
composition in mmx etc.
> (Although now I think of it -- those are maybe plain
asm files again. I'll
> check.)
Yes it's plain asm files, so we don't want them  I am still
working on the 
library that would allow us to have the same implementation
for mmx/sse and 
normal instruction.

-- 
Cyrille Berger
_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop

On Thursday 01 February 2007, Cyrille Berger wrote:

> Which means, that we would need to have two plugins
implementation, one for
> opengl and one for C++/mmx ?

Unfortunately, yes. I still think there are quite a few
places where using 
opengl makes sense, but given the inaccuracy of opengl
calculations, those 
will either be just for display, or at the user's option.

I'm envisioning something like this:

* For convolution; the convolution painter asks the
colorspace whether it has 
got a shader for convolution & whether the user allows
opengl to do the 
convolving (option from the preferences dialog). If so,
convolution is done 
through opengl, if either of these prerequisites are not
fulfilled, 
convolution is done on the cpu. 

This means we need an extra method on KoColorSpace,
something like:
  QByteArray getShader(const QString & shaderType);
or 
  GLuint getShader(enumShader shader); (for precompiled
shaders)

Convolution then proceeds to get the textures from the paint
device in a 
to-be-specified manner and convolves them using the given
shader. For getting 
a texture from a paintdevice, we can either use readBytes
(which can still be 
optimized a lot!) or write code inside image/tiles that
creates a texture for 
a given rect and returns it. The big thing, apparently, is
to create textures 
sparingly and write to them often, because changing an
existing texture is 
much cheaper. See our opengl context class.

* For optional filters. There's stuff you can do in opengl
that's simply not 
feasible on the cpu, like fluid simulation. There are bound
to be others. 
These use the same to-be-specified api to get textures from
the paint device, 
and write them back.

* For display. If the user is satisfied with a lightning
fast but 
inaccurate-as-hell display, we can simply put every layer
into the texture 
map, perhaps as L*a*b floats, or even converted to rgba8,
and use shaders or 
OpenGL's own blending functions to blend them. Whether this
would work 
depends a lot on the graphics card, available memory, image
size and so on, 
but it could be fun to code.

* For composition. We could have an alternative path in
KisPainter::bitBlt 
that, when enabled & a shader is available for the given
colorspace, blends 
the specified area using opengl. Whether that will give us a
speed advantage 
remains to be seen, but i would like to give it a try. 

* For visualisation. Canvas height & shadow, wetness and
more: these will be 
much easier, faster and prettier to do with opengl.

There may be a couple of other possibilities that would be
interesting, and I 
may be overlooking something: I'm only now really diving
into the field of 
using the gpu for more than just first-person shoot-em-ups
. But
I've 
already got as far as having my opengl environment setup
without showing a 
new window! Tonight or tomorrow I might be converting a
chunk of image to a 
texture and who knows -- process it.

-- 
Boudewijn Rempt 
http://www.va
ldyas.org/fading/index.cgi

_______________________________________________
kimageshop mailing list
kimageshopkde.org
http
s://mail.kde.org/mailman/listinfo/kimageshop