local convergence problems?

 	Hey all.  I've been playing with the new dev snapshot
(with 16-bit 
support) and it's fixed one part of my problem, but not the
other.  I've 
been looking at the generated profiles with iccexamin, and I
think I've 
finally figured out what's causing my issues.  When I
generate a profile 
with the local convergence enabled, there seems to be a
relatively 
abrubtly changing areas.  It's difficult to describe, and I
haven't 
completely convinced myself that it's not just a series of
bad IT8.7 test 
target shots.  It's so weird that I haven't ruled out bugs
either.

 	Don't know if I can be anymore help, but I could at least
provide 
the test target shot and see if anyone else could explain
it.

Thanks,
-Cory

-- 

************************************************************
*************
* Cory Papenfuss                                            
           *
* Electrical Engineering candidate Ph.D. graduate student   
           *
* Virginia Polytechnic Institute and State University       
           *
************************************************************
*************



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

On Wednesday 25 January 2006 12:47 pm, Cory Papenfuss wrote:
>  	Hey all.  I've been playing with the new dev snapshot
(with 16-bit
> support) and it's fixed one part of my problem, but not
the other.  I've
> been looking at the generated profiles with iccexamin,
and I think I've
> finally figured out what's causing my issues.  When I
generate a profile
> with the local convergence enabled, there seems to be a
relatively
> abrubtly changing areas.  It's difficult to describe,
and I haven't
> completely convinced myself that it's not just a series
of bad IT8.7 test
> target shots.  It's so weird that I haven't ruled out
bugs either.
>
>  	Don't know if I can be anymore help, but I could at
least provide
> the test target shot and see if anyone else could
explain it.
>
> Thanks,
> -Cory

There is a problem with local convergence extrapolation that
results in local 
reversals (abrupt localized changes in the curves) .  There
is a section in 
the help files that describes the problem.  There is also a
bug report that 
has been opened on this issue.  Please see
 
http://s
ourceforge.net/tracker/index.php?func=detail&aid=1296658
&group_id=146038&atid=764370  

The problem was worse before 16 bit target image support was
added but that 
did not completely fix the problem.    My testing indicates
that it only 
happens in the darkest parts of images and is worse in areas
that have highly 
saturated very dark colors (dark reds and yellows seem to be
the worst).  It 
does not show up at all in images with lower contrast levels
as far as I can 
tell since these images will not have areas dark enough to
show the problem.

I have not had a chance to dig into the code to figure out
what is happening 
but I think that this is related to the fact that digital
cameras, 
particularly those with larger sensor sites like DSLRs, 
have a gamut that is 
considerably larger than what is represented on an IT8.7
chart.  In other 
words there are significant areas of the cameras gamut where
there is little 
or no data from the IT8.7 image and local convergence
extrapolation in affect 
tries too hard to make these areas "right" and the
curves end up with 
discontinuities.   

So far most of my work on LPROF has been to consolidate the
user interface and 
to remove redundancies from the code.  Little has been done
to the 
underlaying profile generation algorithms at his point.  The
current 
recommendation is to not use local convergence extrapolation
with wide gamut 
devices as this will result in smoother curves that do not
exhibit the 
problem.  I suspect that the problem is for the most part
limited to those 
who are working with raw images since raw processing is
cabable of pulling 
way more gamut/dynamic range out of images then would be
typical for images 
that are processed in camera.  

Cory - When I look at the curves in the LPROF profile
checker I can't really 
see the discontinuities in the curves even though I know
that these exist.  I 
have ICCExamin 0.38 and I don't see anyplace to inspect the
curves as such.  
Could you fill me in on how you are doing this?

Hal




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

Am 25.01.06, 13:59 -0800 schrieb Hal V. Engel:

> Cory - When I look at the curves in the LPROF profile
checker I can't really 
> see the discontinuities in the curves even though I
know that these exist.  I 
> have ICCExamin 0.38 and I don't see anyplace to inspect
the curves as such.  
> Could you fill me in on how you are doing this?

The mtf1/2 curves are visible in the A2B and B2A tables.
Select one 
table from the tag list (window below the menubar).

Below the tag list window a mtf1/2 selector appears. The
first entry shows 
an overview of the whole mtf table. The second the matrix,
followed by the 
input curves. After this the 3D tables (3D Lut's).

There you have a right button menu to select the table
channel, navigate 
and tune appearance like background colour or show the
spectral curve.

And as last entry in the mtf selector comes the output curve
of that 
table.

regards
Kai-Uwe Behrmann
                                + development for color
management 
                                + imaging / panoramas
                                + email: ku.bgmx.de
                                + http://www.behrmann.name



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

> There is a problem with local convergence extrapolation
that results in local
> reversals (abrupt localized changes in the curves) . 
There is a section in
> the help files that describes the problem.  There is
also a bug report that
> has been opened on this issue.  Please see
>
> http://s
ourceforge.net/tracker/index.php?func=detail&aid=1296658
&group_id=146038&atid=764370
>
> The problem was worse before 16 bit target image
support was added but that
> did not completely fix the problem.    My testing
indicates that it only
> happens in the darkest parts of images and is worse in
areas that have highly
> saturated very dark colors (dark reds and yellows seem
to be the worst).  It
> does not show up at all in images with lower contrast
levels as far as I can
> tell since these images will not have areas dark enough
to show the problem.
>
> I have not had a chance to dig into the code to figure
out what is happening
> but I think that this is related to the fact that
digital cameras,
> particularly those with larger sensor sites like DSLRs,
 have a gamut that is
> considerably larger than what is represented on an
IT8.7 chart.  In other
> words there are significant areas of the cameras gamut
where there is little
> or no data from the IT8.7 image and local convergence
extrapolation in affect
> tries too hard to make these areas "right"
and the curves end up with
> discontinuities.
>
> So far most of my work on LPROF has been to consolidate
the user interface and
> to remove redundancies from the code.  Little has been
done to the
> underlaying profile generation algorithms at his point.
 The current
> recommendation is to not use local convergence
extrapolation with wide gamut
> devices as this will result in smoother curves that do
not exhibit the
> problem.  I suspect that the problem is for the most
part limited to those
> who are working with raw images since raw processing is
cabable of pulling
> way more gamut/dynamic range out of images then would
be typical for images
> that are processed in camera.
>
> Cory - When I look at the curves in the LPROF profile
checker I can't really
> see the discontinuities in the curves even though I
know that these exist.  I
> have ICCExamin 0.38 and I don't see anyplace to inspect
the curves as such.
> Could you fill me in on how you are doing this?
>
> Hal
>

 	Good to know that what I'm describing isn't just due to my

ignorance of how the whole system works.  Like I said, it's
only been 
recently that I've been able to analyze the profiles enough
to narrow it 
down to the localized convergence parameter.  The built-in
LPROF profile 
checker doesn't show any of the badness really... dE always
looked good. 
Just for the record, the problem is worse as you say with
RAW files and 
linear data... probably due to the reduced numerical
contrast of linear 
data.

 	As far as what I used to notice the problem... first it
was the 
little graphical curve in UFRAW's input histogram plot. 
What nominally 
looks like a gamma curve ("fuzzy" due to the
matrix) had large jumps in 
the curve in the brightest area of the histogram.  The
resulting RAW 
conversion had strange artifacts in the brightest areas of
the image.

 	Yesterday, I figured out how to load up multiple profiles
in 
iccexamin (0.38) and compare the gamuts.  You can see the
discontinuities 
there, but only if you turn on the Spektral line and show
the 3D surface 
in color (not gray).  If you do that, there is a projection
onto the Lab 
plane that shows discontinuities.

 	Behrmann's answer to your question was another feature of 
iccexamin I was unaware of.  I didn't know there was a way
to look into 
the matrix before.  Now that I've done that a bit, you can
definately see 
some funny stuff going on with the local-convergence-enabled
profile. 
Without it, all the slices are pretty much grescale cubes. 
With it, there 
are some awfully strange discontinuous "blocky"
shapes in all of the 
different slices.

 	All of the curves (TRC, shaper, etc) have always looked
good... 
nominally linear or gamma as appropriate.  Just FYI, I am
using a Pentax 
*ist-DS DSLR, so the sensor likely has the larger gamut than
the IT8.7 can 
adequately profile.  I've even found the spectral response
curves for the 
Sony sensor inside... 

-Cory

-- 

************************************************************
*************
* Cory Papenfuss                                            
           *
* Electrical Engineering candidate Ph.D. graduate student   
           *
* Virginia Polytechnic Institute and State University       
           *
************************************************************
*************



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

> The problem was worse before 16 bit target image
support was added but that
> did not completely fix the problem.    My testing
indicates that it only
> happens in the darkest parts of images and is worse in
areas that have highly
> saturated very dark colors (dark reds and yellows seem
to be the worst).  It
> does not show up at all in images with lower contrast
levels as far as I can
> tell since these images will not have areas dark enough
to show the problem.
>

 	I would also like to add (having played with a bunch of
different 
profiles in the past few hours) that the low-luminance
measurements end up 
with significantly more error than the high luminance
measurements when 
using linear data.  It almost implies that the curve-fitting
optimization 
implies log-spacing on the data in its error minimization. 
Possible?

 	BTW, iccexamin is becoming cooler all the time.  I was
unaware of 
the matrix visualization and the measurement display
capability.  Much 
easier to see in 3-space which swatches don't match well.

-Cory

  --

************************************************************
*************
* Cory Papenfuss                                            
           *
* Electrical Engineering candidate Ph.D. graduate student   
           *
* Virginia Polytechnic Institute and State University       
           *
************************************************************
*************



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

On Thursday 26 January 2006 05:51 am, Cory Papenfuss wrote:
snip
>
>  	Good to know that what I'm describing isn't just due
to my
> ignorance of how the whole system works.  Like I said,
it's only been
> recently that I've been able to analyze the profiles
enough to narrow it
> down to the localized convergence parameter.  The
built-in LPROF profile
> checker doesn't show any of the badness really... dE
always looked good.
> Just for the record, the problem is worse as you say
with RAW files and
> linear data... probably due to the reduced numerical
contrast of linear
> data.
>
>  	As far as what I used to notice the problem... first
it was the
> little graphical curve in UFRAW's input histogram plot.
 What nominally
> looks like a gamma curve ("fuzzy" due to the
matrix) had large jumps in
> the curve in the brightest area of the histogram.  The
resulting RAW
> conversion had strange artifacts in the brightest areas
of the image.

I have stayed away from using the matrix in UFRAW as I don't
think this gives 
good results in a color managed workflow.   This was not
based on any 
objective data but more on how I subjectively felt about the
results when I 
tested with the matrix turned on.   I found this to be true
even when I used 
another profiler (ProfilePrism).   So this in not LPROF
specific.   This may 
also explain why you are seeing artifacts in the bright
areas and I am not 
with local convergence turned on.  I should document this in
the LPROF help 
file UFRAW section.

I should add that profiles generated by LPROF (without local
convergence) and 
ProfilePrism give results that are only very slightly
different.  
Subjectively the PP profiles are very slightly more
saturated and that is  
only difference that I can detect.  Overall tonality, white
balance, shadow 
detail, highlight detail ... are almost identical.  Of
course it would be a 
simple matter for a UFRAW user to make adjustments in how
much saturation (or 
contrast, shadow detail, brightness) the profile will add to
the final result 
when profiling with LPROF by adjusting the saturation level
in UFRAW in the 
opposite direction when converting the IT8 target.  So with
a slight 
adjustment I could make the LPROF and PP profiles give
subjectively identical 
results.

>
>  	Yesterday, I figured out how to load up multiple
profiles in
> iccexamin (0.38) and compare the gamuts.  You can see
the discontinuities
> there, but only if you turn on the Spektral line and
show the 3D surface
> in color (not gray).  If you do that, there is a
projection onto the Lab
> plane that shows discontinuities.

This helped and I now know what I am looking at.  The
differences between a 
non-local convergence profile and a local convergence
profile are very clear 
when viewed this way.

snip

>  	All of the curves (TRC, shaper, etc) have always
looked good...
> nominally linear or gamma as appropriate.  Just FYI, I
am using a Pentax
> *ist-DS DSLR, so the sensor likely has the larger gamut
than the IT8.7 can
> adequately profile.  I've even found the spectral
response curves for the
> Sony sensor inside... 

The curves have always looked good and that is one of the
reasons that I have 
not looked into this in more detail.  That is I did not have
a good way to 
visualize what was happening so it would have been difficult
to tell if 
changes to the code had the desired affect.   I am going to
dig into the code 
to see if I might be able to get this fixed as part of the
1.11 series.  This 
might take a while since I have not looked that this part of
the code in 
detail and I suspect that the changes will not be trivial.

My D70 uses the same sensor as do most, if not all, of the 6
mega pixel DSLRs 
with the exception of the Canon DSLRs.

>
> -Cory


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

> I have stayed away from using the matrix in UFRAW as I
don't think this gives
> good results in a color managed workflow.   This was
not based on any
> objective data but more on how I subjectively felt
about the results when I
> tested with the matrix turned on.   I found this to be
true even when I used
> another profiler (ProfilePrism).   So this in not LPROF
specific.   This may
> also explain why you are seeing artifacts in the bright
areas and I am not
> with local convergence turned on.  I should document
this in the LPROF help
> file UFRAW section.
>
 	Ooops.  I guess I wasn't clear.  The matrix I was
referring to is 
the LUT in the profile, not the "use color matrix"
checkbox from within 
UFRAW.  I think I finally (and recently) figured out what
that's supposed 
to do.  It enables the RGB->RGB transform matrix from
dcraw (who got it 
from Adobe) *before* it does any of the color management
stuff.  In other 
words, it's only really useful if you're going to do a
default sRGB 
non-color-managed conversion.  For any real profiles (like
the ones I've 
been playing with generated by LPROF), the checkbox should
be off.

> I should add that profiles generated by LPROF (without
local convergence) and
> ProfilePrism give results that are only very slightly
different.
> Subjectively the PP profiles are very slightly more
saturated and that is
> only difference that I can detect.  Overall tonality,
white balance, shadow
> detail, highlight detail ... are almost identical.  Of
course it would be a
> simple matter for a UFRAW user to make adjustments in
how much saturation (or
> contrast, shadow detail, brightness) the profile will
add to the final result
> when profiling with LPROF by adjusting the saturation
level in UFRAW in the
> opposite direction when converting the IT8 target.  So
with a slight
> adjustment I could make the LPROF and PP profiles give
subjectively identical
> results.
>
 	I don't trust myself to do much subjective adjustments. 
  The 
profile I got with LPROF without local convergence yields a
very similar 
result to that which I ripped out of a BibblePro install. 
That's what 
I'd been using so far.

>>
>>  	Yesterday, I figured out how to load up multiple
profiles in
>> iccexamin (0.38) and compare the gamuts.  You can
see the discontinuities
>> there, but only if you turn on the Spektral line
and show the 3D surface
>> in color (not gray).  If you do that, there is a
projection onto the Lab
>> plane that shows discontinuities.
>
> This helped and I now know what I am looking at.  The
differences between a
> non-local convergence profile and a local convergence
profile are very clear
> when viewed this way.
>
 	... in particular the much larger gamut (at least in my
case). 
Without local convergence, the 3D space is more or less
contained within 
the Lab space.  With local convergence on, the gamut is
significantly 
larger (and more broken).

> snip
>
>>  	All of the curves (TRC, shaper, etc) have always
looked good...
>> nominally linear or gamma as appropriate.  Just
FYI, I am using a Pentax
>> *ist-DS DSLR, so the sensor likely has the larger
gamut than the IT8.7 can
>> adequately profile.  I've even found the spectral
response curves for the
>> Sony sensor inside... 
>
> The curves have always looked good and that is one of
the reasons that I have
> not looked into this in more detail.  That is I did not
have a good way to
> visualize what was happening so it would have been
difficult to tell if
> changes to the code had the desired affect.   I am
going to dig into the code
> to see if I might be able to get this fixed as part of
the 1.11 series.  This
> might take a while since I have not looked that this
part of the code in
> detail and I suspect that the changes will not be
trivial.
>
> My D70 uses the same sensor as do most, if not all, of
the 6 mega pixel DSLRs
> with the exception of the Canon DSLRs.
>
 	The matrix view (NOT gamut view) is the most telling. 
Pretty 
funky on the broken profiles, but very smooth on the
non-local convergence 
ones.

-Cory

-- 

************************************************************
*************
* Cory Papenfuss                                            
           *
* Electrical Engineering candidate Ph.D. graduate student   
           *
* Virginia Polytechnic Institute and State University       
           *
************************************************************
*************



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user

On Thursday 26 January 2006 12:07 pm, Cory Papenfuss wrote:
snip

>  	Ooops.  I guess I wasn't clear.  The matrix I was
referring to is
> the LUT in the profile, not the "use color
matrix" checkbox from within
> UFRAW.  I think I finally (and recently) figured out
what that's supposed
> to do.  It enables the RGB->RGB transform matrix
from dcraw (who got it
> from Adobe) *before* it does any of the color
management stuff.  In other
> words, it's only really useful if you're going to do a
default sRGB
> non-color-managed conversion.  For any real profiles
(like the ones I've
> been playing with generated by LPROF), the checkbox
should be off.

Thanks for clearing this up.

>
> > I should add that profiles generated by LPROF
(without local convergence)
> > and ProfilePrism give results that are only very
slightly different.
> > Subjectively the PP profiles are very slightly
more saturated and that is
> > only difference that I can detect.  Overall
tonality, white balance,
> > shadow detail, highlight detail ... are almost
identical.  Of course it
> > would be a simple matter for a UFRAW user to make
adjustments in how much
> > saturation (or contrast, shadow detail,
brightness) the profile will add
> > to the final result when profiling with LPROF by
adjusting the saturation
> > level in UFRAW in the opposite direction when
converting the IT8 target. 
> > So with a slight adjustment I could make the LPROF
and PP profiles give
> > subjectively identical results.
>
>  	I don't trust myself to do much subjective
adjustments.    The
> profile I got with LPROF without local convergence
yields a very similar
> result to that which I ripped out of a BibblePro
install.  That's what
> I'd been using so far.

One of the features of ProfilePrism is that it allows you to
tweak your 
profiles to change contrast, saturation, gamma (labled
brightness in PP) and 
color balance.  This is useful in PP for doing printer
profiles since it uses 
a scanner as it's "measurement tool" and needs
this to compensate for the 
scanner.  These same settings can also be used when
profiling a camera or 
scanner.  But when I use PP I set these to not make any
changes (default 
settings).   I was mainly pointing out that the differences
between the 
results from LPROF and PP (with default settings) are very
small and that 
LPROF/UFRAW users can influence the profile much like a PP
user can but that 
this is done in a slightly different way.  I think that the
results I am 
getting without doing this are actually sightly better than
PP as I think the 
default setting in PP give a very slightly over saturated
result.

snip

>
>  	The matrix view (NOT gamut view) is the most telling.
 Pretty
> funky on the broken profiles, but very smooth on the
non-local convergence
> ones.

Exactly,  this will be very helpful when working on fixing
this as I will be 
able to tell if a code change has made this worse or better
or not changed 
anything.

>
> -Cory


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep
through log files
for problems?  Stop!  Download the new AJAX search engine
that makes
searching your log files as easy as surfing the  web. 
DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/
sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Lcms-user mailing list
Lcms-userlists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lcms-user