Re: Chinese Traditional appearance -- mixed weights?

On Fri, 2007-10-19 at 21:34 -0400, Behdad Esfahbod wrote:
> Hi Owen,
> 
> I've had this message of yours from last September in
my inbox for quite
> a while, because I couldn't congest it the first time. 
Tonight I gave
> it another try and I still don't follow.  Do you mind
clarifying what
> the fontconfig change is about?  More inline.
> 
> On Sat, 2006-09-30 at 11:55 -0400, Owen Taylor wrote:
> > 
> > The font selection algorithm for older versions of
fontconfig is:
> > 
> >  1) Explicitly specified non-generic fonts, in the
order specified
> >    (generic fonts being things like
"sans-serif", "serif",
"monospace")
> > 
> >  2) Fonts from the generic in use matching the
language tag of the text 
> >    sorted by the order that those fonts appear in
the generic.
> >    The generic in use is determined in  priority
order:
> > 
> >      - If one was given explicitly, that generic
> >      - If a explicitly specified font is
recognized as belong to
> >        a generic class, that generic. (So, Arial
pulls in sans-serif,
> >        Courier, serif)
> >      - Otherwise, sans-serif
> >        
> >  3) Other fonts matching the language tag, in
random order.
> > 
> >  4) Other fonts from the generic in use.
> > 
> >  5) Other fonts in random order.
> 
> Ok, these make sense.  And the final list is pruned,
right?

Yes, I omitted that detail because it makes no user-visible
difference.

> > If the language tag refinement scheme ended up
with no language tag,
> > then 2) and 3) are skipped... this is what is
going on when your
> > language is en-US.
> > 
> > The fix in the most recent versions of fontconfig
referred to above
> > is that 2) and 3) are replaced with:
> > 
> >  2') The best font that matches the language tag
of the text chosen as
> >      the first language-tag matching font of:
> > 
> >        - Fonts from the generic use, in the order
specified 
> >        - Other fonts in random order
> 
> I don't understand this one.
> 
> And which version of fontconfig was this introduced
in?

commit 5cf8c5364f1b7a676f52b480fa55c571cadc6fda
Author: Keith Packard <keithpkeithp.com>
Date:   Tue Dec 14 00:12:25 2004 +0000

    I changed FcFontSetSort to respect the generic aliases
better in
    the face of language matching.
    What I did was to ammend the strict sort order used by
FcFontSort
    so that it 'satisfies' the language specified in the
pattern by
    locating the  best matching font supporting each pattern
language
    and then ignores language in the remaining fonts for
purposes of
    matching.

    So, when asking for 'sans:lang=en', you'll get an
English font
    first, and then the remaining fonts sorted with respect
to the
    'sans' alias alone -- pushing Kochi fonts ahead of
other
    English-supporting Han fonts.

    reviewed by: Owen Taylor <otaylorredhat.com>

I don't know what version of fontconfig that corresponds
to.

> > The advantage of that is that once we know what
font we prefer for,
> > say, Greek, a language tag of Greek won't then
won't reorder Russian
> > fonts to prefer fonts that also happen to cover
Greek.
> 
> And I don't get this either.  So, you are saying that
if we ask for
> fonts with lang=Greek but look for fonts supporting
Russian in the
> result it will work as good as doing the same with
lang=Russian?

So, say we have three fonts:

 Nice font covering Russian
 Nice font covering Greek
 Ugly font covering Greek and Russian

Sorted in that order in the sans-serif alias in fonts.conf.

The way that fontconfig used to work, FcFontSort with
lang=gr would
have given us:

 Nice font covering Greek
 Ugly font covering Greek and Russian
 Nice font covering Russian

Because the Greek and Russian font was considered to be
"better" than
the Russian only font. This forced us to do language tag
refinement.

But the observation is that once Greek has been
"satisfied" by one font,
there is no reason for fontconfig to reorder the rest of
fonts, so newer
versions of fontconfig will instead give us instead:

 Nice font covering Greek
 Nice font covering Russian
 Ugly font covering Greek and Russian

And we get reasonable results with the simple lang=gr
language tag.

- Owen


_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

Attached is a screen shot of a screen generated using Pango
with GTK 2.2.4
in Chinese Traditional.  Does the Chinese look correct?  We
currently have
no one who reads Chinese, but it looks to us like we're
getting mixed
weights (boldness) for different characters in the same line
in many
cases.  Does it do this automatically when the selected font
size doesn't
have all the characters, or is there some other problem? 
The selected
language using pango_context_set_language is "zh";
the behavior is the
same using the more specific "zh-TW".  Appearance
at larger font sizes or
in other languages including Chinese Simplified is much more
uniform.  I
tried installing the AR PL New Sung font recommended at
http://www.unifont.
org/fontguide/ and placed it ahead of all other fonts
in all the lists in my fonts.conf file and it seemed to make
no
difference.  Any suggestions would be appreciated.
_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

On Wed, 2006-09-27 at 16:47 -0500, Boncek, John wrote:
> Attached is a screen shot of a screen generated using
Pango with GTK 2.2.4
> in Chinese Traditional.  Does the Chinese look correct?
 We currently have
> no one who reads Chinese, but it looks to us like we're
getting mixed
> weights (boldness) for different characters in the same
line in many
> cases.  Does it do this automatically when the selected
font size doesn't
> have all the characters, or is there some other
problem?  The selected
> language using pango_context_set_language is
"zh"; the behavior is the
> same using the more specific "zh-TW". 
Appearance at larger font sizes or
> in other languages including Chinese Simplified is much
more uniform.  I
> tried installing the AR PL New Sung font recommended at
> http://www.unifont.
org/fontguide/ and placed it ahead of all other fonts
> in all the lists in my fonts.conf file and it seemed to
make no
> difference.  Any suggestions would be appreciated.

It's really hard to comment on versions of GTK+ and Pango
that were
released over 3 years ago (effectively more like 4, since
2.2.0 was
released in Dec 2002, and 2.2.4 is just a bug-fix of that);
I have
trouble even remembering what font systems were supported at
that time.

The screenshots show multiple fonts mixed together. If you:

 a) Have your text tagged with the language zh-tw
 b) Have some font on your system that fontconfig identifies
as
    zh-tw ('fc-list :lang=zh-tw')
 c) Have not explicitly specified a Simplified Chinese or
Japanese
    font for the text

That won't happen

But maybe you are using X core fonts? Maybe you are using a
pre-fontconfig version of Xft? 
					- Owen


_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

Owen Taylor wrote:
> It's really hard to comment on versions of GTK+ and
Pango that were
> released over 3 years ago (effectively more like 4,
since 2.2.0 was
> released in Dec 2002, and 2.2.4 is just a bug-fix of
that); I have
> trouble even remembering what font systems were
supported at that time.
> 
> The screenshots show multiple fonts mixed together. If
you:
> 
>  a) Have your text tagged with the language zh-tw
>  b) Have some font on your system that fontconfig
identifies as
>     zh-tw ('fc-list :lang=zh-tw')
>  c) Have not explicitly specified a Simplified Chinese
or Japanese
>     font for the text
> 
> That won't happen
> 
The same thing happens with Simplified Chinese in the
default setup of
both older and recent versions of Gnome. For example it
happens with
Gnome 2.14.2 on Gentoo and also Ubuntu Dapper.

attached is a screenshot of some Chinese and English in
gedit using
monospace 16pt. The same problem occurs with sans but the
font mixing
doesn't occur with serif. Nor does it occur if I choose a
Chinese font
but then the English often looks very ugly. As I often need
to work on
documents in mixed languages just choosing a Chinese is not
terribly
satisfactory. Also gnome uses monospace and serif and this
problem
causes the Chinese to look really ugly and will definitely
turn off some
new users of gnome.

I have always assumed that this is a problem with the
default gnome
setup for fontconfig but have never found the time to
actually find and
fix the problem.
_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

On Fri, 2006-09-29 at 12:30 +0800, Greg Aumann wrote:
> Owen Taylor wrote:
> > It's really hard to comment on versions of GTK+
and Pango that were
> > released over 3 years ago (effectively more like
4, since 2.2.0 was
> > released in Dec 2002, and 2.2.4 is just a bug-fix
of that); I have
> > trouble even remembering what font systems were
supported at that time.
> > 
> > The screenshots show multiple fonts mixed
together. If you:
> > 
> >  a) Have your text tagged with the language zh-tw
> >  b) Have some font on your system that fontconfig
identifies as
> >     zh-tw ('fc-list :lang=zh-tw')
> >  c) Have not explicitly specified a Simplified
Chinese or Japanese
> >     font for the text
> > 
> > That won't happen
> > 
> The same thing happens with Simplified Chinese in the
default setup of
> both older and recent versions of Gnome. For example it
happens with
> Gnome 2.14.2 on Gentoo and also Ubuntu Dapper.
> 
> attached is a screenshot of some Chinese and English in
gedit using
> monospace 16pt. The same problem occurs with sans but
the font mixing
> doesn't occur with serif. Nor does it occur if I choose
a Chinese font
> but then the English often looks very ugly. As I often
need to work on
> documents in mixed languages just choosing a Chinese is
not terribly
> satisfactory. [...]
> 
> I have always assumed that this is a problem with the
default gnome
> setup for fontconfig but have never found the time to
actually find and
> fix the problem.

Do you think that Pango should do frequency analysis of
incoming text
to guess the particular Han-using language it is in
(allowing for the
possibility of multiple Han-using languages in a single
Paragraph, 
also allowing for paragraphs that are too short to do any
such
analysis)?

If not, then you have two possibilities:

A) Provide Pango with the information about the language you
are
   displaying (via the locale environment variables, or via
markup...
   Pango doesn't support Plane 14 language tags, and
considering
   how reluctantly they are part of Unicode, is unlikely to
do
   so in the future.)

B) Configure a universal Han font to be preferred over
language
   specific Han fonts in your fonts.conf

   Note that there is no available freely redistributable
Han font
   and also universal Han fonts are not always preferred for
   writing particular languages, so it's really not
surprising 
   that fontconfig doesn't come configured that way out of
the 
   box.
  
This isn't really applicable to the previous poster's
question, 
since apparently language information was supplied.

> Also gnome uses monospace and serif and this problem
> causes the Chinese to look really ugly and will
definitely turn off some
> new users of gnome.					

Not quite sure what you mean here; GNOME defaults to the
system
"sans-serif" alias for most things, and to the
system "monospace" alias
for a few things ... gnome-terminal, gedit ... where a
monospace font is
expected. 

(My assumption is that new users of GNOME are mostly not
reading Chinese
documents when running in an English locale..)

					- Owen


_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

Owen Taylor wrote:
> Do you think that Pango should do frequency analysis of
incoming text
> to guess the particular Han-using language it is in
(allowing for the
> possibility of multiple Han-using languages in a single
Paragraph, 
> also allowing for paragraphs that are too short to do
any such
> analysis)?
> 
of course not. As you obviously expect.

> If not, then you have two possibilities:
>
> A) Provide Pango with the information about the
language you are
>    displaying (via the locale environment variables, or
via markup...

I tried setting the locale to different values to see what
the effect
would be. If I use zh_CN.UTF-8 then it uses the same Chinese
font for
all the text regardless of being mono, sans or serif. I
don't get the
mixing of two fonts. If I use zh_TW.UTF-8 I get mixing of
the same two
fonts as the English locale but I get a lot more of the
"italic" looking
one and much less of the "sans serif" looking one.
A Japanese locale
gives me the same results as an English one. So obviously
setting the
language makes a difference. That was something that I
didn't realise.
But setting the language to a Han locale doesn't necessarily
solve the
mixing fonts problem. In fact if I use a Taiwanese locale
the text is
more simplified characters than if I use an English locale!

How does pango determine which fonts to use for different
languages? I
can't see anything language specific in fonts.conf.

Also how can I untangle the effects of the font sets? I want
to figure
out which glyphs are coming from which fonts so I can adjust
fonts.conf
properly i.e. also understand what is going on. When I
change the font
in gedit or gcharmap to a specific one it still is bringing
in glyphs
from other fonts. Is going through all my font files in
fontforge and
comparing appearances the only way?

_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

On Sat, 2006-09-30 at 13:35 +0800, Greg Aumann wrote:
> Owen Taylor wrote:
> > Do you think that Pango should do frequency
analysis of incoming text
> > to guess the particular Han-using language it is
in (allowing for the
> > possibility of multiple Han-using languages in a
single Paragraph, 
> > also allowing for paragraphs that are too short to
do any such
> > analysis)?
> > 
> of course not. As you obviously expect.
> 
> > If not, then you have two possibilities:
> >
> > A) Provide Pango with the information about the
language you are
> >    displaying (via the locale environment
variables, or via markup...
> 
> I tried setting the locale to different values to see
what the effect
> would be. If I use zh_CN.UTF-8 then it uses the same
Chinese font for
> all the text regardless of being mono, sans or serif. I
don't get the
> mixing of two fonts.
>
>  If I use zh_TW.UTF-8 I get mixing of the same two
> fonts as the English locale but I get a lot more of the
"italic" looking
> one and much less of the "sans serif" looking
one. A Japanese locale
> gives me the same results as an English one. So
obviously setting the
> language makes a difference. That was something that I
didn't realise.
> But setting the language to a Han locale doesn't
necessarily solve the
> mixing fonts problem. In fact if I use a Taiwanese
locale the text is
> more simplified characters than if I use an English
locale!

I don't really have enough familiarity with Chinese to sort
out
"simplified only" from "traditional
only" characters visually. But from
what you describe:

 zh-cn: consistent font
 ja-jp: mixed fonts
 en-us: mixed fonts
 zh-tw: mixed fonts

It really sounds like you have a simplified Chinese
document. The
difference between ja-jp/en-us and zh-tw is likely because
the first
Han font in your fontconfig configuration is a Japanese
font.

> How does pango determine which fonts to use for
different languages? I
> can't see anything language specific in fonts.conf.

The supported languages of a font are determined
automatically by
fontconfig based on the codepoints included in the font;
that is
then one factor that is taken into account by fontconfig's
font
selection algorithm

The determination of the language tag is done, on the other
hand
by Pango. Pango first determines a script for each character
in
the text it is rendering, and also determines a first-pass
language
tag by looking at what was specified by the application, and
if
nothing was specified, by the locale. (*)

It then knows what languages are compatible with what
scripts (That is,
it know that text in the Arabic script can't be English) and
uses
that information to "refine" the language tags. If
it finds a language
tag that isn't plausible it replaces it with:

 - A default language tag for the script when there is
something
   obvious and more-or-less uncontroversial. (Arabic for the
Arabic
   script, Greek for the Greek script)
 - No language tag if an appropriate language tag can't be
guessed.
   (Han script)

(This language tag refinement process is mainly meant to
deal with
pathologies  of the fontconfig font selection scheme that
have been
solved in the most recent versions of fontconfig, so
possibly will be
dropped in future versions of Pango)

> Also how can I untangle the effects of the font sets? I
want to figure
> out which glyphs are coming from which fonts so I can
adjust fonts.conf
> properly i.e. also understand what is going on. When I
change the font
> in gedit or gcharmap to a specific one it still is
bringing in glyphs
> from other fonts. Is going through all my font files in
fontforge and
> comparing appearances the only way?

The font selection algorithm for older versions of
fontconfig is:

 1) Explicitly specified non-generic fonts, in the order
specified
   (generic fonts being things like "sans-serif",
"serif", "monospace")

 2) Fonts from the generic in use matching the language tag
of the text 
   sorted by the order that those fonts appear in the
generic.
   The generic in use is determined in  priority order:

     - If one was given explicitly, that generic
     - If a explicitly specified font is recognized as
belong to
       a generic class, that generic. (So, Arial pulls in
sans-serif,
       Courier, serif)
     - Otherwise, sans-serif
           
 3) Other fonts matching the language tag, in random order.

 4) Other fonts from the generic in use.

 5) Other fonts in random order.

If the language tag refinement scheme ended up with no
language tag,
then 2) and 3) are skipped... this is what is going on when
your
language is en-US.

The fix in the most recent versions of fontconfig referred
to above
is that 2) and 3) are replaced with:

 2') The best font that matches the language tag of the text
chosen as
     the first language-tag matching font of:

       - Fonts from the generic use, in the order specified 
       - Other fonts in random order

The advantage of that is that once we know what font we
prefer for,
say, Greek, a language tag of Greek won't then won't reorder
Russian
fonts to prefer fonts that also happen to cover Greek.

Sorry for the excessive level of detail here; I couldn't
figure out
a good place to stop and simplify. Though, trust me, there's
still
a fair a bit of complexity I *didn't* get to above. 

						Owen

(*) Not quite true, but GTK+ always specifies the locale as
a fallback
    language tag, so effectively that's what you'll see for
using
    Pango through GTK+.

_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

On Sat, 2006-09-30 at 13:35 +0800, Greg Aumann wrote:

> Also how can I untangle the effects of the font sets? I
want to figure
> out which glyphs are coming from which fonts so I can
adjust fonts.conf
> properly i.e. also understand what is going on. When I
change the font
> in gedit or gcharmap to a specific one it still is
bringing in glyphs
> from other fonts. Is going through all my font files in
fontforge and
> comparing appearances the only way?

Ah, actually, I guess you weren't asking about algorithm
here. If 
you right-click on a character in gucharmap, it will show
you the
font it comes from.

You can also see the font that fontconfig prefers for a
particular
generic/language tag pair with, something like:

 $ fc-match sans-serif:lang=zh-tw
 uming.ttf: "AR PL ShanHeiSun Uni"
"Regular"
						- Owen


_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

Owen Taylor wrote:
> It really sounds like you have a simplified Chinese
document. The
> difference between ja-jp/en-us and zh-tw is likely
because the first
> Han font in your fontconfig configuration is a Japanese
font.
> 
It is a simplified Chinese document.

> Sorry for the excessive level of detail here; I
couldn't figure out
> a good place to stop and simplify. Though, trust me,
there's still
> a fair a bit of complexity I *didn't* get to above.

> 
Actually I appreciate very much the detailed but clear
explanation.
Thanks for taking the time. I am going to need to chew on it
for a while
though, and go through my fontconfig setup examining the
fonts to try
and sort out what is going on.

Greg
_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list

I am chinese,
Actually,in redhat linux, both RedHat and Fedora Core,
many applications's  display are ugly.


On 10/4/06, Greg Aumann <Greg_Aumannsil.org> wrote:
> Owen Taylor wrote:
> > It really sounds like you have a simplified
Chinese document. The
> > difference between ja-jp/en-us and zh-tw is likely
because the first
> > Han font in your fontconfig configuration is a
Japanese font.
> >
> It is a simplified Chinese document.
>
> > Sorry for the excessive level of detail here; I
couldn't figure out
> > a good place to stop and simplify. Though, trust
me, there's still
> > a fair a bit of complexity I *didn't* get to
above. 
> >
> Actually I appreciate very much the detailed but clear
explanation.
> Thanks for taking the time. I am going to need to chew
on it for a while
> though, and go through my fontconfig setup examining
the fonts to try
> and sort out what is going on.
>
> Greg
> _______________________________________________
> gtk-i18n-list mailing list
> gtk-i18n-listgnome.org
> 
http://mail.gnome.org/mailman/listinfo/gtk-i18n-list
>
_______________________________________________
gtk-i18n-list mailing list
gtk-i18n-listgnome.org

http://mail.gnome.org/mailman/listinfo/gtk-i18n-list