Re: Fw: ia64: race flushing icache in do_no_page path

Could you please confirm that I understand correctly what is
in the:

Dual-Core Update to the Intel Itanium 2 Processor Reference
Manual...

"2.3.3.2 L2 Caches
...
Any coherence request to identify whether a cache line is in
the processor
will invalidate that line from the L2I cache."

This makes sure that the DMAs invalidate the L2L cache.

"2.7.4 Instruction Cache Coherence Optimization
Coherence requests of the L1I and L2I caches will invalidate
the line if
it is in the cache. Montecito allows instruction requests on
the system
interface to be filtered such that they will not initiate
coherence
requests of the L1I and L2I caches. This will allow
instructions to be
cached at the L1I and L2I levels across multiple processors
in a coherent
domain. This optimization is enabled by default, but may be
disabled by
PAL_SET_PROC_FEATURES bit 5 of the Montecito feature_set
(18)."

Machines star up whit bit 5 = 0, reading instruction pages
via
NFS has to flush them from L2I.


I was wondering if instead of modifying do_no_page() and
Co., should
not we make nfs_readpage() be DMA-like?
(No possible regression for most of the page I/O-s.)
I.e. it should be the responsibility of a file system to
make sure it
supports instruction pages correctly. The base kernel should
provide
such file systems with an architecture dependent macro...

Thanks,

Zoltan Menyhart
-
To unsubscribe from this list: send the line
"unsubscribe linux-ia64" in
the body of a message to majordomovger.kernel.org
More majordomo info at  http://vge
r.kernel.org/majordomo-info.html

On Wed, 04 Jul 2007 16:24:38 +0200
Zoltan Menyhart <Zoltan.Menyhartbull.net> wrote:
> Machines star up whit bit 5 = 0, reading instruction
pages via
> NFS has to flush them from L2I.
> 
In our test, we confirmed that this can be fixed by flushing
L2I just before 
SetPageUptodate() in NFS.

> 
> I was wondering if instead of modifying do_no_page()
and Co., should
> not we make nfs_readpage() be DMA-like?
> (No possible regression for most of the page I/O-s.)
> I.e. it should be the responsibility of a file system
to make sure it
> supports instruction pages correctly. The base kernel
should provide
> such file systems with an architecture dependent
macro...
> 
IMHO, for example, race in cooy-on-write  (was fixed by Tony
Luck) has to be
fixed by MemoryManagement layer.
And only a race in do_no_page() seems to be able to be fixed
by FS layer.

BTW, can we know whether a page is filled by DMA or not  ?

Thanks,
-Kame

-
To unsubscribe from this list: send the line
"unsubscribe linux-ia64" in
the body of a message to majordomovger.kernel.org
More majordomo info at  http://vge
r.kernel.org/majordomo-info.html

KAMEZAWA Hiroyuki wrote:
> On Wed, 04 Jul 2007 16:24:38 +0200
> Zoltan Menyhart <Zoltan.Menyhartbull.net> wrote:
> 
>>Machines star up whit bit 5 = 0, reading instruction
pages via
>>NFS has to flush them from L2I.
>>
> 
> In our test, we confirmed that this can be fixed by
flushing L2I just before 
> SetPageUptodate() in NFS.

I can agree.
We can be more permissive: it can be done anywhere after the
new
data is put in place and before nfs_readpage() or
nfs_readpages()
returns.

I saw your patch http://marc.info/?l=linux-mm&m=118352909826277&
;w=2
that modifies e.g. mm/memory.c and not the NFS layer.

Have you proposed a patch against the NFS layer?

>>I was wondering if instead of modifying do_no_page()
and Co., should
>>not we make nfs_readpage() be DMA-like?
>>(No possible regression for most of the page
I/O-s.)
>>I.e. it should be the responsibility of a file
system to make sure it
>>supports instruction pages correctly. The base
kernel should provide
>>such file systems with an architecture dependent
macro...
>>
> 
> IMHO, for example, race in cooy-on-write  (was fixed by
Tony Luck) has to be
> fixed by MemoryManagement layer.

I can agree.
Note that it has not got much performance impact from our
point
of view because there are not too many COW paves with
executable code...

> And only a race in do_no_page() seems to be able to be
fixed by FS layer.

If it were do_no_page() that would fix the problem, then it
should know
where the page comes from in order not to flush the I-cache
in vain.
do_no_page() just calls vma->vm_ops->nopage() that
goes down to
fs_op->readpage() / fs_op->readpages().

On the other hand, these latter routines do not know why
they fetch
the page(s). Note that a page can be mapped more than one
times, with
different permission bits.
As a consequence, these routines will flush the I-cache in
vain in
most of the cases.

I prefer a performance impact to some non DMA based file
systems
and adding nothing to the path for the majority of the
cases.

> BTW, can we know whether a page is filled by DMA or not
 ?

Well, the file systems based on block devices use DMA
transfer
(with the exception of using bounce buffers, in that case it
is the
responsibility of the bounce buffering layer to flush the
I-cache).

Network based file systems require revision and update...

Note that only the processors with separate L2I require
attention, the L1I is guaranteed to be flushed when you
change
the corresponding TBL entry.

The base kernel should provide a macro / service (that cares
for the
processor model) for the file systems.

Thanks,

Zoltan Menyhart
-
To unsubscribe from this list: send the line
"unsubscribe linux-ia64" in
the body of a message to majordomovger.kernel.org
More majordomo info at  http://vge
r.kernel.org/majordomo-info.html

On Thu, Jul 05, 2007 at 10:57:00AM +0200, Zoltan Menyhart
wrote:
> KAMEZAWA Hiroyuki wrote:
> >In our test, we confirmed that this can be fixed by
flushing L2I just 
> >before SetPageUptodate() in NFS.
> 
> I can agree.
> We can be more permissive: it can be done anywhere
after the new
> data is put in place and before nfs_readpage() or
nfs_readpages()
> returns.
> 
> I saw your patch http://marc.info/?l=linux-mm&m=118352909826277&
;w=2
> that modifies e.g. mm/memory.c and not the NFS layer.
> 
> Have you proposed a patch against the NFS layer?

  This really doesn't look like a job for the file system
layer.
That would require all sorts of file system readpage
routines to
be modified to handle memory management details that are
already
handled by the memory.c functions.  The do_no_page code is
already
dealing with the necessary icache flushing operations.  It
just
happens to be doing it with a bad race condition for ia64.

-- 
Mike Stroyan <mikestroyan.net>
-
To unsubscribe from this list: send the line
"unsubscribe linux-ia64" in
the body of a message to majordomovger.kernel.org
More majordomo info at  http://vge
r.kernel.org/majordomo-info.html