amd64_set_gsbase()

Hi,

I have a "small" amd64 program that makes havy use
of LDT (%GS to be more
specific). The trouble is, in a multithreaded environment,
the selector
value gets lost (or reset?).

The code *always* segfaults with this stack:
  4 LWP 100126  0x0000000800dec07c in select () from
/lib/libc.so.6
* 3 Thread 0x517000 (runnable)  0x000000080055cfbc in ?? ()
  2 Thread 0x517400 (LWP 100125)  0x0000000800c0d85c in
pthread_testcancel () from /lib/libpthread.so.2
  1 Thread 0x517800 (runnable)  0x0000000800d5d000 in
makecontext () from /lib/libc.so.6

at this instruction:
  0x000000080055cfbc:     mov    %gs:0x10,%r11

  (gdb) p $gs
  $1 = 0

I've been reading on the net something about the kernel not
preserving the GS
across syscalls (or stmh). Is this true? and if so, is there
a known workaround?

I'm on a FreeBSD 6.2-STABLE-200706 (AMD64) machine.

Thanks,

-- 
Mihai Donțu
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

On Monday 08 October 2007 02:35 pm, Mihai Donțu wrote:
> Hi,
>
> I have a "small" amd64 program that makes
havy use of LDT (%GS to
> be more specific). The trouble is, in a multithreaded
environment,
> the selector value gets lost (or reset?).
>
> The code *always* segfaults with this stack:
>   4 LWP 100126  0x0000000800dec07c in select () from
/lib/libc.so.6
> * 3 Thread 0x517000 (runnable)  0x000000080055cfbc in
?? ()
>   2 Thread 0x517400 (LWP 100125)  0x0000000800c0d85c
in
> pthread_testcancel () from /lib/libpthread.so.2 1
Thread 0x517800
> (runnable)  0x0000000800d5d000 in makecontext () from
> /lib/libc.so.6
>
> at this instruction:
>   0x000000080055cfbc:     mov    %gs:0x10,%r11
>
>   (gdb) p $gs
>   $1 = 0
>
> I've been reading on the net something about the kernel
not
> preserving the GS across syscalls (or stmh). Is this
true? and if
> so, is there a known workaround?
>
> I'm on a FreeBSD 6.2-STABLE-200706 (AMD64) machine.

Yes, you are correct.  A short version is "don't do
that".  A long 
version goes like this.  %fs and %gs are not preserved while
context 
switching on amd64.  In fact, you should not use
amd64_set_gsbase() 
directly.  If you *really* have to mess up with base
addresses, you 
have to use sysarch(2) syscall, i.e.,
sysarch(AMD64_SET_GSBASE, 
args).  However, it only changes the base address via MSR,
i.e., %gs 
itself has no meaning.

Jung-uk Kim
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

On Monday 08 October 2007, Jung-uk Kim wrote:
> Yes, you are correct.  A short version is "don't
do that".  A long 
> version goes like this.  %fs and %gs are not preserved
while context 
> switching on amd64.

But this makes emulation software such as Wine a lost hope,
doesn't it?
Because Windows apps access the Thread Information Block
(TIB) via %gs
(%fs on ia32).

Anyway, my so called "small" program is actually a
Win64 emulator and
I need the segment selector to "stay put" across
syscalls. It works
like a charm on single threaded apps, but as soon as I spawn
a thread,
all hell breaks loose 

I've managed to come up with something that *kind of* works.
It goes
like this:

void my_handler( int s )
{
    if ( s == SIGSEGV ) {
        if ( get_gs() == 0 ) {
            amd64_set_gsbase();
        } else {
            signal( SIGSEGV, SIG_DFL );
        }
    }
}

int my_init( void )
{
    /* alloc TIB memory and initialize */

    amd64_set_gsbase( lpTIB );
    signal( SIGSEGV, my_handler );

    return 0;
}

but after a series of dlopen()-s, my_handler() is called
without %gs
being zero and without a valid fault (the handler does not
get recalled
after signal( SIGSEGV, SIG_DFL ). I'm still working on this
aspect ...

> In fact, you should not use amd64_set_gsbase() 
> directly.  If you *really* have to mess up with base
addresses, you 
> have to use sysarch(2) syscall, i.e.,
sysarch(AMD64_SET_GSBASE, 
> args).

I found this:
/usr/src/lib/libc/amd64/sys/amd64_set_gsbase.c:32
"
int
amd64_set_gsbase(void *addr)
{
        return (sysarch(AMD64_SET_GSBASE, &addr));
}
"

and this (man 2 sysarch()): "The sysarch() system call
should never be
called directly by user programs.  Instead, they should
access its
functions using the architecture-dependent library."

Who am I suppose to believe? 

> However, it only changes the base address via MSR,
i.e., %gs  
> itself has no meaning.

Maybe, but the selector loaded in %gs *does* have meaning.

Anyway, the thing is I _have_ to make this work. I'll keep
you posted ;)

-- 
Mihai Donțu
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

On Monday 08 October 2007 06:00 pm, Mihai Donțu wrote:
> On Monday 08 October 2007, Jung-uk Kim wrote:
> > Yes, you are correct.  A short version is
"don't do that".  A
> > long version goes like this.  %fs and %gs are not
preserved while
> > context switching on amd64.
>
> But this makes emulation software such as Wine a lost
hope, doesn't
> it? Because Windows apps access the Thread Information
Block (TIB)
> via %gs (%fs on ia32).

It was discussed many times on freebsd-emulation.

> Anyway, my so called "small" program is
actually a Win64 emulator
> and I need the segment selector to "stay put"
across syscalls. It
> works like a charm on single threaded apps, but as soon
as I spawn
> a thread, all hell breaks loose 

Yup, that's expected. 

> I've managed to come up with something that *kind of*
works. It
> goes like this:
>
> void my_handler( int s )
> {
>     if ( s == SIGSEGV ) {
>         if ( get_gs() == 0 ) {
>             amd64_set_gsbase();
>         } else {
>             signal( SIGSEGV, SIG_DFL );
>         }
>     }
> }
>
> int my_init( void )
> {
>     /* alloc TIB memory and initialize */
>
>     amd64_set_gsbase( lpTIB );
>     signal( SIGSEGV, my_handler );
>
>     return 0;
> }
>
> but after a series of dlopen()-s, my_handler() is
called without
> %gs being zero and without a valid fault (the handler
does not get
> recalled after signal( SIGSEGV, SIG_DFL ). I'm still
working on
> this aspect ...

That does not work, i.e., %gs vs. base address mapping is
not 
preserved on FreeBSD/amd64 as I said.  You can probably
maintain some 
mapping table, though.

> > In fact, you should not use amd64_set_gsbase()
> > directly.  If you *really* have to mess up with
base addresses,
> > you have to use sysarch(2) syscall, i.e.,
> > sysarch(AMD64_SET_GSBASE, args).
>
> I found this:
/usr/src/lib/libc/amd64/sys/amd64_set_gsbase.c:32
> "
> int
> amd64_set_gsbase(void *addr)
> {
>         return (sysarch(AMD64_SET_GSBASE, &addr));
> }
> "
>
> and this (man 2 sysarch()): "The sysarch() system
call should never
> be called directly by user programs.  Instead, they
should access
> its functions using the architecture-dependent
library."
>
> Who am I suppose to believe? 

Sorry, my bad. :-(

> > However, it only changes the base address via MSR,
i.e., %gs
> > itself has no meaning.
>
> Maybe, but the selector loaded in %gs *does* have
meaning.

In long mode, we don't really care about segment registers. 
While 
implementing TLS for Linuxulator, I had to do the following
hack, for 
example:

http://docs.freebsd.org/cgi/mid.cgi?200703300006.
l2U06LA1075891

Under Linux and Windows, they do preserve segment registers
vs. base 
addresses mapping for backward compatibility, AFAIK with
some 
performance penalty.

Jung-uk Kim

> Anyway, the thing is I _have_ to make this work. I'll
keep you
> posted ;)
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

On Tuesday 09 October 2007, Jung-uk Kim wrote:
> In long mode, we don't really care about segment
registers.  While 
> implementing TLS for Linuxulator, I had to do the
following hack, for 
> example:
> 
> http://docs.freebsd.org/cgi/mid.cgi?200703300006.
l2U06LA1075891
> 
> Under Linux and Windows, they do preserve segment
registers vs. base 
> addresses mapping for backward compatibility, AFAIK
with some 
> performance penalty.

Ah! But you are doing your magic _in the kernel_. I don't
have this luxury 
I have to do everything in user space (as a normal user) on
an out-of-the-box
FreeBSD (-stable).

I have *one* more question: maybe I don't fully understand
the hole BASE thing,
but since the FreeBSD kernel does not preserve %gs and %fs,
what is the purpose
of amd64_set_XXbase()?

Thanks,

-- 
Mihai Donțu
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

On Tue, Oct 09, 2007 at 03:48:51AM +0300, Mihai Don??u
wrote:
> On Tuesday 09 October 2007, Jung-uk Kim wrote:
> > In long mode, we don't really care about segment
registers.  While 
> > implementing TLS for Linuxulator, I had to do the
following hack, for 
> > example:
> > 
> > http://docs.freebsd.org/cgi/mid.cgi?200703300006.
l2U06LA1075891
> > 
> > Under Linux and Windows, they do preserve segment
registers vs. base 
> > addresses mapping for backward compatibility,
AFAIK with some 
> > performance penalty.
> 
> Ah! But you are doing your magic _in the kernel_. I
don't have this luxury 
> I have to do everything in user space (as a normal
user) on an out-of-the-box
> FreeBSD (-stable).

just to note things... actually you have the luxury. we have
made special hacks
to let wine works well (thr_kill2) I see no reason why
special support for wine64
in kernel would be left out.

just my 2 cents

roman
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

ON TUESDAY 09 OCTOBER 2007 02:48:51 MIHAI DON?U WROTE:
> I HAVE *ONE* MORE QUESTION: MAYBE I DON'T FULLY
UNDERSTAND THE HOLE
> BASE THING, BUT SINCE THE FREEBSD KERNEL DOES NOT
PRESERVE %GS AND
> %FS, WHAT IS THE PURPOSE OF AMD64_SET_XXBASE()?

THE %FS, %GS REGISTERS AND FSBASE AND GSBASE MSRS ARE
SEPARATE
REGISTERS. WHEN YOU WRITE %GS:OFFSET, YOU ACTUALLY GET
(GSBASE+OFFSET),
SO THE ACTUAL VALUE OF %GS DOESN'T MATTER.

THERE ARE TWO WAYS TO SET GSBASE. ONE IS BY USING THE
PRIVILEGED
INSTRUCTION WRMSR TO SET GSBASE DIRECTLY (FULL 64BIT BASE
ADDRESS),
WHICH IS WHAT AMD64_SET_GSBASE() EXPOSES TO USERLAND. THE
OTHER IS BY
LOADING A DESCRIPTOR SELECTOR IN %GS IN WHICH CASE GSBASE
WILL BE SET
TO THE BASE ADDRESS (ONLY 32BIT BASE ADDRESS) OF A
DESCRIPTOR ENTRY IN
EITHER THE GDT OR LDT.

TO GET BACK TO WHAT YOU ARE TRYING TO DO, BECAUSE %GS ISN'T
PRESERVED,
I THINK YOU SHOULD AVOID WRITING TO IT AND INSTEAD STRICTLY
USE
AMD64_SET_GSBASE(). BUT FROM WHAT YOU'VE WRITTEN, I'M
GUESSING YOU'RE
ALREADY DOING THIS, SO THE NEXT THING TO TRY IS TO CREATE
THREADS WITH
PTHREAD_SCOPE_SYSTEM OR USE LIBTHR INSTEAD OF LIBPTHREAD,
BECAUSE IF
I'M NOT MISTAKEN, PTHREAD_SCOPE_PROCESS IN LIBPTHREAD
DOESN'T PRESERVE
GSBASE EITHER.
_______________________________________________
FREEBSD-EMULATIONFREEBSD.ORG MAILING LIST
HTTP://LISTS.FREEBSD.ORG/MAILMAN/LISTINFO/FREEBSD-EMULATION
TO UNSUBSCRIBE, SEND ANY MAIL TO
"FREEBSD-EMULATION-UNSUBSCRIBEFREEBSD.ORG"

On Wednesday 10 October 2007, Tijl Coosemans wrote:
> On Tuesday 09 October 2007 02:48:51 Mihai Donțu
wrote:
> > I have *one* more question: maybe I don't fully
understand the hole
> > BASE thing, but since the FreeBSD kernel does not
preserve %gs and
> > %fs, what is the purpose of amd64_set_XXbase()?
> 
> The %fs, %gs registers and fsbase and gsbase MSRs are
separate
> registers. When you write %gs:offset, you actually get
(gsbase+offset),
> so the actual value of %gs doesn't matter.
> 
> There are two ways to set gsbase. One is by using the
privileged
> instruction wrmsr to set gsbase directly (full 64bit
base address),
> which is what amd64_set_gsbase() exposes to userland.
The other is by
> loading a descriptor selector in %gs in which case
gsbase will be set
> to the base address (only 32bit base address) of a
descriptor entry in
> either the GDT or LDT.

Invaluable info. Thanks! 

> To get back to what you are trying to do, because %gs
isn't preserved,
> I think you should avoid writing to it and instead
strictly use
> amd64_set_gsbase(). But from what you've written, I'm
guessing you're
> already doing this, so the next thing to try is to
create threads with
> PTHREAD_SCOPE_SYSTEM or use libthr instead of
libpthread, because if
> I'm not mistaken, PTHREAD_SCOPE_PROCESS in libpthread
doesn't preserve
> gsbase either.

Well, I'm am not setting (loading) %gs, I *only* do
amd64_set_gsbase() and
expect that *all* instructions such as:
    mov %gs:0x10,%rax
to be valid (not segfault). I don't really care what the
value of %gs is
as long as *all* the instructions as the above work and
access the memory
specified in amd64_set_gsbase( addr ).

I was under the (wrong) impression that the value of %gs is
important,
that's why I wanted it preserved, but if you say:
> [...] When you write %gs:offset, you actually get
(gsbase+offset),
> so the actual value of %gs doesn't matter.
then I don't care if %gs' value gets lost over context
switches as long as
    mov %gs:0x10,%rax
and other such instructions, work.

However, it turns out that amd64_set_gsbase() is not enough
:( Either:
a) someone *does* set %gs (and is not me);
b) the 'gsbase' gets lost;

The thing is I've ported my emulator to Linux and there I
use modify_ldt()
and then some __asm__ voodoo to load %gs, because (quote
from man):
- "ARCH_SET_GS is disabled in some kernels."
- "Context switches for 64-bit segment bases are rather
expensive. It may
   be a faster alternative to set a 32-bit base using a
segment selector
   by setting up an LDT with modify_ldt(2) [...]"

Anyhoo, I'll try to use 'libthr' and see if this helps.

Thanks, again!

-- 
Mihai Donțu
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"

On Wednesday 10 October 2007, Tijl Coosemans wrote:
> > To get back to what you are trying to do, because
%gs isn't preserved,
> > I think you should avoid writing to it and instead
strictly use
> > amd64_set_gsbase(). But from what you've written,
I'm guessing you're
> > already doing this, so the next thing to try is to
create threads with
> > PTHREAD_SCOPE_SYSTEM or use libthr instead of
libpthread, because if
> > I'm not mistaken, PTHREAD_SCOPE_PROCESS in
libpthread doesn't preserve
> > gsbase either.

> Anyhoo, I'll try to use 'libthr' and see if this
helps.

... and success! Indeed: 'amd64_set_gsbase()' + 'libthr.so'
= love. 'libpthread.so'
is a no-no 

I'm not out of the woods yet, I still have some crashes, but
I suspect that's just
bad programming on my side.

Thanks,

-- 
Mihai Donțu
_______________________________________________
freebsd-emulationfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-emu
lation
To unsubscribe, send any mail to
"freebsd-emulation-unsubscribefreebsd.org"