FYI: An FMA related project just proposed on
opensolaris-discuss.
Folowup-To on this email is set to opensolaris-discuss sun.com
Gavin
-------- Original Message --------
Subject: [osol-discuss] Project proposal: full FMA support
for generic x86 machine-check architecture
Date: Thu, 10 May 2007 10:32:04 +0100
From: Gavin Maltby <Gavin.MaltbySun.COM>
To: Open Solaris <opensolaris-discussopensolaris.org>
The Project
This project proposes improving the generic support of x86
CPUs that implement the Machine Check Architecture (MCA).
It
also proposes to create a model-specific plugin interface
whereby
model-specific MCA implementation may supplement and
extend
the generic support rather than stand alone as today.
Finally, the
project proposes to redesign the cpu-module interface such
that
the MCA support it provides will be usable in both
virtualized and bare-metal environments.
Background
Prior to the "amd64 FMA" project (putback in
February 2006)
support for MCA was woeful, with no intelligent enabling
of
error detectors, no polling for errors that do not
produce a machine check, and a clumsy machine-check
handler
which barfed an incomplete and raw dump of error telemetry
to
the console before panic. There was no model-specific MCA
support whatsoever.
The "amd64 FMA" project, putback in February
2006, delivered
detailed model-specific support for AMD family 0xf
revisions B to E
(also known as AMD Opteron/Athlon64/Turion64, K8 etc;
revisions
B to E all have DDR1 memory) and moderately improved
generic
MCA support.
The AMD family family 0xf revs B to E support included:
- performing MCA and model-specific configuration
beyond what the BIOS had setup
- enabling all MCA banks and machine-check on all
uncorrectable error types
- an error poller for errors that do not machine-check
(such as correctable memory errors)
- an error handler that collects all telemetry and
bakes it into a model-specific error report which
is dispatched for logging and diagnosis via a robust
mechanism that will carry telemetry across a
panic/reboot
cycle
- detailed knowledge of model-specific error types, able
to produce an ereport error class from the raw
telemetry
and to specify the appropriate ereport payload members
to include for that error class
- a memory-controller driver able to discover all aspects
of memory configuration and to map system addresses to
memory resource
- a topology enumerator to describe cpu and memory
topology
and properties to the topology library, for use in
diagnosis software
- eversholt diagnosis rules to diagnose all telemetry
- corresponding fault classes and sun.com/msg knowledge
articles
The generic support, applied to anything other than the AMD
models
above including all Intel processors and any AMD models
outside the
model range above, included:
- generic MCA detector and error type enabling
- a machine check exception handler which dumps all
MCA telemetry to the console, still in raw form
- no error poller
- no preparation of ereports (even for panic events), and
therefore no diagnosis of any kind
The above support was backported to Solaris 10 Update 2.
The model-specific AMD support was updated in build 51 to
include
AMD family 0xf revision F and G support. Those revisions
use
DDR2 memory, and most work was required in the
memory-controller
driver. This additonal support has been backported to
Solaris 10
Update 4 (pending release at this time).
The Problem
1) The first problem is that the generic cpu MCA support is
light
on features:
- it lacks an error poller; we won't log memory
correctable ECC errors, for example
- it does not raise ereports for logging and diagnosis,
even for errors that result in a panic
- on an error that results in a panic at best you have
some raw, uninterpreted error register values and no
error classification or interpretation; if the system
lacks console logging no evidence at all may remain
when the system resets.
This generic support is applied to all Intel processors,
and to
AMD processors not handled by already-delivered AMD
model-specific
support. Where a platform with full model-specific
support
is upgradeable to new processors that drop in to existing
sockets
and where the new processor family/model/stepping is no
longer
covered by the existing model-specific support, the
platform
after upgrade will regress to the dumb generic cpu
support.
So for all Intel and for new AMD processors there is a
window of
OpenSolaris builds during which MCA support is very light -
it
takes some time to implement full model-specific support
for a cpu.
Improving generic cpu support will raise the base level of
support
for all processors that implement MCA/MCE features. Purely
from the
Sun point of view, and not a concern for OpenSolaris
itself, the
delay in introducing new model-specific support is further
stretched in having to backport to a suitable Solaris 10
Update
release; it is also possible that products based on a new
cpu model can revenue-release before corresponding full
FMA
support for the cpu model is available in Solaris 10.
2) The second problem is that the current cpu module
implementations (generic and AMD family 0xf) and the
cpu module interface itself are completely invalildated
when
not running natively. A separate project will be proposed
for
FMA/Xen support, but generic cpu MCA support and the cpu
module
interface changes required for the current project
proposal
should be made with that requirement in mind.
3) A third problem is that the current cpu module design
leads to substantial code duplication across
model-specific
implementations (e.g., implementing an error poller in
each).
This should not be necessary given the common base MCA that
they
adhere to.
Key Objectives
1) Enrich the generic cpu support to include an error
poller,
error type classification (as far as the architectural
definition
allows), raising of ereports capturing all the telemetry
for
logging and some degree of diagnosis (again as much as is
possible
generically).
2) Design a new cpu model-specific MCA module interface
which
will supplement generic cpu MCA support to avoid code
duplication
and improve maintainability. This new interface should be
rich enough to allow the refactoring and redelivery of
exising
AMD model-specific support, and should be designed with
upcoming
Intel model-specific support in mind.
3) Identify vendor-specific but nevertheless fairly
widely-
applicable (to many models from that vendor) features that
can be captured in an intermediate level of model-specific
support for new CPU models from that vendor. For example
in
the absence of a fully-featured memory-controller driver we
may
be unable to determine the full memory topology and
provide
full error address to memory resource translation, but
both
AMD and Intel in recent models have hardware registers
that
can identify where errors have occured (albeit at a lower
resolution than a full memory-controller driver may be able
to
provide). Such support could provide a level of
"generic AMD"
and "generic Intel" support for new cpu releases
that is
better than the lowest common denominator generic support,
and which is later not loaded in favour of full
model-specific
support once that is available.
4) Redesign the cpu module interface to cope with running
both
natively and in a virtualized setting, Xen Solaris domain
0
in particular. The new interface should not assume any
1:1
relationship between Solaris logical cpus (cpu_t
structures)
and real physical cores on the system, should not assume
that
being bound to a Solaris cpu means we can perform reads
from
hardware registers all from the same real core, etc.
Instead the points at which we acquire telemetry and other
hardware details should be well-defined, and the
information
gathered via calls into a newly-defined native or
virtualized layer.
Implementation of the virtualized layer itself is left to
another
project.
5) The project deliverables must include some degree of
hardware
error-injection, or of error simulation if that is not
possible.
Existing test harnesses should be extended to include
generic
cpu scenarios.
6) As with any FMA project, this should include an FMA
portfolio
detailing all events, diagnosis thereof, ASRU boundaries
etc.
To date there are no open FMA portfolios so those outside
of Sun
will have little idea of what this involves - that will
become
clear during the progres of the project.
7) This project will not deliver full model-specific
support
for any Intel cpus nor for upcoming "AMD family
0x10" processors.
It should, however, proceed with the needs of those
projects
in mind and should colloborate with those projects to
understand
those needs.
Community Participation
Participation and contribution from the OpenSolaris
community is
encouraged. This project will use public documentation
from both
AMD and Intel, available from the websites of those
companies.
As such, there is no vendor NDA or similar barrier to
community
participation. It may be necessary for some design
decisions to
be made by or in conjunction with those with access to NDA
documentation, but these are expected to be the exception.
Error injector sources are currently in the closed tree.
Otherwise
all applicable source is available in OpenSolaris.
The event registry is now available externally to Sun, so
there is no
reason that the FMA portfolio cannot be prepared and
published
externally as part of this project.
Gavin
--
Gavin Maltby, Solaris Kernel Development.
_______________________________________________
opensolaris-discuss mailing list
opensolaris-discussopensolaris.org
--
Gavin Maltby, Solaris Kernel Development.
_______________________________________________
fm-discuss mailing list
fm-discussopensolaris.org
|
