(ISF) Seeking a model for "data standards"

Hello all,

First, a tip of the hat to Deborah for hosting and monitoring this echange.

I am working with the American Friends Service Committee in Philadelphia and one project is to establish a standard database platform across their organization worldwide. We are currently evaluating hosted CiviCRM as a candidate.

An aspect of the project will be the ability to consolidate some database that now stand alone, and to streamline the process of exchanging records across databases that will remain independent.

A key to success will be the ability to standardize data - or at least to provide a recommended set of data standards.

By this I am referring to issues like:

always separate First and Last Name
addresses should adhere to postal standards

other issues:

what set of prefixes to provide in a drop down?
what set of suffixes?

etc.

This seems like the classic "reinvent the wheel"; project and I know that Development Departments do fabulous careful work in this realm.

If anyone had a Guide they used and would be willing to share it? Or if anyone knows of a great online resource? I would greatly appreciate it.

We would use it as a stating point and adapt it to our specific needs.

Best,

Robert Leming
Rock River Star
Philadelphia

Messages in this topic (1) Reply (via web post) | Start a new topic

.

In my mind, the question of data standards has become far more interesting than database schema (or even application integration). Because the best tools that we use to manage our data (CiviCRM certainly included) allow us to aggregate as well as publish data in a way that can completely separate content from form, we can now talk about data in terms of what discrete items do we need. Additionally, data standards for tings like person records no longer need a group of people to sit in a room to agree on what fields represent the data standard and in what format. These questions are being solved more organically. Essentially, their are two environments that an organization has to consider for how to represent their data.

1) Private Data Environment: this is where questions of form, function, workflow, tools (CiviCRM, Salesforce.com, DIA, ConActive, etc) and content all come in to play. These are personal decision that hinge on the individual requirements of a specific organization.

2) Public Data Environment: This involves questions of content primarily. (Obviously form is involved relative to how the data will be displayed on a web page or other publication but these decisions are completely divorced from the content decisions.) Using eh example of a person record, there are many places than one could go to see how a person is "described". Certainly hcard (http://microformats.org/wiki/hcard) standard is a great place to start and microformats in general (http://microformats.org/wiki/microformats) are great place to check first for what to include in a public data set. Another good place to look is Google Base. Google Base is an attempt by Google to allow detailed searches against explicitly defined data sets. Given Google's position in the search engine world, it is worth taking a long hard look any time they decl are something as a standard for how to represent a data set. (http://base.google.com/base/s2?a_n0=people+profiles&a_y0=9&hl=en&gl=US). Then the default fields that exist in any of the major CRM-ish tools (mentioned above) are worth considering as a standard as these field sets were created with best practices in mind.

Te next question would be the specifics of how to get the data out of your CRM tool and then "publish" this data elsewhere (web page as Microformat, Google Base, another "central" DB, etc). In my mind, they ability of a CRM-ish tool to accomplish this sort of task should be a defining characteristic for choosing that tool. Does the tool allow, via an API (best case scenario), access to the application source code, or other add-on functionality, the ability access and publish your data anywhere you want.

Do not use closed systems. They are bad for you.

-----original message-----
>>I am working with the American Friends Service Committee in Philadelphia and one project is to establish a standard database platform across their organization worldwide. We are currently evaluating hosted CiviCRM as a candidate. An aspect of the project will be the ability to consolidate some database that now stand alone, and to streamline the process of exchanging records across databases that will remain independent. A key to success will be the ability to standardize data - or at least to provide a recommended set of data standards.>>

Messages in this topic (2) Reply (via web post) | Start a new topic

.

Hi Robert,

First off, I think the advice you've received from Mr. Fuller and from Steve is right (ha, a pun you see). But, that said, I think you're also asking for more practical advice. And, here we run into the great problem with standards - there are so many of them to choose from. (consequently, they're not really standards at all, unfortunately). The great, unified, nonprofit data standard has been a quest of mine for years - Don Pedro and I, well, we're still tipping at windmills.

There have been advances - and I have a strange feeling that we'll see some activity around this quite soon (he said with a wink). The one previous attempt - called OPX for Open Philanthropic eXchange - kind of died an untimely death. It was, in my opinion, too soon to market, and not driven by demand. Moreover, it was specifically focused on donors, gifts, volunteers, members and events.

Direct to your question(s):

For data transport (between systems) XML is the only standard. XML is terrific because it is a language for describing not only data, but the data's structure as well. You may know this, but XML is fairly simple. It uses parameter/value pairs. I once heard it described in relation to HTML. HTML is a language for describing what a page looks like, XML is a language for describing what a page contains. It is, in a nutshell, known as "structured data."; It is terrific.

Here's a simple example:

<First Name>&lt;Gavin>

<Last Name> <Clabaugh>

<Type&gt; <Chump&gt;

&lt;SubType&gt; <Fool&gt;

That's a parameter/value pair. It describes the field (first name) and then it provides the value (Gavin), etc, right down to my essential
chumpness.

Using this structure, XML can get fairly complicated fairly quickly, but it easily can be used to provide not only data values, but the structure of the data itself - making it easy to communicate between applications.

Now.. with that out of the way, as I read your question, I actually see two questions:

1) are there standards governing data structure, and;

2) are there standards governing value choices.

So, as an example, let's look at a "people" record (since you used it in your message):

Standard fields might include:

Honorific

First Name

Middle Initial

Last Name

Generational Qualifier (GQ)

Suffix

Standard value choices for those might include:

Honorific: Mr. Ms. Mrs. Dr. Rev. Hon. Brother, Sister, Major, Sgt, Lt., etc.

GQ: Jr. II, III, IV,

Suffix: Esq, Ph.d,

So, where to go for standards - as there have been many standards - I let the content drive the standard to some extent. For people and organizations: I'd look at hCard (you may know this a vCard). Look here for the spec and it's explanation: http://microformats.org/wiki/hcard. The vCard spec will do the same thing.

And, for events, calendar stuff and the like, I'd look at the same place for the hCalendar microformat. You may know that as iCal. Look here: http://microformats.org/wiki/hcalendar

Here, too, is where the folks at CiviCRM and Salesforce and the host of other folks [you know who you are] might open their standards up and provide the world with the fields, and value choices they've built into their systems. It would be easy enough to ascertain by looking... but, it would also be easy enough for them to just list them someplace. Wouldn't that be grand, now. Anyway, I digress from the practical.

For the VALUEs question, strangely, I go further afield. This question existed before there were things like XML, and HTML, and 'puters.

I personally use AP Stylebook. There are others, but I like the AP one. It lists how and when you use various things, like the abbreviation for "Brother" (as in Brother Xavier) is not Bro, nor is the abbreviation for Sister (as in Sister Mary Elephant), "Sis."

There are dozens of choices here. I like the AP, but others like the Chicago Manual of Style, while others may like the Oxford Guide. The "value" choices are pretty similar, as are the abbreviations and usages guidelines.

Look here: http://www.amazon.com/Stylebooks-Publication-Manuals/lm/2B8DXK6QE2T0B

Hope this is helpful or have I missed the boat and misunderstood your questions?

Regards

Gavin

-----original message-----
>&gt;I am working with the American Friends Service Committee in Philadelphia and one project is to establish a standard database platform across their organization worldwide. We are currently evaluating hosted CiviCRM as a candidate. An aspect of the project will be the ability to consolidate some database that now stand alone, and to streamline the process of exchanging records across databases that will remain independent. A key to success will be the ability to standardize data - or at least to provide a recommended set of data standards.>>

Messages in this topic (4) Reply (via web post) | Start a new topic

.

Robert,

As for the data interchange standard, then XML is the only viable option. Regarding the content, the waters muddy quickly. However, two initiatives are clearly in the lead:

1. HUD's Homeless Management Information System data integration group: http://www.hmis.info/resources.asp?cvid=156&ccid=1

2. AIRS Taxonomy for the human services: http://www.211taxonomy.org/

Have fun,

Dale

Dale Fitch, Ph.D., LMSW
Assistant Professor
University of Michigan
School of Social Work
1080 South University, Rm 2794
Ann Arbor, MI 48109-1106
(734)763-6286
(734)763-3372 (FAX)
dale%40umich.edu">daleumich.edu
http://www.ssw.umich.edu/faculty/profile-dale.html

Messages in this topic (6) Reply (via web post) | Start a new topic

.

I definitely think that an OPX type solution is on the horizon. APIs and other integrations are one of the biggest trends I've seen in the last few years and I believe as these databases evolve from peer to peer, something or someone is going to see a profit in an organization of that data. Unfortunately, I think it's the profit motive that will bring something like this together. I have been working on integrations since 97 and custom integrations mainly between Blackbaud's fundraising products and other CRMs ticketing systems etc. I've found that as more people want it, someone always comes in and develops a high end interface between the products. Eventually something will come along to integrate it for an entire database suite. Though I'd love to see something free and easy enough to install without $1000+ of consulting.

Messages in this topic (7) Reply (via web post) | Start a new topic

.