How to integrate C/C++ library in Hadoop Map/Reduce

Hello, 
  We had text processing libraries, which developed by
C/C++, and would like
to
  use Hadoop to process large data set. It looks like not
much talking about
this issue
  (to call C/C++ library under Hadoop Map/Reduce Java
codes)
  Any comment(or examples) to share how to do that?  Thanks
for your help.

ChaoChun
-- 
View this message in context: http://www
.nabble.com/How-to-integrate-C-C%2B%2B-library-in-Hadoop-Map
-Reduce-tf4278023.html#a12176819
Sent from the Hadoop Users mailing list archive at
Nabble.com.

Hi,

ChaoChun Liang wrote:
> Hello, 
>   We had text processing libraries, which developed by
C/C++, and would like
> to
>   use Hadoop to process large data set. It looks like
not much talking about
> this issue
>   (to call C/C++ library under Hadoop Map/Reduce Java
codes)
>   Any comment(or examples) to share how to do that? 
Thanks for your help.
> 

I believe the hadoop *pipes* is what you need... it's coming
up in the 
0.14.0 release in next few days:

http://svn.
apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/ha
doop/mapred/pipes/package.html

Here is some context: http:
//issues.apache.org/jira/browse/HADOOP-234

Arun

> ChaoChun

Arun C Murthy wrote:
> 
> I believe the hadoop *pipes* is what you need... it's
coming up in the 
> 0.14.0 release in next few days:
> 
> http://svn.
apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/ha
doop/mapred/pipes/package.html
> 

Why not use JNI? It looks like will be easy for the
implementation.

ChaoChun
-- 
View this message in context: http://www
.nabble.com/How-to-integrate-C-C%2B%2B-library-in-Hadoop-Map
-Reduce-tf4278023.html#a12341115
Sent from the Hadoop Users mailing list archive at
Nabble.com.

On Aug 26, 2007, at 8:28 PM, ChaoChun Liang wrote:

> Why not use JNI? It looks like will be easy for the
implementation.

One of the desires was to minimize the impact on the C++
code. In  
particular, requiring a JVM to be linked in with the
application is  
pretty heavy. Additionally, the JNI interfaces are really
expensive  
to move large amounts of data across.

-- Owen