Storing data packets using Hadoop

Hi all ,


Is it possible to store the 10000 data packets comming from
sockets using
Hadoop if so please reply me .


Thanks in advance


Dhayalan.G
-- 
View this message in context: http://www.nabble.com/Storing-da
ta-packets-using-Hadoop-tf4573828.html#a13055486
Sent from the Hadoop Dev mailing list archive at
Nabble.com.

Sure. It is possible, as long as you exclude Hadoop traffic
itself and 
you are not expecting to saturate the network interfaces
with combined 
traffic. Also because of the way data is written to the DFS,
you need a 
large buffer between capture and the client writing to DFS,
other wise 
you might drop packets. I would say the buffer should be at
least 2 to 3 
times the block size (default block size is 64MB).

Raghu.

Dhn wrote:
> Hi all ,
> 
> 
> Is it possible to store the 10000 data packets comming
from sockets using
> Hadoop if so please reply me .
> 
> 
> Thanks in advance
> 
> 
> Dhayalan.G

Raghu Angadi wrote:
> 
> 
> Thnaks,
> Following are my requirement please let me know is this
possible using
> hadoop
> Store raw datapackets - 1000 packets in 1 min  in to a
file say abc.dat
> For every min the file will be changed. new datapackets
should be appended
> to the file
> Search for the latest record for the specified 
truckid
> Should start other functionality parallely with the
critical message in
> the file record
> Process packet info and store the info in the RDBMS.
>  
> Sure. It is possible, as long as you exclude Hadoop
traffic itself and 
> you are not expecting to saturate the network
interfaces with combined 
> traffic. Also because of the way data is written to the
DFS, you need a 
> large buffer between capture and the client writing to
DFS, other wise 
> you might drop packets. I would say the buffer should
be at least 2 to 3 
> times the block size (default block size is 64MB).
> 
> Raghu.
> 
> Dhn wrote:
>> Hi all ,
>> 
>> 
>> Is it possible to store the 10000 data packets
comming from sockets using
>> Hadoop if so please reply me .
>> 
>> 
>> Thanks in advance
>> 
>> 
>> Dhayalan.G
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Storing-da
ta-packets-using-Hadoop-tf4573828.html#a13071369
Sent from the Hadoop Dev mailing list archive at
Nabble.com.

Dhn wrote:
> 
> Thnaks, 
> Following are my requirement please let me know is this
possible using
> hadoop 
> Store raw datapackets - 1000 packets in 1 min  in to a
file say abc.dat 
> For every min the file will be changed. new datapackets
should be appended
> to the file 
> Search for the latest record for the specified  truckid

> Should start other functionality parallely with the
critical message in
> the file record 
> Process packet info and store the info in the RDBMS. 
> 
> 
> Raghu Angadi wrote:
>> 
>> 
>> Thnaks,
>> Following are my requirement please let me know is
this possible using
>> hadoop
>> Store raw datapackets - 1000 packets in 1 min  in
to a file say abc.dat
>> For every min the file will be changed. new
datapackets should be
>> appended to the file
>> Search for the latest record for the specified 
truckid
>> Should start other functionality parallely with the
critical message in
>> the file record
>> Process packet info and store the info in the
RDBMS.
>>  
>> Sure. It is possible, as long as you exclude Hadoop
traffic itself and 
>> you are not expecting to saturate the network
interfaces with combined 
>> traffic. Also because of the way data is written to
the DFS, you need a 
>> large buffer between capture and the client writing
to DFS, other wise 
>> you might drop packets. I would say the buffer
should be at least 2 to 3 
>> times the block size (default block size is 64MB).
>> 
>> Raghu.
>> 
>> Dhn wrote:
>>> Hi all ,
>>> 
>>> 
>>> Is it possible to store the 10000 data packets
comming from sockets
>>> using
>>> Hadoop if so please reply me .
>>> 
>>> 
>>> Thanks in advance
>>> 
>>> 
>>> Dhayalan.G
>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Storing-da
ta-packets-using-Hadoop-tf4573828.html#a13071379
Sent from the Hadoop Dev mailing list archive at
Nabble.com.