Messages In This Digest (14
Messages)
Messages
- 1a.
-
Posted by: "Frank Smith" fsmith hoovers.com?Subject= Re%3A%20delay%20tape%20write%3F">
fsmith hoovers.com
Wed Aug 22, 2007 8:58 am (PST)
Paul Bijnens wrote:
& gt; On 2007-08-22 01:35, Frank Smith wrote:
>> Also, on a somewhat related note, if taper were to wait until
>> all the dumps finished, and runtapes is greater than 1, could there
>> be a taperorder parameter similar to dumporder that would do a
>> largest-fit write to tape, and avoid trying to write a dump too
>> large to fit in the remaining space on a tape? I haven't dug in
>> the code, but taper just seems to work in a FIFO fashion, and
>> writes until it hits EOT and then starts anew on the next tape with
>> the failed dump. For runtapes = 2, largest fit would be sufficient,
>> but for runtapes >=3 perhaps another strategy would be better,
>> although possibly not by enough to justify the extra code.
>
> You're describing the parameter "taperalgo", I believe.
Thanks, that sounds like what I was asking for. Evidently I missed
it when that feature was added.
> Because there is no "delayed" tapewriting currently, you can
> only influence the result of the taperalgo by tuning the
> "dumporder" and the number of dumpers ("inparallel") as well.
> See my explanation in:
>
> http://wiki.zmanda.com/index.php/How_To:Fill_tapes_to_100%25
>
> Having an option to delay the tapewriting somehow, would make
> this a lot simpler, especially for those extremely large and
> fast tapes currently on the market.
Yes, a delayed start to taper would certainly make it easier to
better fit the dumps on a tape to maximize tape usage. I've tried
playing with the dumporder but I still often end up wasting tape
with Amanda trying to write a large dump when there is only a medium
amount of space left on the current tape. I'll try adding in the
taperalgo setting to see if that helps, but it seems that might
not be a complete solution until delayed taping is implemented.
Thanks again,
Frank
--
Frank Smith fsmith%40hoovers.com">fsmith hoovers.com
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501
fsmith  hoovers.co m?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1b.
-
Posted by: "Dustin J. Mitchell" dustin zmanda.com?Subject= Re%3A%20delay%20tape%20write%3F">
dustin zmanda.com
Wed Aug 22, 2007 9:31 am (PST)
On Wed, Aug 22, 2007 at 10:54:22AM -0500, Frank Smith wrote:
> Yes, a delayed start to taper would certainly make it easier to
> better fit the dumps on a tape to maximize tape usage. I've tried
> playing with the dumporder but I still often end up wasting tape
> with Amanda trying to write a large dump when there is only a medium
> amount of space left on the current tape. I'll try adding in the
> taperalgo setting to see if that helps, but it seems that might
> not be a complete solution until delayed taping is implemented.
This problem is significantly less serious if you use split dumps.
Split dumps effectively set the maximum size of an on-tape file, which
becomes the maximum amount of wasted space on a tape. So if you have
800G tapes, and your split size is, say, 5G, then you will always fill
at least 795G of the first tape in your run (assuming you're using
runtapes > 1).
It's interesting to note that Amanda has transitioned from a state where
the tapes were the slow part of the equation to the opposite. In its
original design, Amanda tried to get the taper up and running as soon as
possible (potentially starting with dumps already in holding, and then
greedily taping dumps as they are completed) on the assumption that the
taper would be the major consumer of backup time.
Dustin
--
Dustin J. Mitchell
Storage Software Engineer, Zmanda, Inc.
http://www.zmanda.com/
dustin  zmanda.com ?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1c.
-
Posted by: "Matt Hyclak" hyclak math.ohiou.edu?Subject= Re%3A%20delay%20tape%20write%3F">
hyclak math.ohiou.edu
Wed Aug 22, 2007 9:48 am (PST)
On Wed, Aug 22, 2007 at 11:28:42AM -0500, Dustin J. Mitchell enlightened us:
> > Yes, a delayed start to taper would certainly make it easier to
> > better fit the dumps on a tape to maximize tape usage. I've tried
> > playing with the dumporder but I still often end up wasting tape
> > with Amanda trying to write a large dump when there is only a medium
> > amount of space left on the current tape. I'll try adding in the
> > taperalgo setting to see if that helps, but it seems that might
> > not be a complete solution until delayed taping is implemented.
>
> This problem is significantly less serious if you use split dumps.
> Split dumps effectively set the maximum size of an on-tape file, which
> becomes the maximum amount of wasted space on a tape. So if you have
> 800G tapes, and your split size is, say, 5G, then you will always fill
> at least 795G of the first tape in your run (assuming you're using
> runtapes > 1).
>
Some of us are still running 2.4.5 
And the problem I described in my previous mail is rare - once a quarter
when I do archival full dumps of everything. Normal day-to-day operation is
to let backups spool to holding disk for a week, so autoflush does its thing
and by the time the big dumps are done, taper is still flushing old dumps.
> It's interesting to note that Amanda has transitioned from a state where
> the tapes were the slow part of the equation to the opposite. In its
> original design, Amanda tried to get the taper up and running as soon as
> possible (potentially starting with dumps already in holding, and then
> greedily taping dumps as they are completed) on the assumption that the
> taper would be the major consumer of backup time.
>
Writing to tape is usually something you can do even during peak production
hours. It's getting all the data to holding disk that is in the time crunch
- especially if there are DBs that have to go offline, etc. This leads to as
many parallel processes as you can reasonably do with your hardware. Drive
space is cheap, bandwidth is cheap, and processing power is cheap - writing
to tape will always be slow, but generally doesn't interrupt production
processes if it's just shoving data down the scsi bus. From this
perspective, an admin only has a few hours to get dumps done, but nearly 24
hours to get it all to tape (assuming you don't actually want to *restore*
anything, or happen to have another drive to do so with).
My $.02.
Matt
--
Matt Hyclak
Department of Mathematics
Department of Social Work
Ohio University
(740) 593-1263
hyclak  math.ohiou .edu?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1d.
-
Posted by: "Matt Hyclak" hyclak math.ohiou.edu?Subject= Re%3A%20delay%20tape%20write%3F">
hyclak math.ohiou.edu
Wed Aug 22, 2007 10:03 am (PST)
On Wed, Aug 22, 2007 at 10:54:22AM -0500, Frank Smith enlightened us:
> Paul Bijnens wrote:
> > On 2007-08-22 01:35, Frank Smith wrote:
> >> Also, on a somewhat related note, if taper were to wait until
> >> all the dumps finished, and runtapes is greater than 1, could there
> >> be a taperorder parameter similar to dumporder that would do a
> >> largest-fit write to tape, and avoid trying to write a dump too
> >> large to fit in the remaining space on a tape? I haven't dug in
> >> the code, but taper just seems to work in a FIFO fashion, and
> >> writes until it hits EOT and then starts anew on the next tape with
> >> the failed dump. For runtapes = 2, largest fit would be sufficient,
> >> but for runtapes >=3 perhaps another strategy would be better,
> >> although possibly not by enough to justify the extra code.
> >
> > You're describing the parameter "taperalgo", I believe.
>
> Thanks, that sounds like what I was asking for. Evidently I missed
> it when that feature was added.
>
> > Because there is no "delayed" tapewriting currently, you can
> > only influence the result of the taperalgo by tuning the
> > "dumporder" and the number of dumpers ("inparallel") as well.
> > See my explanation in:
> >
> > http://wiki.zmanda.com/index.php/How_To:Fill_tapes_to_100%25
> >
> > Having an option to delay the tapewriting somehow, would make
> > this a lot simpler, especially for those extremely large and
> > fast tapes currently on the market.
>
> Yes, a delayed start to taper would certainly make it easier to
> better fit the dumps on a tape to maximize tape usage. I've tried
> playing with the dumporder but I still often end up wasting tape
> with Amanda trying to write a large dump when there is only a medium
> amount of space left on the current tape. I'll try adding in the
> taperalgo setting to see if that helps, but it seems that might
> not be a complete solution until delayed taping is implemented.
>
Agreed - I have my setup dump the largest DLEs first, and tape the largest
DLEs first, but it always ends up that there are 3 or 4 very large DLEs
(e.g. user home directories) and the remaining inparallel processes start
dumping smaller DLEs which finish long before the 3 of 4 large DLEs and
taper starts taping as soon as one is complete. I get a lot of start/stop on
the tape drive this way, and if dumping enough data end up wasting tape as
well.
The delayed taping, either by size or time (e.g. start taping 1 hour after
dumps have started) would be useful. Size probably moreso than time, but
I'll take either at this point.
Matt
--
Matt Hyclak
Department of Mathematics
Department of Social Work
Ohio University
(740) 593-1263
hyclak  math.ohiou .edu?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1e.
-
Posted by: "Jon LaBadie" jon jgcomp.com?Subject= Re%3A%20delay%20tape%20write%3F">
jon jgcomp.com
Wed Aug 22, 2007 12:20 pm (PST)
On Tue, Aug 21, 2007 at 09:09:24AM -0700, Christopher McCrory wrote:
> Hello...
>
> I seem to remember a feature request for delaying the start of writing to tape until X%/Xbytes are done dumping and ready to write. It was on sourceforge? zmanda wiki? I cannot seem to find it now. Does anyone know where it was?
>
Good memory, it was 2 years ago.
Orion Poplawski submitted a 2.4.5 patch called 'taperwait'. This was not
incorporated as Jean-Louis felt a more comprehensive solution was possible
and needed. Here are JLM's initial comments on OP's submission:
== From: Jean-Louis Martineau < martinea%40iro.umontreal.ca">martinea iro.umontreal.ca>
== Date: Wed, 7 Sep 2005 13:31:06 -0400
== Subject: not taperwait
==
== I Orion,
==
== Your taperwait patch might be useful but it doesn't solve some of the problem
== in the taper scheduling.
==
== eg. 1- I don't want to use a tape every day, but I want to delay taping.
== 2- I don't want to use a tape every day, but I want to start taping as soon as posible.
== 3- My dump size is 120% of my tape size. I don't want to use 2 tapes every day.
== 4- A want to keep many days of dump on holding disk before I write them to tape (for faster recovery).
==
==
== I think that amanda should fill a tape as fast as it can when it started
== to write to a tape, the question is
==
== When amanda should start to write to a new tape?
==
== I will introduce two new options: taperstart and taperflush
==
== taperstart x% y%: default (0 0)
== - start a new tape if x% is already dumped and on holding disk and
== if y% is already dumped or scheduled. (x<y)
== - a high x will improve tape fillment.
==
== taperflush [0|1]: default (1)
== - if 1, then flush all dump on HD to tape at the end of an amdump run.
== - if 0, don't flush them.
== - Will leaves between 0 and x% of dump on holding disk.
== - Must be use with autoflush.
== - Need high values for taperstart.
==
== - In all case, if a dump direct to tape is needed, a new tape will be started
== - If there is not enough space on holding disk, then a new tape will be
== started.
==
== your taperwait is equivalent to:
== taperstart 99999 99999
== taperflush 1
==
== amanda default setup:
== taperstart 0 0
== taperflush 1
==
== example 1: I don't want to use a tape every day, but I want to delay taping
== after the dump are done.
== if the dump_size is smaller than tape_size, then it is possible to not
== use a tape at every day:
== runtapes 1
== taperstart 100 100
== taperflush 0
== autoflush 1
== If the dump_size is 30% of the tape_size, then a tape will be used
== every 3 or 4 days
== Current workaround: don't use tape when amdump and run amflush regularly.
==
== example 2: I don't want to use a tape every day, but I want to start taping
== as soon as posible.
== runtapes 1
== taperstart 0 100
== taperflush 0
== autoflush 1
== If the dump_size is 30% of the tape_size, then a tape will be used
== every 3 or 4 days
== No current workaround.
==
== example 3: My dump size is 120% of my tape size. I don't want to use 2 tapes
== every day.
== runtapes 2
== taperstart 100 100 (or 0 100) (or 50 100)
== taperflush 0
== autoflush 1
== A second tape will be needed every 5 day.
== Current workaround: set runtapes to 1 and run amflush regularly.
==
== example 4: A want to keep many days of dump on holding disk before a write them
== to tape (for faster recovery).
== runtapes 1
== taperstart 600 600 (or 0 600) (or 111 600)
== taperflush 0
== autoflush 1
== You need a big holding disk. You will always have more than 5 tapes on HD.
==
==
== Do anyone have other taper scheduling requirement that we can't fix
== with these new options.
==
== Any comment?
==
== We can also introduce a new taperalgo (optimized) because using the biggest
== is not always the best.
==
== Jean-Louis
==
> Writing 500M to a 800G tape is irritating. I would be willing to sponsor a developer to work on this. Any takers?
If no one is able to implement JLM's suggestion, perhaps this is another candidate
for a GSOC project?
jl
--
Jon H. LaBadie jon%40jgcomp.com">jon jgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)
jon  jgcomp.com ?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1f.
-
Posted by: "Jon LaBadie" jon jgcomp.com?Subject= Re%3A%20delay%20tape%20write%3F">
jon jgcomp.com
Wed Aug 22, 2007 12:26 pm (PST)
On Wed, Aug 22, 2007 at 12:43:20PM -0400, Matt Hyclak wrote:
> On Wed, Aug 22, 2007 at 11:28:42AM -0500, Dustin J. Mitchell enlightened us:
> > > Yes, a delayed start to taper would certainly make it easier to
> > > better fit the dumps on a tape to maximize tape usage. I've tried
> > > playing with the dumporder but I still often end up wasting tape
> > > with Amanda trying to write a large dump when there is only a medium
> > > amount of space left on the current tape. I'll try adding in the
> > > taperalgo setting to see if that helps, but it seems that might
> > > not be a complete solution until delayed taping is implemented.
> >
> > This problem is significantly less serious if you use split dumps.
> > Split dumps effectively set the maximum size of an on-tape file, which
> > becomes the maximum amount of wasted space on a tape. So if you have
> > 800G tapes, and your split size is, say, 5G, then you will always fill
> > at least 795G of the first tape in your run (assuming you're using
> > runtapes > 1).
> >
>
> Some of us are still running 2.4.5 
>
> And the problem I described in my previous mail is rare - once a quarter
> when I do archival full dumps of everything. Normal day-to-day operation is
> to let backups spool to holding disk for a week, so autoflush does its thing
> and by the time the big dumps are done, taper is still flushing old dumps.
>
> > It's interesting to note that Amanda has transitioned from a state where
> > the tapes were the slow part of the equation to the opposite. In its
> > original design, Amanda tried to get the taper up and running as soon as
> > possible (potentially starting with dumps already in holding, and then
> > greedily taping dumps as they are completed) on the assumption that the
> > taper would be the major consumer of backup time.
> >
>
> Writing to tape is usually something you can do even during peak production
> hours. It's getting all the data to holding disk that is in the time crunch
> - especially if there are DBs that have to go offline, etc. This leads to as
> many parallel processes as you can reasonably do with your hardware. Drive
> space is cheap, bandwidth is cheap, and processing power is cheap - writing
> to tape will always be slow, but generally doesn't interrupt production
> processes if it's just shoving data down the scsi bus. From this
> perspective, an admin only has a few hours to get dumps done, but nearly 24
> hours to get it all to tape (assuming you don't actually want to *restore*
> anything, or happen to have another drive to do so with).
As you can tape during the day, might a crontab entry of "amdump ; amflush"
be a suitable workaround?
For the restore, while it doesn't help with older dumps, amrecover will
work with the dumps on holding disk.
--
Jon H. LaBadie jon%40jgcomp.com">jon jgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)
jon  jgcomp.com ?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1g.
-
Posted by: "Frank Smith" fsmith hoovers.com?Subject= Re%3A%20delay%20tape%20write%3F">
fsmith hoovers.com
Wed Aug 22, 2007 12:53 pm (PST)
Dustin J. Mitchell wrote:
& gt; On Wed, Aug 22, 2007 at 10:54:22AM -0500, Frank Smith wrote:
>> Yes, a delayed start to taper would certainly make it easier to
>> better fit the dumps on a tape to maximize tape usage. I've tried
>> playing with the dumporder but I still often end up wasting tape
>> with Amanda trying to write a large dump when there is only a medium
>> amount of space left on the current tape. I'll try adding in the
>> taperalgo setting to see if that helps, but it seems that might
>> not be a complete solution until delayed taping is implemented.
>
> This problem is significantly less serious if you use split dumps.
> Split dumps effectively set the maximum size of an on-tape file, which
> becomes the maximum amount of wasted space on a tape. So if you have
> 800G tapes, and your split size is, say, 5G, then you will always fill
> at least 795G of the first tape in your run (assuming you're using
> runtapes > 1).
Yes, that would be more space efficient, but I'm one of the old-fashioned
types that wants to be able to do bare-metal restores using basic commands,
and my understanding was that split dumps added complexity in this scenario.
I recently had to do such a recovery on one of my Amanda servers, and
I think this ability is one of Amanda's best features.
Frank
>
> It's interesting to note that Amanda has transitioned from a state where
> the tapes were the slow part of the equation to the opposite. In its
> original design, Amanda tried to get the taper up and running as soon as
> possible (potentially starting with dumps already in holding, and then
> greedily taping dumps as they are completed) on the assumption that the
> taper would be the major consumer of backup time.
>
> Dustin
>
--
Frank Smith fsmith%40hoovers.com">fsmith hoovers.com
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501
fsmith  hoovers.co m?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1h.
-
Posted by: "Frank Smith" fsmith hoovers.com?Subject= Re%3A%20delay%20tape%20write%3F">
fsmith hoovers.com
Wed Aug 22, 2007 12:55 pm (PST)
Jon LaBadie wrote:
& gt;
> As you can tape during the day, might a crontab entry of "amdump ; amflush"
> be a suitable workaround?
>
How can you get amdump to not write to tape? I know you can
specify an invalid tape device in the config, or not have a tape
in your drive, to prevent amdump from writing to tape, but either
of those would also prevent amflush from working.
Frank
--
Frank Smith fsmith%40hoovers.com">fsmith hoovers.com
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501
fsmith  hoovers.co m?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1i.
-
Posted by: "Paul Bijnens" Paul.Bijnens xplanation.com?Subject= Re%3A%20delay%20tape%20write%3F">
Paul.Bijnens xplanation.com
Wed Aug 22, 2007 1:26 pm (PST)
On 2007-08-22 21:48, Frank Smith wrote:
> Jon LaBadie wrote:
>> As you can tape during the day, might a crontab entry of "amdump ; amflush"
>> be a suitable workaround?
>>
>
> How can you get amdump to not write to tape? I know you can
> specify an invalid tape device in the config, or not have a tape
> in your drive, to prevent amdump from writing to tape, but either
> of those would also prevent amflush from working.
In recent versions (> 2.5.0 I believe) you can override amanda.conf
parameters on the commandline with "-o name=value"
Assuming we have one tape:
amdump -o tapedev=/no/such/tape -o reserve=0 daily; amflush daily
I'm not sure how to "comment out" an option, so if you use a changer
you might configure amanda.conf with some non-existing tapedev
and override that with a tpchanger option on amflush:
amdump daily; amflush -o tpchanger=chg-scsi daily
(the other changer parameters may be configured in amanda.conf,
because they get consulted only if the tpchanger parameter is set).
(Untested -- about to leave on holiday )
--
Paul Bijnens, xplanation Technology Services Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens%40xplanation.com">Paul.Bijnens xplanation.com
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, : , KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***********************************************************************
Paul.Bijne ns xplanation.com?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1j.
-
Posted by: "Matt Hyclak" hyclak math.ohiou.edu?Subject= Re%3A%20delay%20tape%20write%3F">
hyclak math.ohiou.edu
Wed Aug 22, 2007 1:45 pm (PST)
On Wed, Aug 22, 2007 at 10:10:55PM +0200, Paul Bijnens enlightened us:
> >Jon LaBadie wrote:
> >>As you can tape during the day, might a crontab entry of "amdump ;
> >>amflush"
> >>be a suitable workaround?
> >>
> >
> >How can you get amdump to not write to tape? I know you can
> >specify an invalid tape device in the config, or not have a tape
> >in your drive, to prevent amdump from writing to tape, but either
> >of those would also prevent amflush from working.
>
> In recent versions (> 2.5.0 I believe) you can override amanda.conf
> parameters on the commandline with "-o name=value"
>
> Assuming we have one tape:
>
> amdump -o tapedev=/no/such/tape -o reserve=0 daily; amflush daily
>
> I'm not sure how to "comment out" an option, so if you use a changer
> you might configure amanda.conf with some non-existing tapedev
> and override that with a tpchanger option on amflush:
>
> amdump daily; amflush -o tpchanger=chg-scsi daily
>
> (the other changer parameters may be configured in amanda.conf,
> because they get consulted only if the tpchanger parameter is set).
>
> (Untested -- about to leave on holiday )
See previous note about still running 2.4.5 
My current mode of operations is to eject (but not remove) a tape after
amdump. I can still mt load it if I need it. Then, I only stick the tape in
once a week to collect my dumps for the week. That's no problem.
I suppose I could do full dumps without a tape in as well, and amflush it
the next day, but it'd be nice not to have to.
Matt
--
Matt Hyclak
Department of Mathematics
Department of Social Work
Ohio University
(740) 593-1263
hyclak  math.ohiou .edu?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1k.
-
Posted by: "Chris Hoogendyk" hoogendyk bio.umass.edu?Subject= Re%3A%20delay%20tape%20write%3F">
hoogendyk bio.umass.edu
Wed Aug 22, 2007 1:46 pm (PST)
Fran k Smith wrote:
> Jon LaBadie wrote:
>
>> As you can tape during the day, might a crontab entry of "amdump ; amflush"
>> be a suitable workaround?
>>
>>
>
> How can you get amdump to not write to tape? I know you can
> specify an invalid tape device in the config, or not have a tape
> in your drive, to prevent amdump from writing to tape, but either
> of those would also prevent amflush from working.
my cron entries are:
0 15 * * 1-5 /usr/local/sbin/amcheck -m daily
45 0 * * 2-6 /usr/local/sbin/amdump daily
45 0 * * 0-1 /usr/local/sbin/amdump daily -o reserve=100 -o
tapedev=/dev/rmt/0n -o tpchanger=""
The weekend dumps are incremental only to holding disk.
The monday dump (Tuesday morning just after midnight) autoflushes those
along with its own dump.
You could do something similar with all the dumps going to holding and
amflush running off cron.
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) (*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
< hoogendyk%40bio.umass.edu">hoogendyk bio.umass.edu>
---------------
Erdös 4
hoogendyk  bio.umass. edu?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1l.
-
Posted by: "Jean-Louis Martineau" martineau zmanda.com?Subject= Re%3A%20delay%20tape%20write%3F">
martineau zmanda.com
Wed Aug 22, 2007 2:11 pm (PST)
The problem with the taperstart /taperflush features is that it require a
rewrite of the protocol between the driver and taper, that's why it is
not yet done.
The new protocol is done in the zmanda internal tree, this code will
eventually be merge to the community tree.
After that, it will be easy to implement the taperstart/taperflush features.
Jean-Louis
Jon LaBadie wrote:
> On Tue, Aug 21, 2007 at 09:09:24AM -0700, Christopher McCrory wrote:
>
>> Hello...
>>
>> I seem to remember a feature request for delaying the start of writing to tape until X%/Xbytes are done dumping and ready to write. It was on sourceforge? zmanda wiki? I cannot seem to find it now. Does anyone know where it was?
>>
>>
>
> Good memory, it was 2 years ago.
>
> Orion Poplawski submitted a 2.4.5 patch called 'taperwait'. This was not
> incorporated as Jean-Louis felt a more comprehensive solution was possible
> and needed. Here are JLM's initial comments on OP's submission:
>
> == From: Jean-Louis Martineau < martinea%40iro.umontreal.ca">martinea iro.umontreal.ca>
> == Date: Wed, 7 Sep 2005 13:31:06 -0400
> == Subject: not taperwait
> ==
> == I Orion,
> ==
> == Your taperwait patch might be useful but it doesn't solve some of the problem
> == in the taper scheduling.
> ==
> == eg. 1- I don't want to use a tape every day, but I want to delay taping.
> == 2- I don't want to use a tape every day, but I want to start taping as soon as posible.
> == 3- My dump size is 120% of my tape size. I don't want to use 2 tapes every day.
> == 4- A want to keep many days of dump on holding disk before I write them to tape (for faster recovery).
> ==
> ==
> == I think that amanda should fill a tape as fast as it can when it started
> == to write to a tape, the question is
> ==
> == When amanda should start to write to a new tape?
> ==
> == I will introduce two new options: taperstart and taperflush
> ==
> == taperstart x% y%: default (0 0)
> == - start a new tape if x% is already dumped and on holding disk and
> == if y% is already dumped or scheduled. (x<y)
> == - a high x will improve tape fillment.
> ==
> == taperflush [0|1]: default (1)
> == - if 1, then flush all dump on HD to tape at the end of an amdump run.
> == - if 0, don't flush them.
> == - Will leaves between 0 and x% of dump on holding disk.
> == - Must be use with autoflush.
> == - Need high values for taperstart.
> ==
> == - In all case, if a dump direct to tape is needed, a new tape will be started
> == - If there is not enough space on holding disk, then a new tape will be
> == started.
> ==
> == your taperwait is equivalent to:
> == taperstart 99999 99999
> == taperflush 1
> ==
> == amanda default setup:
> == taperstart 0 0
> == taperflush 1
> ==
> == example 1: I don't want to use a tape every day, but I want to delay taping
> == after the dump are done.
> == if the dump_size is smaller than tape_size, then it is possible to not
> == use a tape at every day:
> == runtapes 1
> == taperstart 100 100
> == taperflush 0
> == autoflush 1
> == If the dump_size is 30% of the tape_size, then a tape will be used
> == every 3 or 4 days
> == Current workaround: don't use tape when amdump and run amflush regularly.
> ==
> == example 2: I don't want to use a tape every day, but I want to start taping
> == as soon as posible.
> == runtapes 1
> == taperstart 0 100
> == taperflush 0
> == autoflush 1
> == If the dump_size is 30% of the tape_size, then a tape will be used
> == every 3 or 4 days
> == No current workaround.
> ==
> == example 3: My dump size is 120% of my tape size. I don't want to use 2 tapes
> == every day.
> == runtapes 2
> == taperstart 100 100 (or 0 100) (or 50 100)
> == taperflush 0
> == autoflush 1
> == A second tape will be needed every 5 day.
> == Current workaround: set runtapes to 1 and run amflush regularly.
> ==
> == example 4: A want to keep many days of dump on holding disk before a write them
> == to tape (for faster recovery).
> == runtapes 1
> == taperstart 600 600 (or 0 600) (or 111 600)
> == taperflush 0
> == autoflush 1
> == You need a big holding disk. You will always have more than 5 tapes on HD.
> ==
> ==
> == Do anyone have other taper scheduling requirement that we can't fix
> == with these new options.
> ==
> == Any comment?
> ==
> == We can also introduce a new taperalgo (optimized) because using the biggest
> == is not always the best.
> ==
> == Jean-Louis
> ==
>
>
>> Writing 500M to a 800G tape is irritating. I would be willing to sponsor a developer to work on this. Any takers?
>>
>
> If no one is able to implement JLM's suggestion, perhaps this is another candidate
> for a GSOC project?
>
> jl
>
martineau  zmanda.com ?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1m.
-
Posted by: "Dustin J. Mitchell" dustin zmanda.com?Subject= Re%3A%20delay%20tape%20write%3F">
dustin zmanda.com
Wed Aug 22, 2007 3:30 pm (PST)
On Wed, Aug 22, 2007 at 04:11:07PM -0400, Jon LaBadie wrote:
> A more elegant solution for recent versions is "configuration override":
>
> amdump ... -o tapedev=/dev/null ...
>
> though I've neither used nor tried this feature.
There was some discussion of this in late June, too, IIRC. You will
also need "-o tpchanger=" to disable any changer (which might otherwise
supply a tape device).
Dustin
--
Dustin J. Mitchell
Storage Software Engineer, Zmanda, Inc.
http://www.zmanda.com/
dustin  zmanda.com ?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
- 1n.
-
Posted by: "Jon LaBadie" jon jgcomp.com?Subject= Re%3A%20delay%20tape%20write%3F">
jon jgcomp.com
Wed Aug 22, 2007 3:35 pm (PST)
On Wed, Aug 22, 2007 at 02:48:26PM -0500, Frank Smith wrote:
> Jon LaBadie wrote:
> >
> > As you can tape during the day, might a crontab entry of "amdump ; amflush"
> > be a suitable workaround?
> >
>
> How can you get amdump to not write to tape? I know you can
> specify an invalid tape device in the config, or not have a tape
> in your drive, to prevent amdump from writing to tape, but either
> of those would also prevent amflush from working.
>
You are right.
Could do it crudely by editing the amanda.conf entry before and after amdump.
A more elegant solution for recent versions is "configuration override":
amdump ... -o tapedev=/dev/null ...
though I've neither used nor tried this feature.
--
Jon H. LaBadie jon%40jgcomp.com">jon jgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)
jon  jgcomp.com ?Subject=Re%3A%20delay%20tape%20write%3F">
Reply to sender
|
amanda-hackers@yahoogroups.com?Subject= Re%3A%20delay%20tape%20write%3F">
Reply to group
|
Reply via web post
Messages in this topic
(17)
|