List Info

Thread: Digest Number 1251




Digest Number 1251
country flaguser name
United States
2007-08-23 09:00:53

Messages In This Digest (14 Messages)

1a.
Re: delay tape write? From: Frank Smith
1b.
Re: delay tape write? From: Dustin J. Mitchell
1c.
Re: delay tape write? From: Matt Hyclak
1d.
Re: delay tape write? From: Matt Hyclak
1e.
Re: delay tape write? From: Jon LaBadie
1f.
Re: delay tape write? From: Jon LaBadie
1g.
Re: delay tape write? From: Frank Smith
1h.
Re: delay tape write? From: Frank Smith
1i.
Re: delay tape write? From: Paul Bijnens
1j.
Re: delay tape write? From: Matt Hyclak
1k.
Re: delay tape write? From: Chris Hoogendyk
1l.
Re: delay tape write? From: Jean-Louis Martineau
1m.
Re: delay tape write? From: Dustin J. Mitchell
1n.
Re: delay tape write? From: Jon LaBadie

Messages

1a.

Re: delay tape write?

Posted by: "Frank Smith" fsmithhoovers.com?Subject= Re%3A%20delay%20tape%20write%3F"> fsmithhoovers.com

Wed Aug 22, 2007 8:58 am (PST)

Paul Bijnens wrote:
> On 2007-08-22 01:35, Frank Smith wrote:
>> Also, on a somewhat related note, if taper were to wait until
>> all the dumps finished, and runtapes is greater than 1, could there
>> be a taperorder parameter similar to dumporder that would do a
>> largest-fit write to tape, and avoid trying to write a dump too
>> large to fit in the remaining space on a tape? I haven't dug in
>> the code, but taper just seems to work in a FIFO fashion, and
>> writes until it hits EOT and then starts anew on the next tape with
>;> the failed dump. For runtapes = 2, largest fit would be sufficient,
>>; but for runtapes >=3 perhaps another strategy would be better,
>> although possibly not by enough to justify the extra code.
>
> You're describing the parameter "taperalgo", I believe.

Thanks, that sounds like what I was asking for. Evidently I missed
it when that feature was added.

> Because there is no "delayed" tapewriting currently, you can
> only influence the result of the taperalgo by tuning the
> "dumporder" and the number of dumpers ("inparallel") as well.
> See my explanation in:
>
> http://wiki.zmanda.com/index.php/How_To:Fill_tapes_to_100%25
>
> Having an option to delay the tapewriting somehow, would make
>; this a lot simpler, especially for those extremely large and
> fast tapes currently on the market.

Yes, a delayed start to taper would certainly make it easier to
better fit the dumps on a tape to maximize tape usage. I've tried
playing with the dumporder but I still often end up wasting tape
with Amanda trying to write a large dump when there is only a medium
amount of space left on the current tape. I'll try adding in the
taperalgo setting to see if that helps, but it seems that might
not be a complete solution until delayed taping is implemented.

Thanks again,
Frank

--
Frank Smith fsmith%40hoovers.com">fsmithhoovers.com
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501

1b.

Re: delay tape write?

Posted by: "Dustin J. Mitchell" dustinzmanda.com?Subject= Re%3A%20delay%20tape%20write%3F"> dustinzmanda.com

Wed Aug 22, 2007 9:31 am (PST)

On Wed, Aug 22, 2007 at 10:54:22AM -0500, Frank Smith wrote:
> Yes, a delayed start to taper would certainly make it easier to
> better fit the dumps on a tape to maximize tape usage. I've tried
> playing with the dumporder but I still often end up wasting tape
>; with Amanda trying to write a large dump when there is only a medium
> amount of space left on the current tape. I'll try adding in the
> taperalgo setting to see if that helps, but it seems that might
> not be a complete solution until delayed taping is implemented.

This problem is significantly less serious if you use split dumps.
Split dumps effectively set the maximum size of an on-tape file, which
becomes the maximum amount of wasted space on a tape. So if you have
800G tapes, and your split size is, say, 5G, then you will always fill
at least 795G of the first tape in your run (assuming you're using
runtapes > 1).

It's interesting to note that Amanda has transitioned from a state where
the tapes were the slow part of the equation to the opposite. In its
original design, Amanda tried to get the taper up and running as soon as
possible (potentially starting with dumps already in holding, and then
greedily taping dumps as they are completed) on the assumption that the
taper would be the major consumer of backup time.

Dustin

--
Dustin J. Mitchell
Storage Software Engineer, Zmanda, Inc.
http://www.zmanda.com/

1c.

Re: delay tape write?

Posted by: "Matt Hyclak" hyclakmath.ohiou.edu?Subject= Re%3A%20delay%20tape%20write%3F"> hyclakmath.ohiou.edu

Wed Aug 22, 2007 9:48 am (PST)

On Wed, Aug 22, 2007 at 11:28:42AM -0500, Dustin J. Mitchell enlightened us:
> > Yes, a delayed start to taper would certainly make it easier to
> > better fit the dumps on a tape to maximize tape usage. I've tried
> > playing with the dumporder but I still often end up wasting tape
>; > with Amanda trying to write a large dump when there is only a medium
> > amount of space left on the current tape. I'll try adding in the
> > taperalgo setting to see if that helps, but it seems that might
> > not be a complete solution until delayed taping is implemented.
>
> This problem is significantly less serious if you use split dumps.
> Split dumps effectively set the maximum size of an on-tape file, which
> becomes the maximum amount of wasted space on a tape. So if you have
>; 800G tapes, and your split size is, say, 5G, then you will always fill
>; at least 795G of the first tape in your run (assuming you're using
> runtapes > 1).
>

Some of us are still running 2.4.5

And the problem I described in my previous mail is rare - once a quarter
when I do archival full dumps of everything. Normal day-to-day operation is
to let backups spool to holding disk for a week, so autoflush does its thing
and by the time the big dumps are done, taper is still flushing old dumps.

> It's interesting to note that Amanda has transitioned from a state where
> the tapes were the slow part of the equation to the opposite. In its
> original design, Amanda tried to get the taper up and running as soon as
> possible (potentially starting with dumps already in holding, and then
>; greedily taping dumps as they are completed) on the assumption that the
> taper would be the major consumer of backup time.
>

Writing to tape is usually something you can do even during peak production
hours. It's getting all the data to holding disk that is in the time crunch
- especially if there are DBs that have to go offline, etc. This leads to as
many parallel processes as you can reasonably do with your hardware. Drive
space is cheap, bandwidth is cheap, and processing power is cheap - writing
to tape will always be slow, but generally doesn't interrupt production
processes if it's just shoving data down the scsi bus. From this
perspective, an admin only has a few hours to get dumps done, but nearly 24
hours to get it all to tape (assuming you don't actually want to *restore*
anything, or happen to have another drive to do so with).

My $.02.

Matt

--
Matt Hyclak
Department of Mathematics
Department of Social Work
Ohio University
(740) 593-1263

1d.

Re: delay tape write?

Posted by: "Matt Hyclak" hyclakmath.ohiou.edu?Subject= Re%3A%20delay%20tape%20write%3F"> hyclakmath.ohiou.edu

Wed Aug 22, 2007 10:03 ;am (PST)

On Wed, Aug 22, 2007 at 10:54:22AM -0500, Frank Smith enlightened us:
> Paul Bijnens wrote:
> > On 2007-08-22 01:35, Frank Smith wrote:
> >> Also, on a somewhat related note, if taper were to wait until
> >> all the dumps finished, and runtapes is greater than 1, could there
> >> be a taperorder parameter similar to dumporder that would do a
> >> largest-fit write to tape, and avoid trying to write a dump too
> >> large to fit in the remaining space on a tape? I haven't dug in
> >> the code, but taper just seems to work in a FIFO fashion, and
> >> writes until it hits EOT and then starts anew on the next tape with
>; >> the failed dump. For runtapes = 2, largest fit would be sufficient,
> >> but for runtapes >=3 perhaps another strategy would be better,
> >> although possibly not by enough to justify the extra code.
> >
> > You're describing the parameter "taperalgo", I believe.
>
> Thanks, that sounds like what I was asking for. Evidently I missed
> it when that feature was added.
>
> > Because there is no "delayed" tapewriting currently, you can
> > only influence the result of the taperalgo by tuning the
> > "dumporder" and the number of dumpers ("inparallel") as well.
> > See my explanation in:
> >
> > http://wiki.zmanda.com/index.php/How_To:Fill_tapes_to_100%25
> >
> > Having an option to delay the tapewriting somehow, would make
>; > this a lot simpler, especially for those extremely large and
> > fast tapes currently on the market.
>
> Yes, a delayed start to taper would certainly make it easier to
> better fit the dumps on a tape to maximize tape usage. I've tried
> playing with the dumporder but I still often end up wasting tape
>; with Amanda trying to write a large dump when there is only a medium
> amount of space left on the current tape. I'll try adding in the
> taperalgo setting to see if that helps, but it seems that might
> not be a complete solution until delayed taping is implemented.
>

Agreed - I have my setup dump the largest DLEs first, and tape the largest
DLEs first, but it always ends up that there are 3 or 4 very large DLEs
(e.g. user home directories) and the remaining inparallel processes start
dumping smaller DLEs which finish long before the 3 of 4 large DLEs and
taper starts taping as soon as one is complete. I get a lot of start/stop on
the tape drive this way, and if dumping enough data end up wasting tape as
well.

The delayed taping, either by size or time (e.g. start taping 1 hour after
dumps have started) would be useful. Size probably moreso than time, but
I'll take either at this point.

Matt

--
Matt Hyclak
Department of Mathematics
Department of Social Work
Ohio University
(740) 593-1263

1e.

Re: delay tape write?

Posted by: "Jon LaBadie" jonjgcomp.com?Subject= Re%3A%20delay%20tape%20write%3F"> jonjgcomp.com

Wed Aug 22, 2007 12:20 ;pm (PST)

On Tue, Aug 21, 2007 at 09:09:24AM -0700, Christopher McCrory wrote:
> Hello...
>
> I seem to remember a feature request for delaying the start of writing to tape until X%/Xbytes are done dumping and ready to write. It was on sourceforge? zmanda wiki? I cannot seem to find it now. Does anyone know where it was?
>;

Good memory, it was 2 years ago.

Orion Poplawski submitted a 2.4.5 patch called 'taperwait'. This was not
incorporated as Jean-Louis felt a more comprehensive solution was possible
and needed. Here are JLM's initial comments on OP's submission:

== From: Jean-Louis Martineau < martinea%40iro.umontreal.ca">martineairo.umontreal.ca&gt;
== Date: Wed, 7 Sep 2005 13:31:06 -0400
== Subject: not taperwait
==
== I Orion,
==
== Your taperwait patch might be useful but it doesn't solve some of the problem
== in the taper scheduling.
==
== eg. 1- I don't want to use a tape every day, but I want to delay taping.
== 2- I don't want to use a tape every day, but I want to start taping as soon as posible.
== 3- My dump size is 120% of my tape size. I don't want to use 2 tapes every day.
== 4- A want to keep many days of dump on holding disk before I write them to tape (for faster recovery).
==
==
== I think that amanda should fill a tape as fast as it can when it started
== to write to a tape, the question is
==
== When amanda should start to write to a new tape?
==
== I will introduce two new options: taperstart and taperflush
==
== taperstart x% y%: default (0 0)
== - start a new tape if x% is already dumped and on holding disk and
== if y% is already dumped or scheduled. (x<y)
== - a high x will improve tape fillment.
==
== taperflush [0|1]: default (1)
== - if 1, then flush all dump on HD to tape at the end of an amdump run.
== - if 0, don't flush them.
== - Will leaves between 0 and x% of dump on holding disk.
== - Must be use with autoflush.
== - Need high values for taperstart.
==
== - In all case, if a dump direct to tape is needed, a new tape will be started
== - If there is not enough space on holding disk, then a new tape will be
== started.
==
== your taperwait is equivalent to:
== taperstart 99999 99999
== taperflush 1
==
== amanda default setup:
== taperstart 0 0
== taperflush 1
==
== example 1: I don't want to use a tape every day, but I want to delay taping
== after the dump are done.
== if the dump_size is smaller than tape_size, then it is possible to not
== use a tape at every day:
== runtapes 1
== taperstart 100 100
== taperflush 0
== autoflush 1
== If the dump_size is 30% of the tape_size, then a tape will be used
== every 3 or 4 days
== Current workaround: don't use tape when amdump and run amflush regularly.
==
== example 2: I don't want to use a tape every day, but I want to start taping
== as soon as posible.
== runtapes 1
== taperstart 0 100
== taperflush 0
== autoflush 1
== If the dump_size is 30% of the tape_size, then a tape will be used
== every 3 or 4 days
== No current workaround.
==
== example 3: My dump size is 120% of my tape size. I don't want to use 2 tapes
== every day.
== runtapes 2
== taperstart 100 100 (or 0 100) (or 50 100)
== taperflush 0
== autoflush 1
== A second tape will be needed every 5 day.
== Current workaround: set runtapes to 1 and run amflush regularly.
==
== example 4: A want to keep many days of dump on holding disk before a write them
== to tape (for faster recovery).
== runtapes 1
== taperstart 600 600 (or 0 600) (or 111 600)
== taperflush 0
== autoflush 1
== You need a big holding disk. You will always have more than 5 tapes on HD.
==
==
== Do anyone have other taper scheduling requirement that we can't fix
== with these new options.
==
== Any comment?
==
== We can also introduce a new taperalgo (optimized) because using the biggest
== is not always the best.
==
== Jean-Louis
==

> Writing 500M to a 800G tape is irritating. I would be willing to sponsor a developer to work on this. Any takers?

If no one is able to implement JLM's suggestion, perhaps this is another candidate
for a GSOC project?

jl
--
Jon H. LaBadie jon%40jgcomp.com">jonjgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)

1f.

Re: delay tape write?

Posted by: "Jon LaBadie" jonjgcomp.com?Subject= Re%3A%20delay%20tape%20write%3F"> jonjgcomp.com

Wed Aug 22, 2007 12:26 ;pm (PST)

On Wed, Aug 22, 2007 at 12:43:20PM -0400, Matt Hyclak wrote:
&gt; On Wed, Aug 22, 2007 at 11:28:42AM -0500, Dustin J. Mitchell enlightened us:
> > > Yes, a delayed start to taper would certainly make it easier to
> > > better fit the dumps on a tape to maximize tape usage. I've tried
&gt; > > playing with the dumporder but I still often end up wasting tape
>; > > with Amanda trying to write a large dump when there is only a medium
&gt; > > amount of space left on the current tape. I'll try adding in the
> > > taperalgo setting to see if that helps, but it seems that might
&gt; > > not be a complete solution until delayed taping is implemented.
> >
> > This problem is significantly less serious if you use split dumps.
&gt; > Split dumps effectively set the maximum size of an on-tape file, which
&gt; > becomes the maximum amount of wasted space on a tape. So if you have
>; > 800G tapes, and your split size is, say, 5G, then you will always fill
>; > at least 795G of the first tape in your run (assuming you're using
&gt; > runtapes > 1).
> >
>;
> Some of us are still running 2.4.5
>
> And the problem I described in my previous mail is rare - once a quarter
> when I do archival full dumps of everything. Normal day-to-day operation is
> to let backups spool to holding disk for a week, so autoflush does its thing
&gt; and by the time the big dumps are done, taper is still flushing old dumps.
&gt;
> > It's interesting to note that Amanda has transitioned from a state where
&gt; > the tapes were the slow part of the equation to the opposite. In its
> > original design, Amanda tried to get the taper up and running as soon as
> > possible (potentially starting with dumps already in holding, and then
>; > greedily taping dumps as they are completed) on the assumption that the
> > taper would be the major consumer of backup time.
&gt; >
>
> Writing to tape is usually something you can do even during peak production
> hours. It's getting all the data to holding disk that is in the time crunch
&gt; - especially if there are DBs that have to go offline, etc. This leads to as
> many parallel processes as you can reasonably do with your hardware. Drive
&gt; space is cheap, bandwidth is cheap, and processing power is cheap - writing
> to tape will always be slow, but generally doesn't interrupt production
> processes if it's just shoving data down the scsi bus. From this
>; perspective, an admin only has a few hours to get dumps done, but nearly 24
> hours to get it all to tape (assuming you don't actually want to *restore*
> anything, or happen to have another drive to do so with).

As you can tape during the day, might a crontab entry of "amdump ; amflush&quot;
be a suitable workaround?

For the restore, while it doesn't help with older dumps, amrecover will
work with the dumps on holding disk.

--
Jon H. LaBadie jon%40jgcomp.com">jonjgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)

1g.

Re: delay tape write?

Posted by: "Frank Smith" fsmithhoovers.com?Subject= Re%3A%20delay%20tape%20write%3F"> fsmithhoovers.com

Wed Aug 22, 2007 12:53 ;pm (PST)

Dustin J. Mitchell wrote:
&gt; On Wed, Aug 22, 2007 at 10:54:22AM -0500, Frank Smith wrote:
&gt;> Yes, a delayed start to taper would certainly make it easier to
>&gt; better fit the dumps on a tape to maximize tape usage. I've tried
&gt;> playing with the dumporder but I still often end up wasting tape
>;> with Amanda trying to write a large dump when there is only a medium
&gt;> amount of space left on the current tape. I'll try adding in the
>> taperalgo setting to see if that helps, but it seems that might
&gt;> not be a complete solution until delayed taping is implemented.
>
> This problem is significantly less serious if you use split dumps.
&gt; Split dumps effectively set the maximum size of an on-tape file, which
&gt; becomes the maximum amount of wasted space on a tape. So if you have
>; 800G tapes, and your split size is, say, 5G, then you will always fill
>; at least 795G of the first tape in your run (assuming you're using
&gt; runtapes > 1).

Yes, that would be more space efficient, but I'm one of the old-fashioned
types that wants to be able to do bare-metal restores using basic commands,
and my understanding was that split dumps added complexity in this scenario.
I recently had to do such a recovery on one of my Amanda servers, and
I think this ability is one of Amanda's best features.

Frank

>
> It's interesting to note that Amanda has transitioned from a state where
&gt; the tapes were the slow part of the equation to the opposite. In its
> original design, Amanda tried to get the taper up and running as soon as
> possible (potentially starting with dumps already in holding, and then
>; greedily taping dumps as they are completed) on the assumption that the
> taper would be the major consumer of backup time.
&gt;
> Dustin
&gt;

--
Frank Smith fsmith%40hoovers.com">fsmithhoovers.com
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501

1h.

Re: delay tape write?

Posted by: "Frank Smith" fsmithhoovers.com?Subject= Re%3A%20delay%20tape%20write%3F"> fsmithhoovers.com

Wed Aug 22, 2007 12:55 ;pm (PST)

Jon LaBadie wrote:
&gt;
> As you can tape during the day, might a crontab entry of "amdump ; amflush&quot;
> be a suitable workaround?
>

How can you get amdump to not write to tape? I know you can
specify an invalid tape device in the config, or not have a tape
in your drive, to prevent amdump from writing to tape, but either
of those would also prevent amflush from working.

Frank

--
Frank Smith fsmith%40hoovers.com">fsmithhoovers.com
Sr. Systems Administrator Voice: 512-374-4673
Hoover's Online Fax: 512-374-4501

1i.

Re: delay tape write?

Posted by: "Paul Bijnens" Paul.Bijnensxplanation.com?Subject= Re%3A%20delay%20tape%20write%3F"> Paul.Bijnensxplanation.com

Wed Aug 22, 2007 1:26 pm (PST)

On 2007-08-22 21:48, Frank Smith wrote:
&gt; Jon LaBadie wrote:
&gt;> As you can tape during the day, might a crontab entry of "amdump ; amflush&quot;
>&gt; be a suitable workaround?
>>;
>
> How can you get amdump to not write to tape? I know you can
> specify an invalid tape device in the config, or not have a tape
>; in your drive, to prevent amdump from writing to tape, but either
&gt; of those would also prevent amflush from working.

In recent versions (> 2.5.0 I believe) you can override amanda.conf
parameters on the commandline with "-o name=value"

Assuming we have one tape:

amdump -o tapedev=/no/such/tape -o reserve=0 daily; amflush daily

I'm not sure how to "comment out" an option, so if you use a changer
you might configure amanda.conf with some non-existing tapedev
and override that with a tpchanger option on amflush:

amdump daily; amflush -o tpchanger=chg-scsi daily

(the other changer parameters may be configured in amanda.conf,
because they get consulted only if the tpchanger parameter is set).

(Untested -- about to leave on holiday )

--
Paul Bijnens, xplanation Technology Services Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens%40xplanation.com">Paul.Bijnensxplanation.com
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?"; ... YES ... Phew ... I'm out *
***********************************************************************

1j.

Re: delay tape write?

Posted by: "Matt Hyclak" hyclakmath.ohiou.edu?Subject= Re%3A%20delay%20tape%20write%3F"> hyclakmath.ohiou.edu

Wed Aug 22, 2007 1:45 pm (PST)

On Wed, Aug 22, 2007 at 10:10:55PM +0200, Paul Bijnens enlightened us:
> >Jon LaBadie wrote:
&gt; >>As you can tape during the day, might a crontab entry of "amdump ;
> >>amflush"
> >>be a suitable workaround?
> >>
> >
>; >How can you get amdump to not write to tape? I know you can
> >specify an invalid tape device in the config, or not have a tape
>; >in your drive, to prevent amdump from writing to tape, but either
&gt; >of those would also prevent amflush from working.
>
> In recent versions (> 2.5.0 I believe) you can override amanda.conf
> parameters on the commandline with "-o name=value"
&gt;
> Assuming we have one tape:
&gt;
> amdump -o tapedev=/no/such/tape -o reserve=0 daily; amflush daily
&gt;
> I'm not sure how to "comment out" an option, so if you use a changer
> you might configure amanda.conf with some non-existing tapedev
> and override that with a tpchanger option on amflush:
>
> amdump daily; amflush -o tpchanger=chg-scsi daily
&gt;
> (the other changer parameters may be configured in amanda.conf,
> because they get consulted only if the tpchanger parameter is set).
&gt;
> (Untested -- about to leave on holiday )

See previous note about still running 2.4.5

My current mode of operations is to eject (but not remove) a tape after
amdump. I can still mt load it if I need it. Then, I only stick the tape in
once a week to collect my dumps for the week. That's no problem.

I suppose I could do full dumps without a tape in as well, and amflush it
the next day, but it'd be nice not to have to.

Matt

--
Matt Hyclak
Department of Mathematics
Department of Social Work
Ohio University
(740) 593-1263

1k.

Re: delay tape write?

Posted by: "Chris Hoogendyk" hoogendykbio.umass.edu?Subject= Re%3A%20delay%20tape%20write%3F"> hoogendykbio.umass.edu

Wed Aug 22, 2007 1:46 pm (PST)



Frank Smith wrote:
&gt; Jon LaBadie wrote:
&gt;
>>; As you can tape during the day, might a crontab entry of "amdump ; amflush&quot;
>&gt; be a suitable workaround?
>>;
>>;
>
> How can you get amdump to not write to tape? I know you can
> specify an invalid tape device in the config, or not have a tape
>; in your drive, to prevent amdump from writing to tape, but either
&gt; of those would also prevent amflush from working.

my cron entries are:

0 15 * * 1-5 /usr/local/sbin/amcheck -m daily
45 0 * * 2-6 /usr/local/sbin/amdump daily
45 0 * * 0-1 /usr/local/sbin/amdump daily -o reserve=100 -o
tapedev=/dev/rmt/0n -o tpchanger="";

The weekend dumps are incremental only to holding disk.

The monday dump (Tuesday morning just after midnight) autoflushes those
along with its own dump.

You could do something similar with all the dumps going to holding and
amflush running off cron.

---------------

Chris Hoogendyk

-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) (*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst

< hoogendyk%40bio.umass.edu">hoogendykbio.umass.edu>

---------------

Erdös 4

1l.

Re: delay tape write?

Posted by: "Jean-Louis Martineau" martineauzmanda.com?Subject= Re%3A%20delay%20tape%20write%3F"> martineauzmanda.com

Wed Aug 22, 2007 2:11 pm (PST)

The problem with the taperstart/taperflush features is that it require a
rewrite of the protocol between the driver and taper, that's why it is
not yet done.
The new protocol is done in the zmanda internal tree, this code will
eventually be merge to the community tree.
After that, it will be easy to implement the taperstart/taperflush features.

Jean-Louis

Jon LaBadie wrote:
&gt; On Tue, Aug 21, 2007 at 09:09:24AM -0700, Christopher McCrory wrote:
&gt;
>>; Hello...
>>
>> I seem to remember a feature request for delaying the start of writing to tape until X%/Xbytes are done dumping and ready to write. It was on sourceforge? zmanda wiki? I cannot seem to find it now. Does anyone know where it was?
>;>
>;>
>
> Good memory, it was 2 years ago.
>;
> Orion Poplawski submitted a 2.4.5 patch called 'taperwait'. This was not
> incorporated as Jean-Louis felt a more comprehensive solution was possible
> and needed. Here are JLM's initial comments on OP's submission:
>
> == From: Jean-Louis Martineau < martinea%40iro.umontreal.ca">martineairo.umontreal.ca&gt;
> == Date: Wed, 7 Sep 2005 13:31:06 -0400
&gt; == Subject: not taperwait
> ==
> == I Orion,
&gt; ==
> == Your taperwait patch might be useful but it doesn't solve some of the problem
> == in the taper scheduling.
> ==
> == eg. 1- I don't want to use a tape every day, but I want to delay taping.
> == 2- I don't want to use a tape every day, but I want to start taping as soon as posible.
> == 3- My dump size is 120% of my tape size. I don't want to use 2 tapes every day.
>; == 4- A want to keep many days of dump on holding disk before I write them to tape (for faster recovery).
> ==
> ==
> == I think that amanda should fill a tape as fast as it can when it started
> == to write to a tape, the question is
> ==
> == When amanda should start to write to a new tape?
&gt; ==
> == I will introduce two new options: taperstart and taperflush
> ==
> == taperstart x% y%: default (0 0)
> == - start a new tape if x% is already dumped and on holding disk and
> == if y% is already dumped or scheduled. (x<y)
> == - a high x will improve tape fillment.
> ==
> == taperflush [0|1]: default (1)
> == - if 1, then flush all dump on HD to tape at the end of an amdump run.
>; == - if 0, don't flush them.
&gt; == - Will leaves between 0 and x% of dump on holding disk.
&gt; == - Must be use with autoflush.
> == - Need high values for taperstart.
> ==
> == - In all case, if a dump direct to tape is needed, a new tape will be started
> == - If there is not enough space on holding disk, then a new tape will be
> == started.
> ==
> == your taperwait is equivalent to:
> == taperstart 99999 99999
&gt; == taperflush 1
> ==
> == amanda default setup:
&gt; == taperstart 0 0
> == taperflush 1
> ==
> == example 1: I don't want to use a tape every day, but I want to delay taping
&gt; == after the dump are done.
&gt; == if the dump_size is smaller than tape_size, then it is possible to not
> == use a tape at every day:
>; == runtapes 1
> == taperstart 100 100
> == taperflush 0
> == autoflush 1
> == If the dump_size is 30% of the tape_size, then a tape will be used
>; == every 3 or 4 days
>; == Current workaround: don't use tape when amdump and run amflush regularly.
> ==
> == example 2: I don't want to use a tape every day, but I want to start taping
&gt; == as soon as posible.
> == runtapes 1
> == taperstart 0 100
> == taperflush 0
> == autoflush 1
> == If the dump_size is 30% of the tape_size, then a tape will be used
>; == every 3 or 4 days
>; == No current workaround.
> ==
> == example 3: My dump size is 120% of my tape size. I don't want to use 2 tapes
&gt; == every day.
>; == runtapes 2
> == taperstart 100 100 (or 0 100) (or 50 100)
>; == taperflush 0
> == autoflush 1
> == A second tape will be needed every 5 day.
>; == Current workaround: set runtapes to 1 and run amflush regularly.
> ==
> == example 4: A want to keep many days of dump on holding disk before a write them
>; == to tape (for faster recovery).
> == runtapes 1
> == taperstart 600 600 (or 0 600) (or 111 600)
>; == taperflush 0
> == autoflush 1
> == You need a big holding disk. You will always have more than 5 tapes on HD.
> ==
> ==
> == Do anyone have other taper scheduling requirement that we can't fix
> == with these new options.
> ==
> == Any comment?
> ==
> == We can also introduce a new taperalgo (optimized) because using the biggest
> == is not always the best.
&gt; ==
> == Jean-Louis
> ==
>
>
>>; Writing 500M to a 800G tape is irritating. I would be willing to sponsor a developer to work on this. Any takers?
>>
>
> If no one is able to implement JLM's suggestion, perhaps this is another candidate
> for a GSOC project?
>
>; jl
>

1m.

Re: delay tape write?

Posted by: "Dustin J. Mitchell" dustinzmanda.com?Subject= Re%3A%20delay%20tape%20write%3F"> dustinzmanda.com

Wed Aug 22, 2007 3:30 pm (PST)

On Wed, Aug 22, 2007 at 04:11:07PM -0400, Jon LaBadie wrote:
&gt; A more elegant solution for recent versions is "configuration override&quot;:
>;
> amdump ... -o tapedev=/dev/null ...
>
> though I've neither used nor tried this feature.

There was some discussion of this in late June, too, IIRC. You will
also need "-o tpchanger=" to disable any changer (which might otherwise
supply a tape device).

Dustin

--
Dustin J. Mitchell
Storage Software Engineer, Zmanda, Inc.
http://www.zmanda.com/

1n.

Re: delay tape write?

Posted by: "Jon LaBadie" jonjgcomp.com?Subject= Re%3A%20delay%20tape%20write%3F"> jonjgcomp.com

Wed Aug 22, 2007 3:35 pm (PST)

On Wed, Aug 22, 2007 at 02:48:26PM -0500, Frank Smith wrote:
&gt; Jon LaBadie wrote:
&gt; >
> > As you can tape during the day, might a crontab entry of "amdump ; amflush&quot;
> > be a suitable workaround?
> >
>
> How can you get amdump to not write to tape? I know you can
> specify an invalid tape device in the config, or not have a tape
>; in your drive, to prevent amdump from writing to tape, but either
&gt; of those would also prevent amflush from working.
>

You are right.

Could do it crudely by editing the amanda.conf entry before and after amdump.

A more elegant solution for recent versions is "configuration override&quot;:

amdump ... -o tapedev=/dev/null ...

though I've neither used nor tried this feature.

--
Jon H. LaBadie jon%40jgcomp.com">jonjgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)

Recent Activity
Visit Your Group
SPONSORED LINKS
Best of Y! Groups

Check out the best

of what Yahoo!

Groups has to offer.

Yahoo! Groups

Your one stop

for beauty & fashion

tips and advice.

Yahoo! Groups

Real Food Group

Share recipes

and favorite meals.

Need to Reply?

Click one of the "Reply" links to respond to a specific message in the Daily Digest.

Create New Topic | Visit Your Group on the Web
Yahoo! Groups
Change settings via the Web (Yahoo! ID required)
Change settings via email: amanda-hackers-normal@yahoogroups.com?subject=Email Delivery: Indiviual Email">Switch delivery to Individual | Switch format to Traditional
Visit Your Group | Yahoo! Groups Terms of Use | amanda-hackers-unsubscribe@yahoogroups.com?subject=Unsubscribe"> Unsubscribe
[1]

about | contact  Other archives ( Real Estate discussion Medical topics )