Sync files locally?

What's the best way to sync files locally?

I'm running rsync over ssh to backup files from a server. 
Now I'm setting 
up a new server, so I rsynced to it as well.  The files end
up in a subdir 
of my home directory, because my account is being used for
the rsync.  I've 
been using cp -R to put them in the right place on the
server and the chmod 
and chown to get the perms right, but I'm thinking there has
to be a way to 
sync the files locally so that only the ones that have
changed or are new 
have to be copied to the right place.

I can use find to set the perms, but I'm not sure how to
sync the files. 
This is unix, so there's got to be a built-in utility that
does this, but I 
can't seem to find it.

Paul Schmehl (paulsutdallas.edu)
Adjunct Information Security Officer
The University of Texas at Dallas
http://www.utdal
las.edu/ir/security/

On Sep 27, 2006, at 5:40 PM, Paul Schmehl wrote:
> What's the best way to sync files locally?
>
> I'm running rsync over ssh to backup files from a
server.

rsync (as root) works just fine to perform local copies
preserving  
permissions and symlinks and so forth.  Other people use a
tar  
pipeline, or dump/restore, or cpio...

-- 
-Chuck

_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

In the last episode (Sep 27), Paul Schmehl said:
> What's the best way to sync files locally?
> 
> I'm running rsync over ssh to backup files from a
server.  Now I'm
> setting up a new server, so I rsynced to it as well. 
The files end
> up in a subdir of my home directory, because my account
is being used
> for the rsync.  I've been using cp -R to put them in
the right place
> on the server and the chmod and chown to get the perms
right, but I'm
> thinking there has to be a way to sync the files
locally so that only
> the ones that have changed or are new have to be copied
to the right
> place.

As long as your rsync path has a leading /, it should start
at the root
filesystem.  e.g. "rsync mydir otherserver:mydir"
creates ~/mydir ,
"rsync mydir otherserver:/mydir" creates /mydir
(assuming you have
permission).
 
> I can use find to set the perms, but I'm not sure how
to sync the
> files.  This is unix, so there's got to be a built-in
utility that
> does this, but I can't seem to find it.

You could use rsync to do this, but It'd probably be easier
to get your
remote rsync to drop the files in the right directory in the
first place.
 
-- 
	Dan Nelson
	dnelsonallantgroup.com
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

--On Wednesday, September 27, 2006 18:04:40 -0400 Charles
Swiger 
<cswigermac.com> wrote:

> On Sep 27, 2006, at 5:40 PM, Paul Schmehl wrote:
>> What's the best way to sync files locally?
>>
>> I'm running rsync over ssh to backup files from a
server.
>
> rsync (as root) works just fine to perform local copies
preserving
> permissions and symlinks and so forth.  Other people
use a tar pipeline,
> or dump/restore, or cpio...

I wasn't aware of that.  Thanks, Charles.

Paul Schmehl (paulsutdallas.edu)
Adjunct Information Security Officer
The University of Texas at Dallas
http://www.utdal
las.edu/ir/security/

--On Wednesday, September 27, 2006 17:04:39 -0500 Dan Nelson

<dnelsonallantgroup.com> wrote:
>
> You could use rsync to do this, but It'd probably be
easier to get your
> remote rsync to drop the files in the right directory
in the first place.
>
Hmmm...maybe I don't fully understand rsync.  The files are
from a 
webserver and owned by www:www.  The rsync runs under my
account, so the 
perms are changed to mine.  Maybe there's a switch with
rsync that would 
allow me to sync directly?

I'll poke around the man page.  Thanks.

Paul Schmehl (paulsutdallas.edu)
Adjunct Information Security Officer
The University of Texas at Dallas
http://www.utdal
las.edu/ir/security/

Paul Schmehl wrote:
> What's the best way to sync files locally?
> 
> I'm running rsync over ssh to backup files from a
server.  Now I'm setting 
> up a new server, so I rsynced to it as well.  The files
end up in a subdir 
> of my home directory, because my account is being used
for the rsync.  I've 
> been using cp -R to put them in the right place on the
server and the chmod 
> and chown to get the perms right, but I'm thinking
there has to be a way to 
> sync the files locally so that only the ones that have
changed or are new 
> have to be copied to the right place.
> 
> I can use find to set the perms, but I'm not sure how
to sync the files. 
> This is unix, so there's got to be a built-in utility
that does this, but I 
> can't seem to find it.

It's really a question of reading far enough in the
man-page. I have 
eventually arrived at the options

   -Cptuvaz

C for CVS omissions (exclude *.bak, *.tmp, *~ etc)
p for preserve permisions
t for preserving time
u for update (using mtime I suppose)
a for archive - recursive
z for compression

and with that, there should be no reason to sync into a
separate dir and 
then copy.

Cheers, Erik
-- 
Ph: +34.666334818                      web: http://www.locolomo.org
X.509 Certificate: http
://www.locolomo.org/crt/8D03551FFCE04F0C.crt
Key ID:
69:79:B8:2C:E3:8F:E7:BE:5D:C3:C3:B1:74:62:B8:3F:9F:1F:69:B9
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

Paul Schmehl wrote:

> What's the best way to sync files locally?
>
> I'm running rsync over ssh to backup files from a
server.  Now I'm 
> setting up a new server, so I rsynced to it as well. 
The files end up 
> in a subdir of my home directory, because my account is
being used for 
> the rsync.  I've been using cp -R to put them in the
right place on 
> the server and the chmod and chown to get the perms
right, but I'm 
> thinking there has to be a way to sync the files
locally so that only 
> the ones that have changed or are new have to be copied
to the right 
> place.
>
> I can use find to set the perms, but I'm not sure how
to sync the 
> files. This is unix, so there's got to be a built-in
utility that does 
> this, but I can't seem to find it.

Well, rsync.  When the paths are both local it won't use ssh
since it 
doesn't need it.  Not sure why you think the files have to
end up under 
your home dir.  Something like

    rsync -a --delete /path1/ /alt/path2

will make an exact copy under /alt/path2, updating only
things that need 
it.  Done as root it would do all the preservation of users,
modes etc 
regardless of ownership.

You also have --flags if you need it but it doesn't appear
to work with 
schg flag - maybe nouulnk and variants either.


Other than that you can use tar with pipes, which was the
traditional 
method Before Rsync (TM).  But obviously not very efficient
for keeping 
a tree up-to-date.

These days cp -Rp as root should also do a one-time copy but
won't 
preserve any hard links.

--Alex






_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

Paul Schmehl wrote:
> --On Wednesday, September 27, 2006 17:04:39 -0500 Dan
Nelson 
> <dnelsonallantgroup.com> wrote:
>>
>> You could use rsync to do this, but It'd probably
be easier to get your
>> remote rsync to drop the files in the right
directory in the first place.
>>
> Hmmm...maybe I don't fully understand rsync.  The files
are from a 
> webserver and owned by www:www.  The rsync runs under
my account, so the 
> perms are changed to mine.  Maybe there's a switch with
rsync that would 
> allow me to sync directly?
> 
> I'll poke around the man page.  Thanks.

1) rsync changes behaviour depending on whether or not you
include a 
trailing / from the man-page:

        rsync -av /src/foo /dest
        rsync -av /src/foo/ /dest/foo

works the same way, (and sync locally in this example). Read
the man 
page, there are lots of examples.

2) The options -t and -p preserve time and permissions
respectively. 
Ownership will always change to the user running rsync
unless you run as 
root. This has nothing to do with rsync, you can't run chown
as an 
ordinary user.

You can preserve the group if you're in that group on the
destination host.

3) The files you are syncing - should they be writeable by
www? For 
security, you may really want something like this:

   -rw-r----- user:www    file

and have user do the rsync. If you really need to have www
write to the 
file, set group permissions +w.

Personally, I have all my web-pages owned by me, and let
apache access 
read-only. Log-files generated are owned by apache, but I
don't back 
these up since they are continuously generated and rotated
out. The 
config file is also only readable by apache, I don't want
the risk of 
some exploit committing changes to the config file.

It might be useful to have a separate dir where apache can
store files 
with write permissions and owned by apache. But this depends
on the web 
apps you deploy.

IIRC to run rsync over ssh the user doing the syncing must
have shell 
access, running your sync as root is not desirable, it MAY
be preferred 
to have it run as www to preserve owner also, at least you
can restrict 
access for www.

Cheers, Erik
-- 
Ph: +34.666334818                      web: http://www.locolomo.org
X.509 Certificate: http
://www.locolomo.org/crt/8D03551FFCE04F0C.crt
Key ID:
69:79:B8:2C:E3:8F:E7:BE:5D:C3:C3:B1:74:62:B8:3F:9F:1F:69:B9
_______________________________________________
freebsd-questionsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-que
stions
To unsubscribe, send any mail to
"freebsd-questions-unsubscribefreebsd.org"

--On September 28, 2006 9:33:39 AM +0200 Erik Norgaard 
<norgaardlocolomo.org> wrote:
>
> 1) rsync changes behaviour depending on whether or not
you include a
> trailing / from the man-page:
>
>         rsync -av /src/foo /dest
>         rsync -av /src/foo/ /dest/foo
>
That I know.

> works the same way, (and sync locally in this example).
Read the man
> page, there are lots of examples.
>
> 2) The options -t and -p preserve time and permissions
respectively.
> Ownership will always change to the user running rsync
unless you run as
> root. This has nothing to do with rsync, you can't run
chown as an
> ordinary user.
>
> You can preserve the group if you're in that group on
the destination
> host.
>
> 3) The files you are syncing - should they be writeable
by www? For
> security, you may really want something like this:
>
>    -rw-r----- user:www    file
>
They're user:www, but the user isn't me.  

I could go into the reasons for that, but it doesn't really
matter.

> and have user do the rsync. If you really need to have
www write to the
> file, set group permissions +w.
>
No, I don't need that and don't want it either.  The owners
of the site 
own the files, and I know how to use sudo.  
>
> IIRC to run rsync over ssh the user doing the syncing
must have shell
> access, running your sync as root is not desirable, it
MAY be preferred
> to have it run as www to preserve owner also, at least
you can restrict
> access for www.
>
I use ssh with keys so I can cron the job, but I'll have to
do the extra 
step of fixing perms and ownership after copying the rsynced
files over. 
No big deal.  I'll script that as well.

Paul Schmehl (paulsutdallas.edu)
Adjunct Information Security Officer
The University of Texas at Dallas
http://www.utdal
las.edu/ir/security/