Hi,

What's the best way to perform an overnight copy of a directory structure from several remote servers back to a central server ?

It's possible that some files may be open for use (locked) and it would be nice to be able to notify a particular user whenever specific files could not be copied.

Thanks,

Jim

If you can map remote servers as network shares then the easiest way is to schedule run of "xcopy [source] [dest] /E /R /F /K /X /Y /C > copy.log" command with output redirected to a log file. You can then make a script job checking the log file and searching for errors then emailing these errors to you.

If you cannot map remote shares then you can have 24x7 agents installed on remote servers and use FileTransferEx command or SyncRemoteDir commands to copy remote files. Bring remote files to working directories first and then either use local xcopy (using the same method as described above) or write a script job to copy file by file and email every error.

: Hi,

: What's the best way to perform an overnight copy of a directory structure
: from several remote servers back to a central server ?

: It's possible that some files may be open for use (locked) and it would be
: nice to be able to notify a particular user whenever specific files could
: not be copied.

: Thanks,

: Jim

: If you can map remote servers as network shares then the easiest way is to
: schedule run of "xcopy [source] [dest] /E /R /F /K /X /Y /C >
: copy.log" command with output redirected to a log file. You can then
: make a script job checking the log file and searching for errors then
: emailing these errors to you.

: If you cannot map remote shares then you can have 24x7 agents installed on
: remote servers and use FileTransferEx command or SyncRemoteDir commands to
: copy remote files. Bring remote files to working directories first and
: then either use local xcopy (using the same method as described above) or
: write a script job to copy file by file and email every error.

Hi,
Thanks for the input - I was aware of the Xcopy option but I was hoping that someone out there had already scripted for 24x7 a (fairly common) solution for directory copy/replication that could be completely contained within the 24x7 Automation environment.

Still Hopefull ;)

Jim

Jim,

Most people use simple SyncFTPDir or SyncRemoteDir and combinations to replicate files from one source computer to one target computer. I think in your case you also can use this method first replicating files to a local "working" directory to avoid potential file locks and sharing conflicts and then from there copy file by file (or directory by directory) to the main destination. The script to do such thing should be fairly simple. I can help you with this, but I need to know how you want to deal with file name conflicts (e.g. same file residing on multiple servers and copied to the same destination).

By the way, examples of using SyncXXXDir and file-by-file process (Dir with LoopUntil/LoopWhile) are available in the on-line help.

: Hi,
: Thanks for the input - I was aware of the Xcopy option but I was hoping that
: someone out there had already scripted for 24x7 a (fairly common) solution
: for directory copy/replication that could be completely contained within
: the 24x7 Automation environment.

: Still Hopefull ;)

: Jim

Hi again,

Thanks for the feedback and the pointers to the online help entries which I will have a look at.

I do not forsee a filname conflict situation arising as the intention was to copy the remote directory structure and files from the remote servers which would all be in the form of g:\site_toplev\fold1\fold2\...\foldn\*.* to a central server under a directory structure of v:\central_toplev\folder \fold1\fold2\...\foldn\*.*
I would probably wish to copy the existing data in the centralised folder structure to a local backup area before deleting/overwriting the areas and copying in the new files each night.
It's interesting that you mention the "file access conflict" issue as I was a little unclear about how 24x7 would handle this - is the implication that by using the copy functions you mentioned to do a "local" copy first on each server then this would resolve any "file open/in use" problems ?
As you can probably tell I'm totally new to this stuff from a 24x7 perspective so any assistance is greatly appreciated.
I presume that the reporting of copy/job errors to named individuals is pretty straightforward within 24x7 as well ?

Regards,

Jim

: Jim,

: Most people use simple SyncFTPDir or SyncRemoteDir and combinations to
: replicate files from one source computer to one target computer. I think
: in your case you also can use this method first replicating files to a
: local "working" directory to avoid potential file locks and
: sharing conflicts and then from there copy file by file (or directory by
: directory) to the main destination. The script to do such thing should be
: fairly simple. I can help you with this, but I need to know how you want
: to deal with file name conflicts (e.g. same file residing on multiple
: servers and copied to the same destination).

: By the way, examples of using SyncXXXDir and file-by-file process (Dir with
: LoopUntil/LoopWhile) are available in the on-line help.

It all depends on the place of a conflict. If in "source" directory on server all files are readable (not locked for reading) then your automation script would be as simple as

// make copies to local work folders
SyncLocalDir "N1:\\fold1\fold2\\...\\foldn", "v:\\central_toplev\\folder1", True, True, True, True
SyncLocalDir "N2:\\fold1\fold2\\...\\foldn", "v:\\central_toplev\\folder2", True, True, True, True
...
SyncLocalDir "Nn:\\fold1\fold2\\...\\foldn", "v:\\central_toplev\\folderN", True, True, True, True

Here you would replace N1...Nn with share names. In case if you don't have permanent shares, you can set/reset them dynamically, but from my expirience setting/changing drive mappings is not reliable, one day it works other it doesn't, restart the server then it works again and suddenly stops working.
That's why I would recommend using either SyncRemoteDir if you have an agent or master scheduler on each server or using SyncFTPDir if you have FTP servers running.

In all cases you can play with last 4 parameters in Sync...Dir, which are "copy new files", "update changed files", "delete missing files", and finally "copy subdirectories recursively"

If you set the "delete missing files" parameter to False then in theory you can replicate all servers to the common central directory, but again this is fine only if you don't expect conflicts (file locks) occurring in that central directory. In case if you do then use 2 step process: 1 - replicas to "working" directory (all servers); 2 – replicas or copies from "working" directories to the central directory.

To have the process continue after and error and get an email notice use the following method:

Dim LastError, string

SERVER1:
OnErrorGoTo ERROR1
SyncLocalDir ….
GoTo SERVER2

ERROR1:
GetLastError LastError
MailSend "me@myserver.com", "password", "Error occurred while replicating files from server 1", LastError

SERVER2:
OnErrorGoTo ERROR2
SyncLocalDir ….
GoTo SERVER3

ERROR2:
GetLastError LastError
MailSend "me@myserver.com", "password", "Error occurred while replicating files from server 2", LastError

……

SERVERn:
OnErrorGoTo ERRORn
SyncLocalDir ….
Exit

ERRORn:
GetLastError LastError
MailSend "me@myserver.com", "password", "Error occurred while replicating files from server n", LastError

Again, Sync…Dir are high-level file replication methods. If you need fine-tuned error handling and file processing options you can use low level methods DirEx, FileTransferEx, FileCopyEx, xcopy.exe, FileCopy (http://www.softtreetech.com/24x7/archive/44.htm) and some other.

: Hi again,

: Thanks for the feedback and the pointers to the online help entries which I
: will have a look at.

: I do not forsee a filname conflict situation arising as the intention was to
: copy the remote directory structure and files from the remote servers
: which would all be in the form of g:\site_toplev\fold1\fold2\...\foldn\*.*
: to a central server under a directory structure of
: v:\central_toplev\folder \fold1\fold2\...\foldn\*.*
: I would probably wish to copy the existing data in the centralised folder
: structure to a local backup area before deleting/overwriting the areas and
: copying in the new files each night.
: It's interesting that you mention the "file access conflict" issue
: as I was a little unclear about how 24x7 would handle this - is the
: implication that by using the copy functions you mentioned to do a
: "local" copy first on each server then this would resolve any
: "file open/in use" problems ?
: As you can probably tell I'm totally new to this stuff from a 24x7
: perspective so any assistance is greatly appreciated.
: I presume that the reporting of copy/job errors to named individuals is
: pretty straightforward within 24x7 as well ?

: Regards,

: Jim

Hi,
Thanks again for the swift response - very helpfull and impressive support :)

I will no go away and read up on the suggested options and see what develops - this looks exactly like the sort of thing I was looking for :)

Thanks again,

Jim

: It all depends on the place of a conflict. If in "source" directory
: on server all files are readable (not locked for reading) then your
: automation script would be as simple as

: // make copies to local work folders
: SyncLocalDir "N1:\\fold1\fold2\\...\\foldn",
: "v:\\central_toplev\\folder1", True, True, True, True
: SyncLocalDir "N2:\\fold1\fold2\\...\\foldn",
: "v:\\central_toplev\\folder2", True, True, True, True
: ...
: SyncLocalDir "Nn:\\fold1\fold2\\...\\foldn",
: "v:\\central_toplev\\folderN", True, True, True, True

: Here you would replace N1...Nn with share names. In case if you don't have
: permanent shares, you can set/reset them dynamically, but from my
: expirience setting/changing drive mappings is not reliable, one day it
: works other it doesn't, restart the server then it works again and
: suddenly stops working.
: That's why I would recommend using either SyncRemoteDir if you have an agent
: or master scheduler on each server or using SyncFTPDir if you have FTP
: servers running.

: In all cases you can play with last 4 parameters in Sync...Dir, which are
: "copy new files", "update changed files", "delete
: missing files", and finally "copy subdirectories
: recursively"

: If you set the "delete missing files" parameter to False then in
: theory you can replicate all servers to the common central directory, but
: again this is fine only if you don't expect conflicts (file locks)
: occurring in that central directory. In case if you do then use 2 step
: process: 1 - replicas to "working" directory (all servers); 2 –
: replicas or copies from "working" directories to the central
: directory.

: To have the process continue after and error and get an email notice use the
: following method: Dim LastError, string

: SERVER1: OnErrorGoTo ERROR1
: SyncLocalDir ….
: GoTo SERVER2

: ERROR1: GetLastError LastError
: MailSend "me@myserver.com", "password", "Error
: occurred while replicating files from server 1", LastError

: SERVER2: OnErrorGoTo ERROR2
: SyncLocalDir ….
: GoTo SERVER3

: ERROR2: GetLastError LastError
: MailSend "me@myserver.com", "password", "Error
: occurred while replicating files from server 2", LastError

: ……

: SERVERn: OnErrorGoTo ERRORn
: SyncLocalDir ….
: Exit

: ERRORn: GetLastError LastError
: MailSend "me@myserver.com", "password", "Error
: occurred while replicating files from server n", LastError

: Again, Sync…Dir are high-level file replication methods. If you need
: fine-tuned error handling and file processing options you can use low
: level methods DirEx, FileTransferEx, FileCopyEx, xcopy.exe, FileCopy (
: http://www.softtreetech.com/24x7/archive/44.htm ) and some other.

Hi again,

Finally got round to looking for the SyncXXXDir examples you mentioned as being available in the "on-line help" but can't find anything other than the command syntax descriptions in the on-line help from within 24x7 scheduler - wher else can I look for examples ?

Thanks,

Jim

: Jim,

: Most people use simple SyncFTPDir or SyncRemoteDir and combinations to
: replicate files from one source computer to one target computer. I think
: in your case you also can use this method first replicating files to a
: local "working" directory to avoid potential file locks and
: sharing conflicts and then from there copy file by file (or directory by
: directory) to the main destination. The script to do such thing should be
: fairly simple. I can help you with this, but I need to know how you want
: to deal with file name conflicts (e.g. same file residing on multiple
: servers and copied to the same destination).

: By the way, examples of using SyncXXXDir and file-by-file process (Dir with
: LoopUntil/LoopWhile) are available in the on-line help.

There is an example button displayed in the WinHelp button bar just below the menu bar. Open topic of interest and then click this button.

: Hi again,

: Finally got round to looking for the SyncXXXDir examples you mentioned as
: being available in the "on-line help" but can't find anything
: other than the command syntax descriptions in the on-line help from within
: 24x7 scheduler - wher else can I look for examples ?

: Thanks,

: Jim