About a week ago we had noticed the image server we were using for the album art covers was failing. Looking into it we found that the image server must be under a high load and there really is not too much we could do about that since we do not own or control that server so we decided to add a new image server of our own to even out the load from all the album art covers. During the process though we had to shift the load somewhere since the image server was down about 80% of the time. I set up a small private server that handled the image requests until we could get this all worked out.
The setup of the new image server went rather smoothly except I could not decide which would be faster, archive 3.8 GB, upload it, unpack it or to just upload the un-compressed images eliminating 2 steps. Well now 3.8 GB uncompressed uploaded via ftp was the initial choice. Now keep in mind I have what “Comcast” calls 12MB connection speed. This was sold to me as 16MB+ but that’s another story. I pay almost 100 bucks a month for that. So anyway we started the ftp transfer. A day and a half later not even 20% complete :/ Kind of expected that though.
A friend of mine presented me with an option that he said should work out nicely “IF” I were using Linux/Linux like he. Well my friend is a very Linux minded fellow but I thought well we both know I have a Win/Lin setup but he knowing me presented the idea anyway since I have a habit of converting Linux type operability to Windows machines.
Well this idea of his turned out that converting or altering anything outside of a few config files and permissions was not even necessary. We set up RSYNC to synchronize the directories and files from my home/private server to the new image server that now had been partially uploaded via the ftp. RSYNC worked beautifully. The job still took longer than I would have preferred (16 hours including setup time) but completed without issue.
Everything I was finding for setting up RSYNC appeared as though I would need a client and server to make it all work. Turns out that was not even the case. I already had SSH setup on both the host and at home. I had been using putty to connect to SSH and this would have to change but yet again not an issue. Cygwin is also something I have been using for many years now and with it you can use SSH. So I had to create a new key pair.
After making the keys I then needed to put the public key in place on the host
scp id_rsa.pub user@host.com:~/.ssh/new_key
which is secure copy [public key] [you]@[yourhost]:[your home dir]/.ssh/[the new key]
Once this file was on the host I then log into the host and su to root. (Turns out to be an issue later). So now I then ran a few commands from within the ~/.ssh directory.
cat new_key
cat new_key >> authorised_keys
This added the new key to the authorized keys file.
Back at home open up Cygwin, try to ssh hoping that this time our key pair is used right because we want to be able to run bash scripts locally to take care of automatically synchronizing sites we need this to be done without any password hint added so we are never prompted to login which would cause our script(s) to fail. This is another place I failed at first creating this extra password hint when generating the key pair was not wanted.
Anyway with the keys done proper I was able to ssh to the host with Cygwin and not be prompted for a password. Looking good, Success so far 
Next I don’t want to go and attempt syncing the remaining 80% of the images just to find it did not work so next we perform a little test.
I go to the host and create a directory named test in the web root and add a few text files with random data in them. I then create a directory locally named test and put some files into it and also create a few directories inside it with files and so on. We want to make sure that
A: the newly created local directories/files get uploaded to the host.
B: the files on the host do not get overwritten unless changes have been made.
In my local user home directory I have an alias setup to a directory named scripts. Any bash file in it becomes accessible at the command line by simply typing its name from any location in the shell window. The bash file for this I named sync_serv. Below I will include the portion related to RSYNC. I am keeping the automation part to myself and leave you to write your own. This is what was left of the original code. Most of the options we had originally intended are not available on windows like –chmod=Dg=rwxs,Fgu=rw,Fo=r –no-times –human-readable –no-owner –no-group which is unfortunate but here is the script.
#!/bin/bash
rsync \
–verbose –archive –update –backup –recursive –checksum –rsh=ssh \
/cygdrive/D/test/ \
user@host.tld:/full_path_2/web/test/
echo RSYNC Completed
exit
Our test completed successfully. But wait! We do now have a few issues that must be taken care of. Permissions are all messed up. Since the RSYNC options we wanted were not available and the fact that we did this as a different user than the one who should own the files all we get is a 403 when trying to access any file that was RSYNC’ed yet the ones that were already in place are totally accessible.
So we log into the host again. CD to the web root. Then we need to do the following commands.
chown –recursive –verbose theusername.thegroupname .
find . -type d -print|xargs chmod 0775
find . -type d -print|xargs chmod g+s
find . -type f -print|xargs chmod 0644
Then all files and directories get set with the proper owner/group and default permissions. So everything in our test went well and now to perform the real SYNC. We got that going, about 12 hours later it completed with one error which I caused. Remember the test directory and that we tested the sync with it. Well that was inside the covers directory when we did this. The first time RSYNC ran on the local machine it built a list of directories and files. The test directory and its files were in the list. I deleted the directory and it’s contents after the rsync started and had built it into the list so we see this at the end.
rsync error: some files could not be transferred (code 23) at /home/lapo/packaging/tmp/rsync-2.6.6/main.c(791)
RSYNC Completed
Next for good measures we do not want everyone hot linking our album covers so we add the following in an htaccess file in the new image servers web root. This stops anyone from directly accessing the images and also from using them in their own pages. Only yourdomain.tld can access them.
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://yourdomain.tld/.*$ [NC]
RewriteRule \.(png|gif|jpg)$ – [F]
ErrorDocument 403 /index.php
<a href="http://asylum-et.com/browser.stats/"><img src="http://asylum-et.com/image.png?r=255" alt="Browser Stats"/></a>
<a href="http://asylum-et.com/browser.stats/"><img src="http://asylum-et.com/image.png?r=255&g=255" alt="Browser Stats"/></a>
<a href="http://asylum-et.com/browser.stats/"><img src="http://asylum-et.com/image.png?r=255&g=255&b=255" alt="Browser Stats"/></a>
<img src="http://asylum-et.com/image.png?r=0&g=0&b=255" alt="Browser Stats">