All posts by zaid

Optimizing and Compiling js/css files in php

In the last month, my team  in the company have been working on applying new theme to old project that we have, this project is more than 3 years old and it is written in relatively old technology (symfony 1.4/Propel ORM).

I wanted to find an automated method to optimize the javascript and stylesheet files that are served for this project (similar to the functionality of assetic in symfony2) , so I write a couple of files to automate this optimization, which do the following:

  1. Scan stylesheet folder and Optimize files using CssMin project:
    which compress the css file by removing whitespaces and comments, then minify.
  2. Scan javascript folder and Optimize  using google closure compiler:
    which parses javascript files, and convert it into better optimized form, as the closure page states:

    It parses your JavaScript, analyzes it, removes dead code and rewrites and minimizes what’s left. It also checks syntax, variable references, and types, and warns about common JavaScript pitfalls.

    note: I used CSSMin and google closure compiler, since they have the least dependencies, so I can utilize without installing additional packages, however, other options like Grunt or UglifyJs are really powerful but require npm to be installed.
  3. Creating new unique file names, through md5(resource_file_size) and copy it to the destination folder with the new name.
    This will prevent the browser caching for the modified files.

    note: other method for preventing browser cache is “cache busting” where you append changable query string to the resource file like

    <link rel="stylesheet" type="text/css" media="screen" href="/css/style.css?v=8" />
  4. Adding the association between the original file and the compiled file name to an array that will be used for rendering the resource path.

 

and here is the code of the task that performs the optimization:

<?php
include('CSSMinify.php'); //download CSS Min from http://code.google.com/p/cssmin/
class optimizeResourcesTask{

    private static $RESOURCES_ASSOCIATION = array();  //array to hold  mapping between original files and compiled ones
    
    private $source_js_folder = 'js/'; //the relative path of the source files for javascript directory, it will be scanned and its individual js files will be optimized
    private $target_js_folder = 'js-compiled/';    //result js files will be stored in this directory
    
    //target and source CSS folders preferred to be on the same folder level, 
    //otherwise path rewrite should be handled in the contents of the css files
    private $source_css_folder = 'css/'; //the relative path of the source files for stylesheet directory, it will be scanned and its individual css files will be optimized
    private $target_css_folder = 'css-compiled/';  //result css files will be stored in this directory   
    
    //path of the file that will hold associative array containing mapping between original files and compiled ones
    private $resource_map_file = '_resource_map.php';
    


    public function run() {
        // initialize the database connection

        $css_dir = __DIR__ .'/'. $this->source_css_folder;
        $this->optimizeCSSResources($css_dir);

        $js_dir = __DIR__.'/'.$this->source_js_folder;
        $this->optimizeJSResources($js_dir);

        $this->writeMappingData();
        
        
        $this->cleanupOldData($this->target_css_folder, 'css');
        $this->cleanupOldData($this->target_js_folder, 'js');
    }

    /**
     * iterating over the CSS directory and optimizing all of its contents
     * every single CSS file found it this directory will be passed to optimizeOneCSS() method in order to optimize
     * @param string $dir
     */
    protected function optimizeCSSResources($dir = null) {
        if (is_null($dir)) {
            $dir = __DIR__ . '/'.$this->source_css_folder;
        }

        if ($handle = opendir($dir)) {
            while (false !== ($entry = readdir($handle))) {
                if ($entry != "." && $entry != "..") {
					
                    if (is_dir($dir . $entry)) {
                        $this->optimizeCSSResources($dir . $entry);
                    } else {
                        $this->optimizeOptimizeOneCSS($dir . $entry);
                    }
                }
            }
        }
    }


    /**
     * optimize one CSS file by using CSSMin library to minify the contents of the file
     * generate new file name using hash of its file contents
     * add the new file name association to $RESOURCES_ASSOCIATION static variable in order to write resource association array later
     * @link "http://code.google.com/p/cssmin/" CSSMin documentation
     * @param string $file css file absolute path to minify
     */
    protected function optimizeOptimizeOneCSS($file) {
	
        print('trying to optimize css file ' . $file. chr(10));
        $info = pathinfo($file);
        if ($info['extension'] == 'css') {
            $optimized_css = CssMin::minify(file_get_contents($file));

            $target_css_dir_absolute = __DIR__ . '/' . $this->target_css_folder;
            if (!is_dir($target_css_dir_absolute)) {
                mkdir($target_css_dir_absolute);
                chmod($target_css_dir_absolute, 0777);
            }

            $new_name = md5($optimized_css) . '.css';
            file_put_contents($target_css_dir_absolute .  $new_name, $optimized_css);


            $file_relative_path = str_replace(__DIR__ , '', $file);
			
            self::$RESOURCES_ASSOCIATION[$file_relative_path] = '/' . $this->target_css_folder .  $new_name;

            print('CSS FILE: ' . $file . ' has been optimized to ' . $target_css_dir_absolute .  $new_name. chr(10));
			
        } else {
            print("skipping $file from optimization, not stylesheet file, just copying it". chr(10));
            
            $file_relative_path = str_replace(__DIR__ . $this->source_css_folder, '/', $file);
            
            $target_css_dir_absolute = __DIR__ . '/' . $this->target_css_folder .dirname($file_relative_path);
            
            if (!is_dir($target_css_dir_absolute)) {
                mkdir($target_css_dir_absolute);
                chmod($target_css_dir_absolute, 0777);
            }
            
            copy($file, $target_css_dir_absolute.'/'.basename($file));
        }
    }
	
	
    /**
     * iterating over the JS directory and optimizing all of its files contents'
     * every single JS file found it this directory will be passed to optimizeOneJS() method in order to optimize/minimize
     * @param string $dir
     */
    protected function optimizeJSResources($dir = null) {

        if (is_null($dir)) {
            $dir = __DIR__ . '/'.$this->source_js_folder;
        }
        print('getting JS inside ' . $dir. chr(10));

        if ($handle = opendir($dir)) {
            while (false !== ($entry = readdir($handle))) {
                
                if ($entry != "." && $entry != "..") {

                    if (is_dir($dir . $entry)) {
                        $this->optimizeJSResources($dir .  $entry);
                    } else {
                        $file_path = $dir . $entry;
                        $pathinfo = pathinfo($file_path);
                        if($pathinfo['extension']=='js'){
                            $this->optimizeOneJS($file_path);
                        }else{
                            print($file_path.' is not passed to optimization, its not a valid js file'. chr(10));
                        }
                    }
                }
            }
        }
    }

    /**
     * optimize one JS File using "Google Closure Compiler", 
     * store the optimized file in target directory named as hash of the file contents
     * add the new file name association to $RESOURCES_ASSOCIATION static variable in order to write resource association array later
     * @link  "https://developers.google.com/closure/compiler/docs/gettingstarted_api" "Google Closure Compiler API"
     * @param string $file js file absolute path to optimize/minify
     */
    protected function optimizeOneJS($file) {
	
        print("trying to optimize js ". $file. chr(10));

        $post_fields = array(
            'js_code' => file_get_contents($file),
            'compilation_level' => 'SIMPLE_OPTIMIZATIONS',
            'output_format' => 'text',
            'output_info' => 'compiled_code',
        );



        $ch = curl_init("http://closure-compiler.appspot.com/compile");
        curl_setopt($ch, CURLOPT_POST, count($post_fields));
        curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post_fields));
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        $optimized_js = curl_exec($ch);
        
        if(strpos($optimized_js,'Error(22): Too many compiles performed recently.') !==false){ //Google Closure API returned error on too many compilation
            trigger_error($file.' is failed to be compiled, skipped...');
            return;
        }
        
        curl_close($ch);

        $target_js_dir_absolute = __DIR__ . '/' . $this->target_js_folder;
        if (!is_dir($target_js_dir_absolute)) {
            mkdir($target_js_dir_absolute);
            chmod($target_js_dir_absolute, 0777);
        }

        $new_name = md5($optimized_js) . '.js';
        file_put_contents($target_js_dir_absolute . '/' . $new_name, $optimized_js);


        $file_relative_path = str_replace(__DIR__ , '', $file);
        self::$RESOURCES_ASSOCIATION[$file_relative_path] = '/' . $this->target_js_folder . $new_name;

        print('JS FILE: ' . $file . ' has been optimized to ' . $new_name. chr(10));
    }

    /**
     * write $resources_map array stored in $RESOURCES_ASSOCIATION into $resource_map_file file that will be used in generating
     * the association between original JS/CSS files and the optimized/minimized ones
     */
    protected function writeMappingData() {
        $str = "<?php \$resources_map = array(";
        foreach (self::$RESOURCES_ASSOCIATION as $original_file => $optimized_file) {
            $str .= "'$original_file'=>'$optimized_file', " . chr(10);
        }
        $str .= "); ";

        $f = fopen(__DIR__ . '/' . $this->resource_map_file, 'w+');
        fwrite($f, $str);
        fclose($f);

        echo 'mapping data written to ' . $this->resource_map_file . chr(10);
    }

    /**
     * this function will remove any file that exists $target_js_folder and $target_css_folder
     * and doesnot exist in $RESOURCES_ASSOCIATION array, most probably that were generated from old builds and not used anymore
     * @param $dir the relative path of the directory to cleanup
     * @param $extension_to_filter the extension that is going to be cleaned (either css or js), the idea is to ignore cleaning static resources like font files, ex. woff, eot
     */
    protected function cleanupOldData($dir, $extension_to_filter){
        $dir_absolute = __DIR__.'/'.$dir;
		
        if ($handle = opendir($dir_absolute)) {
            while (false !== ($entry = readdir($handle))) {
                if ($entry != "." && $entry != "..") {
                    
                    if (is_dir($dir_absolute .  $entry)) {
                        $this->cleanupOldData($dir.$entry, $extension_to_filter);
                    }else{
                        $file_path = $dir_absolute .  $entry;
                        $pathinfo = pathinfo($file_path);
                        print('examining   /'.$dir .  $entry. chr(10));
                        
                        //including the packup files deletion
                        if(in_array($pathinfo['extension'], array($extension_to_filter, $extension_to_filter.'~')) && !in_array('/'.$dir . $entry, self::$RESOURCES_ASSOCIATION)){                            
                            unlink($file_path);
                            print($file_path.' is deleted....'. chr(10));
                        }
                    }
                }
            }
        }        
    }
}


$task = new OptimizeResourcesTask();
$task->run();
echo 'optimization done...';

after running this class, it will generate “_resource_map.php” file that contains array to store mapping between original resources and compiled ones, its contents will be similar to this:

<?php $resources_map = array('/css/main.css'=>'/css-compiled/d41d8cd98f00b204e9800998ecf8427e.css', 
'/css/redmondjquery-ui-1.8.14.custom.css'=>'/css-compiled/d41d8cd98f00b204e9800998ecf8427e.css', 
'/css/style.css'=>'/css-compiled/f866be09baee73d596cb578b02d37d29.css', 
'/js/jquery-1.5.1.min.js'=>'/js-compiled/6c1b3f8d121bfefdad82fb4854a8f254.js', 
'/js/jquery-ui-1.8.14.custom.min.js'=>'/js-compiled/e34d1750b1305e35327964b7f0ea6bb9.js', 
'/js/jquery.cookie.js'=>'/js-compiled/08bf7e471064522f8e45c382b2b93550.js', 
'/js/jquery.easing-1.3.pack.js'=>'/js-compiled/0301f5ff89729b3c0fc5622b7633f4b8.js', 
'/js/jquery.fancybox-1.3.4.js'=>'/js-compiled/cb707a9b340d624510e1fa27d3692f0e.js', 
'/js/jquery.fancybox-1.3.4.pack.js'=>'/js-compiled/f58ec8d752b6148925d6a3f14061c269.js', 
'/js/jquery.min.js'=>'/js-compiled/5ee7bdd2dbbdec528925cb61c3010598.js', 
'/js/jquery.validate.min.js'=>'/js-compiled/9d28b87b0ec7b4e3195665adbd6918be.js', 
); 

now we need a function to get the optimized version of the files (in production environment only):

<?php 
function resource_path($file){
    global $config;
	if($config['env'] == 'prod'){ //serve compiled resource only on production environment
		include '_resource_map.php';
		if(isset($resources_map[$file])){
			return $resources_map[$file]; //return compiled version of the file
		}
	}
	return $file;
} ?>

and here is the use of example css/js file in “header.php”:

<link href="<?php echo resource_path('/css/style.css') ?>" rel="stylesheet" type="text/css" />
<script src="<?php echo resource_path('/js/jquery.min.js') ?>" type="text/javascript" ></script>

now once you render the page in production environment, the optimized css/js will be served instead of the original ones, as follows:
<link href="/css-compiled/f866be09baee73d596cb578b02d37d29.css" rel="stylesheet" type="text/css" />
<script src="/js-compiled/5ee7bdd2dbbdec528925cb61c3010598.js" type="text/javascript" ></script>

Now everything works good and you can serve optimized versions of your resource files with minimal effort upon each update on your website. Whenever there is some amendments to the website theme,  I would only run optimizeResourcesTask to optimize files and serve them automatically in production environments.

I used this code for my project s that written in native php or old symfony version, but as I mentioned  earlier there is some frameworks like symfony2 assetic that perform similar functionality with long list of optimizers available.

MySQL Slave monitor in Bash

I have several slave setups for different sites, that I use for several purposes, like: taking packups from the slave, or using it for reporting/read only operations for masters that have high write query rates.

But, in the last week I found that a couple of the slaves have been stop for some time without me noticing that (one of them was not working for about a month), so that mean that I don’t have valid packups for the last month! So I thought that I must automate monitoring the slave, so I could receive notification if the slave stopped working for a reason or another.

Why Slave Stop working?
MySQL replication works to synchronize master data to slave database in real time asynchronously, so the master data is copied as it is to the slave through executing the same write queries (stored in the master BinLog) to the slave, so any error that occur in one of these queries may cause the slave to halt.

For example: one time a client of mine tried to submit very large article in his news portal, so in order to get this working I increased the value of max_allowed_packet directive without applying the same configuration to the slave, and that caused the slave SQL_THREAD to stop working.

How to check slave is working or not?
The basic command to check the status of your slave is

SHOW SLAVE STATUS\G

this command will give you summery information about your slave, the most useful pieces of information here is:

  1. Slave_SQL_Running : indicates whether the thread responsible for writing SQL queries is running normally (any error on one of the queries that it executes will stop the thread and it will set value for Last_SQL_Error and Last_SQL_Errno)
  2. Slave_IO_Running: indicates whether the thread responsible for receiving commands sent from master and write them to relay log in the slave is working.
  3. Seconds_Behind_Master: how much the slave is delayed from the master (usually it should minimum, if this value is continuously increasing, this indicates a problem even if both SQL_THREAD and IO_THREAD are running)

So, I searched on the internet to see if there are some scripts that perform that check, and I found a couple of useful links but it didn’t contain all the functionality that I needed, so I wrote my own bash script that monitor the slave and send an alert once there is problem in replicating data from the master.
The script below do the following:

  1. extracts the information about slave using SHOW SLAVE STATUS command in mysql
  2. checks the problems about slave (using the three piece of information discussed above), and write them to a log file in the tmp directory
  3. if there is a problem, send the contents of the log file to the DBA in order to for him to fix.
#!/bin/bash
server_name="Slave #1 - Test DB" #change this in order to indicate which slave you are monitoring now
admin_email='[email protected]' #email of the database administrator to recieve notification

# change mysql credentials in the following commands if you running monitor using a user other than root
sql_thread_running=$(mysql -e "show slave status\G" | awk -F":" '/Slave_SQL_Running/ { print $2 }' | tr -d ' ')
io_thread_running=$(mysql -e "show slave status\G" | awk -F":" '/Slave_IO_Running/ { print $2 }' | tr -d ' ')
seconds_late=$(mysql -e "show slave status\G" | awk -F":" '/Seconds_Behind_Master/ { print $2 }' | tr -d ' ')
seconds_late=$(($seconds_late+0))

if [ "$sql_thread_running" = "No" ] || [ "$io_thread_running" = "No" ] || [ $seconds_late -gt 3600 ]; then

log_file="/tmp/log_slave_status_$(date +%m-%d-%Y-%H:%M)"
echo "Slave status report on $(date +%m-%d-%Y-%H:%M)" >> $log_file
echo "Error in slave on $server_name" >> $log_file
if [ "$sql_thread_running" = "No" ]; then
echo "SQL Thread not running" >> $log_file
fi

if [ "$io_thread_running" = "No" ]; then
echo "IO thread not running" >> $log_file
fi

if [ $seconds_late -gt 3600 ]; then #formattting how the latency of the slave behind master should be displayed
display_late="$seconds_late seconds"
if [ $seconds_late -gt 60 ]; then
display_late="$display_late = $(($seconds_late/60)) minutes"
fi

if [ $seconds_late -gt 3600 ]; then
display_late="$display_late = $(($seconds_late/3600)) hours"
fi

echo "slave is behind master by $display_late" >> $log_file
fi

#echo send alerts to the admin
mail -s "Slave status on $server_name" $admin_email < $log_file
echo "Slave not running, alerts sent to the admins..."
else
echo "slave is running normally, no problem detected :)"
fi

you can download the bash script from here

 

Put bash in crontab
The final step is to put this simple script in crontab in order to check the status periodically,
just open crontab file using

crontab -e

And I added it to run each half an hour, as follows:

# some environments require that you set the shell path inside crontab in order to run properly
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
*/30 * * * * /usr/local/sbin/monitor_slave.sh

 

Now, whenever there is problem in one of my slave, a message will be delivered to my email giving me a summery information about the problem, so I am not worried anymore about my slave(s) status.

Fate Zero

I normally don’t like fantasy anime, and would prevent them and seek for slice of life anime that has an experience some how closer to normal people.
However, unlike I expected at first, Fate Zero became one of the favorite animes due to its great storyline.

* note: this anime produced after “Fate Stay Night“, however, you can watch it without watching “Fate Stay Night” without spoiling anything, in fact I did that, and I was glad that I did 🙂

The story of this anime take place before 10 years from “Fate Stay Night”  anime, telling the story of the 4th holy grail war. Seven mages will summon seven spirits in order to fight and the winner will obtain the reward which is the “Holy Grail” that grants the wish of the victor (even if he asked a miracle).

The spirits represents a historical heroes that each of them have some ambition or wish that he couldn’t accomplish in his past life, so they have a strong motive to win the holy grail in to complete their ambition. So there are the King arthur/Saber, and Alexander the Great/Rider and other heroes

The beauty of this anime is the combination of different characters, each mage has its own philosophy in the life that drive him to put his life on the line in order to get the grail. and each hero spirit that fight with the mage, has its own idols and ambitions that form his/her “King Path”.

Regarding the animation, the anime art is too good, the fighting scenes and music are just great.

Usagi Drop

One of the best anime shows that I have ever watched. This anime illustrates the true life of a foster parent with his child with their worries, happy times and normal life routines.

This anime talks about parenthood. It begins when the main character “Daikichi” gets to his grand father funeral. Surprisingly, the family found that the late grand father has a daughter named “Rin” from his mistress, when the family discussed what to do with this little girl, rather that getting her into orphanage like the family almost settled on,  Daikichi decided to raise her at his house.

Life of Daikichi will not be the same anymore, there is new responsibility that imposed a change in his life style.

Its simply smoothing anime with simple yet attractive day to day life of normal people.

Hi all,

Hello,

this is my personal blog, where I can write whatever comes to my mind. Content topics vary from technical subjects (specially PHP,MySql, Symfony), to totally carefree topics of anime, movies…. and whatever 🙂