All posts by zaid

Laravel collections usage with crawler links

Recently I have been maintaining a package for crawling website internal links called Arachnid, this package crawls website links, and extract information about each page including: title, meta tags, h1 tags, status code along with other info, it returns link information as an array. I was searching for a convenient way to extract meaningful summary information from the library output, and that’s when I come a cross Laravel Collections, and it was really useful in my case.

Basic Information

Arachnid library returns information about website links similar to the structure below:

 
  "/" => array:14 [▼
    "original_urls" => array:1 [ …1]
    "links_text" => array:1 [ …1]
    "absolute_url" => "http://zrashwani.com/"
    "external_link" => false
    "visited" => true
    "frequency" => 1
    "source_link" => "http://zrashwani.com"
    "depth" => 1
    "status_code" => 200
    "title" => "Z.Rashwani Blog - I write here whatever comes to my mind"
    "meta_keywords" => "Zeid Rashwani, Zaid Rashwani, zrashwani, web development, LAMP, PHP, mysql, Linux, Symfony2, apache, DBA"
    "meta_description" => "Zeid Rashwani personal blog, mostly contains technical topics related to web development using, LAMP, Linux, Apache, MySQL, PHP, and other open source technologies"
    "h1_count" => 3
    "h1_contents" => array:3 [ …3]
  ]
  "" => array:7 [▼
    "original_urls" => array:3 [ …3]
    "links_text" => array:2 [ …2]
    "visited" => false
    "dont_visit" => true
    "external_link" => false
    "source_link" => "http://zrashwani.com"
    "depth" => 1
  ]
  //... other links

As shown, each element store information about link including:
* original_urls: original url before normalization.
* links_text: text inside <a> tag for the link to appear in.
* absolute_url: absolute version of the url.
* visited: whether this link is visited/crawled or not.
* frequency: frequency on the link appearing in the website.
* source_link: the page which this link first crawled from.
* depth: on which level/depth the link is found.
* status_code: status code of the page (ex. 200, 404…etc.).
* title, meta description, meta keywords: meta information about the page.
* h1_count, h1_contents: number of <h1> tags, and their contents.

Main Functions

Laravel collections offer large set of functions, mainly it is wrapper over functional methods like: filter, map, reduce, each.
The main methods that I used in my case are the following:

  • filter: filter out any elements of an array that you don’t want.
  • map: transform each item in an array into something else.
  • each: loop over collection elements.

Those methods represents the main functional building blocks, which are wrappers over native php functions like: array_walk, array_filter, array_map but with cleaner OOP approach that can easily be used with method chaining.  Also these functions are higher order functions that take a function callback as a parameter.

As a basic example the link collection will be like following example:

<?php 
$url = 'http://zrashwani.com/'; //website url to be crawled 
$crawler = new \Arachnid\Crawler($url, 3);

$crawler->traverse();
$links = $crawler->getLinks();
$collection = collect($links);

Manipulate using Collections

1. Arrange links by page source:
So each array is arranged parent page URL is displayed as key, and value will be links info that parent page contains using groupBy function:

<?php
$linksBySource = $collection->groupBy('source_link');

The output will be an associative array, with each source page as key, and collection of links that this page contains, similar to the result below:
LinksCollection {#276 ▼
  #items: array:18 [▼
    "" => LinksCollection {#261 ▶}
    "http://zrashwani.com/" => LinksCollection {#117 ▼
      #items: array:144 [▶]
    }
    "http://zrashwani.com/category/technical-topics/" => LinksCollection {#84 ▼
      #items: array:2 [▶]
    }
    "http://zrashwani.com/category/anime-reviews/" => LinksCollection {#124 ▶}
    "http://zrashwani.com/about-me/" => LinksCollection {#168 ▶}
    "http://zrashwani.com/wcf-ssl-service-with-php/" => LinksCollection {#207 ▶}
    "http://zrashwani.com/tag/php/" => LinksCollection {#212 ▶}
    "http://zrashwani.com/author/zaid/" => LinksCollection {#295 ▶}
    "http://zrashwani.com/sonata-admin-bundle-multiple-connection/" => LinksCollection {#213 ▶}
    "http://zrashwani.com/tag/database/" => LinksCollection {#52 ▶}
    "http://zrashwani.com/tag/symfony/" => LinksCollection {#140 ▶}
    "http://zrashwani.com/applying-version-stamp-symfony-sonataadmin/" => LinksCollection {#239 ▶}
    "http://zrashwani.com/server-sent-events-example-laravel/" => LinksCollection {#145 ▶}
    "http://zrashwani.com/pagination-optimization-symfony2-doctrine/" => LinksCollection {#162 ▶}
    "http://zrashwani.com/tag/mysql/" => LinksCollection {#282 ▶}
    "http://zrashwani.com/materialized-views-example-postgresql-9-3/" => LinksCollection {#219 ▶}
    "http://zrashwani.com/simple-web-spider-php-goutte/" => LinksCollection {#270 ▶}
    "http://zrashwani.com/page/2/" => LinksCollection {#218 ▶}
  ]
}

2. Get External Links:
The external links can be retrieved by using filter function  according to “external_link” key as below:

<?php
$externalLinks = $collection->filter(function($link_info){
            return isset($link_info['external_link'])===true 
                       && $link_info['external_link']===true;
        });
        

The output will include all external links in the website as below:
LinksCollection {#64 ▼
  #items: array:112 [▼
    "http://www.wewebit.com" => array:14 [▼
      "original_urls" => array:1 [▶]
      "links_text" => array:1 [▶]
      "absolute_url" => "http://www.wewebit.com"
      "external_link" => true
      "visited" => true
      "frequency" => 1
      "source_link" => "http://zrashwani.com/"
      "depth" => 1
      "status_code" => 200
      "title" => "WeWebit - Web Development specialists"
      "meta_keywords" => "Wewebit, Website, Mt4, Forex web design , Development, Forex website design ,Forex web development ,forex bo system , Forex Backoffice Systems ,forex CRM , FX toolskit, forex toolskit , Forex client area , forex client cabinet , members cabinet , forex IB system , ecommerce website development"
      "meta_description" => "Web Development Company in jordan providing development and design services, including Forex Solutions , News Portals , Custom Web Applications, online e-commerce solutions"
      "h1_count" => 2
      "h1_contents" => array:2 [▶]
    ]
    "http://jo.linkedin.com/pub/zaid-al-rashwani/14/996/180/" => array:10 [▼
      "original_urls" => array:1 [▶]
      "links_text" => array:1 [▶]
      "absolute_url" => "http://jo.linkedin.com/pub/zaid-al-rashwani/14/996/180/"
      "external_link" => true
      "visited" => false
      "frequency" => 1
      "source_link" => "http://zrashwani.com/"
      "depth" => 1
      "status_code" => 999
      "error_message" => 999
    ]
    "https://twitter.com/zaid_86" => array:14 [▶]
    ...
  ]
}    

3. Filter and search links by depth:
This useful for getting the links that first appeared in the nth level in the website, for example getting links in depth=3 in the website as below:

<?php
$depth = 3;
$depth2Links = $collection->filter(function($link) use($depth){
            return isset($link['depth']) && $link['depth'] == $depth;
        });
        

The output will be similar to this:
LinksCollection {#734 ▼
  #items: array:141 [▼
    "/introduction-to-sphinx-with-php-part2/" => array:8 [▶]
    "/category/technical-topics/page/2/" => array:8 [▶]
    "/anime-watched-in-summer-2015/" => array:8 [▶]
    "/anime-fall-2015-shows/" => array:8 [▶]
    "/anime-winter-2015-watch-list/" => array:8 [▶]
    "/anime-winter-2014-watch-list/" => array:8 [▶]
    

or you can get simple statistics about how many links in each level in the website by combining groupBy function with mapWithKeys function – which is same as map function but returns key/value pair – as below:

<?php
$linksGroupedByDepth = $collection->groupBy('depth')
        ->mapWithKeys(function($depthGroup,$depth){
            return [$depth =>$depthGroup->count()];
        });
        

it will display how many links exist in each site level:
LinksCollection {#824 ▼
  #items: array:4 [▼
    0 => 1
    1 => 75
    2 => 300
    3 => 141
  ]
}

4. Get Broken links:
The broken links can be retrieved by filtering items according to status code, success pages have status code between 200 and 299, so anything else will be considered as broken link, as below:

<?php 
$brokenLinks = $collection->filter(function($link){
    return isset($link['status_code']) && 
            ($link['status_code'] >= 300 || $link['status_code'] <200);
});

or better broken links can be grouped according to the page where link exists, using groupBy function, extract only summary information using map function:
 <?php
        $brokenLinksBySource = $collection->filter(function($link){
            return isset($link['status_code']) && 
                    ($link['status_code'] >= 300 || $link['status_code'] <200);
        })->map(function($link){
           return [
                'source_page' => $link['source_link'],
                'link'        => $link['absolute_url'],
                'status_code' => $link['status_code'],
                'links_text'  => $link['links_text'],
               ];
        })
        ->unique('link')
        ->groupBy('source_link'); 
        

5. Getting pages that have no title or h1 tags:
This is useful for SEO purposes, and can be done using filter method:

<?php
$linksWithMissingTitle = $collection->filter(function($link_info){
           return empty($link_info['title']); 
        });

<?php
//getting pages with no <h1> tags
$missingH1Pages = $collection->filter(function($link_info){
           return $link_info['h1_count']==0; 
        });
        

The output will contain all pages with no <h1> tag as below
LinksCollection {#823 ▼
  #items: array:2 [▼
    "http://wordcomat.com" => array:14 [▼
      "original_urls" => array:1 [▶]
      "links_text" => array:1 [▶]
      "absolute_url" => "http://wordcomat.com"
      "external_link" => true
      "visited" => true
      "frequency" => 2
      "source_link" => "http://zrashwani.com/simple-web-spider-php-goutte/"
      "depth" => 2
      "status_code" => 200
      "title" => ""
      "meta_keywords" => ""
      "meta_description" => ""
      "h1_count" => 1
      "h1_contents" => array:1 [▶]
    ]
    "http://matword.com" => array:14 [▼
      "original_urls" => array:1 [▶]
      "links_text" => array:2 [▶]
      "absolute_url" => "http://matword.com"
      "external_link" => true
      "visited" => true
      "frequency" => 4
      "source_link" => "http://zrashwani.com/simple-web-spider-php-goutte/"
      "depth" => 2
      "status_code" => 200
      "title" => ""
      "meta_keywords" => ""
      "meta_description" => ""
      "h1_count" => 0
      "h1_contents" => []
    ]
  ]
}

6. Getting duplicate titles between different URLs:
This is useful to see if you have any different pages that have same title – which may effect your SEO negatively by combining several methods of filter,groupBy,unique and map as following:

<?php
        $duplicateTitlePages = $collection
                   ->filter(function($linkInfo){
                       return $linkInfo['visited']===true;
                   })                   
                   ->groupBy('title',true)
                   ->unique('absolute_url')        
                   ->filter(function($links){
                       return count($links)>1;
                   })->map(function($linkGroup){
                       return $linkGroup->map(function($linkInfo,$uri){
                            return $uri;
                       })->values();
                   });

The output will be a collection with the duplicate title as a key, along with URLs that have that title as following output:
LinksCollection {#652 ▼
  #items: array:1 [▼
    "Z.Rashwani Blog - I write here whatever comes to my mind" => LinksCollection {#650 ▼
      #items: array:2 [▼
        0 => "http://zrashwani.com/"
        1 => "/"
      ]
    }
  ]
}

More

There are many other ways to use Laravel collections method combinations to get useful information from such link info array, like: getting links with Internal server errors or links with long/short meta description…etc. And the more you are used to using collections the more useful information that can be retrieved.

WCF SSL Service with PHP

We had a task recently that required our team – me with my colleague Ahmad to write php code to integrate with existing WCF webservice that includes attaching SSL certificates to requests. The application used to integrate with third-party banking system using a form of B2B web service.
In the following post, the main steps we used to write PHP code compatible with WCF:

Existing .net application

The original code was written in C# using WCF webservice over SOAP, it was attaching SSL certificate in PFX format -that includes all certificate chain -, and a separate private key file (as .key format).

PFX format (PKCS#12) is binary format which is usually used in windows to export/import SSL certificates; it stores certificate, intermediate certificates – if there is any – and private key in one file that can be encrypted and signed.

In original C# code, they defined a class that inherits SoapHttpClientProtocol which was used to add SSL certificates to soap request.

Generating SSL files

The existing code was attaching ssl certificate in .pfx format file, so I converted it to .pem format (which is the standard format for openssl) and extracted the key as separate file using openssl commands as following:

openssl pkcs12 -in certificate.pfx -nocerts -out key.pem -nodes
openssl pkcs12 -in certificate.pfx -nokeys -out certificate.pem
openssl rsa -in key.pem -out certificate.key 

Those commands will generate public ssl certificate (*.pem) and private ssl key (*.key) file.

To make sure the generated certificates are correct for php, I wrote this basic function to test:
<?php
function validatePublicPrivateKeys($public_key_file, $private_key_file) {
    $public = openssl_pkey_get_public(file_get_contents($public_key_file)); 
    $public_error = openssl_error_string();
    if(!empty($public_error)){
        echo "Error in public key:".$public_error."\n";
    }else{
        echo "public key is valid\n";
    }

    $private = openssl_pkey_get_private(file_get_contents($private_key_file), 'passphrase-here');
    $private_error = openssl_error_string();
    if(!empty($private_error)){
        echo "Error in private key:".$private_error;
    }else{
        echo "private key is valid\n";
    }
}

Extending SoapClient Class

Normally SSL certificate can be used in php SOAP request by setting `local_cert` parameter in SoapClient Constructor. however I found this option somehow limited, because there is no ability for a private key to be attached as separate file in the request.
so what we did is to extend soap client and override __doRequest method to be based on curl to send soap request as HTTP message as following:

class MySoapClient extends \SoapClient{

      public function __doRequest($request, $location, $action, $version, $one_way = FALSE) {
            $curl = curl_init($location);
            //setting curl options and data here...
            //...
            
    }
}      

Depending in SOAP version, override to curl header values is needed, in my case version used is SOAP 1.2, so the headers will be as following:
    $curl = curl_init($location);
    $headers = array(
        "Content-type: test/xml;charset=\"utf-8\";action=\"" . $location . '/' . $action . "\"",
        "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
        "Cache-Control: no-cache",
        "Pragma: no-cache",
        "Content-length: " . strlen($request),
    ); 
        
    curl_setopt($curl, CURLOPT_HEADER, TRUE);
    curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);        

Attaching SSL certificates

I set ssl certificate public file, key file – generated earlier – and passphrase to curl request as following:

    curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 2);
        
    curl_setopt($curl, CURLOPT_SSLKEYPASSWD, 'passphrase');
    curl_setopt($curl, CURLOPT_SSLKEY, 'your-private-key.key');
    curl_setopt($curl, CURLOPT_SSLCERT, 'your-certificate.pem');

In case curl error “Peer certificate cannot be authenticated with known CA certificates” appeared – which is usually happens in windows- you shall download CA certificate bundle from Mozilla – or other trusted source -, and save it to your system and set CURLOPT_CAINFO option in curl:
curl_setopt($curl, CURLOPT_CAINFO, “C:\full-path-to\cacert.pem”);
or better set it globally in your php.ini:
curl.cainfo=c:\full-path-to\cacert.pem

The final class implementation will be like the following snippet:

<?php
class MySoapClient extends \SoapClient{
      public $certificate_ssl_location = "/full/path/to/your-ssl-public-certificate";
      public $private_key_location = "/full/path/to/your-ssl-private-certificate";
      public $ssl_passphrase = "password";
      public $ca_cert_file = "/full/path/to/ca-file";
      
      public function __doRequest($request, $location, $action, $version, $one_way = FALSE) {

        // Call via Curl and use the timeout
        $curl = curl_init($location);

        $headers = array(
            "Content-type: test/xml;charset=\"utf-8\";action=\"" . $location . '/' . $action . "\"",
            "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
            "Cache-Control: no-cache",
            "Pragma: no-cache",
            "Content-length: " . strlen($request),
        ); //SOAPAction: your op URL

        curl_setopt($curl, CURLOPT_VERBOSE, TRUE);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
        curl_setopt($curl, CURLOPT_POST, TRUE);
        curl_setopt($curl, CURLOPT_POSTFIELDS, $request);
        curl_setopt($curl, CURLOPT_HEADER, TRUE);
        curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);

        curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 2);
        
        curl_setopt($curl, CURLOPT_SSLKEYPASSWD, $this->ssl_passphrase);
        curl_setopt($curl, CURLOPT_SSLKEY, $this->private_key_location);
        curl_setopt($curl, CURLOPT_SSLCERT, $this->certificate_ssl_location);
        curl_setopt($curl, CURLOPT_CAINFO, $this->ca_cert_file);

        $response = curl_exec($curl);

        if (curl_errno($curl)) {
            throw new Exception(curl_error($curl));
        }
        curl_close($curl);

        $soap_start = strpos($response, "<soapenv:Envelope");
        $soap_response = substr($response, $soap_start);

        if (!$one_way) {
            return $soap_response;
        }
    }
}

you can call soap request normally by initializing object from MySoapClient class (instead of native SoapClient class)
and now the integration with secured WFC web service works fine 🙂

Anime watched in summer 2015

Summer of 2015 was not a good anime season for me, it has few interesting shows, but mostly was not as good as previous season’s.

Here is a list of the shows that I watched in that season, ranked according to my opinion from least to most enjoyable:

  • 5. Working!!!

    working-3

    Genres : Comedy, Slice of Life, Work Life
    Brief : Third season for working anime, that revolves around high school students working in family restaurant with comedy oriented atmosphere along with awkward relationships and funny characters.

    My opinion : third season of the same funny comedy, this season contained noticeable character development.

  • 4. Aoharu x Kikanjuu

    English Name: Aoharu x Machine Gun

    Aoharu-x-Kikanjuu-anime-730x730

    Genres : Action, Comedy, Shounen
    Brief : Hotaru Tachibana is a girl usually dressed as a boy, who accidentally enters the world of survival games and joins “Top Gun Gun” team along with two members.

    My opinion : Funny story that revolves around survival games and friendship between team members with light comedy, it is funny yet enjoyable show.

  • 3. Akagami no Shirayuki-hime

    English Name: Snow White with the Red Hair

    red-hair-princess

    Genres : Drama, Fantasy, Romance, Shoujo
    Brief : Shirayuki is a pharmacist with apple-red hair, becomes friend with prince Zen after rescuing her from her kingdom prince who demands that she become his concubine.

    My opinion : at first I thought this a traditional story between beautiful cheerful girl and a young prince, but as the story progress, I found other sides for each character, it is slight romance and good storyline.

     

  • 2. Rokka no Yuusha

    English Name: Rokka: Braves of the Six

    rokka-anime

    Genres : Action, Adventure, Fantasy, Magic, Mystery
    Brief : When the Demon God about to awakens, the Goddess of Fate selects six heroes and give them with the power to save the world. Adlet, is one of those “The Heroes of the Six Flowers.”. However, as the heroes gather, there was seven of them instead of six, one is the impostor and the doubts gives to Adlet to be the “fake” one.

    My opinion : fantasy show with good fighting scenes, the mystery of finding the seventh was really tricky and shocking at the same time.

  • 1. Charlotte

    Charlotte_anime

    Genres : Drama, School, Super Power
    Brief : some teenage boys and girls who wield special powers that disappears after they become adults, gathered in a school for students with supernatural abilities. Yuu Otosaka joins the school counsel to gather all guys with super powers to refrain them from using their abilities.

    My opinion : Beginning as slow based standard super powers story, good animation with lovely characters and have some surprising events as the story goes on.

Anime watched in fall 2015

Fall of 2015 was  good anime season for me, it has some interesting anime series.

Here is a list of the shows that I watched in that season, ranked according to my opinion from least to most enjoyable:

  • 5. Dance with Devils


    anime fall 2015 - Dance_with_Devils

    Genres : Demons, Romance, Shoujo, Supernatural
    Brief :  Ritsuka Tachibana is high school student that got summoned by student counsel which consistent of demon male characters searching for forbidden item called “Grimoire”. Many events take place after that starting with her mother being kidnapped,  conflict between vampire and demo worlds came to surface.
    My Opinion: it is an average anime with good animation and supernatural setting, but I found the story lacks interesting plot, also the musical part seems weird to me, it didn’t fit well in anime
  • 4. Owari no Seraph: Nagoya Kessen-hen

    English Name: Seraph of the End: Battle in Nagoya


    anime fall 2015 - Owari no Seraph Nagoya Kessen-hen

    Genres : Action, Drama, Shounen, Supernatural, Vampire
    Brief : Second season of Seraph of the End in which Yuichiro continues to learn and gain more power in order to save Mikaela. But humans get information that vampires were to launch an attack on Tokyo to exterminate exterminate Japan  Imperial Demon Army, so – to get ahead of vampires – the army launch their battle in Nagoya.

    My opinion : I found the second season is much better than the first one, it contains many interesting fights and it exposes some conspiracies between human and vampire worlds, hope next seasons  will get better.

  • 3. Kidou Senshi Gundam: Tekketsu no Orphans

    English Name: Mobile Suit Gundam: Iron-Blooded Orphans


    anime fall 2015 - Mobile_Suit_Gundam_IRON-BLOODED_ORPHANS_Poster

    Genres : Action, Drama, Mecha, Sci-Fi, Space
    Brief : The story takes place in Mars after 300 years of Calamity War, where earth government is the dominant over Mars economy. Independence movement began in Mars and outer colonies. Orga, Mikazuki and their comrades are young kids that are working for security company caught into many Mecha fights protecting Kudelia Aina BERNSTEIN who works for her people independence.

    My opinion : One of the best anime shows in the season, great animation, story line and soundtrack, I haven’t watched a Gundam show since “Gundam Wings” in late 1990’s, but this one is really good.

     

  • 2. Haikyuu!! 2


    anime fall 2015 - haikyuu!! season2

    Genres : Action, Adventure, Fantasy, Magic, Mystery
    Brief : Second season of Haikyuu, Karasuno training more after losing to Aoba Johsai, the team members go to training camp in Tokyo, and each of them improve his skills, working toward going to finals.

    My opinion : This is great show, I liked the logical improvement in team skill level by practicing and observing other teams, character relationships and background.

  • 1. One-Punch Man


    anime fall 2015 - one-punch-man

    Genres : Action, Comedy, Parody, Sci-Fi, Seinen, Super Power, Supernatural
    Brief : Saitama is hero of “fun” who became super hero after 3 years of training. Although he is not famous among other super heroes, he is capable of defeating any enemy by using only one punch.

    My opinion I though this show is boring in the first episode, but it turned hilarious show that is really funny, with unexpected jokes and amusing characters.

Save

Save

Save

Technical Books I read in 2015

2015 was a good year, I have read several books in it, some of them I studied cover-to-cover, and selected some interesting parts from others.
here is a list of those books:

SQL Anti-Patterns

Avoiding the Pitfalls of Database Programming

After I watched Percona webinar “How to Avoid Even More Deadly MySQL Dev Mistakes“, I decided that I must read this book. It explained many bad (or even incorrect) practices of SQL – some of them are widely spread in many frameworks and projects – like: polymorphic association and Metadata Tribbles.
The structure of discussing each topic is really useful; explaining theoretical concept, how to recognize, legitimate use and solution of each antipattern.
This book helped me improve many SQL related parts of my projects.

Modern PHP

This book explains the most recent features and tools of PHP language and community. It covers many parts that are not restricted to php development,
including: standards, profiling, testing, hosting, deployment with introduction to interesting related topics like: HHVM and vagrant …etc.

Linux and Linux Administration Handbook

This book is a complete guide to Linux administration, it is highly detailed book that I didn’t read all of it; but I selected several chapters that are interesting and contains important details to me as a developer.
Some of the sections that I read:

  • scripting and the shell (included introduction to server side scripting in bash, python and perl)
  • booting and shutdown (GRUB, user modes and startup scripts like upstart)
  • Controlling processes (processes components, states and monitoring tools)

I think I will get back to this book in near future to study many other chapters.

GIT Workbook

This book is relatively small book, but it contains excellent practical information about GIT. I think every developer working with GIT can benefit from reading this book.

some parts that I found interesting: rebase, rebase interactive, GIT bisect for fault finding and submodules.
I wish if this book get updated soon to include more advanced stuff about GIT.

 

I have skim read some other e-books like “Percona MySQL Performance Optimization” along with regular reading of PHPArch magazine,
hopefully I will be able to read and learn more in 2016.

Using Sonata admin bundle with multiple connections

Last month I was participating in developing solution which provides reporting features using Symfony that deals with data partitioned over multiple MySQL servers. Data was sharded among two reporting databases that have identical structure, each database store trading history for a separate company branch/trading system.
So basically, there is a bundle that deals with multiple connections – thus different entity managers- dynamically. Handling those connections in Symfony is straight forward, but for sonata admin bundle, it needed some work to get it work.

Project Structure

The project contains bundle called WebitReportingBundle which have the entities related to reporting database, I mainly use entity called MT4Trades for this post (it holds trades information performed by clients)
This bundle uses two entity managers, depending on the application context, here is portion of config.yml containing entity managers/connection mappings:

    orm:
        default_entity_manager: default
        entity_managers:
            default:
                connection: default
                mappings:
                    FOSUserBundle: ~
                    ApplicationSonataUserBundle: ~
                    SonataUserBundle: ~
                    AppBundle: ~                    
            reporting_real:
                connection: reporting_real
                mappings:
                    WebitReportingBundle: ~
            reporting_real2:
                connection: reporting_real2
                mappings:
                    WebitReportingBundle: ~

In the config file above, there are two extra entity managers (real_reporting & real_reporting2), each handles connection to separate database.

In Symfony, the entity manager can be specified with doctrine repository simply as following:

<?php

namespace Webit\ReportingBundle\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\Controller;

class DefaultController extends Controller {
    
    public function IndexAction(Request $request) {
        //retrieve trades from server 1
        $obj_server1 = $this->getDoctrine()
            ->getRepository('WebitReportingBundle:MT4Trades', 'reporting_real')
            ->findOneBy(['ticket'=>42342]);
                
        //retrieve trades from server 2
        $obj_server1 = $this->getDoctrine()
            ->getRepository('WebitReportingBundle:MT4Trades', 'reporting_real2')
            ->findOneBy(['ticket'=>123123]);
    }
}

but when I wanted to do the same on sonata admin module, there was no easy way to handle that.

How Sonata Admin Bundle handle multiple connections

When you create sonata admin file for certain entity with “SonataDoctrineORMAdminBundle“, “Admin” class will determine the entity manager to use by calling “getManagerForClass()” method in the associated “ModelManager” object.

As in the following code, which is taken from ModelManager class inside sonataDoctrineORMAdminBundle:

<?php 
namespace Sonata\DoctrineORMAdminBundle\Model;

class ModelManager implements ModelManagerInterface, LockInterface
{
    public function getEntityManager($class)
    {
        if (is_object($class)) {
            $class = get_class($class);
        }

        if (!isset($this->cache[$class])) {
            //detect entity manager based on class name automatically
            $em = $this->registry->getManagerForClass($class);

            if (!$em) {
                throw new \RuntimeException(sprintf('No entity manager defined for class %s', $class));
            }

            $this->cache[$class] = $em;
        }

        return $this->cache[$class];
    }
}

However, this method will be an issue for entities when dealing with sharded databases, so in my case, the first matching entity manager will be always selected – according to mappings defined in config.yml – and there is no direct way to specify the preferred entity manager for the admin class.

Solution

In order to work around this problem, I performed the following:

  1. Define custom model manager extending the ModelManager class, and override methods “getEntityManager” and “createQuery” to use specific entity manager if it is defined in the class, as following:
    <?php 
    namespace Webit\ReportingBundle\Model;
    
    use Sonata\DoctrineORMAdminBundle\Model\ModelManager as BaseModelManager;
    use Sonata\DoctrineORMAdminBundle\Datagrid\ProxyQuery;
    
    class CustomModelManager extends BaseModelManager
    {
        /* @var $emName stores the preferred entity manager name */
        protected $emName;
        
        /**
         * set preferred entity manager name to be used in ModelManager
         * @param string $name
         */
        public function setEntityManagerName($name){        
            $this->emName = $name;
        }
    
        /**
         * {@inheritdoc}
         */    
        public function createQuery($class, $alias = 'o') {
            //adding second parameter to getRepository method specifying entity manager name
            $repository = $this->getEntityManager($class)
                               ->getRepository($class,$this->emName);
    
            return new ProxyQuery($repository->createQueryBuilder($alias));
        }
        
        /**
         * {@inheritdoc}
         */
        public function getEntityManager($class) {        
            if (is_object($class)) {
                $class = get_class($class);
            }                
            if (isset($this->cache[$class]) === false) {
                //return fixed value if preferred entity manager name specified
                if (isset($this->emName) === true) {
                    $this->cache[$class] = $this->registry->getEntityManager($this->emName);
                } else {
                    $this->cache[$class] = parent::getEntityManager($class);
                }
            }
    
            return $this->cache[$class];
        }
    }
    
  2. Add new model manager as a service and associate it with the admin service entries of the entity:
    services:        
        reporting.model_manager:
             class: Webit\ReportingBundle\Model\CustomModelManager
             arguments:
                 - '@doctrine'
    
        reporting.admin.trades:
            class: Webit\ReportingBundle\Admin\MT4TradesAdmin
            tags:
               - { name: sonata.admin, manager_type: orm, group: webit.admin.group.reporting, label: "Trades (server 1)", pager_type: "simple" }
            arguments: [null,Webit\ReportingBundle\Entity\MT4Trades, WebitReportingBundle:Admin\MT4Trades]        
            calls:
               - [setModelManager, ['@reporting.model_manager'] ]  
               
    
        reporting.admin.trades2:
            class: Webit\ReportingBundle\Admin\MT4Trades2Admin
            tags:
               - { name: sonata.admin, manager_type: orm, group: webit.admin.group.reporting, label: "Trades (server 2)", pager_type: "simple" }
            arguments: [null,Webit\ReportingBundle\Entity\MT4Trades, WebitReportingBundle:Admin\MT4Trades]        
            calls:
               - [setModelManager, ['@reporting.model_manager'] ]
               
                       
    Here I am having two admins for the same entity, each admin shall use different connection.
  3. Define the entity manager name inside the admin classes by overriding setModelManager method, as following:
    <?php
    
    namespace Webit\ReportingBundle\Admin;
    
    use Sonata\AdminBundle\Admin\Admin;
    
    /**
     * admin for MT4Trades (Server 1)
     */
    class MT4TradesAdmin extends Admin{
    
        protected $emName = 'reporting_real';
        protected $baseRouteName = 'admin_webit_reporting_mt4trades';    
        protected $baseRoutePattern = 'admin-reporting-mt4trades-server';
         
        public function setModelManager(\Sonata\AdminBundle\Model\ModelManagerInterface $modelManager) {
            parent::setModelManager($modelManager);
            //override setModelManager to specify the preferred entity manager for this admin class
            $modelManager->setEntityManagerName($this->emName); 
        } 
        
        //remaining admin class methods...
    }
       
        
       

    and here the other admin class that is connecting to the second database:
    <?php
    
    namespace Webit\ReportingBundle\Admin;
    
    /**
     * admin for MT4Trades (Server 2)
     */
    class MT4Trades2Admin extends MT4TradesAdmin{
    
        protected $emName = 'reporting_real2'; //second entity manager will be used
        
        //specify route name and pattern to get two admin classes work with same entity
        protected $baseRouteName = 'admin_webit_reporting_mt4trades2';    
        protected $baseRoutePattern = 'admin-reporting-mt4trades-server2';
         
        //...
    }

    I extended second admin class from the first, so they have identical configurations/logic, with different connection used.

Conclusion

Now, I have two identical admin classes for the same entity, each using different entity manager correctly. and if additional server is deployed in the future, I just have to define new admin classes -along with new connection/entity manager – with placing the correct $emName value.

multi-connection-sonata

Anime watched in spring 2015 season

Spring season of 2015 has many interesting shows that I find interesting. Here is a list of the shows that I watched in that season, ranked according to my opinion from least to most enjoyable:

  • 7. Nisekoi:

    English Name: False Love

    w1280_Nisekoi_2nd_Season_backdrop

    Genres :  Comedy, Romance, School, Shounen, Harem
    Brief : second season of Nisekoi, where Raku is the center of his story with his locket given to him by his childhood girlfriend, the same characters exists in this season, with few new female characters appearing.

    My opinion : first season was much better, the same story and incidents occurring in this season with little advancements or plot, I force myself to watch many episodes hopping in will get better, but it didn’t 🙁

  • 6. Owari Seraph

    English name: Seraph of the End: Vampire Reign

    3d745e151fb24686d50cf4173f52916d

    Genres :     Action, Drama, Shounen, Vampire, Supernatural
    Brief : A mysterious virus appeared on Earth which killed every infected human except children, Hyakuya Yuuichirou is a kid taken with his ‘family’ from orphanage as livestock by the vampires who seems to rule the remaining humans. Hyakuya goal in life is to kill vampires.

    My opinion : Slow based first season that has some entertaining fights – and other boring parts – , the end of this season reveals that this is just the beginning, I guess it will turn into much better show and greator plot in the next season which is scheduled in October.

  • 5. Arslan Senki

    English Name: The Heroic Legend of Arslan

    arslan senkai

    Genres : Action, Adventure, Drama, Fantasy, Historical, Supernatural
    Brief : Arslan is 14 years old prince of Pars, who lost almost everything at war against neighboring nation “Lusitania”, he strive with his friends to reclaim his kingdom and build new better country. This anime is re-make of popular show from 1990s.

    My opinion : Interesting story that have very good animation and nice characters, the storyline is somehow realistic and has a some charm based on historical elements, hopefully it will have advancements in plot as the story advance. Although I prefer the old anime character design better.

  • 4. Ore Monogatari!!

    English Name: My Love Story!!

    Ore-Monogatari

    Genres : Comedy, Romance, Shoujo
    Brief : Gouda Takeo is a freshman in high school with giant body, not a good looking  guy so he is not popular with girls, yet he is very nice person with many friends. One day Takeo save a girl called Yamato in the bus from a pervert, and his romance life begins.

    My opinion : Very nice shoujo anime that worth watching, it is rare to have a romance story without the male protagonist being handsome. This anime has a good heartwarming story about the life on Takeo, his relationship with his girlfriend Yamato , his best friend Sunakawa along with many other characters. It really emphasize on the known idea: don’t judge people by their appearances.

  • 3. Plastic Memories

    plastic memory

    Genres : Drama, Romance, Sci-Fi
    Brief : Tsukasa Mizugaki starts working in SAI Corporation which specializes in manufacturing, management and terminating service of expired Andriods that have human emotions known as “Giftia”. He meets young Giftia named Isla and starts working with her in the same team, and so their relation begins…

    My opinion : Very nice Sci-fi story that have somehow deep idea behind it, what would you handle it if your lifespan was predetermined? it is impressive anime with beautiful artwork, nice characters and enjoyable story.

     

  • 2. Fate/stay night: Unlimited Blade Works 2nd Season

    Fate-stay-night-Unlimited-Blade-Works-Simulcast-Ein-Blick-in-die-erste-Episode

    Genres :  Action, Fantasy, Magic, Shounen, Supernatural
    Brief : Remake of popular fate/stay night. It is about group of magician “masters” who summon heretic spirits of the past called “servants”, fighting to win the holy grail; a mysterious device that can grant the wish of the winner. Unlimited blade works focus on the story of Shirou and Archer.

    My opinion : Great action fantasy anime, being produced by Ufotable, it has really beautiful artwork, animation and deep philosophy behind the story. Each one of the masters and servants has his complex personality along with interesting background story. Fate/stay night UBW is excellent anime, but to be honest, fate zero still is the superior.

     

  • 1. Yahari Ore no Seishun Love Comedy wa Machigatteiru. Zoku

    English Name: My Teen Romantic Comedy is Wrong As I Expected

    yahari -zoko

    Genres : Comedy, Romance, School
    Brief : Hachiman Hikigaya the loner master is back, although he is not interested in making any special school experiences, his relationships with “Services Club” colleagues Yukinoshita and Yuigahama advance, and his character starts to develop slowly.

    My opinion : This is absolutely the best anime I watched in this year. RomCom-Romantic Comedy is impressive show, with great story line, interesting character development and clever conversations. This second season is one of the best emotional intelligent shows I have watched in a while, I really hope there will be a third season.

     

Applying version stamp to Symfony & SonataAdmin

I spend a lot of time in my job in developing and maintaining backoffice systems to process some workflow logic for Forex portals. Our systems are developed using Symfony2 and depend highly on SonataAdminBundle with deep customization to impose business rules for Forex companies.

Recently some data inconsistency appeared in a system of one of our clients, and after digging into logs, I found that the cause of the problem is two users processing same user application in almost same time range, and that caused one edit operation to override the other one, and many other undesired consequences occurs after that.

so in order to fix this issue, and prevent it from happening again, I worked on adding “Version Stamps” to my entity to maintain offline data consistency within the application, and I would like to share here what I learned.

 

Version Stamps and offline consistency

Version Stamps is a field that changes every time a writing operation is performed on the data, it is used to ensure that no one else has changed data of that row before applying your modification.
There is several ways to implement version stamps, and the simplest way is an integer value which is noted on the read, its value is compared to the submitted data before write, and once validated, the write operation take place with increasing version stamp value.
Let’s say there is a form bind to an entity in Symfony application, version stamp column will be present in the entity and added as a hidden field in the form, once submitted version stamp value submitted will be compared to the one in the database currently, to ensure that no other edit is performed on that entity -in the time between displaying your form initially and submitting it-, if the condition fails, the update operation will be rejected, thus by adding error via constraint.

Implementation in SonataAdmin

The implementation of this concept is quite simple as following:
In the desired entity, I applied those modifications:

  1. Define new field to hold stamp value.
  2. Mark the field to be a version stamp, using doctrine @Version annotation; this will cause doctrine to update versionStamp field every time the object is persisted to the database
<?php

namespace Webit\ForexCoreBundle\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * RealProfile
 *
 * @ORM\Table(name="forex_real_profile")
 * @ORM\Entity(repositoryClass="Webit\ForexCoreBundle\Repository\RealProfileRepository")
 * @ORM\HasLifecycleCallbacks()
 */
class RealProfile
{
    
    /**
     * @var integer
     *
     * @ORM\Column(name="id", type="integer", nullable=false)
     * @ORM\Id
     * @ORM\GeneratedValue(strategy="IDENTITY")
     */
    private $id;
    
    /**
     * @var integer
     *
     * @ORM\Version 
     * @ORM\Column(name="version_stamp", type="integer")
     */
    private $versionStamp;
    
    /* Other columns, getters, setters here */

    
}

In Admin class, the following is added:

  1. In configureFormFields() method, the version stamp is added as hidden field, also I set mapped option to false, to prevent persisting its value along the form. version stamp value must be modified only via PreUpdate() method inside the entity.
    <?php
    
    namespace Webit\ForexCoreBundle\Admin;
    
    use Sonata\AdminBundle\Admin\Admin;
    use Sonata\AdminBundle\Datagrid\ListMapper;
    use Sonata\AdminBundle\Datagrid\DatagridMapper;
    use Webit\ForexCoreBundle\Entity\RealProfile;
    
    class RealAccountsAdmin extends Admin
    {
        protected function configureFormFields(\Sonata\AdminBundle\Form\FormMapper $formMapper)
        {
            $formMapper->add('versionStamp','hidden',array('attr'=>array("hidden" => true, 'mapped'=>false)))
            //other fields and groups...
        }
    }
  2. Here is the important point, which is validating version_stamp posted from the form against the one that is already saved in the database. There is two methods to apply that, one if by using doctrine locking mechanism, and the other is using sonata inline validation to add extra validation layer by implementing validate() method in order to apply additional validation layer.

    option 1:

    class RealAccountsAdmin extends Admin
    {
        /**
         * {@inheritdoc}
         */    
        public function getObject($id)
        {
            $uniqid = $this->getRequest()->query->get('uniqid');
            $formData = $this->getRequest()->request->get($uniqid);        
            
            $object = $this->getModelManager()->find($this->getClass(), 
                    $id, 
                    \Doctrine\DBAL\LockMode::PESSIMISTIC_WRITE, 
                    $formData['versionStamp']);
            
            foreach ($this->getExtensions() as $extension) {
                $extension->alterObject($this, $object);
            }
    
            return $object;
        }
    
        /**
         * {@inheritdoc}
         */      
        public function update($object) {
            try{
                parent::update($object);
            }catch(\Doctrine\ORM\OptimisticLockException $e) {
                $this->getConfigurationPool()->getContainer()->get('session')
                        ->getFlashBag()->add('sonata_flash_error', 'someone modified the object in between');
            }
        }

    This approach will take advantage of doctrine locking support. Here is brief explanation:

    • I have overridden getObject() method in admin class, to add two extra parameters for getModelManager()->find() method;
      third parameter indicates locking type, I used here LockMode::PESSIMISTIC_WRITE
      fourth parameter represents the expected version stamp value -to compare with database value before flushing.
    • I have overridden update($object) method so I catch OptimisticLockException exception and add error flash message to handle it

    option 2:
    In this approach, I used sonata inline validation to detect the form as invalid before even trying to persist and flush the object to the database:

        /**
         * {@inheritdoc}
         */
        public function validate(\Sonata\AdminBundle\Validator\ErrorElement $errorElement, $object) { 
            //get all submitted data (with non-mapped fields)
            $uniqid = $this->getRequest()->query->get('uniqid');
            $formData = $this->getRequest()->request->get($uniqid);
            $submitted_version_stamp = $formData['versionStamp'];        
            
            $em = $this->getConfigurationPool()
                  ->getContainer()->get('doctrine')->getManager();
            
            //get up-to-date version stamp value from the database
            $class_name = get_class($object);        
            $q = $em->createQuery("select partial o.{id,versionStamp} from $class_name o"
                                    . " where o.id=".$object->getId());        
            $saved_data = $q->getArrayResult();        
            $saved_version_stamp = $saved_data[0]['versionStamp'];
            
            //compare version stamps and add violation in case it didn't match
            if($saved_version_stamp != $submitted_version_stamp){
                $errorElement->addViolation('Record data seems outdated, probably someone else modified it, please refresh and try again.')->end();
            }
        }
        
    Here is more details about the operations performed inside this method:

    • To get versionStamp value submitted via form inside that method, I used:
      $uniqid = $this->getRequest()->query->get('uniqid');
      $formData = $this->getRequest()->request->get($uniqid);
      $submitted_version_stamp = $formData['versionStamp'];
    • To get an updated value of versionStamp that is stored in the database, I used doctrine query that retrieve partial object
      $em = $this->getConfigurationPool()
                 ->getContainer()->get('doctrine')->getManager();
              
      $class_name = get_class($object);        
      $q = $em->createQuery("select partial o.{id,versionStamp} from $class_name o"
              . " where o.id=".$object->getId());        
      $saved_data = $q->getArrayResult();        
      $saved_version_stamp = $saved_data[0]['versionStamp'];

      *If you retrieve the whole object from the database again, it will cause many issues specially if doctrine result cache is enabled.
    • Then compare the two values with each other, if those values are not equal, an error shall be appear to the user, and that is performed by calling $errorElement->addViolation() method
      if($saved_version_stamp != $submitted_version_stamp){
          $errorElement->addViolation('Record data seems outdated, probably someone else modified it, please refresh and try again.')->end();
      }

That’s all, now I can perform some basic test.

Test Solution

In order to verify that this mechanism solved the issue, I emulated the inconsistency behavior, by opening sonata admin edit page on two browsers, and then try to modify the data and submitted on each browser consequently.
The browser that got submitted last, will not save data and error will appear
Record data seems outdated, probably someone else modified it, please refresh and try again.”
versionstamp test
In that way, the inconsistency is prevented by stopping the second user from overriding the information modified by the other user.

At last

“Version Stamp” approach helped me in preventing data inconsistency in my Symfony/SonataAdmin application, hope it will help others who face similar scenario. I would like to know if anyone else has other idea or better way to handle that issue.

Anime Winter 2014 Watch list

Here is a list of the anime shows that I watched in winter season 2014 (in fact two of those shows are left from the previous season), ranked according to my opinion from least to most enjoyable shows:

  • 7- D-Frag

    D-Frag anime winter 2014

    Genres : Comedy, School, Seinen.
    Brief : The story about Kazama Kenji who is a high school delinquent who is forced somehow to join “game” club with four girls, so he found himself spending time with the club members playing various strange yet funny games.
    My opinion : This anime is a light comedy anime that does not have any story, I enjoyed some of the episodes, but Its not my favorite comedy style.
  • 6- Toaru Hikuushi e no Koiuta (Pilot Love Song)

    pilot love song - winter anime 2014

    Genres : Adventure, Drama, Romance.
    Brief : Story about a prince who’s parents were killed in revolution and forced to live as normal person. Then, he joins military journey in order to become a pilot along with the same girl that led the revolution against his family.
    My opinion : good drama anime with some good battle scenes, however, I found the plot not deeply engaging.
  • 5- Golden Time

    golden time anime - fall/winter 2014

    Genres : Comedy, Romance, Seinen.
    Brief : Banri is a freshman in the university who lost his previous memories in an accident after high school graduation, the story evolves around his relationships with his girlfriend, previous crush and other friends.
    My opinion : good love and drama anime, characters are somehow realist with real concerns and problems. Its not one of the best in those genres but still worth watching.
  • 4- Mikakunin de Shinkoukei (Engaged to unidentified)

    Engaged to the unidentified - winter anime 2014

    Genres : Slice of Life, Comedy, Romance.
    Brief : On the sixteenth birthday of young girl Kobeni, she surprised that she has a fiance, as decided by her granpa, and this fiance will live in their house along with his little sister. the story about lovely relationship between the couple, and their life activities in the school and house.
    My opinion : I really enjoyed my time watching this anime, funny and lovely relationship along many comedy scenes.
  • 3- Nisekoi (Fake Love)

    nisekoi anime winter 2014

    Genres : Comedy, Romance, Shounen, Harem.
    Brief : Raku Ichijou is a high school student and son of gang leader in his city, he is forced to fake being boyfriend to Kirisaki who is daughter for another gang in the city. This puts him into weird situation especially that he has crush on his classmate Onodera.
    My opinion : Enjoyable show, made me laugh all the time about the awkward fake relationship between Ichijou and Kirisaki. Highly recommend it for anyone who loves comedy love stories.
  • 2- Tokyo Ravens

    3f70d3a4ba7219cbe5822b8839a1ef171368822754_full

    Genres : Comedy, School, Shounen, Super Power, Supernatural.
    Brief : In an era where magic have major influence in the society, Harutora, who is descendant of prestigious Onmyoji family, has no talent or interest in magic. However, appearance of his cousin Natsume changes his opinion, and make his join Onmyo school.
    My opinion : great anime, unexpected twists in the story along with excellent fighting scenes and plot. I hope they produce second season soon.
  • 1- Gin No Saji 2nd Season

    silver-spoon - winter anime 2014

    Genres : Comedy, School, Shounen, Slice of Life.
    Brief : second season of “silver spoon” show, that continue story of Hachiken, who is high school student that joins agricultural school to leave his family problems behind. The anime shows the daily activities of farm people and along with challenges that face them.
    My opinion : this anime is very amusing and has great story line. It is both realistic, emotional and somehow inspiring. I think it’s the best slice of live anime I have watched in a while.