Tag Archives: automation framework

Why having versioned documentation

Heya,

very often we have our project’s piece of code under a versioning system. This has proven to be of real help. It helps the developer to observe the changes, remember the reason behind them and code just for the differences.

The documentation should obey the same rules.

“My taller friend” pointed out that the documentation is split, by functionality, in at least two categories. One would be to describe the characteristics of an entity or process. The second to describe a list of checks to be made in order to validate an entity.

By combining those two we look at the ideal documentation as follows. A dynamical part, with unchecked checks that is included automatically after each clone of the previous version. A static part that is being altered from one version to another.

 

Real life simplified example:

We have a a folder that can contain an infinite depth of folders and an infinite number of files.

We work hard enough to create a good enough script that validates that each folder has the require image and name based on the documentation provided by the stakeholder.

We copy our folder onto a new location and add a few folders and files.
Now we have two ways of writing some documentation in order to help the automation:

We work hard enough, again, to parametrize the initial script based on the re-written documentation.
OR
We copy the first script and just alter the changes.

 

The versioned documentation allows the team to adapt faster without over-thinking the technical solution.

I would like to read your opinions,
Gabi

Page Objects – My friend is not insane

Hello reader,

I am writing this as I am thinking of a work colleague who tries to propose the page objects notion. To get a grip of his effort versus the general acceptance of it imagine Don Quijote versus the windmills.
As in any other post we shall start by understanding the problem at hand.

Baseline:

When we write automated tests, in almost any keyword driven framework, we must implement an action for a selector/locator. This set of key:value will be under a name. This name is specific for an area of a page of the application. If it is a website it will be a webpage, if it is a software program it will be the state of a window. From now on, this will be called a “screen”.

This screen contains multiple key:values, one for each specific action. Please note the “specific” word and think about it. So far we have screens with multiple specific key:values.

On the web, it is very likely that a screen will feature more pages at once. Let’s look at this image:

websiteLayout

From this image it is obvious that most of the screens will have the “Header”,”Categories” and the “Footer” more or less present. Some slight changes will occur for the logged/anonymous areas, except that it is always the same. The “Dynamic content” is the actual driver of the screen. This area is the reason why the users are receiving the info.

Another problem is related to the duplicated code. Every time we write something twice there is a 50% chance that on update we forget about the other piece. Also there is a 100% chance of having to work twice to maintain.

The last problem refers to tests versioning. We like to complain that xxx and yyy changed the locators; however we do little to avoid it. If the project is aiming for a difficult release it is very likely that an older backup of the codebase will be kept. This is why our test should be aware of the version required to be run.

Problem:

How do we keep the functionalities grouped in such a way that we do not have duplicated code and we maintain a versioning system?

My answer:

1) We look around in the project and draw a map, similar to the one I made earlier. Don’t fall into the trap of going too deep with the granularization. I bet that it will take an amount of time unjustifiable to the product manager. This will give us an idea of what is fixed.

2) We create a sketch of the screens and write in that sketch the name of the areas discovered earlier together with their particularities for this step. Those will be our page objects.

3) We add to the sketch the specific actions and validations.

4) In the runner class we receive as many variables as page objects and areas there are. Those variables should start from 1. They represent the version of the scripts.

5) We code each piece of the areas and take into consideration the flags that will cover the states and the version.

6) We code each of the page objects taking into consideration the input data and the version.

7) We go out and drink

This method of automation can be applied under any type of framework and has a very good return of investment. It allows the team to maintain the tests as fast as possible.

Bonus:

It allows the tester, or the PM in case of BDD, to ask questions if the requirements skip areas that used to exist before.

For instance, in the mockups a list can hold 3 products without a scroll bar.  After we look at the tests we can point out the fact that the scroll bar is not present anymore on more than 3 products. This will raise a question that will get clarified into a red scroll bar.

Have fun,
Gabi

 

Behat – External selectors file – Definition of useless or genius

Hello,

Before we start, thank you Mario for listening to my idea and coming up with a better one :).

Last night I have finished implementing a feature into Behat on which I have mixed feelings about. It allows the team to store the selectors in an external file. Now, this sounded great at first and I did not vouch against. Mostly I was curios how it can be done.

I am saying it is a bad idea because sending parameters to the FeatureContext constructor can be done through several different ways:

  • – an array of parameters via the behat.yml that can be extended through import to include a different file
  • – a multi-dimensional array via the Scenario Outline and the Examples table
  • – a normal file include in FeatureContext

However, none of those actually inserts the values into the scenarios at runtime, replacing the keywords. This is when I get to say that it is genius.

But again, this should not be used in the first place. The whole purpose of BDD (in this context) is to be a tool that provides documentation for the stakeholders replacing a tests managements tool. Else we should not have used Gerkin to begin with. But what if the the target is the QA person, if so it makes sense. However, we are testing a framework built on top of Magento that we implement for the clients. Now it gets back to be a bad idea. The clients will not understand jack from our tests. On the other hand, since we share some of the code base but we implement custom functionalities on top of it, we want to maintain or selectors and values in a decoupled spot and not work on the code all the time. But the .feature files are quite decoupled as they are. Uhmm… reasons reasons.

I will let you reader to meditate upon using it or not and if you do use the code, please drop a comment why. Thank you in advance unknown friend.

We will start with a top down overview of the implementation. The xxxx.feature file looks like this:


1
2
3
4
5
6
7
8
9
10
11
  Scenario Outline: Invalid user login
    Given I am on homepage
    And I follow "Log In"
    And I fill in "email" with "<__email__>"
    And I fill in "pass" with "<__password__>"
    And I press "<__button__>"
    Then I should see "<__messageBody__>"

  Examples:
    |  |
    |  |

The current implementation needs to have the empty table at the end in order for Behat to generate an array at runtime. Probably this can be fixed in the code. The <> are regular placeholders. They are substitute with the values from the Examples table at runtime. The “__” (double underscore) are used in order to ensure some kind of differentiation between our keys and the table already existing.

In FeatureContext.php I have created this method that will be loaded on the @BeforeFeature hook:


1
2
3
4
5
6
7
8
9
    /**
     * @BeforeFeature
     */

    public static function prepare(\Behat\Behat\Event\FeatureEvent $event)
    {
      $feature = $event->getFeature();
      $exampleLoader = new ExamplesLoader();
      $exampleLoader->replaceExamples($feature);
    }

The class for this is a file called ExamplesLoader.php, located in the Bootstrap folder. This does not have to be loaded or anything because Behat automatically loads all the classes from that folder. Since the logic is in here I will post it based on functionality.

This will iterate through our scenarios and for each scenario will get the examples. In will return an array with the number of elements equal to the number of rows in the Examples table. In the current implementation it works with two.


1
2
3
4
foreach ($feature->getScenarios() as $scenario) {
            $examples = $scenario->getExamples();
// all the other pieces of code will go in here. Leave it blank.
}

This piece will glue together the current working directory ( getcwd() and the name of the file where the selectors/locators exist ). They will be glued by the DIRECTORY_SEPARATOR so it works on every operating system. Please note that the working directory is where behat.yml is located, not where the current file exists. This string will exist in the “$filePath” variable.
The “$holder” will contain a bi-dimensional read from the “tsv” file just read. If you want to read a file with a different separator please read the PHP documentation for fgetcsv(). The second argument is the separator. Also, if the examples table is longer an iteration inside the $holder[$row] is required because we want to have data for all the rows, not just two.


1
2
3
4
5
6
7
8
9
10
11
12
$filePath = join(DIRECTORY_SEPARATOR, array(getcwd(), 'locatorsFile.tsv'));
$holder = array();
$row=0;
if (($handle = fopen($filePath, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, "\t")) !== FALSE) {
        //if you are thinking that it would be better to iterate over many elements,
        //don't later you will use only key:value
        $holder[$row]=array($data[0],$data[1]);
        $row++;
    }
    fclose($handle);
}

The $rows is a variable created by Behat which stores all the values of the Examples table. Each element of this array is an array of what is inside between 2x| (pipe) on that specific row in examples. Basically here is where we want to add our keys and values. Because after we are inserting them, the framework will handle all the logic that there is to come. The setRows($rows) is a method that locks in place this table for tests creation.


1
2
3
4
5
6
7
// Add our global examples
foreach($holder as $value){
    $rows[0][] = $value[0];
    $rows[1][] = $value[1];
}
//and we send the data to the examples table
$examples->setRows($rows);

Now our table will include all the data from locatorsFile.tsv. Here’s how that file looks like on the inside:


1
2
3
4
__button__  Send
__messageBody__ Invalid login or password.
__email__   asdkjasdj@askdjaskjd.com
__password__    asdadasd

Have a nice day,
Bye bye!

Behat Contexts – A stretch of imagination – Episode 3 – Our Poison

Hello,

This will be a short post. Out of all the poisons we have picked the http://www.php.net/traits one for our custom context “classes”. It keeps the OOP idea for later Behat versions and it allows us to have auto-complete in the FeatureContext.php. No auto-complete in the included contexts but since they are for helpers it is ok.

Behat Contexts – A stretch of imagination – Episode 2 – The almost solution

Hello reader,
in the previous post I have presented 4 viable alternatives to our problem.

“How to have multiple contexts in Behat and maintain the OOP structureparameterssupport an expanding number of context files for a scalable application and most of all, have code hinting available in all the contexts.”

None of them quite hit the sweet-spot. From the coding perspective, for PHP 5.3 we have no tools, we I have tried a hack. The idea was to have the smaller contexts extend each other in the IDE, this provides the code hinting, but at runtime replace the code so that it accommodates Behat’s useContext syntax and logic. This should provide a win-win.

in FeatureContext.php


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
public function __construct(array $parameters)
{
    //get the path to bootstrap in a variable
    $pathToBootstrap = join(DIRECTORY_SEPARATOR,array(getcwd(),"features","bootstrap"));

    //get an array with all the file names that have the Context.php in their name
    $filesFromBootstrap = glob(join(DIRECTORY_SEPARATOR,array($pathToBootstrap,"*Context.php")));

    //for every file modify the internal structure in order to behave as expected by Behat
    foreach($filesFromBootstrap as $filename){
        $handler = fopen($filename,"r");
        $finalClass = fread($handler,filesize($filename));
        $customContexts = array("HelperContext","FrontendContext","BackendContext");
        foreach($customContexts as $context){
            if(stristr($finalClass,"extends ".$context)){
                $finalClass = str_replace("extends ".$context,"extends BehatContext",$finalClass);
                $writeHandler = fopen($pathToBootstrap.DIRECTORY_SEPARATOR."temp.php","w");
                fwrite($writeHandler,$finalClass);
                fclose($writeHandler);
                break;
            }
        }
        fclose($handler);
        $className = str_replace(".php","",basename($filename));

        //include the temporary file
        include($pathToBootstrap.DIRECTORY_SEPARATOR.'temp.php');

        //create the context using Behat's implementation
        $this->useContext($className, new $className($parameters));
    }

    $this->login = $this->getSubcontext("login");
}

 

in the “lower” classes the code looks like:
class HelperContext extends MinkContext{ }
class Helper2Context extends HelperContext{ }
so on and so forth. This will provide the code hinting.

 

Note: The code is not complete, currently it crashes with “Fatal error: Cannot redeclare class BackendContext” ” at the temporary file inclussion. This happens because the contexts are already loaded by Behat before the FeatureContext constructor. In order to fix this, for PHP 5.3 we need to create a patch or drop the already initialized object or to modify the included class’ name. For the second I did not test the behavior or stability of the framework.

Feel free to use the code as desired and fix it.

The next post will include the PHP 5.4 “traits” functionality for the helpers contexts.

Have a nice day,

Gabi

Testing framework – EP 2 – What types of automation are there

Hello,

now that we have identified the fact that automation is required let’s have a look at the approaches already available.

  1. Code driven testing
  2. Graphical interface testing (GUI)

The code driven testing involves testing the classes, modules and/or libraries.
The graphical interface testing involves the emulation of keyboard and mouse actions. The output is visible on the screen.

How to chose between one and the other?

This question arises because the GUI projects are composed of code but not all the applications have a GUI.

If the project’s code exposure through visual feedback for both the configuration data (Admin) , manipulation feedback (Frontend) and it’s scope is to facilitate the actions of the visitors, the GUI only approach is enough.

If the configuration data is updated through a file (.tsv/.csv etc), some code driven testing is necessary for the input scenarios.

If all the project is an API, for instance, or its scope is to connect two systems, code driven testing is sufficient.

Based on this article we know for sure which way we should head with our automation process.