Automated visual layout testing

• Is it possible to unit test that something is visually rendered as wanted (for some definition of wanted)? Specifically for HTML and CSS renders your page as "expected"

I would personally imagine such tools allow you to define a test in a declarative manner of how something is supposed to look and then run a complex heuristic on the actual file your testing that determines how "close" it is to your declarative definition.

开发者_如何学JAVA
• Would there be any other ways to do machine testing on, say a HTML file for whether the content is visually consistent.

I'm not asking to almost impossible question of whether something "looks good", but rather whether it looks consistent or whether it matches a given pattern.

• The final question is, if these tools don't exist, are they worth writing? Would tools of such a nature actually give you meaningful results that are not bound by a very large margin of error. I.e. could they accurately predict whether some change to code has broken your visual layout without giving a large amount of false negatives or missing things that are obviously broken.

I would personally imagine such tools allow you to define a test in a declarative manner of how something is supposed to look and then run a complex heuristic on the actual file your testing that determines how "close" it is to your declarative definition.

There is a way to test exactly what you want using Galen Framework, which is a tool I have helped to develop. It is a special tool with its own language which allows to express how the website should look like. It is based on Selenium and can be run in Selenium Grid if you need to check it in different browsers. At the moment what the tool is doing - it is getting the location of specified elements on page and checks them relatively to each other. So for example if you want to check that menu panel is below header and stretches to the width of a browser and has 50 pixel height, you do it like this:

menu
    below: header 5 px
    width: 100% of screen/width
    height: 50px

In case you don't want to assert for exact values you can do it like this

menu
    below: header 0 to 5 px
    width: 90 to 100% of screen/width
    height: 45 to 55px

The tool is usefull for testing responsive design as you can use custom tags to match exact specifications for your needs. For instance if we would like to express that a "side-panel" on desktop should be on the right side of a screen with static width 300 pixels, but on mobile it should drop below the "content" and stretch to complete width of a browser, you do it like this:

@ desktop
---------------------
content
    inside: screen 0px left

side-panel
    inside: screen 0px right
    near: content 0px right
    width: 300px

@ mobile
--------------------
content, side-panel
    width: 100% of screen/width

side-panel
    below: content 0px 

There are not a lot of information about the tool since it is a new one. At the moment there is only one article explaining how to test the responsive web page using Galen Framework http://mindengine.net/post/2013-11-16-tdd-for-responsive-design-or-how-to-automate-testing-of-website-layout-for-different-devices/.

You can also find complete documentation on official website http://galenframework.com

You may try some tools like this: Screenster,mogotest. Both using visual matching for layout testing.

I don't think you need complex image analysis. As was mentioned there will be small differences between browsers anyway. Also something you might dismiss as too small might be very important, such as a small icon changing.

If you want cross browser testing it would be extremely difficult to automate. But you could create a tool to check for unexpected visual changes. Saving a desired DOM state and then comparing it with the changed version could help highlight issues. If you change a CSS class expecting it to only effect on page, running your tests would reveal undesired knockon affects on other pages.

Well, if you are interested in CSS and HTML rendering on several browsers you can automate tests that compare screenshots.

I've been involved in a similar test, first all our CSS are compiled from less and then tested. Basically the process is to generate base screenshots an then when there are changes it compare new screenshots.

Here is a small CSS test project in NodeJS that i created as baseline: https://github.com/adcarabajal/generator-css-testing

Cheers

If you are interested in CSS HTML visual testing, differencify does that for you. It is a nodejs library built on top of chrome headless

If you use Java + Selenium test automation, you can try this plug-in: https://github.com/webrelab/layout_testing

This component allows you to detect changes in the visual display of the page based on a comparison of the current state with the reference one.

This component does not use screenshots. The work is based on the calculation of the values of the attributes of styles, sizes and arrangement of elements.

The component automatically finds all visually significant elements of the screen in the specified area and performs all necessary measurements.

The final question is, if these tools don't exist, are they worth writing?

I don't think so, since it takes seconds to look at the page and see if all the elements are rendered correctly, while you probably will spend at least several weeks writing and testing such tool from scratch.