Testing PHP extensions on Travis-CI

Testing PECL extensions on Travis-CI has always been cumbersome for me; build PHP in different versions and debug/threadsafe flavors, install PECL dependencies and so on, which usually results in a mess of command line scripts, repeated for every extension.

Enter travis-pecl.

This tiny support bin comes with a Makefile and small PHP scripts to generate the build matrix and check your package.xml file. It supports building PHP in a wide variety of flavors, installing PECL dependencies from the PECL website or from bundled pharext packages and running the testsuite with a comprehensive set of commands.

Let's look at a few commands with a properly set up environment from the test matrix:

env:
 - PHP=5.6 enable_debug=yes with_iconv=yes enable_json=yes

Build PHP:

make -f travis/pecl/Makefile php

Install a PECL dependency:

make -f travis/pecl/Makefile pecl PECL=propro

Install a PECL dependency from a pharext package, located in travis/:

make -f travis/pecl/Makefile pharext/raphf-phpng

Build the currently checked out extension;

make -f travis/pecl/Makefile ext PECL=http

Run the testsuite:

make -f travis/pecl/Makefile test


Finally, check out the README and have a look at a couple of complete examples here:

• https://github.com/m6w6/ext-apfd/blob/master/gen_travis_yml.php
• https://github.com/m6w6/ext-http/blob/phpng/gen_travis_yml.php

Evolution of $

Today I realized an important and probably very obvious fact: we all change. Of course, you will say. Surely most of us will agree to that, but the important thing about that is also realizing that we drift away from things we loved. Okay, so what am I actually talking about? :)

I talk about software development, specifically development of and with PHP. We've seen some radical proposals in the nearer past and in deed are still seeing those for PHP, the language. Often, I was inclined to think "cool, that's a great idea, let's improve in that direction."

PHP was and is a huge success, it attracts a lot of awesome people, sometimes becoming unhappy with the language's principles and resulting limitiations, like its simplicity, its forgiving nature and the enigne's effort to come to an end for this request, i.e. to die. Looking at the larger frameworks which have evolved the last few years, they head into a clear direction: strictly object oriented programming with RidiculouslyLongClassNamesLikeFoundInOtherStrictlyObjectOrientedLanguages. Anything non object oriented seems to be verboten. But this is not of what PHP was meant to be and it is very hard to change something that late in the game in such drastical ways. I don't think PHP can change as fast as we do, it's actually much longer around than most of us call ourself programmers.

What I mean to say is, don't be angry with PHP the language, don't force it to change in a way it's not supposed to survive, stop proposing language level changes in a weekly manner, don't be afraid to change yourself. Don't be afraid to advance yourself. Don't be afraid to change the language you program in. BOOM, I said it. Yes, don't be afraid to explore other possibilities that better support your principles instead of fighting them.

I'm not saying go away, I'm saying go ahead, or at least I mean to.

Sorry for the fast write-up, all the typos and errors. Have a nice start into your week!

Hear, hear

I was about to write "in early February" but actually it already was in late January that I stumbled over this tweet:

I'm still ready & willing to hire a #PHP internals coder to work on PHP fulltime. Amazing place to work: http://t.co/GX4wtPSc
— Don MacAskill (@DonMacAskill) January 18, 2013

Fast-forward five months – now I am very excited to be able to announce that the Awesomes over at SmugMug, Inc. have hired me to work full-time on the core of PHP.

Hide bugs.php.net, expect massive amounts of commits, sleep well. Thank you for reading the simple words of the proudest man alive. Thank you SmugMug!

pecl_http-1.7.5

pecl_http-1.7.5 has been released today.
Nearly a year and 170k downloads after the last release (1.7.4 was released April 2nd 2012).

It fixes a single bug:

• Bug #64310 (weak etags W/"abc" are quoted as "W/"abc"")

The same user (thanks Niko), discovered a peculiarity of libcurl:

• If you utilize libcurl's TIMECOND feature through pecl_http's lastmodified request option, libcurl ignores response bodies from servers that do not closely follow the RFC and send a 200 OK response instead of a 304 Not Modified when the condition is unmet.

Slightly more background information is available at the relevant bug report.

Please leave a comment, if you have an opinion about which component's behavior is arguable here.

pecl_http-v2 - http2 or httpi or http-*

I'm pondering to release v2 of pecl_http with a different extension name than "http", but I cannot agree with myself what's worse:

http2

This might be confused with HTTP/2.0

httpi

There's mysqli following this approach, but I think this is pretty odd.

Split it up

Split the package up in several smaller sub-packages, like:

• http-common
• http-env
• http-client
• http-client-curl
• etc.

If you have an opinion, or maybe even a better idea, please leave a comment.

WebSockets Handshake non HTTP conforming?

Update:

Just found a revelatory discussion. The summary of Bjoern Hoehrmann is especially interesting with regards to that topic.

---

While skimming through the new HTML5 WebSocket draft, I noticed the following exemplar HTTP message demonstrating the client message of a WebSocket handshake:

        GET /demo HTTP/1.1
        Host: example.com
        Connection: Upgrade
        Sec-WebSocket-Key2: 12998 5 Y3 1  .P00
        Sec-WebSocket-Protocol: sample
        Upgrade: WebSocket
        Sec-WebSocket-Key1: 4 @1  46546xW%0l 1 5
        Origin: http://example.com

        ^n:ds[4U

To me this looks non conforming to the HTTP spec due to the lack of an indicator that the request contains a message body.

Quoting the 4th paragraph of section 4.3 of RFC2616:

The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer-Encoding header field in the request's message-headers. A message-body MUST NOT be included in a request if the specification of the request method (section 5.1.1) does not allow sending an entity-body in requests. A server SHOULD read and forward a message-body on any request; if the request method does not include defined semantics for an entity-body, then the message-body SHOULD be ignored when handling the request.

Huh?

Looking for PHP Geeks in Austria

You guessed. I'm desperately looking for decent PHP Coders in Vienna, Austria.

The job posting (in german) can be found here:
http://www.inqnet.at/job09.html

Just ping me if you're interested in an face-to-face interview.

PHP commandline highlighter

Just chewed up a command line highlighter for php code.
It's hacky, but it fullfills my needs.

Cheers-

pecl_http and sub-second timeouts

Just a quick info, because I forget this again and again--and it's not noted in the documentation yet either.

Sub-second timeouts are supported by libcurl and thus by pecl_http--yes but only if libcurl is built with (c-)ares support:

<quote url="curl_easy_setopt.html">

If libcurl is built to use the standard system name resolver, that portion of the transfer will still use full-second resolution for timeouts with a minimum timeout allowed of one second.

</quote>

Introducing libcurls multi socket API

So, finally a first beta of pecl_http 1.6 has been released.

This is the first version which supports libcurls multi socket API introduced in 7.16 through libevent.

Here's a not very impressive comparison of the performance of the traditional multi API vs. the new multi socket API:

mike@honeybadger:~/build/php-5.2-debug$ cli 
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php 
    -u http://honeybadger/empty.html

>   1.134667s

mike@honeybadger:~/build/php-5.2-debug$ cli 
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php 
    -u http://honeybadger/empty.html

>   1.151088s

mike@honeybadger:~/build/php-5.2-debug$ cli 
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php 
    -u http://honeybadger/empty.html

>   1.131867s

mike@honeybadger:~/build/php-5.2-debug$ cli 
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php 
    -u http://honeybadger/empty.html -e

>   0.993878s

mike@honeybadger:~/build/php-5.2-debug$ cli 
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php 
    -u http://honeybadger/empty.html -e

>   0.998832s

mike@honeybadger:~/build/php-5.2-debug$ cli 
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php 
    -u http://honeybadger/empty.html -e

>   0.997121s

Above empty.html is, well, empty. The following test requests a 100k file:

mike@honeybadger:~/build/php-5.2-debug$ cli  
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php  
    -u http://honeybadger/100k.bin

>   2.205190s

mike@honeybadger:~/build/php-5.2-debug$ cli  
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php  
    -u http://honeybadger/100k.bin

>   2.210525s

mike@honeybadger:~/build/php-5.2-debug$ cli  
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php  
    -u http://honeybadger/100k.bin

>   2.254281s

mike@honeybadger:~/build/php-5.2-debug$ cli  
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php  
    -u http://honeybadger/100k.bin -e

>   2.007220s

mike@honeybadger:~/build/php-5.2-debug$ cli  
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php  
    -u http://honeybadger/100k.bin -e

>   1.945564s

mike@honeybadger:~/build/php-5.2-debug$ cli  
    ~/cvs/pecl/http/scripts/bench_select_vs_event.php  
    -u http://honeybadger/100k.bin -e

>   1.969575s

So, apparently time savings are not huge, but noticable.

Silence

It's been more than 3 months that I brought back to mind, what was happening in my PHP/HTTP/CURL world, so here we go.

February:

pecl/http with support for persistent handles has been released.

A proposal of mine at internals to change the behaviour of read_property/write_property and get_property_ptr_ptr object handlers has successfully been ignored.

The idea was the following:

---

Wouldn't it be reasonable for the engine to use get_property_ptr_ptr() whenever it wants to modify a property and get rid of using read_property() for write access?

That would make rather simple ops like concatenation, in-/decrementation etc work again with overloaded internal classes. Just make the engine use read_property() followed by a write_property() when there's no get_property_ptr_ptr().

---

Nobody was interested.

March:

I added a comparison object handler to the DateTime class, which means that comparisons of DateTime objects should work in recent PHP5 versions.

I started development of pecl/libetpan, but dropped it again because of pretty huge differences between how the library works and how PHP is supposed to work. What a pity.

A less fine guy, some Omid, wanted me to write a PHP extension which calculates the strength of "poker hands", but he jumped off when I had done the algos.

April:

A fine guy, pretending his name was Andreas Weber, noticed that pecl/http's request method implementation was completely b0rked. Thanks to his report it was fixed in 1.5.3.

All fame Lukas Smith, queried me about what might have changed with my new output layer implementation for PHP6, but I noticed too late that it's been for a conference talk he was preparing, so I feel like I didn't spend enough attention to his questions. Sorry Lukas.

I started to implement libcurl's new multi_socket AKA hiper API in conjunction with libevent, but stumbled across a few problems.

May:

My journey with libcurl's multi_socket API continues, which eventually causes the delay of a first pecl/http 1.6 release candidate.

I tried Drupal for the first time.

---

Well, that'd be it. If it reads like an extrusion of junk, it probably was. ;)

Phar vs World

Recent discussions about inclusion of pecl/phar into the core distribution shows again that we are missing a defined process of handling additions to the PHP main distibution. How many people are really reading through all mails of 100+ message threads?

It's going to be a chaos. Always. Once a developer has got his new, shiny and soon-to-be-world-dominating extension into the core, he'll be a even stronger advocator of the "no-new-extensions" camp. Not to disrespect any work, but this is pure rivalism, masculine--really!

The idea of moving extensions from core to pecl is honorous, but we all know the current problems of this ideology.

Many people have asked me when pecl/http will be included in the main distribution. My answer is short and simple: "Never" (I usually add a tiny sentence, but that's nothing encouraging either).

Ah... and please do me a favour: rather keep your comments to yourself :) (...or toss it into the nonsense thread@internals).

Upgrading to 5.2

So I finally came around to upgrade to PHP 5.2 (I was running 4.4 on the production server until now). I know, what a shame! :)

Anyway the only issue I really had, in spite testing the code really well over time, was with class_exists() and millions of warnings becaus of a missing __autoload(). I blame the people who introduced the second parameter to class_exists() and changing the default behaviour at the very same moment. Actually, it was an annoying but rather easy to fix compatibility brake.

APC in conjunction with PHP-5.2 seems to work very well either, it even feels a lot less memory exhaustive than running APC+PHP-4.

Ah yes, just on a side note, I also upgraded to Apache-2.2 in the same run. Somehow I now have a warm and fuzzy feeling running this up-to-date software, it really was already like a brinck in my stomach. Anyway, what's left is to migrate from the Apache module to the FastCGI SAPI and switching to the worker MPM... but I'm done already.

Something I didn't even know I was missing

$ php --ri extension

Nice.

HTTP, CURL and stuff

News, news, yawn.

Daniel Stenberg, head of the cURL project, accepted a patch for sub second time out support within libcurl. That means that you can use float values as time outs with a millisecond resolution, as soon as you use libcurl >= v7.16.2, that is, a minimum supported time out of 0.001 seconds (which is not reasonable, just in case you wonder). On a side note, you should build libcurl with c-ares to get working sub second DNS lookup time out support, AFAICT.

Clay Loveless, usually killing people softly with his random strings, mashing up APIs or providing compatibility libraries, kindly asked me, if there were persistent connection support in pecl/http.

Is there an option in pecl_http that I'm overlooking that would be the equivalent of STREAM_CLIENT_PERSISTENT?

I was already about to answer that libcurl already does that all for us, realizing that we destroy the used CURL handles at the end of each PHP request at the latest, thus killing any alive connections. This happened about two weeks ago, and pecl/http 1.5, of which RC1 had just been released, will support per process persistent CURL handles. It can be enabled at compile time only, as it's a quite intrusive feature. Yet I'm still unsure about the implementation of this feature and it might change in the future to become a bit more user-friendly.

To use it, there's nothing else to be done than tossing --enable-http-persistent-handles at configure.

pecl_http response performance

I made some small tests to get an idea of the current response performance of pecl/http.

Test victim was a >300k PDF file served by Apache 2.0.55 running the worker MPM.

Full size chart, PDF with stats.

The legend should be read as follows:

php	=	request to a PHP script serving the PDF file by HttpResponse
php-c	=	just like above, cached by ETag
file	=	request to the PDF file served directly by Apache
file-c	=	just like above, cached by ETag

The results surprised me, because it clearly shows that the performance drop for starting the PHP engine and doing the neccessary negotiation to provide caching is not that bad. On the other side, PHPs throughput when actually serving the file is not exciting. The current CVS version of pecl/http contains some tweaks to improve that situation, but it didn't really work out as well as I hoped.

HttpRequestDataShare

There are some news to talk about development of pecl/http.

I recently implemented an interface to the curl-share functionality in form of an HttpRequestDataShare class.

This is what reflection will tell you about it:

mike@honeybadger:~/build/php-5.2-debug$ cli --rc HttpRequestDataShare
Class [ <internal:http> class HttpRequestDataShare implements Countable ] {

  - Constants [0] {
  }

  - Static properties [1] {
    Property [ private static $instance ]
  }

  - Static methods [1] {
    Method [ <internal> static public method singleton ] {

      - Parameters [1] {
        Parameter #0 [ <optional> $global ]
      }
    }
  }

  - Properties [4] {
    Property [ <default> public $cookie ]
    Property [ <default> public $dns ]
    Property [ <default> public $ssl ]
    Property [ <default> public $connect ]
  }

  - Methods [5] {
    Method [ <internal, dtor> public method __destruct ] {

      - Parameters [0] {
      }
    }

    Method [ <internal, prototype Countable> public method count ] {

      - Parameters [0] {
      }
    }

    Method [ <internal> public method attach ] {

      - Parameters [1] {
        Parameter #0 [ <required> HttpRequest $request ]
      }
    }

    Method [ <internal> public method detach ] {

      - Parameters [1] {
        Parameter #0 [ <required> HttpRequest $request ]
      }
    }

    Method [ <internal> public method reset ] {

      - Parameters [0] {
      }
    }
  }
}

Using this class, you can save a fair amount of time with name lookups which the following example shows:

<?php
$s = HttpRequestDataShare::singleton(true);
print_r($s);
for ($i = 0; $i < 10; ++$i) {
    $r = new HttpRequest("http://www.google.com/");
    $s->attach($r);
    $r->send();
    printf("%0.6fn", $r->getResponseInfo("namelookup_time"));
    $s->detach($r);
}
?>

Executing this script without dns data sharing enabled gives the following results:

mike@honeybadger:~/build/php-5.2-debug$ cli  -d"http.request.datashare.dns=0"
    ~/devel/http_rshare.php
HttpRequestDataShare Object
(
    [cookie] =>
    [dns] =>
    [ssl] =>
    [connect] =>
)
0.071296
0.048798
0.049598
0.051545
0.046258
0.052318
0.043769
0.060753
0.049168
0.048568

...and with dns data sharing enabled:

mike@honeybadger:~/build/php-5.2-debug$ cli -d"http.request.datashare.dns=1" 
    ~/devel/http_rshare.php
HttpRequestDataShare Object
(
    [cookie] =>
    [dns] => 1
    [ssl] =>
    [connect] =>
)
0.051945
0.000043
0.000041
0.000040
0.000039
0.000041
0.000041
0.000040
0.000040
0.000041

QED

You can either use a per-process global datashare object created with HttpRequestDataShare::singleton(true) or different instances for your HttpRequest objects. Note that dns datasharing is used autmagically for HttpRequestPool requests. Currently libcurl has implemented cookie and dns data sharing only, trying to enable ssl session or connect sharing will raise a warning.

Be sure to try it out; either directly from CVS or the next release, probably being 1.3.0RC1.

__get() and array rumors

There've been lots of rumors about overloaded array properties lately.

The following code

<?php
class funky {
  private $p = array();
  function __get($p) {
    return $this->p;
  }
}
$o = new funky;
$o->prop["key"] = 1;
?>

will yield:

Notice: Indirect modification of overloaded property funky::$p has no effect

As arrays are the only complex types that are passed by value (resources don't really count here) the solution to described problem is simple: use an object; either an instance of stdClass or ArrayObject will do well, depending if you want to use array index notation.

So the folloiwng code will work as expected, because the ArrayObject instance will pe passed by handle:

<?php
class smarty {
  private $p;
  function __construct() {
    $this->p = new ArrayObject;
  }
  function __get($p) {
    return $this->p;
  }
}
$o = new smarty;
$o->prop["key"] = 1;
?>

I guess most of you already knew, but anyway... ;)

Round up

It's been a long time since I wrote something here, mostly because I got distracted by some real private life recently ;) and due to paid work of course. Therefore I thought I'd round up what has happened behind the scenes in my PHP world.

PHP-6
I rewrote the output control layer for PHP-6 some months ago and I'm about to upgrade ext/zlib to see how it really works out.

PHP-5.2
I didn't contribute that much to this upcoming release. Two things I'd like to mention are a fix for the Apache2 SAPI where each header("Content-Type: aaa/bbb") caused Apache to add output filters for the type to the outgoing filter chain and the addition of the error_get_last() function, which is a convenient accessor to the last occured error without fiddling around with INI(track_errors) and $php_errormsg.

pecl/http
There's official documentation now available online in the PHP manual, yay! :) It's not fully fleshed out, but gives some feeling about the provided functionality and hints on how to use this module.

php|a published an article by me about pecl/http in their Augusts issue!

There have also been three releases since 1.0, the most recent one (1.2) today. See the changes since then outlined below.

Improvements/Additions

• Improved response performance (HttpResponse, http_send API)


• Added http_build_cookie() function


• Added HttpQueryString::mod(array $params) method


• Added ArrayAccess to interfaces implemented by HttpQueryString


• Added HttpMessage::getHeader(string $name) method

Bug Fixes

• Fixed http_parse_cookie() allowed_extras and flags parameters


• Fixed configuration with shared dependencies


• Fixed endless loop in http_build_url("..")


• Fixed HttpResponse::capture() failure if buffered output exceeds 40k


• Fixed HttpQueryString failures with objects as params


• Fixed memory leaks with overloaded classes extending HTTP classes


• Fixed build with gcc-2.95 (Thanks to Alexander Zhuravlev)


• Fixed memory leak in inflate code (Thanks to Thomas Landro Johnsen)

Installing pecl_http

As pecl/http 1.0 has finally been released and I had noticed that it's been packaged already by several projects like PLD, Gentoo and FreeBSD, I wanted to explain what one is going to gain respectively lose by using the different build/configure options for the extension.
The help text of configure for pecl/http should look similar to the following:

  --enable-http           Enable extended HTTP support
  --with-http-curl-requests[=LIBCURLDIR]
                           HTTP: with cURL request support
  --with-http-zlib-compression[=LIBZDIR]
                           HTTP: with zlib encodings support
  --with-http-magic-mime[=LIBMAGICDIR]
                           HTTP: with magic mime response content type guessing
  --with-http-shared-deps  HTTP: disable to not depend on extensions like hash,
                                 iconv and session (when built shared)

If you link the extension source directory into your php tree, you should be aware that these options show up on the end of the list of configure options for extensions, not--as probably expected--in alphabetical order. This is due to a recent change to use config9.m4 because the HTTP extension may depend on several other PHP extensions (hash, iconv, session).

--with-http-curl-requests
This configure option enables request functionality, uses libcurl and is highly recommended to be enabled. The minumum libcurl version required is 7.12.3. Debian/stable currently ships 7.13.2 (no, this is not a typo).

--with-http-zlib-compression
I think this is the most overseen/ignored option. Besides handling of compressed HTTP messages, it also provides superior deflate/inflate functionaly in regards to stability and performance compared to the standard zlib extension. Both http_deflate()/http_inflate() functions and http.deflate/http.inflate stream filters are able to encode/decode all valid gzip, zlib (AKA deflate) and raw deflated data. It requires at lieast libz version 1.2.0.4, while Debian/stable ships 1.2.2, and is also highly recommended to be enabled.

--with-http-magic-mime
This option enables content type guessing for the HttpResponse and HttpMessage classes. It's rather a gimmick and thus not enabled by default. As there's no version information available for libmagic, I don't even know which is the minimum version required but I guess anything coming from a file-4.1x versioned package should work. If you get an empty string as content type for payload which is obviously XML text, check the magic.mime database you use for a broken first XML section. Comment out everything except the SVG detection as other XML types and HTML is handled further down the magic file (noticed on Debian systems). If you changed your magic.mime database, don't forget to regenerate the precompiled version with the `file -C`command.

--with-http-shared-deps
This option controls whether pecl/http will depend on extensions built as dynamically loadable modules. So, if e.g. ext/iconv has been compiled shared, pecl/http relies on ext/iconv to be loaded when itself is going to be loaded. This option is enabled by default.

ext/hash
pecl/http uses ext/hash to generate ETag hashes (else standard PHP MD5, SHA1 or CRC32).

ext/iconv
If ext/iconv is present, the HttpQueryString class provides an xlate() method for charset transformation.

ext/session
http_redirect() can automatically append session information to the redirect URL.

ext/spl
ext/spl cannot be built shared, so pecl/http always uses it if it's enabled. HttpMessage and HttpRequestPool classes implement the interface Countable provided by ext/spl.