a faster alternative to sql-sync

Where I work I probably load environments about 50 times a day. Testing bug fixes, data migrations, reproducing errors, failure analysis, and so on.

Even if I can save 30 seconds with an automated database reload process, it will add up.

There’s been work on improving drush sql-sync, including https://drupal.org/project/drush_sql_sync_pipe

The bottleneck is that drush sql-sync works with temporary files – meaning it has to:

  1. Connect to the remote machine
  2. Perform a sql-dump to a file on the remote machine and compress it
  3. Transfer that file to your machine
  4. Restores the dump to database

The problem with this is that each step is executed consecutively. It would be better if all these steps were performed concurrently. Drush defaults to this method because it is compatible with most systems. If you’re a power user though, you may want a find a faster solution.

What we’d like to do is

  1. Connect to the remote machine
  2. Perform these steps at the same time
    1. Read the file remotely
    2. Compress on the fly
    3. Stream it to your local machine
    4. Uncompress on the fly
    5. Pipe sql to database

I wrote this little script that accomplishes just that and a little extra for dumping locally. The key is piping data instead of saving it temporarily. Note that this only works on Linux/Mac.

Put this script somewhere (maybe ~/bin) and chmod a+x it.

From within your site directory, run fastdump @someAlias

This will

  1. Delete all the local tables (to ensure tables that don’t exist in your source are gone)
  2. Restore the database from an alias
  3. Run updates

But quickly! The next step for this would be making it into a Drush command instead of a shell script.

don’t kill your live site with a sql-sync

We have a shared alias file that represents every site that we work with. For example


@abcstage
@abctest
@abclive

are all valid aliases. Developers would have access to stage and test, while live only works for privileged users.

But, we still want to make sure that no funny business goes on.

Create a file, ~/.drush/policy.drush.inc

This will ensure that nobody can accidentally sql-sync to a live site. You can adjust the criteria as need be.

protecting content profiles in drupal 6

Content profiles in Drupal 6, by default are plain old nodes, so if they are published everyone will have access to them.

This sets up a realm and restricts it to the profile owner.

Pulled from https://drupal.org/node/837220#comment-3147640 – but this is the gist of it.

Then rebuild node access.

xdebug makes PHP hang

I’ve found it’s a common misconception that you can’t have XDebug running all the time without impacting PHP performance.

There’s a few reasons you could be experiencing hangs or delays:

  • You have xdebug.remote_autostart set to 1.

    This will make XDebug try to contact your debug client on every PHP process. Web, console, whatever. Generally a bad idea! You could easily run into multiple requests trying to connect and stalling.

    It’s best to set this to 0, then use a browser extension to toggle your debugging session on or off, like:

    https://addons.mozilla.org/en-US/firefox/addon/easy-xdebug/

    The only time setting this to 1 is a good idea is when there is no other way to send the XDebug session start command (for example, when debugging an application that receives HTTP calls from a 3rd party machine).

    If you are doing console PHP, you can set an environment variable to toggle.
    On – export XDEBUG_CONFIG="idekey=netbeans-xdebug"

    Note it is important to set xdebug.remote_connect_back to 0, and remote_host to a valid host, because in console, XDebug won’t automatically know the location of your debugger from xdebug.remote_connect_back.

    While this is set, PHP run in CLI mode (like drush) will trigger debugging.

  • You had a debugging session open, and opened another one to the same debug client.

    For example, debugging index.php in your browser, opening a new tab, and debugging it again. The 2nd request will stall because your local debugger is busy.

    Suggestions: use the XDebug toggler to enable/disable debugging. You can start debugging in one browser tab, but turn it off for another.

  • Your application makes a URL call to itself, and you had xdebug.remote_autostart set to 1.

    (A combination of the above)

    The best example of this is using Drupal simpletests – you start a PHP process that connects to your debugger, but then the remote HTTP call inside of PHP also tries to connect. Your simpletests will stall indefinitely. Setting xdebug.remote_autostart to 0 ensures that the internal HTTP calls do not trigger XDebug.

    Caveat: Unfortunately, to actually debug a call inside of another call is somewhat complicated. You will have to turn off debugging in your browser/console, then manually inject

    ?XDEBUG_SESSION_START=mykey

    into your internal HTTP calls, disable xdebug.remote_connect_back, and set an xdebug.remote_host.

XDebug becomes incredibly flexible with xdebug.remote_connect_back and xdebug.remote_autostart – especially for debugging live servers where you do not want XDebug to take up any overhead. I’ll make another post on that soon – including bits about debug security and the debug proxy for handling multi-user debugging.

If you’re not set up with xdebug yet check out my quick start Linux + PHP + XDebug howto

building PHP 5.3 on fedora 18/19/20

Fedora 18 is out, and while awesome comes packaged with PHP 5.4 only. I have some older webapps (Drupal 6) which are somewhat incompatible. So we have to build from scratch.

Another alternative is ZendServer CE, which provides architecture independent RPMs, but that isn’t yet built for the version of Apache that comes with Fedora 18 (Apache 2.4)

Make sure to install the libraries that PHP needs to build

yum install pcre-devel ncurses-devel httpd-devel mysql-devel zlib-devel libcurl-devel readline-devel mysql-connector-c++-devel openssl-devel libjpeg-turbo-devel libxml2-devel libpng-devel freetype-devel libmcrypt-devel

Grab the source from PHP.net and extract it. This configure command should get you the PHP you are used to.


'./configure' '--prefix=/usr' '--disable-debug' '--enable-inline-optimization' '--enable-libxml' '--enable-session' '--enable-xml' '--enable-hash' '--with-pear' '--with-apxs2' '--with-layout=GNU' '--enable-filter' '--with-pcre-regex' '--with-zlib' '--enable-simplexml' '--enable-dom' '--with-openssl' '--enable-pdo' '--with-pdo-sqlite' '--with-readline' '--with-iconv' '--with-sqlite3' '--enable-xmlwriter' '--enable-xmlreader' '--with-config-file-scan-dir=/etc/php.d' '--with-mysql=/usr' '--with-libdir=lib64' '--with-pdo-mysql' '--with-gd' '--with-curl' '--enable-zip' '--enable-mbstring' '--enable-soap' '--with-mcrypt' '--enable-gd-native-ttf' '--with-freetype-dir=/usr/lib64' '--with-mysqli' '--with-jpeg-dir=/usr/lib64' '--with-config-file-path=/etc'

make, and make install

jenkins, paraunit, junit, and saucelabs

Jenkins has great support for taking JUnit-style test reports and tracking successes and failures.

Saucelabs has an extension for running parallel tests, but only reports a total pass/fail with aggregated PHPUnit output. This doesn’t help Jenkins since it needs the XML to properly report.

I noticed Jenkins will also take multiple XML results and combine them into a test report. So, I figured that if we made each parallel process generate an XML, we could just get Jenkins to combine them until this functionality gets built into PHPUnit.

And it worked! Let’s modify vendor/bin/paraunit

Change

$cmd = "$PHPUNIT --filter=$testName $testFile";

to

$cmd = "$PHPUNIT --log-junit result-$testName.xml --filter=$testName $testFile";

Then, in Jenkins, add a new “Publish JUnit report” and add all your XML files as a result (e.g., “result-*.xml”)

Make sure to delete the XML results before each test run!

This brings back the reporting functionality that you get running plain PHPUnit with –log-junit 🙂

converting phpunit-selenium assertTextPresent to spinAssert

Just started using SauceOnDemand and was working on adding some new functionality that they offered. One of them is the spinAssert, which is the equivalent of pausing or issuing a waitFor* before asserting something. That’s bad practice because it slows down tests, so spinAssert keeps trying the assertion a set number of times until it fails. This should get rid of our intermittently failing tests!

So I had

But there isn’t really an assertTextPresent in the driver. So here’s the spinAssert alternative:

Ta da!