PHP Activist

Lessons learned:

• PHPng is significantly faster and uses significantly less memory
• objects need more memory than arrays (+ 5-250 percent)
• if array values are numeric, don't save them as strings!
• saving 1M integers takes 33M of memory with PHPng (increase by factor 8)
• saving 1M integers as strings takes 79M of memory with PHPng (increase by factor 20)
• using SplFixedArray can reduce memory usage by 20-100 percent
• avoid big Arrays and Objects in PHP whenever possible
  (don't use file('big_file') or explode("\n", file_get_contents('big_file')), etc.)
• use streams whenever possible (fopen, fsockopen, etc.)
• use generators when available with PHP 5.5 (RFC)
• comparing 32bit to 64bit systems, memory consumption increases by 100-230 percent

If you need to save memory temporarily in your script, you can stringify your array (increasing cpu usage and runtime):

function test(){

$a = array();
for ($i=0; $i<1000000; $i++) $a[] = $i;
$a = implode(',', $a); // or $a = json_encode($a);
echo number_format(memory_get_usage(true)/1048576, 2)."\n";

}
test();
// 7.5 M / 0.64s (php), 10.4 M / 0.55s (hhvm), 7.25 M / 0.20s (phpng)

Here is a small script to show how much memory PHP needs to handle big Arrays:
the total runtime of the script is 15.5s (php), 8.4s (hhvm), 4.6s (phpng)
(tests made with PHP 5.5.9, HHVM 3.2.0-2014-06-06, PHP 5.7.0-dev-2014-06-05, 2.8GHz single-core openvz)

ini_set('memory_limit', '1024M');

function test(){

$a = '';
for ($i=0; $i<1000000; $i++) $a .= '1';
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 1.50 (php), 3.5 (hhvm), 1.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.25 (php), 30.4 (hhvm), 32.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146.25 (php), 80.92 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54.25 (php), 30.56 (hhvm), 16.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.50 (php), 30.56 (hhvm), 33.25 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 80.97 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54.25 (php), 30.58 (hhvm), 16.75 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 61.10 (hhvm), 78.50 (phpng)

$a = new stdclass();
for ($i=0; $i<1000000; $i++) $a->$i = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 237.75 (php), 111.50 (hhvm), 128.25 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 145.5 (php), 61.11 (hhvm), 61.75 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[(string)$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 61.12 (hhvm), 78.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.5 (php), 30.62 (hhvm), 33.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 81.02 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54 (php), 30.63 (hhvm), 17.00 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.5 (php), 30.64 (hhvm), 33.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 81.04 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54 (php), 30.65 (hhvm), 17.25 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 222.5 (php), 30.65 (hhvm), 34.00 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 81.05 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138 (php), 30.66 (hhvm), 17.50 (phpng)


$a = array();
for ($i=0; $i<500000; $i++) $a[$i] = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 214.75 (php), 18.67 (hhvm), 18.25 (phpng)

$a = new stdclass;
for ($i=0; $i<500000; $i++) $a->$i = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 218.25 (php), 41.81 (hhvm), 41.75 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<500000; $i++) $a[$i] = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 176.00 (php), 30.68 (hhvm), 17.50 (phpng)


$a = array();
for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 214.50 (php), 18.68 (hhvm), 18.25 (phpng)

$a = new stdclass;
for ($i=0; $i<500000; $i++) $a->$i = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 218.25 (php), 41.82 (hhvm), 41.75 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 176 (php), 30.69 (hhvm), 17.50 (phpng)

}
test();

More details and technical explanations about PHP's memory usage can be found on nikic's blog.
More details about PHPng can be found here.

Shortly: a string or an integer in PHP are not mapped to a char-array or an integer in C. A more complex structure is used to get from dynamic typing in PHP to static typing in C:

struct _zval_struct {
  zvalue_value value;     // value object
  zend_uint refcount__gc; // number of references to variable
  zend_uchar type;        // type of variable
  zend_uchar is_ref__gc;  // reference? (&$var)
};
typedef union _zvalue_value {
  long lval;              // integer and boolean
  double dval;            // float (double)
  struct {
    char *val;            // string (with zero bytes)
    int len;              // length of the string
  } str;
  HashTable *ht;          // array (hash table)
  zend_object_value obj;  // object
} zvalue_value;

Lessons learned:

• Foreach is 4-5 times faster than For
• Nested Foreach is 2-3 times faster than nested For
• Foreach with key lookup is 2-3 times slower than Foreach without
• C++ is 5-300 times faster than PHP running For/Foreach on Arrays
• HHVM is 2-6 times faster than PHP
• HHVM is currently no alternative to C++
• Javascript is 2-20 times slower than C++/Java running For on nested Arrays

Here is a sample script:

<?php
function test(){
// init arrays
$array = array();
for ($i=0; $i<50000; $i++) $array[] = $i*2;

$array2 = array();
for ($i=20000; $i<21000; $i++) $array2[] = $i*2;

// test1: foreach big-array (foreach small-array)
$start = microtime(true);
foreach ($array as $val) {
  foreach ($array2 as $val2) if ($val == $val2) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

// test1b: foreach big-array (foreach small-array)
$start = microtime(true);
foreach ($array as $val) {
  foreach ($array2 as $val2) if ($val === $val2) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

// test2: foreach small-array (foreach big-array)
$start = microtime(true);
foreach ($array2 as $val2) {
  foreach ($array as $val) if ($val == $val2) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

// test3: foreach big-array (foreach small-array) with key lookup
$start = microtime(true);
foreach ($array as $key=>$val) {
  foreach ($array2 as $key2=>$val2) if ($array[$key] == $array2[$key2]) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

// test4: foreach small-array (foreach big-array) with key lookup
$start = microtime(true);
foreach ($array2 as $key=>$val2) {
  foreach ($array as $val) if ($array[$key] == $array2[$key2]) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

// test5: for big-array (for small-array)
$start = microtime(true);
$count = count($array);
$count2 = count($array2);
for ($key=0; $key<$count; $key++) {
  for ($key2=0; $key2<$count2; $key2++) if ($array[$key] == $array2[$key2]) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

// test6: for small-array (for big-array)
$start = microtime(true);
$count = count($array);
$count2 = count($array2);
for ($key2=0; $key2<$count2; $key2++) {
  for ($key=0; $key<$count; $key++) if ($array[$key] == $array2[$key2]) {}
}
echo number_format(microtime(true)-$start, 2)."s\n";

$array = array();
for ($i=0; $i<1000000; $i++) $array[] = $i*2;

// test7: foreach big-array
$start = microtime(true);
foreach ($array as &$val) $val++;
echo number_format(microtime(true)-$start, 2)."s\n";

// test8: for big-array
$start = microtime(true);
for ($key=0; $key<count($array); $key++) $array[$key]++;
echo number_format(microtime(true)-$start, 2)."s\n";

// test8b: for big-array, doing count() outside the loop!
$start = microtime(true);
$count = count($array);
for ($key=0; $key<$count; $key++) $array[$key]++;
echo number_format(microtime(true)-$start, 2)."s\n";
}
test();

Here are some results from PHP 5.4.4 and HHVM (2014-05-04, QEMU 2.3 GHz, 64bit):

php	hhvm
2.78s	0.47s
2.90s	0.44s
2.97s	0.44s
6.90s	1.36s
6.27s	1.33s
5.83s	1.13s
6.24s	1.15s
0.07s	0.04s
0.24s	0.04s
0.11s	0.03s

Using HHVM instead of PHP gives big improvements.

With Javascript (node.js) you'll get similar values:
example.js (run: node example.js)

var array = [];
for (i=0; i<50000; i++) array.push(i*2);

var array2 = [];
for (i=20000; i<21000; i++) array2.push(i*2);

// js-test1: for big-array (for small array)
var start = new Date().getTime();
for (key=0; key<array.length; key++) {
  for (key2=0; key2<array2.length; key2++) if (array[key] == array2[key2]) {}
}
console.log((new Date().getTime() - start) / 1000); // 1.53s

// js-test2: foreach big-array (foreach small array)
var start = new Date().getTime();
for (key in array) {
  for (key2 in array2) if (array[key] == array2[key2]) {}
}
console.log((new Date().getTime() - start) / 1000); // 6.32s

var array = [];
for (i=0; i<1000000; i++) array.push(i*2);

// js-test3: for big-array
var start = new Date().getTime();
for (key=0; key<array.length; key++) array[key]++;
console.log((new Date().getTime() - start) / 1000); // 0.03s

tested with QEMU 2.3 GHz, node.js v0.10

With C++ (gcc 4.6 win32) you'll also get similar values:
example.cpp (run: g++ -o example example.cpp && ./example)

#include <sys/time.h>
#include <stdio.h>
#include <vector>
using namespace std;

main() {
  struct timeval start, end;

  vector<int> array;
  for(int i=0; i < 50000; i++) array.push_back(i*2);

  vector<int> array2;
  for(int i=20000; i < 21000; i++) array2.push_back(i*2);

  gettimeofday(&start, NULL);
  for (int key=0; key<array.size(); key++)
    for (int key2=0; key2<array2.size(); key2++)
      if (array[key] == array2[key2]) {}

  gettimeofday(&end, NULL);
  printf("%lf\n", (float)(end.tv_sec - start.tv_sec +
    (end.tv_usec - start.tv_usec)/1000000.0)); // 0.61s, 0.00s (-O3)

  vector<int> array3;
  for(int i=0; i < 1000000; i++) array3.push_back(i*2);

  gettimeofday(&start, NULL);
  for(int i=0; i < array3.size(); i++) array3[i]++;
  gettimeofday(&end, NULL);
  printf("%lf\n", (float)(end.tv_sec - start.tv_sec +
    (end.tv_usec - start.tv_usec)/1000000.0)); // 0.009s, 0.001s (-O3)
}

tested with QEMU 2.3 GHz, gcc 4.7

And finally Java (Java 1.7 win64):

example.java (run: javac -g:none example.java && java example)

import java.util.ArrayList;
import java.util.Vector;

public class example {
  public static void main(String[] args) throws Exception {

    ArrayList<Integer> array = new ArrayList<Integer>();
    for (int i = 0; i < 50000; i++) array.add(i*2);

    ArrayList<Integer> array2 = new ArrayList<Integer>();
    for (int i = 20000; i < 21000; i++) array2.add(i*2);

    long start = System.currentTimeMillis();
    for (int key = 0; key < array.size(); key++)
      for (int key2 = 0; key2 < array2.size(); key2++)
        if (array.get(key).equals(array2.get(key2))) {}
    System.out.println((float) (System.currentTimeMillis() - start) / 1000);
    // 0.066s

    ArrayList<Integer> array3 = new ArrayList<Integer>();
    for (int i = 0; i < 1000000; i++) array3.add(i*2);

    start = System.currentTimeMillis();
    for (int i = 0; i < array3.size(); i++) array3.set(i, array3.get(i)+1);
    System.out.println((float)(System.currentTimeMillis() - start) / 1000);
    // 0.164s


    Vector<Integer> varray = new Vector<Integer>();
    for (int i = 0; i < 50000; i++) varray.add(i*2);

    Vector<Integer> varray2 = new Vector<Integer>();
    for (int i = 20000; i < 21000; i++) varray2.add(i*2);

    start = System.currentTimeMillis();
    for (int key = 0; key < varray.size(); key++)
      for (int key2 = 0; key2 < varray2.size(); key2++)
        if (varray.get(key).equals(varray2.get(key2))) {}
    System.out.println((float) (System.currentTimeMillis() - start) / 1000);
    // 1.652s

    Vector<Integer> varray3 = new Vector<Integer>();
    for (int i = 0; i < 1000000; i++) varray3.add(i*2);

    start = System.currentTimeMillis();
    for (int i = 0; i < varray3.size(); i++) varray3.set(i, varray3.get(i)+1);
    System.out.println((float)(System.currentTimeMillis() - start) / 1000);
    // 0.074s
  }
}

tested with QEMU 2.3 GHz, OpenJDK 1.6, 64bit

Lessons learned:

• HHVM runs recursive function calls 13-20 times faster than PHP
• HHVM runs recursive function calls 25-50 percent slower than Javascript
• HHVM runs recursive function calls 4-5 times slower than Java
• HHVM runs recursive function calls 2-8 times slower than C

Here is a sample script using the Ackermann function:

<?php

function ack($n, $m) {
  if ($n == 0) return $m + 1;
    else if ($m == 0) return ack($n - 1, 1);
    else return ack($n - 1, ack($n, $m - 1));
}

function ack_while($n, $m) {
  while ($n != 0) {
    if ($m == 0) $m = 1;
      else $m = ack_while($n, $m - 1);
    $n--;
  }
  return $m + 1;
}

$start = microtime(true);
ack_while(3, 7);
echo number_format(microtime(true)-$start, 4).'s'.PHP_EOL;

$start = microtime(true);
ack(3, 7);
echo number_format(microtime(true)-$start, 4).'s'.PHP_EOL;

Here is a sample script using the Fibonacci function:

<?php

function fib_it($n) {
  $a = 0;
  $b = 1;
  for ($i = 0; $i < $n; $i++){
    $sum = $a+$b;
    $a = $b;
    $b = $sum;
  }
  return $a;
}

function fib_rec($n) {
  if ($n < 3) return 1;
  return fib_rec($n - 1) + fib_rec($n - 2);
}

$start = microtime(true);
fib_it(38);
echo number_format(microtime(true)-$start, 4).'s'.PHP_EOL;

$start = microtime(true);
fib_rec(38);
echo number_format(microtime(true)-$start, 4).'s'.PHP_EOL;

Running ack(3, 7) with PHP 5.4.4, 64bit, 2.3GHz (QEMU):

0.1127s
0.1794s

Running ack(3, 7) with HHVM 3.0/2014.03.27:

0.0089s
0.0141s

Running ack(3, 7) with Javascript (node.js v0.10):

0.013s
0.009s

Running ack(3, 7) with Java (OpenJDK 1.6 64bit):

0.011s
0.006s

Running ack(3, 7) with C (gcc 4.7):

0.002s

Running ack(3, 10) with PHP:

8.6962s
14.8648s

Running ack(3, 10) with HHVM:

Fatal error: Stack overflow in ack.php on line 12

Running ack(3, 10) with Javascript:

0.374s
0.595s

Running ack(3, 10) with Java:

0.077s
0.155s

Running ack(3, 10) with C:

0.085s

Running ack(3, 11) with PHP:

32.4175s
63.0294s

Running ack(3, 11) with HHVM:

Fatal error: Stack overflow in ack.php on line 12

Running ack(3, 11) with Javascript:

RangeError: Maximum call stack size exceeded

Running ack(3, 11) with Java:

0.314s
0.612s

Running ack(3, 11) with C:

0.339s

Running fib(38) with PHP 5.4.4, 64bit, 2.3GHz (QEMU):

0.0000s
14.6734s

Running fib(38) with HHVM 3.1/2014.04.09:

0.0032s
0.7149s

Running fib(38) with Javascript:

0.008s
0.572s

Running fib(38) with Java:

0.0s
0.158s

Running fib(38) with C:

0.092s

ack.c (run: gcc -O3 -o ack.out ack.c && time ./ack.out)

unsigned int ack(unsigned int n, unsigned int m) {
  if (n == 0) return m + 1;
    else if (m == 0) return ack(n - 1, 1);
    else return ack(n - 1, ack(n, m - 1));
}

unsigned int ack_while(unsigned int n, unsigned int m) {
  while (n != 0) {
    if (m == 0) {
      m = 1;
    } else {
      m = ack_while(n, m - 1);
    }
    n--;
  }
  return m + 1;
}

int main(int argc, char* argv[]) {
  ack_while(3, 7);
  ack(3, 7);
}

fib.c (run: gcc -O3 -o fib.out fib.c && time ./fib.out)

unsigned int fib_it(unsigned int n) {
  unsigned int a = 0;
  unsigned int b = 1;
  unsigned int sum;
  unsigned int i;
  for (i = 0; i < n; i++){
    sum = a + b;
    a = b;
    b = sum;
  }
  return a;
}

unsigned int fib_rec(unsigned int n) {
  if (n < 3) return 1;
  return fib_rec(n - 1) + fib_rec(n - 2);
}

int main(int argc, char* argv[]) {
  fib_it(38);
  fib_rec(38);
}

ack.js (run: time nodejs ack.js)

function ack(n, m) {
  if (n == 0) return m + 1;
    else if (m == 0) return ack(n - 1, 1);
    else return ack(n - 1, ack(n, m - 1));
}

function ack_while(n, m) {
  while (n != 0) {
    if (m == 0) {
      m = 1;
    } else {
      m = ack_while(n, m - 1);
    }
    n--;
  }
  return m + 1;
}

var start = new Date().getTime();
ack_while(3, 7);
console.log((new Date().getTime() - start) / 1000 + 's');

start = new Date().getTime();
ack(3, 7);
console.log((new Date().getTime() - start) / 1000 + 's');

fib.js (run: time nodejs fib.js)

function fib_it(n) {
  var a = 0;
  var b = 1;
  var sum = 0;
  for (var i = 0; i < n; i++){
    sum = a + b;
    a = b;
    b = sum;
  }
  return a;
}

function fib_rec(n) {
  if (n < 3) return 1;
  return fib_rec(n - 1) + fib_rec(n - 2);
}

var start = new Date().getTime();
fib_it(38);
console.log((new Date().getTime() - start) / 1000 + 's');

start = new Date().getTime();
fib_rec(38);
console.log((new Date().getTime() - start) / 1000 + 's');

ack.java (run: javac -g:none ack.java && java ack)

public class ack {
  public static int ack(int n, int m) {
    if (n == 0) return m + 1;
      else if (m == 0) return ack(n - 1, 1);
      else return ack(n - 1, ack(n, m - 1));
  }

  public static int ack_while(int n, int m) {
    while (n != 0) {
      if (m == 0) {
        m = 1;
      } else {
        m = ack_while(n, m - 1);
      }
      n--;
    }
    return m + 1;
  }

  public static void main(String[] args) throws Exception {
    long start = System.currentTimeMillis();
    ack_while(3, 7);
    System.out.println((float) (System.currentTimeMillis() - start) / 1000);

    start = System.currentTimeMillis();
    ack(3, 7);
    System.out.println((float)(System.currentTimeMillis() - start) / 1000);
  }
}

fib.java (run: javac -g:none fib.java && java fib)

public class fib {
  public static int fib_it(int n) {
    int a = 0;
    int b = 1;
    int sum;
    for (int i = 0; i < n; i++){
      sum = a + b;
      a = b;
      b = sum;
    }
    return a;
  }

  public static int fib_rec(int n) {
    if (n < 3) return 1;
    return fib_rec(n - 1) + fib_rec(n - 2);
  }

  public static void main(String[] args) throws Exception {
    long start = System.currentTimeMillis();
    fib_it(38);
    System.out.println((float) (System.currentTimeMillis() - start) / 1000);

    start = System.currentTimeMillis();
    fib_rec(38);
    System.out.println((float)(System.currentTimeMillis() - start) / 1000);
  }
}

Lessons learned:

• strpos() is 3-16 times faster than preg_match()
• stripos() is 2-30 times slower than strpos()
• stripos() is 20-100 percent faster than preg_match() with the caseless modifier "//i"
• using a regular expression in preg_match() is not faster than using a long string
• using the utf8 modifier "//u" in preg_match() makes it 2 times slower

Here is a sample script to compare the functions with different string sizes:

<?php

function loop(){

$str_50 = str_repeat('a', 50).str_repeat('b', 50);
$str_100 = str_repeat('a', 100).str_repeat('b', 100);
$str_500 = str_repeat('a', 250).str_repeat('b', 250);
$str_1k = str_repeat('a', 1024).str_repeat('b', 1024);
$str_10k = str_repeat('a', 10240).str_repeat('b', 1024);
$str_100k = str_repeat('a', 102400).str_repeat('b', 1024);
$str_500k = str_repeat('a', 1024*500).str_repeat('b', 1024);
$str_1m = str_repeat('a', 1024*1024).str_repeat('b', 1024);

$b = 'b';
$b_10 = str_repeat('b', 10);
$b_100 = str_repeat('b', 100);
$b_1k = str_repeat('b', 1024);

echo str_replace(',', "\t", ',strpos,preg,preg U,preg S,preg regex,stripos,preg u,'.
  'preg i,preg u i,preg i regex,stripos uc,preg i uc,preg i uc regex').PHP_EOL;

foreach (array($b, $b_10, $b_100, $b_1k) as $needle) {
  foreach (array($str_50, $str_100, $str_500, $str_1k, $str_10k,
    $str_100k, $str_500k, $str_1m) as $str) {

    echo strlen($needle).'/'.strlen($str);

    $start = mt();
    for ($i=0; $i<25000; $i++) $j = strpos($str, $needle); // strpos
    echo "\t".mt($start);
    
    $regex = '!'.$needle.'!';
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg
    echo "\t".mt($start);

    $regex = '!'.$needle.'!U';
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg Ungreedy
    echo "\t".mt($start);

    $regex = '!'.$needle.'!S';
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg extra analysiS
    echo "\t".mt($start);

    $regex = "!b{".strlen($needle)."}!";
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg regex
    echo "\t".mt($start);
    
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = stripos($str, $needle); // stripos
    echo "\t".mt($start);

    $regex = '!'.$needle.'!u';
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg utf-8
    echo "\t".mt($start);

    $regex = '!'.$needle.'!i';
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg i
    echo "\t".mt($start);

    $regex = '!'.$needle.'!ui';
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg i utf-8
    echo "\t".mt($start);

    $regex = "!b{".strlen($needle)."}!i";
    $start = mt();
    for ($i=0; $i<25000; $i++) $j = preg_match($regex, $str); // preg i regex
    echo "\t".mt($start);

    echo PHP_EOL;
  }
  echo PHP_EOL;
}
}

function mt($start=null){
  if ($start === null) return microtime(true);
  return number_format(microtime(true)-$start, 4);
}

loop();

Running with PHP 5.4.4, 64bit, 2.3GHz (QEMU):

	strpos	preg	preg U	preg S	preg regex	stripos	preg u	preg i	preg u i	preg i regex
1/100	0.0052	0.0144	0.0147	0.0172	0.0168	0.0102	0.0180	0.0153	0.0195	0.0143
1/200	0.0058	0.0177	0.0166	0.0159	0.0143	0.0121	0.0229	0.0160	0.0236	0.0159
1/500	0.0061	0.0237	0.0246	0.0236	0.0219	0.0215	0.0394	0.0217	0.0383	0.0212
1/2048	0.0087	0.0589	0.0590	0.0580	0.0359	0.0645	0.1205	0.0499	0.1077	0.0472
1/11264	0.0221	0.2693	0.2672	0.2640	0.2443	0.3115	0.5939	0.3631	0.6803	0.3662
1/103424	0.1867	2.3432	2.3384	2.3575	2.3211	2.9327	5.3040	3.4689	6.4391	3.4750
1/513024	1.0228	11.4006	11.3530	12.0689	11.3858	14.5023	26.0701	17.2658	31.6543	16.9941
1/1049600	2.0988	23.3561	23.3382	23.3811	23.4459	30.2119	52.9937	34.9654	64.4141	35.1033
10/100	0.0055	0.0167	0.0171	0.0166	0.0148	0.0104	0.0199	0.0158	0.0215	0.0150
10/200	0.0053	0.0165	0.0155	0.0180	0.0151	0.0125	0.0222	0.0167	0.0264	0.0175
10/500	0.0065	0.0247	0.0226	0.0219	0.0179	0.0207	0.0391	0.0222	0.0376	0.0236
10/2048	0.0088	0.0590	0.0578	0.0625	0.0371	0.0649	0.1192	0.0503	0.1108	0.0484
10/11264	0.0212	0.2701	0.2652	0.2617	0.2412	0.3068	0.5838	0.3553	0.6652	0.3491
10/103424	0.1542	2.3274	2.2972	2.2938	2.2765	2.7586	5.1789	3.4096	6.3119	3.3939
10/513024	0.7236	11.4388	11.4880	11.4932	11.3312	14.1851	25.7927	17.1388	32.1902	17.5287
10/1049600	1.4951	23.4072	23.2000	23.4175	23.2439	29.6794	53.0526	34.9641	64.9861	35.9626
100/100	0.0063	0.1963	0.1956	0.0186	0.1264	0.0146	0.2922	0.2061	0.2281	0.1397
100/200	0.0060	0.0234	0.0219	0.0219	0.0212	0.0180	0.0296	0.0309	0.0422	0.0235
100/500	0.0071	0.0304	0.0297	0.0291	0.0250	0.0273	0.0452	0.0376	0.0559	0.0310
100/2048	0.0103	0.0671	0.0681	0.0676	0.0428	0.0762	0.1329	0.0665	0.1239	0.0516
100/11264	0.0210	0.2625	0.2623	0.2642	0.2422	0.3067	0.5762	0.3600	0.6774	0.3548
100/103424	0.1474	2.2864	2.2856	2.3114	2.2968	2.7858	5.2261	3.4506	6.3282	3.4003
100/513024	0.7241	11.4054	11.4286	11.3435	11.5313	14.1507	25.6885	17.1702	31.8195	17.1162
100/1049600	1.4736	23.2258	23.4724	23.2595	23.3052	29.4276	53.4051	35.2015	64.3818	34.6691
1024/100	0.0047	0.2033	0.2041	0.0517	0.1042	0.0083	0.2899	0.2092	0.2290	0.1204
1024/200	0.0049	0.0562	0.0554	0.0563	0.2900	0.0109	0.0615	0.5340	0.6496	0.3281
1024/500	0.0049	0.0647	0.0638	0.0637	1.3358	0.0181	0.0761	2.5063	3.2811	1.4076
1024/2048	0.0069	0.0955	0.0954	0.0963	0.0688	0.0887	0.1526	0.1536	0.2367	0.0809
1024/11264	0.0195	0.2974	0.2982	0.3000	0.2817	0.3375	0.6168	0.4718	0.8103	0.3845
1024/103424	0.1524	2.3356	2.3316	2.3477	2.3338	2.8351	5.2727	3.5535	6.4703	3.4583
1024/513024	0.7217	11.4283	11.5377	11.4987	11.4162	14.2862	25.8131	17.1762	33.1633	18.1542
1024/1049600	1.4657	23.3272	23.2205	23.2752	23.6812	29.1804	54.9962	34.9720	64.7529	37.1609

Running with HHVM 3.0/2014.03.27, 64bit, 2.3GHz (QEMU):

	strpos	preg	preg U	preg S	preg regex	stripos	preg u	preg i	preg u i	preg i regex
1/100	0.0028	0.0081	0.0085	0.0079	0.0079	0.0039	0.0124	0.0084	0.0125	0.0085
1/200	0.0021	0.0093	0.0094	0.0092	0.0094	0.0045	0.0167	0.0091	0.0162	0.0088
1/500	0.0021	0.0160	0.0183	0.0168	0.0121	0.0102	0.0341	0.0154	0.0309	0.0138
1/2048	0.0051	0.0541	0.0531	0.0523	0.0302	0.0374	0.1118	0.0396	0.0980	0.0394
1/11264	0.0176	0.2531	0.2537	0.2541	0.2373	0.3415	0.5682	0.3451	0.6602	0.3436
1/103424	0.1781	2.2955	2.2846	2.2814	2.2535	3.3880	5.2139	3.3744	6.2748	3.4123
1/513024	1.0425	11.3477	11.6653	11.4149	11.3018	16.9227	26.7142	17.2595	31.6541	17.2308
1/1049600	2.1241	23.2635	23.2525	23.3054	23.2256	34.9508	52.7012	34.9024	64.5221	34.9430
10/100	0.0013	0.0084	0.0084	0.0086	0.0076	0.0040	0.0126	0.0086	0.0119	0.0077
10/200	0.0014	0.0089	0.0091	0.0091	0.0088	0.0063	0.0158	0.0108	0.0167	0.0095
10/500	0.0020	0.0155	0.0158	0.0157	0.0120	0.0129	0.0308	0.0156	0.0308	0.0147
10/2048	0.0073	0.0500	0.0496	0.0505	0.0295	0.0477	0.1088	0.0410	0.1032	0.0397
10/11264	0.0169	0.2536	0.2522	0.2519	0.2312	0.4572	0.5668	0.3498	0.6614	0.3443
10/103424	0.1829	2.3130	2.2926	2.3003	2.2826	4.5526	5.2329	3.4511	6.4243	3.4623
10/513024	1.0610	11.5358	11.5248	11.4948	11.4198	22.9372	25.9165	17.0674	31.4214	16.9746
10/1049600	2.0783	23.8131	23.7319	23.1988	23.2098	46.5398	53.5480	35.4728	65.4561	35.6284
100/100	0.0012	0.0065	0.0062	0.0066	0.0058	0.0017	0.0112	0.0066	0.0101	0.0059
100/200	0.0013	0.0102	0.0101	0.0103	0.0118	0.0106	0.0172	0.0173	0.0250	0.0126
100/500	0.0018	0.0165	0.0163	0.0166	0.0149	0.0170	0.0320	0.0219	0.0383	0.0176
100/2048	0.0042	0.0505	0.0503	0.0508	0.0318	0.0510	0.1090	0.0473	0.1085	0.0432
100/11264	0.0178	0.2556	0.2535	0.2552	0.2356	0.4605	0.5690	0.3537	0.6825	0.3559
100/103424	0.1764	2.2992	2.3004	2.2966	2.2881	4.5522	5.1924	3.4346	6.2967	3.4354
100/513024	1.0296	11.3657	11.3776	11.3046	11.4125	22.5889	26.0071	16.9886	32.3710	16.9588
100/1049600	2.1064	23.4239	23.3236	23.2174	23.1929	46.4756	53.4129	34.8542	65.2457	35.4284
1024/100	0.0013	0.0116	0.0117	0.0121	0.0060	0.0009	0.0216	0.0122	0.0154	0.0058
1024/200	0.0009	0.0151	0.0151	0.0154	0.0071	0.0009	0.0279	0.0139	0.0201	0.0074
1024/500	0.0009	0.0216	0.0216	0.0219	0.0104	0.0009	0.0426	0.0190	0.0331	0.0132
1024/2048	0.0030	0.0557	0.0556	0.0562	0.0632	0.0939	0.1197	0.1173	0.1947	0.0742
1024/11264	0.0161	0.2624	0.2590	0.2588	0.2659	0.4981	0.5789	0.4164	0.7576	0.3794
1024/103424	0.1467	2.2902	2.3172	2.2994	2.2857	4.5648	5.2119	3.4726	6.3626	3.4485
1024/513024	0.7160	11.3871	11.4345	11.3034	11.4561	22.7561	25.8564	17.0333	31.4014	16.9700
1024/1049600	1.4864	23.1771	23.1033	23.4143	23.3364	47.5513	55.7561	36.7900	64.7846	35.1457

Lessons learned:

• shm/apc are 32-60 times faster than memcached or mysql
• shm/apc are 2 times faster than php file cache with apc
• php file cache with apc is 15-24 times faster than memcached or mysql
• mysql is 2 times faster than memcached when storing more than 400 bytes
• memcached is 2 times faster than mysql when storing less than 400 bytes
• php file cache with apc is 2-3 times faster than normal file cache
• php file cache without apc is 8 times slower than normal file cache

Tests were made with PHP 5.3.10, MySQL 5.5.29, memcached 1.4.13, 64bit, 3.4GHz (QEMU):

shm
0.031 0.020 0.021 0.021 0.026 0.028 0.032 0.042 0.084 0.155 0.290 0.629 0.110
Total: 1.489, Avg: 0.115

apc
0.025 0.025 0.025 0.026 0.031 0.036 0.043 0.060 0.106 0.171 0.328 0.756 0.097
Total: 1.728, Avg: 0.133

memcache
3.116 3.014 3.005 3.072 3.077 3.910 3.929 4.067 4.308 10.371 15.323 25.013 3.281
Total: 85.488, Avg: 6.576

memcache socket
1.736 1.756 1.981 1.780 1.809 1.907 1.941 1.983 2.225 9.368 14.071 24.897 1.979
Total: 67.435, Avg: 5.187

memcached
2.241 2.540 2.713 2.769 2.897 3.249 4.286 5.298 7.729 10.539 16.021 28.060 2.578
Total: 90.919, Avg: 6.994

mysql myisam
3.267 3.291 3.310 3.295 3.700 3.777 3.888 4.078 4.368 6.272 6.930 9.626 3.726
Total: 59.529, Avg: 4.579

mysql memory
3.238 3.360 3.470 3.502 3.310 3.346 3.681 4.108 4.370 6.286 7.279 7.079 3.397
Total: 56.426, Avg: 4.340

file cache
0.593 0.595 0.593 0.609 0.546 0.563 0.574 0.600 0.648 0.800 1.115 1.956 0.775
Total: 9.966, Avg: 0.767

php file cache
0.177 0.176 0.175 0.180 0.187 0.188 0.195 0.210 0.236 0.318 0.479 0.901 0.228
Total: 3.650, Avg: 0.281

(10b  0.1k  0.3k  0.5k  1k    2k    4k    8k    16k   32k   64k   128k  array)

Notes: Numbers in seconds, smaller numbers are better, connection times for memcached and mysql are not counted, file cache was done on tmpfs.

Cold cache, fill apc cache from cron:

All entries in apc cache are kept in shared memory. Shared memory is available to the web server and is not shared with the command line php (php-cli). When the web server is restarted, the cache is empty and refilling the complete cache can create performance problems. To fill the apc cache with a cron job, we use a second script to dump the cache to a binary file and load it inside the web server:

// php cron.php
$data = array("cached"=>1, "hello"=>"world");
apc_store($data);
apc_bin_dumpfile(array(), array_keys($data), "/var/cache/apc_cron.bin");

// bootstrap.php, http://myserver.com/...
if (apc_fetch("cached")===false) apc_bin_loadfile("/var/cache/apc_cron.bin");

echo apc_fetch("hello"); // gives "world"

Related articles:

• MySQL 5.6+ InnoDB Memcached Plugin as a caching layer (from Mike Benshoof)

Here is the test script:

<?php
// memcached -m 64 -s /tmp/m.sock -a 0777 -p 0 -u memcache
// memcached -m 64 -l 127.0.0.1 -p 11211 -u memcache
set_time_limit(3600);
error_reporting(E_ALL);
ini_set("display_errors", 1);
ini_set("apc.enable_cli", 1);
mysqli_report(MYSQLI_REPORT_STRICT | MYSQLI_REPORT_ERROR);

$data = array( // test are range from 10-128,000 bytes
  "1111111110",
  str_repeat("1111111110", 10),
  str_repeat("1111111110", 30),
  str_repeat("1111111110", 50),
  str_repeat("1111111110", 100),
  str_repeat("1111111110", 200),
  str_repeat("1111111110", 400),
  str_repeat("1111111110", 800),
  str_repeat("1111111110", 1600),
  str_repeat("1111111110", 3200),
  str_repeat("1111111110", 6400),
  str_repeat("1111111110", 12800),
  array(0=>"1111111110", "id2"=>"hello world", "id3"=>"foo bar", "id4"=>42)
);

echo "shm\n";
$t = array();
foreach ($data as $key=>$val) $t[] = shm(4000+$key, $val);
echo stats($t);

echo "apc\n";
$t = array();
foreach ($data as $key=>$val) $t[] = apc((string)$key, $val);
echo stats($t);

echo "memcache\n";
$t = array();
$m = memcache_connect("127.0.0.1", 11211);
foreach ($data as $key=>$val) $t[] = memcache($m, (string)$key, $val);
echo stats($t);

echo "memcache socket\n";
$t = array();
$m = memcache_connect("unix:///tmp/m.sock", 0);
foreach ($data as $key=>$val) $t[] = memcache($m, (string)$key, $val);
echo stats($t);

echo "memcached\n";
$t = array();
$m = new Memcached();
$m->addServer("127.0.0.1", 11211);
foreach ($data as $key=>$val) $t[] = memcached($m, (string)$key, $val);
echo stats($t);

/* not in memcached 1.x
echo "memcached socket\n";
$t = array();
$m = new Memcached();
$m->addServer("unix:///tmp/m.sock", 0);
foreach ($data as $key=>$val) $t[] = memcached($m, (string)$key, $val);
echo stats($t);
*/

echo "mysql myisam\n";
$t = array();
$m = new mysqli("127.0.0.1", "root", "", "t1");
mysqli_query($m, "drop table if exists t1.cache");
mysqli_query($m, "create table t1.cache (id int primary key, data mediumtext) engine=myisam");
foreach ($data as $key=>$val) $t[] = mysql_cache($m, $key, $val);
echo stats($t);

echo "mysql memory\n";
$t = array();
mysqli_query($m, "drop table if exists t1.cache");
mysqli_query($m, "create table t1.cache (id int primary key, data varchar(65500)) engine=memory");
foreach ($data as $key=>$val) $t[] = mysql_cache($m, $key, $val);
echo stats($t);

echo "file cache\n";
$t = array();
foreach ($data as $key=>$val) $t[] = file_cache((string)$key, $val);
echo stats($t);

echo "php file cache\n";
$t = array();
foreach ($data as $key=>$val) $t[] = php_cache((string)$key, $val);
echo stats($t);

function stats($t) {
  return "\nTotal: ".number_format(array_sum($t), 3).", ".
    "Avg: ".number_format(array_sum($t) / count($t), 3)."\n\n";
}

function format($num) {
  return number_format($num, 3);
}

function shm($id, $data) {
  if (is_array($data)) {
    $arr = true;
    $data = serialize($data);
  } else $arr = false;
  $len = strlen($data);
  $shm_id = shmop_open($id, "c", 0644, $len);
  shmop_write($shm_id, $data, 0);
  $start = microtime(true);
  if ($arr) {
    for ($i=0; $i<100000; $i++) $v = unserialize(shmop_read($shm_id, 0, $len));
  } else {
    for ($i=0; $i<100000; $i++) $v = shmop_read($shm_id, 0, $len);
  }
  echo format($end = microtime(true)-$start)." ";
  shmop_close($shm_id);
  assert(substr(is_array($v) ? $v[0] : $v, 0, 10)=="1111111110");
  return $end;
}

function apc($id, $data) {
  apc_store($id, $data);
  $start = microtime(true);
  for ($i=0; $i<100000; $i++) $v = apc_fetch($id);
  echo format($end = microtime(true)-$start)." ";
  assert(substr(is_array($v) ? $v[0] : $v, 0, 10)=="1111111110");
  return $end;
}

function memcache($m, $id, $data) {
  memcache_set($m, $id, $data);
  $start = microtime(true);
  for ($i=0; $i<100000; $i++) $v = memcache_get($m, $id);
  echo format($end = microtime(true)-$start)." ";
  assert(substr(is_array($v) ? $v[0] : $v, 0, 10)=="1111111110");
  return $end;
}

function memcached($m, $id, $data) {
  $m->set($id, $data);
  $start = microtime(true);
  for ($i=0; $i<100000; $i++) $v = $m->get($id);
  echo format($end = microtime(true)-$start)." ";
  assert(substr(is_array($v) ? $v[0] : $v, 0, 10)=="1111111110");
  return $end;
}

function mysql_cache($m, $id, $data) {
  $d = is_array($data) ? serialize($data) : $data;
  mysqli_query($m, "insert into t1.cache values (".$id.", '".$d."')");
  $start = microtime(true);
  if (is_array($data)) {
    for ($i=0; $i<100000; $i++) {
      $v = mysqli_query($m, "SELECT data FROM t1.cache WHERE id=".$id)->fetch_row();
      $v = unserialize($v[0]);
    }
  } else {
    for ($i=0; $i<100000; $i++) {
      $v = mysqli_query($m, "SELECT data FROM t1.cache WHERE id=".$id)->fetch_row();
    }
  }
  echo format($end = microtime(true)-$start)." ";
  assert(substr($v[0], 0, 10)=="1111111110");
  return $end;
}

function file_cache($id, $data) {
  file_put_contents($id, is_array($data) ? serialize($data) : $data);
  $start = microtime(true);
  if (is_array($data)) {
    for ($i=0; $i<100000; $i++) $v = unserialize(file_get_contents($id));
  } else {
    for ($i=0; $i<100000; $i++) $v = file_get_contents($id);
  }
  echo format($end = microtime(true)-$start)." ";
  assert(substr(is_array($v) ? $v[0] : $v, 0, 10)=="1111111110");
  return $end;
}

function php_cache($id, $data) {
  $id .= ".php";
  $data = is_array($data) ? var_export($data, 1) : "'".$data."'";
  file_put_contents($id, "<?php\n\$v=".$data.";");
  touch($id, time()-10); // needed for APC's file update protection
  $start = microtime(true);
  for ($i=0; $i<100000; $i++) include($id);
  echo format($end = microtime(true)-$start)." ";
  assert(substr(is_array($v) ? $v[0] : $v, 0, 10)=="1111111110");
  return $end;
}

Refactoring of PHP methods is often difficult:

• syntax errors or non-existing methods are only detected during runtime
• wrong method calls or uninitialized variables are only detected during runtime
• wrong order of parameters often remains undetected
• not enough unit tests to validate the changes
• not many resources for refactoring (time + money)

We can solve most of these issues by using a few tools for syntactic analysis.

You might have already worked with grep to search your code:

# find a function in all PHP files (and sub-directories)
grep -rin "SomeFunc(" *.php

This also finds "doSomeFunc(" as well as strings or documentation containing "SomeFunc(".
If you want to find all occurrences of SomeFunc() with exactly 2 parameters or at least 1 parameter, things get complicated.

Happily, Facebook has released a few tools to run static analysis and source-to-source transformations on PHP code. The package is named pfff and is available on GitHub.

Syntactic grep

Using sgrep from pfff, we can do syntactic searches in the code:

# find all occurrences of SomeFunc() with exactly 2 parameters
sgrep -e "SomeFunc(X,Y)" *.php

# find all occurrences of SomeFunc() with at least 1 parameter
sgrep -e "SomeFunc(X,...)" *.php

# ... is a wildcard for any number of parameters
# X, Y are wildcards for a parameter, X != Y

Syntactic patch

Using spatch, we can use syntactic searches to refactor our code:

# change parameters order from ABC to CAB
spatch -e 's/SomeFunc(A,B,C)/SomeFunc(C, A, B)/' *.php

# drop second parameter
spatch -e 's/SomeFunc(A,B,C)/SomeFunc(A, C)/' *.php

# rename SomeFunc to SomeOtherFunc
spatch -e 's/SomeFunc(...)/SomeOtherFunc(...)/' *.php

Install pfff

There are currently no binary packages, so you need to compile pfff manually on your machine. Here is a small guide to get it done with Ubuntu 12.10:

# ocaml 4.0 is currently only in Debian experimental
echo deb http://ftp.de.debian.org/debian experimental main \
>/etc/apt/sources.list.d/debian_exp.list
apt-get install git build-essential debian-archive-keyring
apt-get update
apt-get install libpcre3-dev libgtk2.0-dev binutils-gold gawk
apt-get install -t experimental ocaml camlp4 ocaml-base ocaml-nox \
ocaml-base-nox ocaml-interp ocaml-compiler-libs
cd /
git clone --depth=1 git://github.com/facebook/pfff.git
cd pfff
./configure
make depend && make && make opt
make install

Note: After compiling pfff, the binaries of sgrep and spatch can be directly copied to other systems without installing or compiling other packages.

Resources:

• pfff GitHub repository
• sgrep manual
• spatch manual

First, we setup a machine to analyze the logs:

# install missing packages (e.g. with Ubuntu 12.10)
apt-get install netcat logtop

# create a ramdisk with 1GB for storing the logs
mkdir /ramdisk
mount -t tmpfs -o nosuid,noexec,noatime,size=1G none /ramdisk

# receive logs on port 8080
ncat --ssl -l -k 8080 > /ramdisk/access.log
# open second terminal
tail -f /ramdisk/access.log | logtop

# clean up the ramdisk from time to time
echo >/ramdisk/access.log

Second, we setup the web servers:

# install missing packages
apt-get install netcat

# send logs to analyzer-ip:8080
tail -f /var/log/apache2/access.log | ncat --send-only --ssl <analyzer-ip> 8080

Besides access.log, we can also monitor other log files using different port numbers.

Let's start watching the requests coming in:

Instead of analyzing each line separately, we can also aggregate all requests by client IPs:

tail -f /ramdisk/access.log | awk -Winteractive '{print $1}' | logtop

Or we can aggregate all requests by URLs:

tail -f /ramdisk/access.log | awk -Winteractive '{print $7}' | logtop

Or filter all requests by a user agent:

# show only iPad
tail -f /ramdisk/access.log | grep iPad | logtop

# don't show Google, Msn, Bing
tail -f /ramdisk/access.log | grep -Ev 'Googlebot|bingbot|msnbot' | logtop

To extend the IP address with its owner, we write a small PHP script:

// whois.php
<?php
$fp = fopen('php://stdin', 'r');
while (!feof($fp)) {
  list($ip, $null) = explode(' ', fgets($fp), 2);
  if (!isset($whois[$ip])) {
    $who = shell_exec('whois '.escapeshellarg($ip));
    preg_match_all('!(?:descr|orgname|organization|country|owner).*:\s+(.+)!im',
      $who, $m);
    $whois[$ip] = ' '.str_pad($ip, 15).'  '.$m[1][0].' '.$m[1][1]."\n";
  }
  echo $whois[$ip];
}
fclose($fp);

and run:

apt-get install whois logtop php5-cli
tail -f /ramdisk/access.log | php whois.php | logtop

To send uptime messages every 5 seconds, we can use:

# @analyzer
ncat --ssl -l -k 8081 > /ramdisk/uptime.log
tail -f /ramdisk/uptime.log
# @webserver
while true; do echo -n `hostname`; uptime; sleep 5; done | ncat --send-only ...
# or free disk space, replace uptime with: df -h | grep sda

To configure a firewall on Ubuntu, we can use ufw:

# start firewall, block incoming connections
ufw enable
# allow incoming connections on port 80
ufw allow 80/tcp
# allow limited connections on port 22 (max. 6 connections in 30 seconds)
ufw limit 22/tcp
# show firewall status
ufw status verbose
ufw show listening

# block all new connections from IP 192.168.1.66
ufw insert 1 deny from 192.168.1.66

# remove blocking rule for IP 192.168.1.66
ufw delete deny from 192.168.1.66

Note: If your servers are connected by a secure network (e.g. VPN), you can skip --ssl and certificates.

Coming next: ncat with ssl-certificates