web

Web Services with PHP & nuSoap - Part 1.1

Introduction

[EDIT] - This is a re-hash of a document I wrote a couple years ago.  There's been changes to the nuSOAP library and I wanted to document the updates relative to the tutorial series.  Also, I need to fix a lot of the broken-links and code listings since I've changes hosting providers since this article was first written.

This article is specific to nuSOAP release 0.9.5 on 2011-01-13.  This tutorial was updated on September 22, 2011.

Once upon a time, I was tasked with developing a web-services by my boss as the integration point between our production application and Sales Force.  At this point, although I'd heard of web services ... kind of ... didn't Amazon use web services for something?  Still, I'd never coded a web-services based application before.

At this point, I have to assume you are unfamiliar with the concept of web services and why you may have to create and provide a web-services offering to your client base.  Web services allow a remote client to access functionality (as defined by you, the programmer) via the standard HTTP interface (normally, port:80) to your application and database services.

Back in the day, networking services (semaphores, pipes, streams, message queues, and other forms of IPC) were custom-written and assigned/slaved to unique networking ports for accessing specific service daemons.  Of course, the internet was a kinder, gentler place back then... and a given server may have had dozens, or even hundreds, of non-standard ports open, listening, waiting, for networking service requests.  Or hacking attempts.

Today, web-services is a replacement to dedicated networking apps - all handled by your web server, and all serviced over the same network port: port 80.  Since port 80 is a standard port and, usually, already open on a web-server, additional security risks by opening new non-standard ports for networking services are averted.  Your web-server, such as Apache, now has the responsibility of processing the request and delivering the results to the client.

The web-services component piece is a collection of functions that you've written that provide remote clients access to your system.  These functions are accessible only via the web services framework and while they may be duplicated from a dedicated and traditional web-based application, the web-services framework is designed as a stand-alone piece of software.  Think of the web-services piece as your application's data-processing layer minus the presentation layer.  The "M" and "C" of the "MVC" model.

Initially, when I was tasked with a similar set of objectives, I initially tried xml-rpc.  This led me down a rabbit-hole that spanned nearly a week of my time with the end-result being abandonment.  I hit road-blocks with xml-rpc over server authentication and passing complex objects.  Exacerbating the issue overall is that xml-rpc seems to be dated technology - I had a hard time locating resources that were recent.

Then I stumbled across nuSOAP - it's free via sourceForge, stable and, using a quote from sourceForge:

"NuSOAP is a rewrite of SOAPx4, provided by NuSphere and Dietrich Ayala. It is a set of PHP classes - no PHP extensions required - that allow developers to create and consume web services based on SOAP 1.1, WSDL 1.1 and HTTP 1.0/1.1."

nuSOAP seemed to have more of everything available: tutorials, examples, articles, blog posts.  When I started my implementation with nuSOAP, the first thing I received help with was server-level authentication.  I was able to immediately get my remote requests validated by the web-server and handed off to the web-services module!

The major selling point, for me, on nuSOAP is that nuSOAP is self-documenting.  As part of the API functionality, nuSOAP generates HTML pages that documents the exposed services via the WSDL  and also provides you with a complete XSLT definition file!

First off, download and install the nuSOAP libraries - I provided a link to the sourceForge site a couple paragraphs ago - and unpack the tarball.  You'll end-up with a directory (mine is called: ./nuSOAP) and, within that directory, is the one file you include: nusoap.php.

There are two pieces to this tutorial -- a server side piece and a client-side piece.  While you can execute both pieces off the same environment (machine), normally you'd use the client remotely to access the API server-side code.

What's Not Covered:

Apache.  Apache configuration for your vhost.  Apache .htaccess.  This article assumes that you've a working server and that you're able to install and access the server files via Apache.  Even if your Apache server is a localhost configuration, access the server-side files via localhost client, traversing the TCP stack locally, is still a valid method for testing your web-services server application.

 

Time to push up our sleeves and start working on the server code...

The Web-Services Server

Today, we're going to write a ping server -- where the server has an exposed service (method) named "ping" which takes a single argument (a string) and returns an array back to the calling client.  The return array contains two associative members: a boolean (which should always be true - otherwise there are other issues...) and a string which is the modification of the original string in order to prove that, yes, we went there and we came back.

Because my project, and I'm doing this project for my new company, is going to represent significant effort, size and complexity, I've broken out the components of nuSOAP request into exterior files because, later, these will become control file which will, in turn, load files that have been organized into a hierarchy friendly to the application's data model.

So, if this file, which I've named index.php, seems small, remember that you're not viewing the dependent files (yet).

[ccne lang="php" line_numbers="on" tab_size="4" width="100%"]

<?php
/**
 *
 */

// setup...
require('./nuSOAP/nusoap.php');
// set namespace and initiate soap_server instance
$namespace = "http://myapi/index.php";
$server = new soap_server('');
// name the api
$server->configureWSDL("myapi");
// assign namespace
$server->wsdl->schemaTargetNamespace = $namespace;
// register services
require('myServiceRegistrations.php');
// load WSDL...
include('mywsdl.php');
// load services code modules...
require('myServicesModules.php');
// create HHTP listener:
$request = isset($HTTP_RAW_POST_DATA) ? $HTTP_RAW_POST_DATA : '';
$server->service($request);
exit();
?>

[/ccne]

So far, so good - let's take a look at what we've just done:

  • we've included the nu_soap library...
  • declared our namespace (which is the URL of the api server)
  • instantiated a new soap_server instance and assigned it to the variable $server
  • initialized the WSDL
  • assigned the namespace variable to the WSDL
  • load and register our exposed services
  • load the WSDL
  • load the service code
  • create the HTTP listener
  • invoke the requested service
  • exit

This (index.php) file is the server-side file that will be invoked for ALL future API calls to the service.  It invokes three control files which, in turn loads the services (WSDL definitions), the WSDL variable definitions (think of these as inputs and outputs to your exposed services), and the actual code for all of the services, and their supporting functions, that you're going to expose via your API.

Side Note:  This is the file you'll reference in Apache when you're (preferably) creating a new Virtual Host for the API.  HTTP requests that resolve to your server will be serviced by Apache which will, in turn, serve the results of this program back to the client.

Next, we're going to define the myServiceRegistrations.php file that is required by index.php.  This file contains the WSDL for each and every exposed service that the API serves.

[ccne lang="php" width="100%" line_numbers="on" first_line="1" tab_size="4"]

<?php
/**
 *
 *
 */

$server->register('ping', array('testString' => 'xsd:string'),
                          array('return' => 'tns:aryReturn'),
                  $namespace, false, 'rpc', 'encoded',
'<p><strong>Summary</strong>: returns response to a ping request from the client.  Response
includes testString submitted.  Used to test Server response/connectivity.
</p>
<p>
<strong>Input</strong>: $testString (type: string) and random collection suffices.
</p>
<p>
<strong>Response</strong>: Service returns an array named $aryReturn with two associative
members: &quot;status&quot; and &quot;data&quot;.<br />$aryReturn[&quot;status&quot;] should
<i>always</i> return true.  (A time-out is an implicit false.)<br />If no value was passed
to the service via $testString, then the value of $aryReturn[&quot;data&quot;] will be
empty.<br />Otherwise it will contain the string: &quot;Rcvd: {yourString} (count)&quot; to
show that the message was received and returned. (count) is a character count of the
passed string, also validating that the passed data was received and processed.
</p>
');

[/ccne]

This PHP code registers a function called "ping" with the nuSOAP $server instance.  The second parameter is the input to function.  Note that all input (and output) parameters have to be declared as an array even if there's only a single value being passed.  Also notice that you have to type-cast the variable being passed using XML datatypes.  For your data definitions, you use one of the 44 built-in datatypes defined in this document: http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/.

(For more information on XSD object and XML schema, please visit: http://ws.apache.org/axis/cpp/arch/XSD_Objects.html.)

The third argument to the method "register" is the data type returned.  This is a complex variable called "aryReturn" -- don't worry about right now, we're going to define this complex-type variable in another configuration file.

The next four arguments are:

  • our $namespace variable we set in index.php
  • boolean false
  • 'rpc' for the call type
  • 'encoded'
Use these values literally.
The last variable is a huge block of HTML.  This block of HTML can be as large, or as small, as you need it to be.  It's the basis for the WSDL documentation that nuSOAP generates for your client-side developers.
When developers hit the server URL, they'll be presented with your API documentation that nuSOAP generates from the WSDL file(s).  As your API grows, exposed methods will be listed in the blue-ish box on the left side of the screen.  Clicking on any of the methods will expose the details (requirements) about that method thus:

Nice, huh?

The second file (that we've included in our source: (mywsdl.php)) is the WSDL file that defines our data structures that are used as either inputs, outputs, or both, to the exposed services.  Another thing I like about SOAP and nuSOAP is that it introduces a layer of strong-typing to the PHP stack.  You can save a bit of debugging time by requiring that you call a method with EXACTLY this and return EXACTLY that.  Anything else tends to generate a system-level error:

[ccne lang="php" width="100%" line_numbers="off"]

[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP Fatal error:  Call to a member function addComplexType() on a non-object in /htdocs/LL2/trunk/services/mywsdl.php on line 9
[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP Stack trace:
[Thu Sep 22 14:36:59 2011] [error] [client ::1] PHP   1. {main}() /htdocs/LL2/trunk/services/ll2wsdl.php:0
[/ccne]

This error message, from the apache error log, is somewhat obfuscated in it's meaning.   I attempted to return only the string, by itself, instead of returning the array (of two elements) that I had told nuSOAP I would return for this service.  This error was generated because the types (between the code and the WSDL) of the return variable (structure) did not exactly match.

If you've only ever coded in a loosely-typed language, like PHP, than this part of SOAP is going to be a bit of a ... transition ... for you.  When we say that something, be it a variable, function, or exposed service, is strongly typed, we're declaring the type of that object and, if the type of the object during run-time does not match, then SOAP will force PHP to throw a fatal as shown in the error log snippet above.

Keep this in-mind as you develop exposed services that are increasingly complex.  Since the error messages tend to point you at your code, at the point of failure, it's easy to forget that that the requirements of the underpinnings (in this case, the WSDL), are the root cause of your PHP fatals.

That being said, let's take a look at the WSDL for our ping service:

[ccne lang="php" width="100%" line_numbers="on" first_line="1" tab_size="4"]

<?php
/**
 * WDSL control structures
 *
 * initially, while in dev, this will be one large file but, later, as the product
 * matures, the plan will be to break out the WSDL files into associative files based
 * on the object class being defined.
 */
$server->wsdl->addComplexType('aryReturn', 'complexType', 'struct', 'all', '',
            array('status' => array('name' => 'status', 'type' => 'xsd:boolean'),
                  'data'   => array('name' => 'data',   'type' => 'xsd:string')));

[/ccne]

We're invoking the nuSOAP method addComplexType to define a structure to the WSDL in our Table Name Space (tns).   To do this, we first define the name of the structure that we're going to use: aryReturn and then we define the composition of that structure.

The declaration for this looks a lot like a standard PHP declaration for an array with the exception of the XSD (XML Schema Definition) appended at the end of each element's declaration.  (See the links I embedded above for explanations and examples of valid XSD.)

XSD provides part of the strongly-typed concept for our structure elements.  We're telling nuSOAP to expect a variable structure containing these named elements of this type.

What we have, then, is an associative array with two elements: 'status' and 'data'.  $aryReturn['status'] and $aryReturn['data'] and they're of type BOOL and STRING respectively.

Note, finally, that this variable structure isn't confined to single-use.  Once we've declared it within our tns, it's available to any exposed service where it's needed.  This is the model for my common error structure -- the boolean indicates success or fail on some service operation and the data component contains the relative diagnostic information.

The third and final file we're including into the server source is the code for the exposed service.  This is where you write the function handlers for your services.  Since you've already defined the input parameters, and the return types for the ping service, there's very little left to do.

[ccne lang="php" width="100%" line_numbers="on" first_line="1" tab_size="4"]

<?php
/**
 * ping
 *
 * service that confirms server availability
 *
 * input: any string
 *
 * output: reformatted string:  "Rcvd: {string} (charcount{string})"
 *
 * @param string $testString
 * @return string
 */
function ping($testString = '') {
    return(array('status' => true, 'data' => 'Rcvd: ' . $testString . '(' . strlen($testString) . ')'));
}

[/ccne]

Note the following in the code for our exposed service:

  • our exposed method is named ping because that's how we registered the service (myServiceRegistrations.php)
  • we're providing a default type-cast for the input param of string in case the service is invoked without input params.
  • we're returning BOOL true and prepending "Rcvd: " to the received string, and appending a character count to prove that the service successfully responded to the client's request.
  • the return structure exactly matches the WSDL declaration: the name of the array elements, and the element types.
If you've correctly installed and referenced (within your PHP) the nuSOAP libraries, then you should be able to load the url of the new server source file into your web browser to see the nuSOAP-generated documentation for your new web services.  Click on the WSDL service function: ping to see a detailed description of the function.

If you're using IE, then clicking on the WSDL link will return the XML.  If you're using Firefox, Chrome or other browsers, clicking on WSDL will display the generated XML for your service.

Now that the server is working on it's own, it remains fairly useless until we can get a client to connect to it invoke it's methods.  Let's work on the client next...

The Web-Services Client

The web services client application will also be written in PHP.

The web client is an application that connects to remote server using the http port 80.  To do so, you'll need the client to be aware of certain bits of information that may, or may not, be required to access the remote server.

In our client, we're not going to require remote authentication -- but I'll take a quick aside andexplain how you would include this, client-side, if your server required .htaccess authentication.

nuSOAP has a client method called setCredentials which allows you to specify your .htaccess username and password and the authentication schema.  It's a single line of code which is normally used to require not only clients to login to access your API, but, once identified, you can limit the set of exposed methods available to individual clients or groups.

For example, if you have a product you've developed in-house, then you'd want full-access for your front-end web/applications servers.  Your PM later decides to open a subset of the API to the general public and a subscription-based set of exposed methods in order to monetize your product.  Finally, the PM also wants to "white-box" the product so that other companies can use it but with their branding and access to isolated or discrete data sets.

What you'd end-up with is several levels of client access to your web services.  Implementation of limiting exposed services would be handled server-side but it would be based on your client's authentication and possibly the subject of a future tutorial...

So, to the client-side code: (name this file: apiTestClient.php)

[ccne lang ="php" width="100%" line_numbers="on" first_line = "1" tab_size="4"]

<?php
// Pull in the NuSOAP code
require_once('./nuSOAP/nusoap.php');
$proxyhost = isset($_POST['proxyhost']) ? $_POST['proxyhost'] : '';
$proxyport = isset($_POST['proxyport']) ? $_POST['proxyport'] : '';
$proxyusername = isset($_POST['proxyusername']) ? $_POST['proxyusername'] : '';
$proxypassword = isset($_POST['proxypassword']) ? $_POST['proxypassword'] : '';
$useCURL = isset($_POST['usecurl']) ? $_POST['usecurl'] : '0';
$client = new nusoap_client('http://{YOURSERVERURLHERE}/index.php', false, $proxyhost, $proxyport, $proxyusername, $proxypassword);
$err = $client->getError();
if ($err) {
	echo '<h2>Constructor error</h2><pre>' . $err . '</pre>';
}
$client->setUseCurl($useCURL);
$client->useHTTPPersistentConnection();
// Call the SOAP method
$result = $client->call('ping', array('testString' => 'Argle'), 'http://localhost');
// Display the result
if (!$result) {
    echo "service returned false";
} else {
    print_r($result);
}
unset($client);
?>

[/ccne]

The first thing we do in our client-side code is to include the nuSOAP libraries.

The next five lines of code read from the local POST environment, testing if you've established a proxy for your web-services server and, if so, populating the proxy variables.

The next line instantiates a nuSOAP client and associates it with our remote server.  Change "YOURSERVERURLHERE" to the name of your apache web server URL where you have the server side code installed.  (e.g.: localhost, myserver.com, etc.)

Note the name of the function call: newsoap_client()...as opposed to using the function soapclient().  This function name is legacy-compatible with PHP 5.0's instantiation call:  new soapclient() - the PHP SOAP extension uses the same instantiation function name as the nuSOAP library.  If you have both installed, (PHP 5.0 SOAP extension, and the nuSOAP libraries), executing the client will return errors as you've overloaded the soapclient() function.  (You're calling the PHP SOAP function with the nuSOAP function parameters.)  Rename the soapclient() function to the back-compatible function: newsoap_client().

The next two lines tell the nuSOAP client to useCurl, when possible and if previously saved and, if possible, to use HTTP persistent connections.

Next, we're going to consume the web-services from the server by making a call to the nuSOAP method: call().  The arguments to this method are:

  1. the name of the service being consumed (ping)
  2. the input string (note:  all inputs must be passed as arrays!)
  3. the namespace URI (optional:  WSDL can override)

Store the results of the web-services to the aptly-named variable $results and evaluate it upon return.  If the client call was not successful, then display an error message.

If the client call was successful, then display the contents of the returned array.

Note that you can run this client code from either a browser using the "file://" option or, if the client source code is accessible to apache, then you can display using a browser.

When I run this client-side software in the browser, I get displayed back:

Array ( [status] => 1 [data] => Rcvd: Argle(5) )

So, how do we know that the client went out and successfully returned from the server with the data?  Simple - in the client source, you will not find the literal "Rcvd:" anywhere in the source.  This data was supplied by the server-side function and returned back to the client.  It's a simple example, but it provides proof-of-concept that we're successfully able to connect to a remote web-server and return data created on the remote server back to the client program.

Let's wrap this up...

Summary:

This tutorial (hopefully) explained what web-services are, and provided you with a practical example of a consumable service: ping().  Such a service would normally be invoked as a means of testing server availability.

We created a web-services server file using the nuSOAP library by defining a complex-structure (an associative array) and registering a method with the nuSOAP server.  The method takes an input parameter which, although it's only a single input parameter, must be built and passed as an array construct to the server method.  Next, we showed that by loading the server source into a browser, we learned the nuSOAP provides built-in documentation for your web-services structures and methods.  Which is a very nice-to-have when you're creating your developer documentation!  Next we created the web-services client source code file which connected to our remote server and invoked the server's method: hellow().  If all worked correctly, you displayed the string returned from the remote server within your browser window.

In the next installment, I'll cover the web-services server-side of this business and we'll see how to create the various methods for accessing a mySQL database, complex structures as input and output parameters to those methods, and general debugging techniques.

Thank-you for your patience - I hope this article helped you.

September 26, 2011

Part 5: Setting-up a Linux Development Machine: Virtual Hosts in Apache2

When I am working on code project, I isolate that project within it's own directory/repository.  Further, it matters not if I'm starting a completely new project, or if I'm branching off the trunk of an existing project.  As a means of imposing order over chaos, I isolate the existing project within it's own sandbox both on the filesystem and via Apache2.

To do so requires an understanding, somewhat, of the mechanics of Apache2, DNS, and your localhost.  A minimal understanding, trust me.

What it, in return, gives you is an isolated view of your code project from the web-server perspective.  Cookies are isolated by domain, your document root is isolated to a single directory/repository, and you not only put your log files, just for that domain, where ever you want but you can also name them anything you want as well.

What I'll provide you with in this installment is a rudimentary understanding of the mechanics behind virtual hosting using Apache2, a template configuration file to get you going, and the basic steps necessary to get the whole mess working.  Let's get started...

When you start a new project, if you're checking it out from a source-code repository, you'll typically assign it to a directory somewhere common.  For example, within your home directory, you may have a folder named "code" and beneath that folder, other folders that describe either the project or the programming language you're working in.  Doesn't really matter as the point is this:  you've isolated your code repository from everything else on your filesystem, right?

It really doesn't matter, to Apache2, where you create your filesystem repository.  As long as the webserver pseudo-user has access permissions to the directory, then you can access the files within that directory via a web browser.  The webserver has to be configured to be told that, for a given domain name, where is the documentRoot for that domain.

Some of you, at this point, may be asking: what's a domain name and why is it important?  Well, a domain name is simply a name you've assigned to the project to keep it separate, at least in your own head, from the other projects you may, or may not, have running on your development machine.  For example, I create a new project called newWidget and it's currently in the 1.4 revision.  I'm ready to branch and write some new features for the product so, using whatever sccs tool, I branch off the trunk and create the 1.5 branch.

I check that branch out to a directory in /lampdev/php/newWidget115.  I now need to do two basic things:

  1. invent some domain name that will be used exclusively for this project and resolve the domain to my localhost
  2. create a virtual host in apache so that apache knows that http://newW115 points to my localhost

The reasons, apart from what we've already discussed, is to keep your local DNS services on your local machine.  If you, before entering any configuration information, entered: http://newW115 into a browser url bar, chances are very good you're going to end-up on a search page (I'm using Chrome) or get some sort of browser error.

So the first step is to define the new domain name (again, given that we're already checked the code out into the aforementioned directory) to the local system so that all requests to that domain are resolved locally through our name services.  To do this, we're going to sudo edit the /etc/hosts file.

This file, /etc/hosts, is the first thing checked whenever your local name services is trying to resolve a host name.  If it finds a host-to-IP alias in this file, all further attempts at resolution are halted as it has successfully resolved the host name.  Edit /etc/hosts to resolve your new domain.  It should look something like this:

[cc lang='bash' line_numbers='false']

127.0.0.1    localhost codemonkey codemonkey.shallop.com codeMonkey.shallop.com newW115

[/cc]

The way /etc/hosts works is that you first list an IP address for the domain to resolve to - in this case, we're using 127.0.0.1 which is TCP/IP speak for your local host.  Next we list all of the domain names that are going to resolve to this IP address.  In the example above, we're resolving localhost, codemonkey, codemonkey.shallop.com, codeMonkey.shallop.com, and the new domain: newW115 all to 127.0.0.1.

Whenever I type one of these domains, for example, in to a web browser URL bar, my local host domain services won't go out to my network name servers to resolve the domain name -- it's telling the requesting service that it's 127.0.0.1.  Note, too, that you can alias multiple domain names to the same machine.

Side Note -- this is how you can blacklist certain domains from your browsing experience.  Simple resolve that domain to 127.0.0.1...but that's an article for another day...

You can also have multiple entries resolving to the same IP address.  It would have been just as correct for me to have listed by /etc/hosts file as:

[cc lang='bash' line_numbers='false']

127.0.0.1     localhost 127.0.0.1     codemonkey 127.0.0.1     codeMonkey 127.0.0.1     codemonkey.shallop.com 127.0.0.1     codeMonkey.shallop.com 127.0.0.1     newW115

[/cc]

Finally, also note that a domain extension isn't really required.  We can name our domain pretty much anything we want and as long as you universally use that spelling (and case), then it will resolve locally.

Now that the domain is resolving locally, the next step is to tell Apache2 how to handle the request.  When you type: http://newW115 at the browser, the browser will query local services and receive a response that the domain is handled locally.  Apache2 will then say: "Oh, if it's local, then were do I go to get the files and stuff?"

The configuration for Apache2 is done with virtual hosting.  Technically, you can do this without virtual hosting -- but you can only do it for one domain.  If you want to locally-host multiple domains, you have to use virtual hosting.

The Apache2 configuration file lives in: /etc/httpd/conf and is named: httpd.conf.  This is the main configuration file for Apache2.  Some installations use a sub-directory, usually called something like: vhostsd.conf, and stores the vhosts.conf file within that directory.  That's ok, too.  Apache2 is versatile that way but, for our purposes, we're going to maintain the virtual host configuration(s) within the main conf file.

However, if you wanted to use a separate file for Virtual Hosting, all you need in your httpd.conf file is the directive:

[cc lang='apache' line_numbers='false']

# Virtual hosts Include conf/extra/httpd-vhosts.conf

[/cc]

At the very end of httpd.conf, there's a section called: Name-Based Virtual hosting.  We're going to append this virtual host configuration to the end of this file.

Allow me to side-step for a quick second.  Consider if we were to install phpMyAdmin locally on our server because this is how we want to administer our mySQL database.  We can install the program files anywhere as phpMyAdmin is just another LAMP application, right?  Were we to do that, then we would need a <Directory> directive to Apache2 telling Apache2 where to look for phpMyAdmin.  The domain for phpMyAdmin would still be localhost, or 127.0.0.1 or whatever else you'd defined in /etc/hosts.  The location of the application can live anywhere and we're using the conf file to tell Apache2 how to find and serve it to us when requested.

[cc lang='apache' line_numbers='false']

Alias /phpMyAdmin "/opt/local/www/phpmyadmin" <Directory "/opt/local/www/phpmyadmin"> Options Indexes MultiViews AllowOverride None Order allow,deny Allow from all </Directory>

[/cc]

What this <Directory> directive simply does is tell Apache2 where to look for phpMyAdmin if I enter something like: http://localhost/phpMyAdmin in the URL bar of my browser.  It's not the same thing as giving phpMyAdmin it's own domain at all.

I do this with a lot of my web applications: phpMyAdmin, mcmon, ajaxmytop, nagios, etc., simply because I don't want to remember where the fill path name is of the applications.  It's easier to type: http://localhost/phpMyAdmin that it is to type: http://localhost/webapps/database/phpMyAdmin.

Ok, so back to domains.  Here's the template for the virtual host we've created in /etc/hosts: newW115:

[cc lang='apache' line_numbers='false']

<VirtualHost *:80> ServerName  newW115 ServerAdmin mshallop@gmail.com DocumentRoot /code/webapps/LAMP/newWidget/1-15

DirectoryIndex  index.php

<Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory /code/webapps/LAMP/newWidget/1-15> Options Indexes FollowSymLinks MultiViews AllowOverride All Order allow,deny allow from all </Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all </Directory>

ErrorLog /var/logs/115_error.log

LogFormat       "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat       "%h %l %u %t \"%r\" %>s %b" common LogFormat       "%{Referer}i -> %U" referer LogFormat       "%{User-agent}i" agent        CustomLog       /var/logs/115_log common        ErrorLog        /var/logs/115_error_log

# Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. LogLevel warn

CustomLog /var/logs/115_access.log combined ServerSignature On

</VirtualHost>

[/cc]

This is a pretty minimal configuration -- but it's the boilerplate template I use for all new domains and it works.  The lines that in boldface are the lines you should change to match your environment.  Note that you can pretty much put files, such as the log files, where ever you wish.  I changed the names from my normal location but, as a rule, I maintain the entire environment outside of the root filesystem.

Once you've made your changes and saved the file, you'll need to restart Apache2 so that it will read the new configuration.  If there are errors in your configuration file, Apache2 will let you know and will refuse to start.  Make sure you've corrected all errors and, once the server successfully restarts, you should be able to type: http://newW115 into your browser URL bar and have that domain resolve locally, and serve files from the directory you specified in the httpd.conf file.

Over time, as you add additional projects and create new code-domains, you can simply add the new <VirtualHost> directives, appending them, to the httpd.conf file as needed.  When you expire and remove hosts and files, don't forget to remove them from the Apache configuration as well.

And that's pretty much it.  This is a simple thing to set-up as we didn't delve into anything that wasn't plain-vanilla.  For example: SSL configurations, .htacces, or the re-write engine.  That's for another day, another article.

Hope this helps...

Part 4: Installing Apache Thrift: Linux Development Environment

Previously, we dealt with getting a working LAMP development environment up and running on a fresh CentOS 6 install.  We next dealt with the installation of PHPStorm and our JDK issues.

In this, and the next issue, I'm going to talk about the Thrift framework and getting it installed and running.

Thrift was originally developed by Facebook, was entered into open source in 2007, and became part of the Apache incubator the next year.

Thrift, according to Apache, is "a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, and OCaml."

What it is in plainspeak is an API framework for your LAMP application.

Why I want it:  I want to use Thrift for our project because of the nature of the project.  (A social-networking concept.)  Because the application will rely heavily on data-storage calls, I've decided to implement the data access layer as an API instead of a more-traditional OOP model.  Thrift, as the API framework, allows me complete freedom on the back-end of the API.  I can implement the API in a variety of languages, although I'll probably use PHP.

Thrift also provides me with a strongly-typed interface to the API.  Like XML-RPC, calls to the API are well-defined beforehand and must comply with the typed definition of both the methods used, and the data exchanged to/from said methods.

My personal experience with Thrift is limited -- I used it as an API for a product concept at a former employer.  The calling application would invoke the API and make requests to the API which, in turn, would do a "bunch of stuff" and return a well-defined "data ball" (a json object) back to the calling stub for processing and display.

The other concept that makes me embrace Thrift as the controller for my LAMP application is that I can completely encapsulate the data layer from the front-end developers.  They do not need to know if the data is stored within mongodb, mysql, or a flat file.  All they need is the data.  The query language is hidden; front-end developers should not need to write data-access code.

I'll talk more about the glories of Thrift later.  For now, let's just get it installed and running...

On our Linux system, we have to do some preliminary installation of packages first.  Luckily, if you hit the Thrift Wiki, you'll find pretty much everything you need to do a successful install.  Be warned, however.  Sparseness of documentation could easily be one of the hallmarks of Thrift.  Read carefully, and then read again before punching the enter key on your keyboard.  Make sure you understand what it is you're about to do.

Ok.  Let's get some non-LAMP development tools installed.  Our first command will be to install most of the pre-requisite packages needed by Thrift:

[cc lang='bash' line_numbers='false']

#  sudo yum install automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel

[/cc]

This  will install the base development packages you're going to need.  Once this has completed, you should also install the open-SSL development libs as the build will fail without it.  (At least, if failed on my install.)

[cc lang='bash' line_numbers='false']

#  sudo yum install openssl-devel.x86_64

[/cc]

Installing this package will also pick-up all the dependent packages you'll need to complete the install.

Next, download the Thrift tarball from the site and move the package somewhere within what will become your DocumentRoot path for Apache2.

[cc lang='bash' line_numbers='false']

#  tar xvzf thrift-0.7.0.tar.gz

[/cc]

Once you've expanded the tarball, cd into the thrift directory and follow the instructions to make the Thrift packages and libraries.  I did this pretty much exactly as told and my installation went without a problem.

At this point, we've only built and installed the Thrift libraries (installed in /usr/lib, I believe...).  In the next installation, we're going to install the PHP src directory and make it visible to our application's docRoot.

Installing a Linux-Based Development Environment - Fedora 15 Fails

I've scrapped my VOIP project with the realization that, advanced DSL or not (whatever THAT is, AT&T), I just don't have the upload bandwidth to support the project to a place of anything usable.

So, after a couple months of looking at an inactive PC sitting on the top of my desk doing absolutely nothing, except irritating my OCD bone, I decided to embark on a new project -- installing a linux-desktop based IDE platform.

Keeping my goals simple, I want a LAMP development environment with the option to later build out to a compiled-based environment/IDE for C/C++.

I am installing on an older PC - an HP Pavilion Slimline S3300f  that initially had Windows Vista installed but has since advanced and evolved to host a variety of various Linux platforms.  The CPU is a AMD Athalon X2 64-bit Dual-Core hosting 2-gb of PC2-5300 SDRAM, a 500-gb hard drive and a 2007 Phoenix BIOS.  Perfect for Linux.

I thought about which flavor of Linux to install - over the years I've used pretty much all of them and finally decided to try something new...new in that I've not used it in quite a while.  I decided on the latest 64-bit Fedora install with the Gnome IDE available from the Fedora Project.  While I normally choose CentOS as my linux-of-choice, I picked Fedora because I wanted to re-visit the distribution.

I've not used Fedora since 2002.  I was never that enthusiastic about Fedora since it seemed, to me at the time, to spend too much time customizing what was, at that time, standard unix configurations.  Well, I really like CentOS, which is based on Fedora, so maybe it's time to put past-prejudices aside and see what's up with one of the most popular distributions in the world.

And, while we're at it, let's eliminate our past-dependencies on KDE and stagger completely free from our comfort zone and go with the Gnome desktop, shall we?  <muted screaming noises>

I downloaded the Fedora x86_64 CD ISO from the Fedora project and booted my Pavilion off the CD.  A desktop soon appeared which, I have to admit, is really nice.  The default boot brings you into a run-time environment chock-full of neat applications which I didn't take the time to explore...unfortunately...but I do like the concept.  This ISO gives you a portable and secure desktop system that, with internet access, provides you with the ultimate, secure, thin-client.

Anyway, I clicked on the install-to-hard-drive option and dove into the Fedora installation program.  I have no screen shots for you  - I am writing this blog on my Mac OS-X bootmaster partition of Windows 7 (Ultimate-64) because I'm too lazy to reboot back into Mac OS X and I'm going to play Rift later...

The Fedora installer took two options, initially -- my language and the storage device choice.  English and local (as, say opposed to SAN) and it paused because I had encrypted, in a previous install, the hard drive.  So, it waiting patiently until I was able to dredge the password out of my abused long-term memory.  Luckily, I had saved it in a keychain...

I then configured standard stuff - machine name (codemonkey), root passwords, and storage option.  I elected to use the entire hard-drive which will wipe whatever crap I previously had installed/stored.  In this, I was also given the option to encrypt the file system (which is cool in a James Bond sort of way but a pain-in-the-ass in RL because you have to remember your password and you do take a CPU hit), and to review the partitioning schema.  I went with no encryption and no review -- nuke it, Chuck.

While it's installing, I wanted to add that this system is using a 17" LCD monitor.  While this type of monitor used to be the shizzle, it's now dwarfed by my iMac's 27" display and the slaved 23" LCD next to that.  On the other hand, I now have 71" of LCD wrapping around my two keyboards...which is pretty cool.  Traditionally, Linux never seemed to give me the same video resolutions on this monitor that Windows was able to.  8x6 or even 1024x768 just doesn't seem to cut it anymore, so I'm curious as to what Redhat will deliver.

The Fedora installer copied the live image to my hard drive fairly quickly.  I noticed that the installer should update it's (C) date and include 2011 - I believe this install was last update just a few days ago...and, the entire time the installer is running as a windowed app beneath the live desktop.  My first linux install was sometime around 1991 under version 0.91 or so -- so the installers have definitely evolved!

After about 10 minutes, I was presented with the reboot message....so I popped the CD out of the tray and rebooted...exiting the installer took me back to the live desktop.  I had to manually power off the machine, reboot, remove the CD from the tray...so I missed the short window to pop back into the BIOS to reset the boot order.

Booting from the hard drive, I was brought to a Welcome screen (very snazzy!) and querying me for more information.  I was asked to create a user account - which I added to the administrator's group assuming that this would bestow me with su privileges.   I then set-up the NTP server and submitted my hardware profile back to the Fedora Project.

Following this, I was presented with the log-in window...and I after entering in my password, I was introduced to my first-ever Gnome desktop environment under Fedora.  (2001 Space Odyssey music playing in my head.)

I'm immediately happy because I'm at 1280x1024 resolution for the first time, ever, in a linux install on this hardware.  I'm going to play around with my desktop for a bit and get used to the UX while finding a comfortable configuration.

The system information informs me that I am running Gnome 3.0.1 with 1.8gb of RAM, and AMD Athalon 64x2 Dual Core Processor 5000+ X2, a Gallium 0.4 on NV4C graphic driver, 64-bit OS and 931.4 gb free on the hard drive.  df -h output shows that I've used 6% of my 50gb root partition and I'm worrying that this won't be enough to support system updates.  Looks like everything will have to be installed into /home.

So, I have the following (default) filesystem set-up:

/ (root) at 50gb with 94% free /boot at 500mb with 91% free /home at 864gb with 99% free /tmp at 50gb with 94% free

Interesting that this window also provides me with update notification which was my next step...installation of updates and installation of new packages...the software has located a staggering 634 updates at 197.3mb...the majority of which appear to be bug-fixes with the remaining packages listed as enhancements.

The package install fails immediately with what I suspect are permissions issues.  I then next learn that Fedora has disabled root logins into Gnome.  Since this is an administrator account, I need to figure out how to install these updates.  I re-login and try to re-install and get another failure, albeit a different failure message.  I try to, and succeed, in installing a single update and the installer informs me I have to re-login for the updates to be effective.  I do so, but it's a bit of an Easter-Egg-hunt as the installer reset my terminal...I have to Ctrl-Alt-F8 to get back to my terminal session and I re-login and restart the software update program.

And I keep getting errors...geh... time for terminal.  I google the issue and discover that it's a known bug in the package kit installer and to use yum instead.  I switch to terminal, su to root and yum update ... The (now) 302 updates I need spin down to the box effortlessly and the entire system is updated in about 10 minutes.

The next step is to identify and install the development components.  Also, I'll need to install some server software (apache2, php5, mysql, mongodb, etc.) for the LAMP development environment.  For now, what I have is a working desktop environment.

I pulled out an old set of Logitech speakers from the garage, rescuing them from an in-progress garage-sale, and now I have working sound which is cool.  I also want to get network shares up and running so that I can copy files back-and-forth across the network.

As a final note to this post, I had a hell of a time trying to get Firefox to connect.  It's a DNS server issue with Fedora 15 and FF that's based on auto-enabling of IPV6 protocols.  To "fix" the issue, simply add google's default nameserver configuration to your /etc/resolv.conf file:

nameserver 8.8.8.8

And any name-server resolution issues in firefox should disappear immediately.

[EDIT]

I'm going to have to trash Fedora 15 as a viable option.  I had the desktop lock-up on me while displaying the application listings following a Chrome install.  When I rebooted, I lost the nameserver configuration that I added to /etc/resolv.conf (above) and had to manually add it so that other network-based applications (browser, chat client) would work.

All in all, there's probably fixes for these issue but at this point I really don't care.  There are other Linux distributions out there that are stable and don't distract you from accomplishing real work with niggling little issues such as what I've experienced.  The application lock-up was unforgivable, to be honest.  If I wanted to experience fun like that, I'd install Vista back on this machine.

I'm really kind of surprised to have this much trouble with a Fedora install considering their reputation for reliability and stability in the enterprise.  Clearly, Fedora isn't ready for a desktop/client market.

I'm in-process of downloading CentOS 6 -- we'll see how that goes in the next installation....

Online Cloud Storage...Which one?

In a recent article, I wrote about cloud storage to use for my source-code repository.  I chose ZumoDrive as the tool to implement this because it allowed filesystem level access to my files from the desktop.  Or, in other words, my desktop sees the ZumoDrive like another physical device attached to my computer. However, in actually using ZumoDrive, I noticed some ... features ... that I wasn't too pleased with.  Instead of capping on just ZumoDrive, I thought I'd offer a narrow perspective on the capabilities and ... features ... of some of the more popular online cloud-based storage options.

My selection process was based on simply whether or not I could use the storage from my Mac.  Let's get started...

The first system tested was Memopal -- this solution is available on all platforms, (Windows, Linux, Mac, Android, iPhone), and offers 2Gb of free storage.  It advertises itself as "online backup and storage" as it archives your files in real-time to their remote servers.

You can browse any of your files online, using a web-browser, and you can also share these files with other users.  Memopal allows transfers of files that are larger than 1Gb, so using this service as a means to copy files to other users, because you can't send large attachments though email, make this a handy solution.

What's also amazing is that you can get 200Gb for only $49 per year.

What's not so amazing is that it doesn't provide you with desktop level access to your files so you can't use the offline storage as a real-time disk file system.  For my needs and purposes, I'm going to pass and un-install the product.

I have been a .mac (or .me) account owner since 2007 and have witnessed several upgrades to the service.  I was in the process of dumping my me.com account when Apple suddenly extended my account until June 30, 2012 in anticipation of the release of their iCloud offering.

The iDisk, as it's referred to via your mobile-me account, is slightly more than 15Gb of online storage which used to cost you $99/year.  (You got other stuff besides the storage which supposedly made it a "deal", but Apple has been forced to re-price their offerings in order to remain strategically competitive with other cloud vendors.)  The iDisk is configurable from your System Preferences menu and, if you're one of the account holders at the time Apple froze the offering, you can no longer upgrade or increase your storage capacity.  What you had is what you have until the iCloud becomes available.

What's good about this storage is that it's accessible as a mount-point (file system access as a device drive) to your system which means you an use it finder, or through any application, to access your files.  It's totally transparent as a remote device.  I also like that I have to manually mount the device to access it so there's never any background "sync" happening to slow my system down when I don't need it.

The downside is the bandwidth limitation of 200Gb of data transfer per month.  If I'm doing a lot of development, I'd imagine I could hit that pretty quickly just checking-in, and then creating and modifying the existing code base.  So I've never tried to use my iDisk for anything other than storing static documents that I don't need clogging up my physical devices.

Because of the bandwidth limitation, Mobile-Me does not satisfy my requirements.

ZumoDrive is the software I initially chose to use as my cloud-storage choice for my source-code control.  ZumoDrive is also the reason why I am writing this article, wishing I had done my due-dilligence in evaluating the software before committing (svn pun) to it.

ZumoDrive offers you 1Gb of free storage which is easily expandable to 2Gb once you complete the "belts" in their "dojo".  Cute.  Basically a test-drive through the product, training in the Dojo advances you through the belts until your max your training at black-belt and you've doubled your storage to 2Gb.

ZumoDrive is software you download and install.  It's available on all desktop and mobile platforms.  ZumoDrive mounts on your desktop as a virtual drive, which meets our purposes of remote file-system storage.  Unfortunately, ZumoDrive caches a copy of your file(s) on your local drive and then updates the remote drive when not in use.  Like, when you're playing World-of-Warcraft and need the network bandwidth because, you know, it's not already laggy enough in capital cities.

If you've followed my previous tutorial and created a TrueCrypt container on your cloud drive, then the real downside of this system makes itself readily apparent.  Uploading a 500Mb file to the server at (average speed) 77Kb/sec is going to take a LONG time.  Changing the cache options to minimize the amount of diskspace stored locally didn't impact this -- the software still sees a single, 500Mb file.

While I love the concept of ZumoDrive -- MobileMe without the bandwidth constraints -- the local caching of the TrueCrypt volume murders the concept since it's doesn't see the files within the TrueCrypt container.

SugarSync is an online cloud storage system that offers a 30-day free trial.  You can get 30Gb of online cloud storage for $5/month or $50/year.  They have a 250Gb plan for $250/year which is sort of funny when you look at what Memopal was charging: $50 for 200Gb...

I'm not going to incur another monthly charge for online storage so, to be honest, I didn't even bother to download and install the product for evaluation.

TeamDrive offers 2Gb of free cloud storage.  From reviewing the product on the website, I knew it wouldn't meet my requirements, but there were enough enticements to the feature-set that I went ahead and downloaded and installed the 100Mb file anyway.  TeamDrive is accessed through a custom-application that's finder-like in it's UX.  You can also access the application my clicking on the relevant icon in the menubar.  TeamDrive offers collaboration and synchronization as it's main features.

The UX is intuitive although window's-like.  Since TeamDrive is primarily collaboration software, it keeps track of the users who are in your team.  Although I didn't think much of the product for what I need, I was encouraged to evaluate the offering because it had a feature I'd not seen before -- the ability to create and host your own TeamDrive server.

I've been using Dropbox for over a year now.  It's my primary means of transferring files between home and work.  I also like the fact that 1Password uses it automagically to synchronize itself.  Dropbox offers 2Gb of free cloud storage that is accessible from pretty much every known appliance available on the market today.  Dropbox provides you with a file-system mount point so that you can access your files via Finder making it perfect for what I need.

The only downside, for me, is that I depend on Dropbox for file storage for other things such that my available bandwidth is only about a 500Mb.  The upgrade costs are prohibitively expensive, especially when compared with other offerings in the industry.  Sorry, Dropbox, but $200/year for 100Gb is not a value-add.

The final note, for DropBox, is that it, too, views an encrypted file container as a single file.  Dropbox's upload/synchronization speed was even worse than ZumoDrive's at 55K/sec.

I also have an Amazon Cloud account -- which is 20Gb of free storage -- because I use Amazon's MP3 cloud on my Android device.  I mention it here because of the phenomenal amount of storage that you get for FREE.  I can only access my cloud storage through the web, which is ok since all I'm storing there are my MP3 files that I've downloaded (DRM Free - hear that, Apple?) from their music service. Unfortunately, I can't access the storage from my desktop...

Other Mentions:

I looked at these products but didn't bother with the evaluation since I could read, from the product descriptions, that they would not satisfy my requirements.

idrive.com -- 5Gb basic (free) solution limited to only back-up and recovery.

syncplicity.com - web-based back-up and recovery tool.  Free, but limited to 2-computer access and 2Gb storage.

spideroak.com - pretty much web-based b&r and file sharing through the web ux.

Conclusions

What I want to do just isn't possible at the current time because of restrictions of my DSL and the way cloud services view a TrueCrypt container.  I've got a pretty good working overview of what's available and I explored a lot of solutions that were pretty damn exciting.  I also think that we're going to see the cloud marketplace evolve rapidly and those companies which are charging significant amounts per megabyte are going to have to rethink their pricing strategies or risk becoming fossil fuel.

I'm also excited by what Apple will bring to the table with iCloud -- I think that we'll be able to have file-system level access to our cloud storage but I'm also sure that the same limitations will apply for synchronization...