In Praise of Hidden Complexity

Girl in front of blackboard

How many of you have a stack of remote controls on your couch to manage your TV, cable box and stereo? Or, perhaps you have a universal control that supports 75% of the features of the native controllers.  And, which TV input goes to the old VCR for watching those dreadful kiddie videos? These are everyday examples of how complexity frustrates our daily lives.

As much as we love simplicity in our gadgets, it’s hard to achieve. The idea of a universal remote is great, but the universe of devices to emulate is just too large to fully cover. 

Likewise, there’s a conflict between openness and complexity. Do we choose the computer or phone that has a multitude of hardware and software suppliers, with the corresponding compatibility issues? Or do we choose the more monolithic system, with assured compatibility but with reduced choices and increased costs?

A More Detailed Example

The examples above were focused on our consumer experience. The cost-benefit analysis of complexity applies equally within the world of technology development. For example, have you ever thought about what happens when you click on a web link in your browser? Here’s a very short version of how the process starts:

  • Your browser will open a connection to the destination server using a transmission control protocol (TCP) socket. TCP is a means of providing an assured connection to the destination, so the web browser doesn’t have to worry about resending lost packets.
  • TCP will formulate an internet protocol (IP) packet that is bound for the destination, asking to open a new connection.
  • IP needs to convert the name of the server (e.g. praysonpate.com) into an IP address (72.167.191.69). To do this, it contacts a domain name service (DNS) server to ask for a lookup. The DNS system is worthy of its own exploration in terms of hidden complexity.
  • To reach the DNS server, the media access control (MAC) layer will ask the network for the MAC address for the DNS server, using either wired or wireless access.
  • The physical (PHY) layer will now send the packet to the DNS server, again using the appropriate access.
  • And so on …

As we see, we have already gone through several layers of protocols just to find out where to send the web request. Why are there so many steps and layers? The answer is that having layers lets us divide up the work and prevent each layer from knowing or caring about the layers above or below. For example, only the PHY layer in the last step had to worry about whether the connection was wired or wireless. That simplifies all of the layers above. Likewise, the use of TCP means that the web browser sends a request and knows that it will either get a valid response or no response. That also simplifies how that application is built.

This type of hidden complexity is common. Your smartphone is far more complicated and powerful than the desktop computers of 20 years ago. Why? Because having that complexity enables desirable features such as downloadable apps that replace a pocketful of appliances.

The Cloud Isn’t Simple – It Just Looks That Way

Another example is the ubiquitous cloud. While the cloud may seem simple when it comes to storing our pictures or running our apps, there’s a ton of hidden machinery running behind the scenes. Modern data centers are filled with thousands of servers in racks, connected by layers of high-speed switches, and powered and cooled by sophisticated systems. Furthermore, you will be sharing a server with others and not even know it. Those in the cloud business have taken on the task of embracing that complexity to provide powerful and cost-effective services, while hiding the complexity from the users.

A similar challenge awaits telco operator who want to use network functions virtualization and software-defined networking to bring the benefits of the cloud to their networks. Embracing these technologies means a lot of re-thinking and re-organizing, which will be painful. The benefit will be the ability to deliver innovative services much more quickly and at a lower cost. It will look easy from the outside, but there will be a lot of hidden complexity inside.

Keep It Simple, Stupid (At Least On The Outside)

As we see from the examples above, complexity is sometimes a by-product of innovation and choice. By embracing the latest and greatest, we sometimes get the added requirement of a more complicated system. That’s OK, as long as we keep the complexity under the hood and make it dead simple on the outside for the user.