Over the last month, I’ve read and listened to people’s description of what Serverless means to them. My conclusion: There’s no single definition of whether something is Serverless or not. Instead, it’s a set of characteristics allowing you to focus on the business logic of your application, rather than the heavy-lifting of managing infrastructure. Any particular service falls somewhere on the Serverless spectrum, based on how well it exhibits these characteristics.

Wait? Serverless is Mainstream?

Yes, it’s definitely moving in that direction. It’s no longer viewed as an exclusive cutting-edge way of developing applications, with fresh-off-the-press videos showing how little effort is required to bootstrap and execute code, all for the low-low price of $0.00001. It still does those things, but many developers now see it as a “tool in their tool belt” to save time and money as they focus on building their application.

To illustrate the maturity path, let’s draw a comparison to Object-Oriented Programming (OOP).

When I was an undergraduate student in the 1990s, we learned Pascal and C as our two main programming languages. We also heard about Smalltalk (from 1972), with objects and messages, but never had a chance to write a program. It all seemed very abstract and I honestly didn’t understand the point of objects. It wasn’t until I wrote a bank teller simulation using the Simula language (from 1962) that it felt natural, and started to make sense.

As a graduate student (late 1990s), I was a regular at the OOPSLA conference, surrounded by academics and industry practitioners who were ahead of the game in Object Oriented Programming and Design. The exhibit halls were flooded with vendors selling their latest tool or technique. I brushed shoulders with people who are now legends, and saw the birth of Java and UML. It was all ground-breaking at the time, but not yet mainstream.

Fast forward to 2024 - The university students I interact with use TypeScript, Python, or Java, each with objects baked into the standard language. The students learn OO programming as the “normal” way of doing things, as if there was never any other way! They take these expectations into industry, and enthusiastically work in OO style. These days, the OOPSLA conference has rebranded into a much smaller format, mostly attended by academics. OOP has become mainstream - the normal way of doing things, rather than something special and buzz-worthy.

Serverless is not as far along the maturity curve as OOP, but it’s heading in the correct direction. It’s not a passing fad that’ll be gone in a few years. There are plenty of Serverless conferences generating buzz, but students are learning to deploy containers or functions-as-a-service to the cloud, without the cognitive burden of SSHing to a Linux host. They’re saving money by paying per request, rather than by instance-hour. It’s becoming a normal and desirable way to operate.

But, back to the topic - what exactly is Serverless? Or, more to the point, what are the characteristics of the services that are referred to as Serverless?

What Do the Experts Say?

The obvious place to start is by looking at what the industry or academic experts are saying. Here are some quotes:

From Wikipedia’s Serverless Computing topic…

Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers. "Serverless" is a misnomer in the sense that servers are still used by cloud service providers to execute code for developers. However, developers of serverless applications are not concerned with capacity planning, configuration, management, maintenance, fault tolerance, or scaling of containers, virtual machines, or physical servers. When an app is not in use, there are no computing resources allocated to the app. Pricing is based on the actual amount of resources consumed by an application.

From the AWS Serverless page…

AWS offers technologies for running code, managing data, and integrating applications, all without managing servers. Serverless technologies feature automatic scaling, built-in high availability, and a pay-for-use billing model to increase agility and optimize costs. These technologies also eliminate infrastructure management tasks like capacity provisioning and patching, so you can focus on writing code that serves your customers.

From Pluralsight…

Serverless is one of those terms that’s used for all sorts of stuff — some legitimate and some marketing fluff. But, in essence, serverless is a design approach that lets you build and run entire applications without having to directly manage servers. 

From a UC Berkeley research paper…

Serverless cloud computing handles virtually all the system administration operations needed to make it easier for programmers to use the cloud. It provides an interface that greatly simplifies cloud programming, and represents an evolution that parallels the transition from assembly language to high-level programming languages

In summary, the goal of Serverless is to allow developers to focus on their application’s business logic, rather than on undifferentiated heavy lifting. Although each service does this differently, I’ve noticed some common characteristics:

• Avoiding the need for command-shell access (the administration is done for you).

• Avoiding the need to size/scale servers, or to be aware of them at all.

• Paying only when constructive work is being performed.

• Thinking about upgrading software

Let’s discuss in more detail.

Command Shell Access

With traditional infrastructure (cloud-based or on-premise), administrators have command-shell access to their servers, typically via SSH or RDP. This is a basic need when you’re responsible for security, upgrades, backups, or other server-side tasks. In contrast, the promise of Serverless is for cloud providers to handle this mundane work for you.

There’s clearly a benefit of having somebody else do the admin tasks, but for many developers this alleviates the need to learn how it’s done in the first place. Most application developers I’ve met prefer to focus their time and skills on developing business logic, without the distraction of skilling-up on IT responsibilities.

Which AWS services fit in this category? Clearly Amazon EC2 is not Serverless because the first thing you do is log into the command line or console. Container services, such as Amazon ECS are closer to being Serverless if you use ECS on AWS Fargate, but if you run ECS containers on EC2 you still have access to the underlying servers. Even with Fargate, it’s possible to run a command shell inside a container, similar to running docker exec on your local machine.

Perhaps the most famous Serverless offering from Amazon is AWS Lambda, providing time-limited general-purpose compute (running on Python, JavaScript, Java etc), without requiring access to the underlying servers. Instead, the developer selects a run-time environment (such as NodeJS), then uploads their code to the service. The service manages the complexity.

Looking further into the list of Serverless offerings, there are many services where command-shell access is not even possible, such as Amazon SQS, Amazon SNS, AWS Step Functions, Amazon EventBridge, and Amazon DynamoDB. All of these services have very specific purposes, but none of them provide access to general-purpose compute, nor a command shell.

Interestingly, we might question whether Amazon RDS Postgres meets this requirement, given that users are not permitted command-shell access to the underlying servers. This is true, but there are other reasons RDS doesn’t fit the Serverless model.

Awareness of Servers and Their Size

Another expectation of Serverless is that you don’t think about servers. That is, you don’t need to worry how large the servers are (RAM and CPU), how much disk space is attached, or how many servers you’ve been allocated. All of those details are managed by the cloud provider.

This is where RDS Postgres immediately fails the Serverless test, since the first thing you do is specify server size. You then worry if you have large enough servers, or whether you’ve over-provisioned and are paying too much. An ideal Serverless environment takes away these concerns, with Amazon DynamoDB going a long way in this direction (you just create a table, and don’t worry about how it’s stored/accessed).

Services such as AWS Lambda or AWS Fargate eliminate the need to worry about server sizing, as they automatically find a suitable server for your workload, allocating new servers whenever necessary to scale. However, at the function/container level, both Lambda and Fargate require you to specify the RAM and CPU needed for running your software. Tweaking these parameters can be time-consuming, given that RAM vs CPU trade-off significantly impacts software performance.

The other Serverless offerings mentioned earlier (SQS, SNS, Step Functions, EventBridge, DynamoDB) fit nicely into this model, since you never think about quantity or size of servers, nor about how much memory is allocated to your software. Of course, these services don’t provide general-purpose compute, which is how they differ from Lambda.

In terms of developer burden, a couple of edge cases are worth mentioning. First, a Lambda function can be impacted by cold-start latency where the service takes slightly longer than normal to scale the underlying server fleet on your behalf. Second, all services have limits/quotas to stop customers scaling up more than anticipated. Both these important scenarios place a burden on developers to be aware that servers are a real thing.

Elasticity and Pay-Per-Use

A significant benefit of Serverless technology, and of cloud computing in general, is paying for what you use. Capacity can be allocated and deallocated almost instantly, and you only pay while those resources are active. This is in stark contrast to the weeks (or months) required to install new equipment in your own data centre, paying the bulk of the cost before the hardware is even plugged in.

The interesting difference with Serverless is that you pay per request (or message/event, or state transition), rather than pre-allocating the resources and paying for them, even if they’re idle. For example, allocating an EC2 instance involves paying for every second the instance is dedicated for your use, even if the server isn’t doing meaningful work. In contrast, Lambda functions only execute when there’s meaningful work to be done, then immediately terminate rather than being idle. There are no charges for Lambda when there’s no work to do.

Anything in the Serverless world should therefore “scale to zero” and not charge you anything when idle. With this in mind, AWS Lambda, Step Functions, EventBridge, SNS, and SQS all fit into this model.

However, it’s debatable whether Amazon DynamoDB or AWS Fargate meet this requirement. For DynamoDB, you’ll always be charged for data storage, even in the absence of read/write requests. For Fargate, you can quickly scale up/down the number of instances, but you’ll still need to pay for idle instances that aren’t currently processing requests.

Upgrades

Although it’s not often discussed in the literature, there’s a cost of thinking about software upgrades. Serverless alleviates the need for upgrading or patching the underlying operating system, but that’s not always true for the run-time environment.

For example, with AWS Fargate, you’re still responsible for managing the Dockerfile, describing which libraries, frameworks, or applications run inside the container. If third-party software is used, there’s ongoing effort to upgrade packages to adopt new features, avoid EOLed versions, or to fix security vulnerabilities.

For AWS Lambda, there’s less effort involved, but with new versions of the runtime environment (e.g. NodeJS) being available every year, there’s ongoing effort to perform software upgrades.

On the other hand, services such as Step Functions, EventBridge, SQS and SNS maintain backward compatibility, not requiring the code or configuration to be updated on a periodic basis.

Event-Driven Architectures

One perspective I’ve seen is that Serverless is closely tied to Event-Driven Architectures (EDA). That is, your software consists of multiple collaborating services, each generating events of their own, and/or reacting to events from other services. In this model, there’s a clear benefit from only paying when there are events to process, or scaling-to-zero when the service is idle.

But, it wouldn’t be fair to say that EDA implies Serverless, or that Serverless implies EDA. In fact, if you have a predictable workload passing through your system, and you rarely scale-to-zero, it can be more efficient to use pre-allocated containers or VMs. In this approach, each service processes a high number of concurrent requests within the same container/VM, rather than starting up a new Function-as-a-Service environment for each request.

Serverless Compute

Another interesting perspective is that Serverless implies general-purpose compute, such as AWS Lambda, where you can run any type of code written in Java, JavaScript, Python, Golang, Rust, etc. In contrast, services such as DynamoDB, EventBridge, SNS or SQS don’t fit this definition of Serverless, as they have very specific modes of operation (fetching/storing data records, or propagating events).

I haven’t discovered the source of this thinking, but it’s likely because databases and message brokers were traditionally managed by dedicated teams (DBAs, or IT), rather than by application developers. From that perspective, we never thought much about servers in the first place.

And My Point Was?

After reading a lot of opinion, I learned that Serverless is not black or white… instead, it’s a colourful spectrum of characteristics making a service easier to use. By eliminating the undifferentiated heavy lifting, developers can focus more on building out their applications.

I’m not the first person to have these thoughts, and many have done a great job of documenting this topic. I encourage you to read/view:

• Mike Roberts / Thoughtworks - A detailed perspective of all the characteristics of Serverless, including the history, the pros, the cons, and forward-looking trends. A long document to read, but it’s worth it.

• Jeremy Daly - A fun presentation to watch, referring to a bunch of other experts who have equally interesting opinions.

• Ben Kehoe - Describes many of the edge cases of the Serverless definition, and includes the idea of it being a spectrum.

Regardless of what Serverless really means, and whether we’re using the word correctly, the characteristics we’ve discussed are super useful, especially if you’d rather focus on business logic. For this reason, I feel the characteristics of Serverless have a bright future, no matter what they’re called.

I’ve recently been intrigued by the dynamics within and between software development communities, and particularly the mindset (mental models and attitudes) presented by each. If you find yourself needing to collaborate with a software development community different from your own, then understanding their culture is important.

This could mean working with a different team inside your own company, contributing code to an open source project, or perhaps selling a tool, library, or service for the community to use. You shouldn’t assume your approach to developing software will be looked upon favourably by other communities.

In this blog post, I’ll provide my definition of “Software Development Community”, explain why it’s important to understand communities (not just your own) and will describe four of the communities I’ve been exposed to in recent years.

What is a Software Development Community?

Back in the good-old-days (sometime before the 2000s) it was common to develop applications from scratch. You either wrote the source code yourself, or searched various FTP sites or Usenet groups for “free” software to incorporate. It was painful to construct a full application, causing you to implement the same code over and over again for each new project.

The 1990s saw the introduction of downloadable language runtimes and compilers, package repositories providing reusable code libraries, and frameworks with pre-architected skeletons for a new application. All of these worked together to make application development far more efficient than ever before.

To give examples, we now have:

• The Java language, with the Maven package repository, and frameworks such as Spring and Akka.

• The JavaScript language, with the NPM package repository, with backend runtime environments such as NodeJS with the Express framework, or front-end frameworks such as React or Angular, running in modern web browsers.

• The Python language, with the PyPi package repository, using the pandas library to perform data science operations, or PyTorch for machine learning, or Django for web applications, or many other libraries for many other purposes (Python is very popular).

• The Ruby language, using the RubyGems repository, and the Rails framework for developing web-based applications.

• Using a range of languages, the Serviceful community creates applications using a collection of pre-built services, typically running on a cloud provider such as Amazon Web Services (AWS).

• And the list goes on…

Each of these development environments has been successful enough that communities formed around them. Senior members of the community contributed content (libraries, documentation, training videos) attracting more consumers, resulting in an ever-growing community. Rather than building applications from scratch, new community members could build upon the frameworks and libraries that were already proven effective.

The Community Mindset

What’s interesting about communities is they share a common mindset, a set of attitudes or beliefs about how software should be developed. This is because communities shape their members toward a common set of values, but also those communities attract similar people in the first place.

For example, a common agreement in the Ruby-on-Rails community is that developing applications should be fun and easy. Therefore, the Rails framework and the corresponding libraries (known as Ruby Gems) are designed to support minimal configuration and therefore “just work”. As a result, it’s possible to construct very impressive Ruby-on-Rails applications with very little code, in a short amount of time.

In the JavaScript community, using NodeJS, all I/O calls are expected to be asynchronous in nature, not allowing the main application thread to block. Any library function using blocking code is frowned upon by the community as it negatively impacts system performance. In contrast, the Ruby-on-Rails community avoids this asynchronous approach, due to the excessive complexity of reasoning about program flow.

Finally, the Serviceful community is comfortable using Infrastructure as Code (such as AWS CloudFormation) to deploy and stitch together services, which is something the Ruby and Rails and NodeJS communities find unnecessarily low-level, therefore shying away from.

Despite their benefits, the downside of communities is that members are at risk of developing a single-minded approach, believing there’s one preferred way to solve a problem. A particular library will be recommended, a specific coding style must be used, or that “everybody does it this way”. This may be true within their community, using their agreed-upon set of values to make a decision, but it won’t always make sense across community lines.

I find it’s rare for developers to be deeply involved in more than one community. It can take years to become an expert in a programming language, a set of libraries, a collection of tools, or a framework, so there’s usually no time to stay actively involved with multiple communities. Not to mention, monitoring the community’s blog posts and YouTube videos requires significant effort. Focusing on one language or framework is therefore common.

Most developers have a community they feel very comfortable with, but will dabble lightly in other languages or frameworks when necessary to solve a problem. While branching out, it’ll take them longer to learn the language syntax, which tool to use, or the specific functionality of the framework they’re now working with. Overall development will be slower, but if motivated enough they’ll eventually finish the task.

Why You Should Care

Simply put, it’s easier to work alongside a community if you take time to understand their values, and the historical context around why they prefer certain approaches.

This can take many forms - you may be collaborating with an arms-length team within your own company, or need to dabble in a code base written in a different language, or perhaps you’re interviewing a job candidate from another community. You might even be a software vendor selling a tool or service into a new market.

Here are some differences to keep in mind:

• Different communities use different words for the same concepts, so being an outsider can be confusing. That is, until you take five minutes to read the documentation, only to realize you’re familiar with the concept but by a different name! Of course, you must also start using those words when interacting with the other community. Is the correct word stream, or channel, or iterator, or …?

• Along those same lines, the frameworks and libraries each have their specific set of APIs, which are similar to those of other frameworks. For example, every framework has a “read” function, but I can never remember the exact name of the function, or the exact parameters. I find my IDE’s auto-suggest functionality to be super helpful for prompting me, but bookmarking the online API documentation is also a must.

• Every development community has an opinion about “speed of execution” versus “speed of writing code”, based on the application they’re building. A C/C++, or Rust programmer spends effort on efficient use of CPU and RAM, with the goal of faster execution. In contrast, Ruby-on-Rails developers want to minimize the time to develop a new feature, being less concerned about execution performance. Both approaches are perfectly valid, and depend on their end customer’s needs.

• Similarly, the style of coding will differ between communities, depending on domain of the application being constructed. Systems-level programming (C/C++, Rust, or sometimes Java) can involve the implementation of complex algorithms using advanced computer science concepts designed to achieve performance and reliability. In contrast, a data scientist using Python, or a business application developer, using Ruby-on-Rails, will use a more domain-focused programming style to meet their customer’s needs.

• Types or no types? This has been an ongoing argument for many years, with different communities preferring different styles. It can be faster to author code if you don’t define types for your variables and don’t need to worry about type mismatch errors. In contrast, many argue that type annotations make their program more solid and less likely to fail, which in the end speeds up development. With types being added to Python and TypeScript (on top of JavaScript), the communities are gradually converging.

• Distributed system versus monoliths? It can be argued that distributed systems (multiple programs communicating via a network) are important when scaling your software. But, it’s equally argued that developing in a monolithic style (all code in a single program image) is far easier to comprehend, and easy to debug. Both groups are correct, but expecting the monolithic community to use a message queue or a distributed workflow service will be challenging.

• Finally, each community has their preferred blogs and video channels where they share knowledge. Unfortunately, there’s a risk of becoming insular because recommendation engines are good at showing you similar content to what you’ve already viewed. It takes a deliberate effort to search for content produced by a different community.

To close off the discussion, let’s finish with a quick survey of some of the communities I’ve been exposed to in recent years. I won’t claim to be an expert in any one of these areas, but I’ve done plenty of dabbling, and have encountered more than my fair share of friction in doing so.

Example: The NodeJS Community

NodeJS is a server-side runtime environment for executing JavaScript code, working uniformly across Linux, MacOS, and Windows. Besides sharing a common language with web browsers, NodeJS provides additional functionality for file system access, network access, and many other traditional server-side features. Although general purpose in nature, NodeJS is commonly used to build light-weight web application servers, using frameworks such as Express.

NodeJS is the most popular framework, according to the 2023 Stack Overflow survey. It can be used for long-running web application backends, text-based command-line tools, or short-lived functions that execute in a Function-as-a-Service environment, such as AWS Lambda. Due to this flexibility, and the prominence of JavaScript as a web-development language, it’s no surprise NodeJS has a large community following.

NodeJS provides a single-thread programming model, which seems counter-intuitive for performance-critical applications. However, due to the Promise-based programming model, NodeJS does an effective job of time-slicing a large amount of work onto that single thread, keeping the CPU busy. This is important to the community members who care about performance.

A “Promise” is a chunk of code to be executed some time in the future, whenever the CPU is available to do the work. They are typically chained together in a sequence, and can be dependent on the completion of other Promises, or on the completion of asynchronous I/O. In contrast, many other programming frameworks block the thread while an synchronous I/O operation is in progress. For NodeJS, we instead schedule a Promise to be executed once the asynchronous I/O is complete, without blocking the overall thread of execution. Experience shows this is a high-performance and low-latency approach to performing work.

Like other communities, NodeJS users share pre-written software libraries, tools, or frameworks in a repository. The NPM Repository contains a large number of packages that are easily downloaded and imported into a user’s application. As you might expect, packages that gain a positive reputation within the community become more popular and are typically held to a higher standard of quality than less popular packages.

In addition to using NPM packages, NodeJS developers are comfortable writing large amounts of custom code. However, due to the somewhat unstructured history of JavaScript (at least until 2015), there’s a wide range of programming styles in effect. A newer object-oriented syntax (with class and method definitions) is available for fans of object-oriented programming, but it’s also possible to write code in a functional style (data is passed as arguments and returned as a result). Even though it’s harder to debug, use of global variables is still commonplace in a lot of software.

There’s no disputing that the NodeJS community is very strong, for a variety of reasons. Sharing the JavaScript language and package repository with frontend (web browser) code is a key advantage. From an efficiency point of view, the use of Promises and the light-weight runtime environment is a key benefit for faster development cycles. This community will likely be active for a long time.

Example: The Ruby on Rails Community

Ruby on Rails (2004) is the web application framework which made the Ruby programming language (1995) famous. You can create a complete web application in minutes by generating a templated application, then incrementally adding new features with just a few commands. The whole premise is that development should be fun and easy, allowing you to focus on your business logic rather than the underlying plumbing (which tends to be the same for every web application).

Rails follows a “convention over configuration” approach where a pre-defined set of standards (such as naming conventions) allows the parts of your application to work together seamlessly. You are welcome to go against these conventions, but doing so makes your coding effort more challenging and could break parts of the framework in unexpected ways. It’s therefore common wisdom to always follow the standards.

For example, if you create a User class as a subclass of the ApplicationRecord class, Rails creates an underlying SQL database table named users, with all the necessary columns. Querying that table is as simple as writing User.where(name: 'David') which invokes the relevant SQL command under the hood.

Overall, the Rails architecture is prescribed for you. Using the Model-View-Controller pattern, developers add their code into the correct classes (again, with naming conventions), allowing the framework to know exactly where to find the relevant code. Once a Rails developer has learned these conventions, navigating the code base is highly efficient (although it can be very challenging for outsiders!)

The Rails philosophy is to write as little code as possible. Developers prefer to solve problems by downloading Ruby Gems (packages), which ideally should just work without much configuration. A one-line solution to a problem (which invokes a Gem) is preferable to a 10-line custom-built solution. Writing a long Ruby method (> 10 lines) is often frowned upon.

The Ruby language is very dynamic, both with its variable type system and with how it allows dynamic interpretation of code. This flexibility allows very powerful programming constructs, such as DSLs (Domain Specific Languages) to be added into a Ruby-based application. On the downside, it’s nearly impossible to know that a Ruby program won’t have any syntax errors, until the code is actually executed. Unit testing has become a vital part of any Ruby on Rails application, to find errors at development time.

Overall, Ruby on Rails is one of the less common frameworks (see the 2023 Stack Overflow survey), but does have a devoted user base who enjoy the flexibility of the Ruby language and the conciseness of adding new functionality to an application. The community’s mindset is has therefore developed around these qualities.

Example: The Serviceful Community

The concept of Serviceful is relatively new, but is loosely-defined as being a programming model connecting together an array of existing services, often running in the cloud (such as AWS). A Serviceful program invokes service APIs, orchestrates the flow of data from one service to another, and handles error conditions that may arise. The community has developed around the tools and patterns used to build Serviceful applications.

There are many characteristics of Serviceful development, including:

1. Existing services provide functionality that we don’t need to implement ourselves. These services are easy to use when managed by a cloud provider.

2. Cloud-based services can scale in an elastic way. That is, you only pay for what you use, and can scale rapidly if demand for that service suddenly increases.

3. You’ll still need to write some custom code, but it’s often quite short. The paradigm of Event Driven Development allows small functions (such as using AWS Lambda) to execute in response to events taking place (a file is uploaded, an HTTP request is received, a timer expires).

4. Communication between services is done via creating and dispatching network messages (e.g. HTTP), rather than calling functions within the same (monolithic) program. This introduces the chance of partial failure if the remote service is unresponsive.

5. Existing application frameworks (such as NodeJS or Ruby and Rails) still have a place in this model, but are considered to just be one of the many services in the overall application (aka “microservices”). Communication with these services is usually done via the HTTP protocol.
   

Much of the time, the Serviceful community is referred to as the Serverless community. Although there are similarities, we shouldn’t exclude services that require knowledge of the underlying servers. Instead, a better comparison would be the Distributed Systems community, which focuses on communication between services over a computer network. The key observation is that many of the community’s challenges (distributed programming, network protocols, resilience) have existed for many decades.

In the Serviceful community, it’s normal to include Infrastructure as Code, such as CloudFormation or CDK, as part of the application. For example, creating and deploying code to AWS Lambda functions, provisioning databases, defining message queues, and specifying event routing rules are an important part of application development. Members of the Serviceful community must possess a strong understanding of cloud concepts, such as services, resources types, and permission rules.

In many companies, the Serviceful community is a partner to other communities, such as NodeJS or Ruby on Rails. That is, the Serviceful community are the members of the “Platform Team” who focus on the cloud infrastructure. Meanwhile, the NodeJS or Ruby on Rails community focus on the business applications that execute on the cloud-based servers, or within the Docker containers or Lambda functions. This separation of responsibility is quite common, especially in larger organizations.

Example: The Data Science Community

The final group I’ll mention is the Data Science community. These developers focus on the hidden meaning buried in large amounts of data. With the internet filling up with more raw data every second, there’s a lot to analyze!

Data Scientists have a statistics or mathematics background, rather than a computer science or engineering education. Their community is focused on languages (such as Python, R, or SQL) that support querying and summarization of data. Their code reads data into tables (rows and columns), performs analysis of the data, then outputs a summary of the key findings.

Common operations for data scientists include translating data from one format to another (such JSON to tabular format), cleaning the data by removing outliers and invalid data cells, querying tabular data to find trends, and joining data from different sources. Many of these operations are performed interactively using command line scripts, or Jupyter Notebooks that allow visual representation of the results.

Members of the Data Science community focus their skills on writing scripts to perform data transformation, often running the scripts on small datasets on their local machine (desktop or laptop). The community is large, and a variety of data manipulation packages have been written and shared widely. One of the most well-known Python libraries is pandas, providing a set of functionality for manipulating tabular data.

Given their statistics background, Data Scientists often rely on Data Engineers to put their code “in production”, using their computer science skill set. That is, instead of running their scripts on their local desktop, a Data Engineer has the skills to automate running the script on a large amounts of data, often using a compute cluster (such as Spark, or AWS Step Functions Distributed Map).

As you can see, the approach and mindset of a data scientist is quite different from those in the other communities we’ve discussed.

Conclusion

The takeaway message is that software development communities exist as a mechanism for sharing tools, libraries, frameworks, and documentation, to help build software applications. They form a common mindset by which the community members can make choices, all with the goal of building software more efficiently.

If we wish to interact with another community (other than our own), it’s important to understand the day to day life of a developer in that community, and the values by which they operate.

Receiving appreciation is a basic human need, with software engineers being no exception. Whether it be promotion, a pay increase, an award, or just a simple “Thank you!”, our self esteem needs validation. When properly thanked, we’re capable of pushing ourselves harder, achieving more, and enjoying work for longer. In contrast, lack of appreciation leads to frustration, resentment, and is one of the main reasons people leave their jobs.

Over the years, I’ve noticed situations where the “Thank you!” felt missing, or was attributed to the wrong person. I want to share some scenarios to explain why this happens, identify who received the praise, then consider who else should have been thanked.

Customer-Facing versus Behind the Scenes

In the software world, there’s work that’s visible to customers, and there’s work that’s “behind the scenes”. The customer-facing work includes frontend UI construction, as well as any backend feature development visible via the UI. In contrast, the behind-the-scenes work involves platform development, security, performance, latency, and system internals, things that are either hidden from the customer, or are less visible.

Both types of development are important, but the customer-facing work is easier to see, easier to understand, and easier to praise. Your colleagues in sales, marketing, developer advocacy, corporate leadership, and customer support, tend to have outgoing personalities and show excitement for each new feature. It’s easy to feel appreciated when so many are discussing your work.

As an example, I recall a UI developer demoing how they moved a button to a more prominent location on the screen. A simple code change, but the impact to the customer was significant, and the cries of “Ship It!” were free-flowing. On that same day, another developer describing their optimization of Docker images was met with polite applause, but an obvious lack of understanding or appreciation from many in the room. The developer clearly noticed.

If you see this in your organization, consider raising the profile of the behind-the-scenes work, describing it in terms important to customers. For example:

• When significantly refactoring the software, publicize how the work increases velocity of new feature development.

• When an improvement is made to the software’s reliability, advertise the expected decrease in system outages (e.g. we’ll achieve 99.99% uptime).

• Likewise, an improvement in performance provides a less sluggish application.

• Finally, an improvement in security directly impacts the software’s compliance report, as well as the customer’s peace of mind.

In other words, identify why people should be thankful for the work that was done. All developers deserve thanks for their work, not just those building highly-visible features.

Being Reactive versus Being Proactive

Escalations are common in the software industry, ranging from individual customer complaints through to complete system outages. No matter what, it’s important to identify the severity of the problem and resolve the issue accordingly. When a developer fixes a problem, they’ll surely be hoping for praise.

Be careful though - praise for reactive behaviour can give the wrong message. For example, John and Mary stay after work to resolve a system failure, working long hours into the night to address the problem. In the morning, customers are happy again and management is appreciative of John and Mary’s work. Should they receive thanks? Of course they should!

But, what about a different scenario where Frank works long hours to test his code, addressing review feedback from his peers. He believes in releasing quality code, putting in effort to ensure positive customer experience. For six whole months, he goes home at the end of the day with no outages, and no customer complaints. Should Frank receive thanks? He should, but often he doesn’t hear a thing!

To make sure praise is given where it’s due, you should always track metrics on the number of problems encountered. Although you should praise those who fight fires, much more appreciation is due to those avoiding fires in the first place (assuming they’re still able to deliver customer value!)

Individual Contributor vs Leader

As a software engineering leader, an important part of your role is to thank the members of your team. They look to you for validation of their effort, often going out of their way to please you, with praise being heavily sought after. It’s important for you to recognize good work, both from individuals as well as the team.

But what if you’re the leader? Who thanks you? You’ve put in countless hours of hard work, running meetings, scheduling the team’s activities, dealing with escalations, supporting low-performing team members, and even reviewing and writing code. Unfortunately, praise for your work can be hard to find.

Should the members of the team be thanking you? I’d like to think so, but the power dynamics makes it hard for individual contributors to feel comfortable praising their leader. After a while, they become accustomed to you performing those tasks, and don’t see it worthy of thanks. Ironically, you’re more likely to receive thanks if you do their work for them, such as finding the source of a complex bug.

Alternatively, should your own manager make an effort to recognize your work? I’d like to think so, but they’re just a single person who’s involved in numerous projects. They might forget to thank you, or in order to avoid bias they’ll thank the whole team, rather than thanking you personally. Unfortunately, that’s not the same feeling.

Small Tasks versus Large Actions

In my experience, it’s more common to receive a “Thank You” if you interact with somebody face-to-face to help with an immediate problem. For example, it’s easy to get praise if you spend time explaining a difficult concept, or if you pair with somebody to fix an annoying bug. It’s minimal effort to thank somebody, so it happens more often.

The challenge lies when your work is longer-term in nature, and people are at arms-length. You might be a leader, or might achieve great things for your company or community, but you rarely hear words of appreciation.

In one example, I once received company-wide praise for helping arrange chairs for an evening meet-up. The task required 15 minutes of my time, but was apparently worthy of a call-out to the whole company. This seemed odd to me, as somebody who spearheaded meet-ups for the past 10 years, spending 100s of hours to organize events. I had noticed the distinct lack of praise for my ongoing leadership, but why was that 15 minutes of chair moving worthy of mention?

In reality, I had received praise for my longer-term actions, but it was “behind my back”. From time to time, I’d overhear mention that I was a well-known leader in the community. I also came across email threads stating “Peter is the organizer of…”. Even though people were not giving thanks to my face, I’d instead been developing a reputation in the community. In the end (after 10 years), I did receive a significant community award, so the thanks eventually arrived.

Demo Presenter versus Innovator

If you’re somebody who likes to innovate, you may have seen the following scenario play out. Imagine you think of a great idea to solve a challenging customer problem, then you explain the solution to your colleague, William, to get his feedback. You draw pictures on a whiteboard, and spend time convincing William it’s worth implementing. William agrees to build a proof of concept, demonstrating the value of the idea.

A few months later, you start hearing about William’s great new idea! Apparently he gave a demo to senior staff members who loved William’s proposal. William is now the hero for solving a challenging problem, and is receiving lots of recognition for doing so. How does this make you feel? It was your idea!

At this point, you might be tempted to say “But it was my idea in the first place!”. In reality, William likely modified your idea and improved upon it, so now it’s unclear who actually deserves the credit. At the very least, you should share the praise.

The solution comes in the form of a document. Rather than white-boarding the idea, with somebody else building the prototype, you should personally write a two-pager document that explains the vision. This document should be quick to write, and can be refined over time, but there’ll be no doubt you had the original idea.

Short-Term Delivery versus Long-term Strategy

Along similar lines to the previous problem. You may be the innovator of a project that ends up being a long-term venture for your company. Imagine you spent six months implementing a prototype, gaining buy-in from leadership, campaigning for budget, and putting together the project plan. Your great idea is finally in motion!

The project ends up taking 18 months to build, with a team of ten developers. They work very hard to productize the idea, building a solid solution to delight customers. You trust the team to do a good job, so you switch your focus to the next big project on the roadmap. You don’t need to be watching over their work.

Of course, when the project is eventually delivered, there’ll be plenty of praise for the whole team. But wait! The first six months of work was entirely your effort, so shouldn’t you receive 25% of the praise? The reality of being an innovator is that it takes a whole team to turn your idea into a viable product, and to support the software in the longer term. That’s not something you could do by yourself. As an innovator, you must share the praise, and also be comfortable with delayed gratification - possibly years later.

At the Project End versus at Each Milestone

A large boost to a software team’s morale is when you find reasons to celebrate. This is easier when you release software on a regular cadence, providing that all-important praise from customers. However, many projects take multiple years to complete, leaving long periods of time with no visible reward.

As any good project manager will tell you, it’s important to break a large project into achievable milestones. This helps with resource allocation and budgeting, but also provides a reason to thank people, and to celebrate. Even if you’ve only got half the work done, that’s a milestone to celebrate.

One of my favourite memories of being a TPM in the IP networking industry was the “ping party” we held when our newly-built network devices first sent a “ping” message across the network. We would always celebrate with ice cream (in California it’s always time for ice-cream!). There would still be several months of development before the product was fully tested and ready for release, but reaching the intermediate milestone was a good excuse to stop, celebrate success, and to thank people!

Conclusion

If you’re working on a software project, and you don’t feel appreciated, I encourage you to think about why. What’s missing? Do you feel ignored or resentful? Is somebody else being praised for the hard work you’re doing, or perhaps their work is less complex but more visible? All are common reasons for feeling unappreciated in the work place.

I won’t claim to have a perfect solution, but I do find that identifying the root cause is the first step to feeling better about the situation. You might be able to change the way you work - to increase the visibility of your contribution, or to raise awareness of how important your work has been to the organization.

Thanks for reading!

The PE role isn’t well defined across the industry, with each company having their own definition of when somebody is ready for promotion. In smaller companies, you likely gained the role by being the most productive developer on the team. In larger companies (such as the FAANG companies) there are strict guidelines on promotion readiness, including leadership qualities, coding skill, breadth and depth of experience, scope of your impact, and the ability to think strategically.

I doubt my observations will appear on a formal job description, but they are early signs of readiness for the role. Specifically:

1. Does the engineer say “we” to mean their team, or the whole organization?

2. Does the engineer raise problems, or do they solve problems?

Let’s see some examples of each…

1. Saying “we” to mean the team, or the organization

As a Principal Engineer, you don’t belong to a single team (of 6-8 developers), but will instead oversee many teams - perhaps an entire organization (100+ people) with multiple managers, directors, and a VP. You must always think about the organization as a whole, without excluding any of the teams, and without “taking sides”.

Therefore… when I hear a Senior Engineer say the following things, I know they’re not ready for the PE role.

• We (on my team) have adopted this approach. You (on your team) might want to try it.

• We made a code change on the backend side, and you’ll need to make a similar change on the frontend.

• We did the work on the JavaScript side, but your team can port it to Kotlin if they want to.

These innocent statements tell me the engineer thinks in terms of we and you, rather than thinking on behalf of the organization as a whole. There’s nothing wrong with team pride, and ownership over what the team builds, but it’s not a good approach for the Principal Engineer to only think from one perspective.

Instead, they should redefine “we” to be the organization as a whole, and to think on behalf of all teams, not just one. For example:

• The platform team has seen good results with this approach, and we should have other teams use it too.

• We made the code change on the backend, and we’ll also need to make a similar change on the frontend.

• We did the necessary work in JavaScript, but we could also make use of a Kotlin implementation.

It’s a subtle difference in wording, but is a good indication the engineer is thinking on behalf of the whole organization, not just a single team.

Well… that’s easy to say, and I’ve encountered problems with achieving this goal. For example, if you’re a newly-minted PE who was once a member of a single team (you were promoted into the role), you’ll likely struggle with perceptions.

1. Other engineers who knew you before your promotion will likely play the “we” and “you” game, and they’ll pigeon-hole you into your previous role. This is not intentional, but they were so comfortable with you in your previous role, it’s hard to adjust. Try working with different teams on different projects to break that perception.

2. If you’re an expert in a specific programming language, and you continue to work in that area, you’ll be closely associated with the teams using that language. Try working in different programming languages to break the perception. Be very public about the code you’re writing so people can see your range of skills.

3. Be careful where you sit in the office - if you sit too close to a single team, others think you’re part of that team. Move around from time to time, or sit near to the senior management who are seen as team-agnostic.

4. Be sure to work on projects that are cross-team, and spend equal amounts of time with each team. Bond with the team members to be accepted as an honorary member, rather than as a outsider.

5. Give presentations to the whole organization on a range of different topics, breaking the perception you’re associated with a single team, or one technology.

But most importantly, when you say “we”, you should be talking about the whole organization, not just a single team.

2. Raising problems versus solving problems

A typical indication an engineer has reached Senior Software Engineer status is they feel comfortable complaining about things. They know the software well enough to understand the weaknesses, and have been a team member long enough to know the organizational problems. They’ve seen the same issues occurring over and over again, and feel confident in voicing their concerns. They’re also able to anticipate problems before they actually occur. A great skill to have!

For example, I’ve heard these complaints many times, including from my own mouth:

• We never spend enough time refactoring our code to remove technical debt.

• Management doesn’t give us enough time to test our code properly.

• Why don’t these new engineers know what they’re doing?!?

Unfortunately, complaints of this nature occur when the engineer starts to be Senior, not when they start to be Principal. Identifying problems is a great skill, but to become a Principal-level thinker, it’s vital to understand the root cause of the issue and to phrase the complaints as solutions:

Let’s rephrase:

• We need to collect metrics to show our feature delivery velocity is slowing down due to poorly-written code. By identifying key areas of code rot, then spending 15% of our time refactoring those areas, we expect to see great than 15% in increased feature development velocity.

• Based on the customer feedback we’ve collected, we have evidence that spending more time writing automated tests before releasing the software will noticeably increase customer satisfaction. We suggest delaying software releases by 2-3 weeks to write more tests, without being tempted to squeeze in new features.

• Our new engineers are encountering friction in their work because they don’t yet have the necessary domain knowledge. I’ll set up some Lunch and Learn sessions to ensure they receive the training they need. I feel that 2-3 of our best engineers can share the training effort, with each delivering two L&L sessions.

Now, those statements are what you expect to hear from a Principal Engineer! Not complaining, but identifying solutions and putting those solutions into action. Even using fancy words (that sound “progressive”) enable your colleagues to trust your opinion more, and be more willing to buy-in to your ideas.

Conclusion

Hopefully these two ideas have been thought-provoking. They’re certainly not the only things indicating you’re ready to be a Principal Engineer, but if you don’t exhibit these qualities, I encourage you to think twice about asking for that promotion. Managers - you should insist on seeing these qualities on a regular basis before allowing a promotion - if you don’t see these skills, then promoting the engineer “because they’re the best we have” or “because they’ll quit if I don’t” is just misleading and not at all helpful for your organization.

Computer technology has dramatically changed, but there’s only a small list of things that really blew my mind. Most changes were incremental, building on top of existing ideas I’d already seen. Sure, the numbers keep on getting larger - CPU speeds and RAM sizes increased by a factor of millions - but that wasn’t too surprising, nor exciting.

But, what was exciting? What were the things that made my jaw drop when I saw them? Let’s go from 1982 to 2023, looking at those milestone events and think about why they were so meaningful. Of course, in 2023 many of these technologies seem old and boring, but at the time they were amazing!

My First Computer - Sinclair ZX81 (1982)

The first computer my family owned was a Sinclair ZX81. It had 1KB of RAM (expandable to 16KB), ran at 3.25MHz clock speed, had 64x44 black-and-white pixel resolution, plugged into the family TV, and used cassette tapes for data storage (roughly 300bps of audio screeches as data loaded).

My first program was to print “Peter” in a continuous loop on the screen.

10 PRINT "PETER"
20 GOTO 10

I’d seen computers on TV shows, and my Dad’s office had a mainframe of some kind, but this was the first time I could program a computer for myself. I wrote software in BASIC, but also taught myself Z80 machine code to write “high-performance” games. Here’s an example of a chess game that we purchased for the ZX81.

Why was this jaw-dropping? Suddenly the world was a different place for me, and I spent hours each day sitting in front of the keyboard, rather than playing outdoors in the yard. Most families didn’t have a computer until 10 years later, and there was no (public) internet yet. The world was still based on knowledge-sharing via paper (magazines, newspapers), with broadcast radio and TV being our main source of news and entertainment.

Life had changed for me, and was about to change for the rest of the world.

Marketing for my Second Computer (1982-83)

Growing up in New Zealand had only one downside that I cared about - we were so far away from the UK or US that our exposure to new technology was via printed magazines (remember, no public internet yet). These magazines were shipped by sea and often arrived in our shops about 2-3 months after being published. Despite the delay, these were our lifeline for learning about new technology.

My next jaw-dropping experience was seeing the following dual-page advertisement for the ZX Spectrum computer (successor to the ZX81 we already owned).

Yes, that’s correct - it was the glossy advertising that changed my life, not the computer itself. It was the promise of colour graphics, higher-resolution (256 x 192 pixels) and sound/music! For at least six months (until early 1983), I would excitedly read the advertisements, study the magazine’s program listings to learn the new variant of BASIC, and dream about the new capabilities of the ZX Spectrum.

Sure, it was great to eventually own a ZX Spectrum, but the jaw-dropping part came from the magazines themselves.

Knight Lore (1984)

The ZX Spectrum was the first computer for many teenagers in the 1980s (at least, in much of Europe, Australia, and New Zealand). Many of those kids are now 50+ years old, and fondly recall the five-minutes of screeching audio as our games loaded off cassette tape. There are even Facebook groups devoted to reminiscing about the good old days.

Knight Lore was a game produced by Ultimate Play the Game, a small UK-based software company. The goal of Knight Lore was to navigate a 3D world, find the ingredients for a magic potion, then place them in the cauldron, all without losing your lives (see walkthrough video).

There were plenty of awesome games that pre-dated Knight Lore, but there was something inspiring about this game. It was a combination of the amazing 3D graphics, the challenging game play, and the marketing. Ultimate Play the Game was known for their full page magazine advertisements providing no detail, other than the game’s title and the company logo. This is what made Knight Lore stand out as jaw-dropping.

Desktop-Switching on the Commodore Amiga (1985)

The Commodore Amiga was an amazing computer when it jumped onto the market. We were impressed by the quality of graphics producing near-realistic images (4096 colours!) , and the sound was approaching the quality of music on a CD (fairly new at the time). We now had the power of full-sized arcade games in our living room, without inserting 50 cents for each game.

For me, the jaw-dropping experience was quite specific. The Amiga Workbench (desktops, icons, files, windows) allowed you to have multiple desktops open, each with different content and graphics resolution. Amazingly, you could grab the top menu bar of the desktop with your mouse pointer, quickly and smoothly dragging it downward to reveal the desktop behind it, or even a portion of the other desktop. These days, this “multiple desktop switching” feature is a standard part of the MacOS or Windows experience.

When developing games on the ZX Spectrum, I was accustomed to the main CPU writing to display memory. At a rate of 50 times/second (TV refresh rate), you’d erase the previous graphics character (e.g. space invader, monster, game piece), then redraw it at the new location in the display memory, therefore providing the illusion of animation. To implement the smooth scrolling of this "desktop switching” feature, we’d need to copy the entire display memory 50 times/second.

With the Amiga, this was the first time I’d experienced dedicated graphics hardware, and the smooth animation was done via instructions to the Graphics Processing Unit (GPU), rather than the main CPU copying bits of data around. This jaw-dropping experience was my introduction to ASICs (Application-Specific Integrated Circuits).

Seeing Europeans Posting on Usenet (1990)

When I first attended university (1989-1992), we had access to both SunOS and VMS-based computer systems. I had no idea how large they were physically, but at 128MB of RAM, that was impressive. We always logged in remotely from dumb terminals, and knew these servers were somehow connected to the internet - mostly a university-based network at that time.

The jaw-dropping moment was my discovery of the Usenet news service, the primary mechanism for sharing news and discussions on the internet (there was no world-wide web at this time). Being in far-off New Zealand, where magazines arrived by sea, I was amazed to log in and instantly read postings from somebody in Europe, or somebody in the USA. Suddenly the world became a much smaller place, and for the low-low price of $5/MB, I too could communicate with the rest of the world.

The Amoeba Operating System (1992)

While I was still an undergraduate student, I had the good fortune of beta testing the Amoeba Operating System. This research OS was built ground-up to be fully distributed, using 128-bit unique IDs to identify each resource in the system (e.g. files, devices, and processes). To access a resource, just provide the 128-bit ID and the system would figure out where on the network the resource was. The developer had no idea whether the resource was local to the current host, on some other host across the network, or whether it had recently moved from one place to another. This resource-location approach provided in the appearance of a single large machine, rather than multiple smaller machines connected via a network.

In contrast, I’d already learned about the Unix system, and how it manages resources on a per-host basis. To access resources on a local machine, you’d make a local system call. However, if the resource is on a remote machine, you must explicitly send a TCP or UDP message to the remote host’s unique IP address, where a server process would access the resource for you. In this sense, accessing a local resource (a system call on the current host) versus a remote resource (via a network message) is very different.

Amoeba was an elegant and full-featured operating system that impacted my thinking about system design, and I’m sad it never got the traction it deserved. Although the elegance of the OS design was eye-opening, it did suffer from performance issues, and wasn’t fully compatible with existing Unix software. In the end, it was eclipsed by the Linux system (as a side note, I had the good fortune of reading the famous Tanenbaum-Torvalds debate of Minix vs Linux as it was happening!)

HTML, The WWW and Mosaic Browser (1993)

When I moved to Canada in 1993 to attend graduate school, I was fortunate to have an early glimpse of the WWW (World-Wide Web). My research lab installed the Mosaic Browser on their servers, allowing access to the fledgling web. I recall the NASA web site, some museum sites (perhaps the Smithsonian?), and the Internet Movie Database (IMDB). We also had the ability to create our own personal web sites, using a new description language known as HTML.

The jaw-dropping moment for me was seeing hyperlinking - the ability to click a link on a web page and instantly be transferred to another page, possibly on a completely different web site. Everything on the internet had a unique URL, and I could easily add my own content for others to link to.

It was exciting, but little did we realize the WWW would become synonymous with “the internet”, and that everybody in 2023 would spend so many hours each day surfing the web

The Java Programming Language (1996) and Java Duke

I first heard of the Java programming language at the OOPSLA ‘96 conference. People were raving about this newly-designed language that addressed many of C++’s weaknesses, and incorporated idea from Smalltalk and LISP. For me, it was the first “new” language I’d seen, as languages like BASIC, C, C++, and Pascal, were well-established by the time I learned them.

The jaw-dropping experience was the ability to execute Java code inside somebody else’s web browser (aka Java Applets) running on top of a virtual machine (VM). It was only a couple of years since I’d learned HTML, but now I could run general-purpose program code inside an end-user’s browser. I’d always loved the idea of “remote procedure calls”, but this was going to a new level.

One example was Java Duke, who had previously appeared as a fixed image on a web page. But now, it was possible to animate Duke, and have him actually wave his hand! My jaw dropped when I first saw that particular web page on the Sun Microsystems site! Gone were the days where HTML pages would load once, then remain fixed until the next page load.

Handheld GPS (1998)

The Global Positioning System (GPS) was first launch in 1978, but exclusively for the US military. In the late 1990s it had become accessible to the public, and I purchased a hand-held receiver unit. This was my first exposure to yet another technology that’s now in everyone’s pockets.

My GPS receiver was very simple, taking about five minutes to find your location. The small LCD display listed the satellites it was syncing with, and after a minimum number (I think it was 3?) it would report your longitude and latitude (no map drawing yet). We didn’t yet have Assisted GPS, so the start-up time was still fairly high.

Around that same time, I attend a university lecture given by the Chief Strategy Officer of Sun Microsystems. He presented a wild idea of having GPS in your car, which would direct you to the nearest McDonalds restaurant and provide you with discount coupons. What an awesome futuristic idea!

I recall riding the local buses and trains with my GPS device in hand, taking note of the coordinates of the local landmarks. Sadly that’s as far as I got with that idea, but it was certainly a revolution in the making.

YouTube’s Storage Size (2005)

One of my grad school colleagues did his research on video streaming, so I was no stranger to the topic. I also had plenty of short videos downloaded to my home computer, which were costly to store given their multi-megabyte file size.

What shocked me about YouTube was the magnitude of the storage required. If anybody could create and upload their own videos, possibly 15-30 minutes in duration, I was perplexed by the number of disks required to store all that data! Wouldn’t it be terabytes of new disk space required every hour?

I’m still not clear how they manage all that storage, but I’ve always imagined a convoy of trucks heading toward each data centre, carrying nothing but disk drives. You’d then require dozens of people working around the clock to plug them in, just to keep up with the amount of data being uploaded. It must be more of a problem these days with modern social media and photo applications, constantly uploading new images to the cloud.

A First Glance at Amazon EC2 (2006)

As far back as 1993, I had installed the Linux operating system on numerous hosts, mostly from floppy disk or CD-ROM. The first distribution I used was Slackware, requiring many hours of installation effort, and a lot of manual typing. Years later I purchased a license for Redhat Linux, and received the CD-ROMs in the mail. Even as recently as 2017, I downloaded an Linux ISO file and ran CentOS in a virtual machine on my desktop computer.

When I started using Amazon EC2 in 2014, I had memories of an article I read in Linux Magazine, probably from 2006. The article talked about the new EC2 service from Amazon allowing you to create a Linux VM within minutes. You would ssh to your new VM to perform your work (no direct console access). Not only was installation time impressive, but you were only charged for the number of minutes the server was running, with no upfront hardware setup cost! When you were done, you’d simply delete the VM and no longer be charged.

I didn’t need EC2 at the time, because I already had my Linux hardware configured and “paid for”. It wasn’t until 2013 that I fully appreciated the value of the cloud - I was working for a company that insisted on their own private data centres, literally requiring months of careful project management and cross-team communication to get a new server running. The cloud changed all of that.

Interactivity of Google Maps

Google Maps has improved dramatically over the years, but the one improvement I remember the most is interactive scrolling. Before this feature, you needed to explicitly click the left, right, up, or down button to see the map scroll to a different location, triggering another page load each time. This was typical of web applications at the time.

One day though, I discovered you could simply drag the map left, right, up, or down, and the page would smoothly scroll, with new parts of the map appearing like magic. In fact, you could scroll infinitely in any direction. We’d now reached the time when JavaScript (running in the browser) could dynamically load data, and redraw parts of the canvas, without reloading the full web page.

I’m really not sure why this feature impressed me so much, but for some reason it did. It didn’t fit my mental model of how web browsers work, so this feature caused me to rethink what was possible.

The iPhone (2007)

It’s clear now that the iPhone has changed the world, but it was hardly the first device of its kind. I remember seeing the Apple Newton in 1993, and I personally owned a Palm Pilot in the early 2000s. Both of these devices used a stylus for data input, and had a crude form of handwriting recognition instead of a keyboard. Although some of the later Palm devices had wireless connectivity, they were generally standalone devices.

The iPhone was jaw-dropping for several reasons, but for me it was the use of finger-swiping gestures instead of the stylus (which I often feared losing). I was amazed the first time I saw somebody in a coffee shop making flicking and pinching motions on their phone screen, with the display updating via smooth animation to give immediate feedback on the operation.

IBM Watson (2011)

With all the news about ChatGPT, I would be remiss in mentioning IBM Watson from 2011. I excitedly watched the episode of Jeopardy where a computer faced off against two human contestants, and did extremely well. What boggled my mind was that Watson had so much knowledge about so many things, and was able to interpret English sentences and respond in mere seconds.

Of course, ChatGPT has taken this one step further, being accessible to the general public. But I’ll never forget the first time I saw IBM Watson in action.

This Person Does Not Exist (2020)

The most recent technology that blew my mind was This Person Does Not Exist. You can literally spend hours looking for new friends! Give it a try!

You’ll have a hard time finding an experienced software engineer who doesn’t have a preferred naming convention. This includes function or method names, classes, files, directories, projects, products, or potentially any concept that we talk about, or any identifier in our source code.

This blog post isn’t about typical naming conventions. We won’t discuss whether to use camelCase or snake_case, or whether an identifier should contain a noun, or a verb. We won’t specify how to pluralize, or whether suffixing with Manager or Util is a bad idea. The Internet is full of blog posts covering those topics.

Instead, we’ll focus on the long-term effects of your naming choices. The identifiers you select today may still be in use 5–10 years from now. Most developers want their software to be used long term, yet they choose identifiers that are not resilient to the passage of time. This discussion is based on more than 20 years experience of seeing these issues in the software industry, so the problems are real!

What’s the impact of non-resilient names? To be honest, there’s plenty of other technical debt you could focus on first, but learning how to select a resilient name avoids confusion in the long-term, once your product is successful and mature.

What in the World Might Change?

To illustrate the passage of time, we’ll use an example of a SaaS application. Imagine that Kitty Incorporated is a small software company with a single web-based product, KittyPics, allowing customers to upload and share their favourite cat photos.

Here are some events in the life of Kitty Incorporated:

• Scaling Web Traffic — Initially KittyPics only required a single web server, with a single disk drive to store images. Due to the site’s growing popularity, there’s now a need for multiple web servers and multiple disks.

• Projects Come and Go — After successfully serving cat pictures, a new internal project (code named Tiger) is started at Kitty Inc. The goal of the Tiger project is to allow upload of cat videos, rather than just photos. The project is active for six months, with all code being added to the same KittyPics code base.

• More Flexibility— Although the original version of KittyPics did one thing (and did it well), countless new features and configuration settings were added over time. Users now have many ways to adjust their experience, requiring updated algorithms in the code.

• Changing Technology — The CTO decides that maintaining a large number of disk drives isn’t cost effective, so a project is initiated to move all cat pictures/videos to Amazon Web Services using S3 Buckets. Similarly, a switch was made from storing meta-data in MySQL, to instead using the more scalable DynamoDB database.

• Kitty Incorporate is Acquired — After successfully operating for two years, Kitty Inc is acquired by a competitor — Cat Pictures Corporation.

• KittyPics is Renamed — After careful market analysis, it’s determine that KittyPics should be renamed to a much trendier name: SabreShots.

These events are common in a software product’s lifecycle. We’ll now focus on how these changes cause our identifiers to become outdated. If they’re outdated, the code becomes harder to read and more costly to maintain, therefore increasing the technical debt load.

How Names Become Confusing or Inconsistent

Let’s look at some real-world examples of how the names we choose can become confusing or inconsistent, largely due to the passage of time. For each example, we’ll see a naming choice that made sense when the software was first written, but stopped making sense at a later time. We’ll also see an example of names that are more resilient to change.

1 — Expect New Implementations, but Avoid Calling Them “New”

Let’s start with a simple example. Imagine our KittyPics software contains an algorithm for automatically positioning cat pictures on the screen. The algorithm has worked well, but we want to add a second algorithm, while still keeping the first as an optional feature.

If we started with the following code:

def layoutAlgorithm { ... }

There’s a natural tendency to define the second algorithm as:

def newLayoutAlgorithm { ... }

Clearly this makes sense at the time when transitioning from old to new algorithm, but the word new becomes less relevant over time. After a few years, developers consider both implementations as “old”, and there’s no hint about what the algorithms actually do. Also, if there was a third implementation added, would it be called muchNewerLayoutAlgorithm?

A better practice is to name the algorithms in a descriptive way to indicate how they differ. For example:

def gridLayoutAlgorithm { ... }
def circularLayoutAlgorithm { ... }

These names remain relevant as time passes, since their purpose is well-described, even years after their “new-ness” fades.

2 — Remember That Names Last Longer Than Projects

Projects and Products differ is significant ways. Projects are a time-bounded (for example, six months in duration) and have the primary purpose of adding a discrete set of features to an existing code base. In contrast, a Product has a much longer life cycle, possibly existing for 5–10 years, or even longer.

It’s important to keep in mind that any name you choose in your software will last beyond the scope of the current project. In our example of the Tiger project (to add cat videos to the KittyPics product), any use of the name Tiger becomes increasingly meaningless after the project finishes.

module Tiger
def tigerAlgorithm {
   count = getTigerCount()
   ...
}

Newly-hired developers will struggle to wrap their minds around what Tiger means, especially as the Tiger project is now ancient history.

Instead, use names that are meaningful after the project has completed, when the newly-added features have simply become part of the long-term product code base. For example:

module VideoManagement
def videoPlacementAlgorithm {
  count = getVideoCount()
  ...
}

This advice may seem obvious, but it’s very common to see modules, classes, algorithms, databases, or DNS names with project names embedded into them. With the passage of time, code bases are littered with temporary project names in their identifiers.

One example of this problem is using the project’s name to identify newly-created software modules. For example, Tiger may become the internal code name for the new software module that handles videos. Over time, people think of the Tiger module, rather than the Tiger project, when they hear the name Tiger, even if the original project had a much wider scope.

3 — Consider That Technology Changes, but Names Don’t Need To

Throughout a product’s lifespan, it’s common to change the underlying technology used. For example, our KittyPics product moved from using MySQL as the primary database, to using the more scalable DynamoDB. Ideally, this change should be transparent to most of the software, but that’s not always the case.

In the first implementation of KittyPics it made sense to write code such as:

mySql.getImageCount()

but that clearly doesn’t make sense when you switch to using DynamoDB. Instead, use a more generic name for your data store, such as:

metaData.getImageCount() 

You’ll still need to modify the internal code for the getImageCount() method, but if you choose your names more carefully, you won’t need to modify the code everywhere the getImageCount() method is called.

4 — Marketing Names Can Change Outside Your Control

The name of your product (or the domain objects within the product), can change due to marketing or sales decisions. In our example, KittyPics was rebranded to SabreShots, with the goal of increasing sales. Likewise, names of domain objects within the product (such as Collection or Image), may change to something new (such as Album or Photo). Experience shows that products can go through 2–3 name changes in a ten year span, largely due to corporate acquisitions or rebranding.

To solve the problem, you might choose to go through a major refactoring exercise to keep the internal code base aligned with the latest external names. This makes sense for small code bases, but for larger products it’ll only be a source of frustration and newly-introduced bugs.

Also, be very cautious about mixing the old name with the new name in the same code base (for example, passing a Photo object into a method expecting an Image object, even if they’re the same thing). Inter-mixing the old and new names can be very confusing for developers, especially when they’re developing in the code base for the first time.

Perhaps the best advice is to avoid adapting the code base at all, but instead be comfortable with internal names being different from external names. That is, always use the original names of Collection or Image, regardless of the current customer-facing name. Of course, developers need to remember the mapping from internal name to external name, but at least the code will be consistent and easier to work with.

5 — Your Company Name Can Also Change

It probably comes as no surprise, but it’s good advice to not include your company’s name in source code identifiers. Mergers and acquisitions are common, and acquired company names disappear into history.

Regardless of whether your company name changes, you should always think twice before using it in identifiers. What value does it add to have Kitty in these names? You probably want more descriptive names than these:

kittyCollectionCount = 100
def displayKittyImage() { ... }
kittyDb.executeSql(...)

Of course, DNS names are an obvious places where the company name is a requirement, so it’s not a hard rule. Interestingly, many smaller companies only sell a single product, so their company name and product name end up being synonymous, with a lot of work involved to break that association as the company grows. Was kitty.com the product, or the company?

6 — Avoid Being Too Specific About a Purpose

When implementing code, there’s a tendency to focus on solving the specific problem you have at that moment. However, as the product changes, there’s a desire to reuse code you wrote in the past. Refactoring has become a common activity for most developers, including the selection of more suitable names.

For example, imagine you had written a PhotoArranger class, providing the ability for users to manually change the order of the photos on the product home page. However, now that the home page also supports videos, the PhotoArranger name becomes confusing. One solution would be to rename the class to PhotoAndVideoArranger, but that gets even more problematic when the ability to add static text and hyperlinks is added to this class (should we now call it PhotoAndVideoAndStaticTextWithHyperlinksArranger ?)

Perhaps the original name should have been something less specific, making it more flexible and resilient to change. Perhaps HomePageArranger? However, something like MediaLayoutArranger would be even more generic, allowing it to be used on other pages (not just the home page).

7 — You’ll Likely Have More Than One

Software is often written with time-to-market as a driving factor, rather than worrying about long-term growth (which may never happen if you don’t get to market quickly). One downside of this approach is that software is designed to support only one of a particular component, rather than many.

In our KittyPics example, the following components needed to go from one instance to many instances:

• The number of web application servers running the software.

• The number of disks used for storing image files.

• The number of countries in which servers are running.

• The number of databases required for storing meta-information.

Aside from all the software design decisions you’ll need to make (such as using an array of server names, rather than a single server name), there are naming issues too.

For example, for API server names, you’d probably start with:

api.kitty.com

but when you need to support servers in multiple countries, what do you call them? The common solution is:

api.kitty.com
api-nz.kitty.com
api-uk.kitty.com
...

This example is certainly on the low-end of the technical debt scale, but it does look odd that there’s no country tag on the first entry, and it might cause confusion in your automatic deployment scripts. To avoid these scenarios, you should keep expansion in mind and add a descriptive tag on the first DNS name you use. In the case of countable things, add a -1 to the name by default, such as api-1.kitty.com.

Summary — How Much Should You Care?

In this blog post we’ve seen a number of different ways in which names fail to be resilient over time, therefore becoming stale, confusing, or inconsistent.

What should you do to fix these problems?

To be honest, although these problems occur in numerous products, in almost every company, people always manage to adjust to the inconsistencies. There’s confusion when internal names don’t match external names, especially for new employees who are wrapping their minds around a new code base. However, given enough time, people always seem to overcome the learning curve and just live with the inconsistencies.

Should we refactor our product to resolve these inconsistencies?

That decision is totally up to you, based on how much effort it’ll be to fix the problem, versus how much pain it’ll be to live with it. This is the ever-present technical debt problem we all face with legacy code. Refactoring an internal class name is a simple and pragmatic improvement, but changing a customer-facing DNS name is a challenge you might choose to ignore.

Perhaps the best advice is to try and use meaningful names when you first write the code. However, no matter how much effort you put into choosing names for the long term, you’ll never quite get it right. In the end, you must be prepared to accept and ignore the remaining problems.

This blog post describes a high-performance micro-service we created at Galvanize to meet the authorization needs of our SaaS platform. This new micro-service, code-named “Authy” (not related to the commercial product of the same name), provides platform-wide management of users and groups, while tracking permissions assigned to all the platform’s domain objects. Amazon’s DynamoDB is used to manage the authorization data in our platform, typically allowing authorization decisions to be made within 20–30ms.

In the past, authorization was handled separately by each service, so different parts of our SaaS platform solved the problem in different ways. With our new Authy service, we’ve unified the approach, providing a consistent model for controlling access to resources. This was especially important as we added new micro-services, and didn’t want all of them to solve the same challenges in their own unique way.

The following diagram shows the authorization workflow. First, a user requests some data from our platform, typically asking for an HTML page, or a JSON response to an API call. The backend services each call upon Authy to validate the current user has access to the resource they’re requesting. Based on Authy’s response, the client either approves or rejects the request.

Note that Authy provides Authorization services, but doesn’t handle Authentication. All HTTP requests from the user are accompanied by a JWT token confirming the user’s identity. Authy then determines whether the resource being accessed (e.g. /projects/1234) is available to the user.

The Requirements

To understand how Authy works, let’s start by learning more about our SaaS product. Galvanize operates in the GRC (Governance, Risk management, and Compliance) space, helping companies be fiscally and socially responsible. This includes fighting accounting fraud, and detecting corporate waste.

From a software perspective, our product is similar to an ERP or CRM solution. It has a beautiful user experience via the web interface, with well-defined REST APIs for programmatic access to data. Multiple tenants access the same platform, but can only see their own organization’s data. Finally, each of our customers potentially has hundreds of users, so we have a powerful user-management system with complex user-defined workflows.

Let’s now learn about the GRC domain model, as well as how users and groups are granted permission to access the domain objects (aka “resources”).

The Domain Model

Our SaaS product has a large number of Domain Objects, representing items of interest to the customer. The most important top-level domain objects include:

• Project — Represents a body of work performed by a user. This includes a schedule for undertaking the project work, Risks identified within the project, Controls to help manage the Risks, Issues identified, and Actions to resolve the Issues.

• Asset — Represents something of interest to the company, such as a third-party vendor, an IT server, or a piece of software.

• Collection — A collection of data tables, each containing many rows of data. For example, the access log from a security system showing when people entered/exited a building.

• Toolkit — A template of product content, such as a pre-defined project with typical risks and controls already defined for common scenarios.

Inside each of these top-level domain objects is a large number of fine-grained domain objects. For example, within a Project, a user would add Risks, Controls, and Issues, each with their own ID number.

The other top-level domain objects, such as Assets, Collections, or Toolkits, are also container-like, holding their own fine-grained domain objects.

Users, Groups, and Organizations

Our SaaS platform supports multiple tenants (aka Organizations), where each customer’s data is fully isolated from that of other customers, even though they reside in the same database. All database queries include an orgId field to ensure the correct data is returned.

Within each organization there are multiple Users, which for large customers will number in the thousands. Most customers make use of Groups to help manage those users.

Users and groups can both be assigned permissions to access various domain objects in the system. For example, User John may be assigned write access to a specific Project, whereas the Finance Group may only have read access. Note that Administrators are a special class of Users who automatically have access to everything.

Domain Objects are Containers for Permissions

When it comes to granting permissions for users or groups, this can only be done on top-level domain objects (such as Projects) which are containers for fine-grained objects, such as Risks or Controls. It is not possible, nor is it desirable, to grant permission on fine-grained objects.

This is enforced for several reasons:

1. Experience shows that when customers can grant user permissions on every domain object, no matter how small, they end up with a lot of support challenges when permissions become confusing to manage.

2. We have billions of fine-grained domain objects in the system, which would explode the number of permissions that Authy must manage.

3. Likewise, the number of calls from a backend service to Authy would be excessive if every domain object must be authorized.

Therefore, we chose to only allow permissions on top-level domain objects, while requiring fine-grained objects to inherit permissions from their parent container. To make this work in a practical way, the permissions provided on the top-level objects are very expressive.

For example, let’s imagine that John has the following permissions, set on a specific Project:

• CAN_CREATE_RISKS — This allows John to create a new (fine-grained) Risk object, but only in the context of this specific Project.

• CAN_CREATE_ISSUES — Similar to the previous example, John can also create Issue objects inside this Project.

Note that John can not create other fine-grained objects, such as Controls, or Actions, since he doesn’t have those permissions on the parent Project.

Next, John may also have permissions to read the existing domain objects in the Project:

• CAN_READ_RISKS — This permits John to read the content of any Risk object in the Project.

• CAN_READ_ISSUE_IF_OWNER — This allows John to read Issues in the Project, but only if his name appears in the owner field of the Issue. He doesn’t have permission to read any other Issues that he’s not the owner of.

This second example illustrates how permissions become more fine-grained, while still only being set on the Project itself. Once we’ve moved fully over to Authy, we anticipate having hundreds of possible permissions. At a minimum, this includes create, read, update, delete, and list permissions for each of the possible fine-grained objects.

Now that we understand the domain object we’re authorizing access to, let’s learn about queries that Authy must respond to.

The Queries

An important goal for Authy is to respond to queries within 20–30ms, so as to avoid negative impact on the end-user response time. Authy’s algorithms were optimized to reduce the run time of a single query, but also to reduce the number of calls necessary to validate an incoming customer request.

Common Queries

The following are typical queries to Authy, based on the type of information a backend service needs when authorizing an incoming request. We did a comprehensive review of our platform’s authorization requirements to ensure that Authy would cover them all.

The common queries are:

1. Give me the personal details for a user — This is needed throughout the platform when displaying a user’s name or sending them email.

2. Give me the list of groups in this organization, and their members — This is used for display purposes only, but is not used for authorization.

3. Give me the list of groups that this user belongs to — Also used for display purposes only, but not for authorization.

4. Can this user perform this operation on this resource (aka domain object)? — This is a very common question, accounting for most of the queries to Authy. The user’s ID comes from the JWT of the original HTTP request, and the resource ID comes from the URL of the request (such as /projects/1234). Finally, the necessary permission (for example, CAN_READ_PROJECT) depends on the semantics of the HTTP operation, such as GET versus PUT.

5. What operations can this user perform on this resource? — This is similar to the previous query, but rather than providing a single yes/no answer, Authy provides the complete set of permissions a user has for that resource (such as [ CAN_READ_PROJECT, CAN_UPDATE_PROJECT, CAN_READ_RISKS, …]. This reduces the need to send multiple requests to inquire about multiple permissions.

6. Which users in the organization can perform this operation on this resource? — This is not used as much as the previous queries, but is very useful to identify users in the organization with a base set of permissions. For example, to populate a drop-down list of users who could be asked to review an Issue object, we’ll ask Authy for all users who have the CAN_REVIEW_ISSUE permission. There’s no point in assigning a user to review something if they don’t have permission to do so.

7. Which resources can this user perform this operation on? — This query is very important for listing the objects a user can act upon. For example, when the user requests a list of Projects, they should only see the Projects they have CAN_READ_PROJECT permission for. Other Projects should not be listed.

Queries That Shouldn’t Be Asked

When designing Authy, we identified several queries that were commonly asked, but ended up being anti-patterns.

1. Is the user an administrator? — In our platform, administrators have the power to perform any operation on any resource in the system. However, instead of explicitly asking Authy whether the user is an administrator, the query should only ask about the specific permission, such as CAN_DELETE_PROJECT. If Authy responds with “Yes”, then it doesn’t matter whether the user is an administrator, or whether they got the permission some other way. All that matters is that they have the permission.

2. Does this group have a specific permission for this resource? — Initially this seemed like an important variation on asking whether a user has permission, but in reality, both users and groups get folded into the same query anyway. For example, although the Sales group might have CAN_READ_ISSUE permission, the request to Authy will actually ask about Frank, the logged-in user who is a member of the Sales group. Therefore, Frank may either get the CAN_READ_ISSUE permission by being part of Sales, or by explicitly being assigned that permission on the parent Project. In either case, he has the permission.

3. Can this user access this Risk? — As mentioned above, a Risk is a fine-grained domain object that resides inside a Project, and Authy doesn’t track permissions on fine-grained objects. To answer this query, the backend service first determines the ID of the Project containing the Risk, then ask whether the CAN_READ_RISK permission is set on the Project.

Now that we understand the domain object, their permissions, and the common authorization queries, it’s now time to dive into the implementation, using the DynamoDB database.

The DynamoDB Solution

When describing how Authy stores permission assignments in DynamoDB, we’ll focus exclusively on these common queries:

1. What operations can the user perform on a specific resource?

2. Who are all the users in the organization who can perform this operation on this resource?

3. Which resources can this user perform this operation on?

Our DynamoDB table has four main attributes of interest, allowing these queries to be answered:

Attribute: orgID

This is the table’s Partition Key, used to separate one organization’s data from another. This is simply an organization number, such as 47.

Attribute: identity

This field, the DynamoDB Sort Key, identifies the user or group the permission assignment is for.

• For a user, the value will be u|<user-id>|<resource>.

• For a group, the value will be g|<group-id>|<resource>.

• To assign the permission to everybody in the organization: *|<resource>.

Note the <resource> suffix is only required because DynamoDB needs each record to have a unique Partition Key / Sort Key combination, so <resource> makes them unique. The next section explains the format for <resource>.

Attribute: resource

This field specifies which resource (aka domain object) the permissions are for. We have multiple top-level domain objects, so we combine the resource’s type with the resource’s ID. The format will be <resource-type>|<resource-id> where <resource-type> is a digit identifying the type of domain object that <resource-id> refers to.

• 0 = The whole organization.

• 1|<project-id>= A Project domain object.

• 2|<collection-id> = A Collection domain object.

• 3|<asset-type-id> = An Asset Type domain object.

• 4|<toolkit-id> = A Toolkit domain object.

This list can easily be expanded to include new top-level domain objects.

Attribute: grantedSet

This is the field in which all the permissions are stored. In other words, the user is granted the set of permissions listed in this field. For the sake of efficiency, we encode all possible permissions into an enumeration:

export enum Permission {
  CAN_CREATE_PROJECT     = 0
  CAN_READ_PROJECT       = 1,
  CAN_UPDATE_PROJECT     = 2,
  CAN_DELETE_PROJECT     = 3,
  CAN_CREATE_RISK        = 4
  CAN_READ_RISK          = 5,
  CAN_UPDATE_RISK        = 6,
  CAN_DELETE_RISK        = 7,
  CAN_READ_RISK_IF_OWNER = 8,
  ...
}

There could in theory be hundreds or thousands of permissions, with any combination of permissions being valid. We therefore introduce the concept of Permission Set allowing multiple permissions to be specified.

In essence, a Permission Set is implemented as a set of binary bits. For example, if a user is allowed to read and update projects, but not to create or delete them, their permission set will be CAN_READ_PROJECT (bit 1) + CAN_UPDATE_PROJECT (bit 2) = 0b0110, which is 6 in decimal.

To support potentially thousands of permissions in this Permission Set, we use a DynamoDB List field type, with each entry being a 32-bit number. List index 0 contains bits 0–31, with list index 1 containing bits 32–63, and so on. For practical purposes, we only have a few hundred permissions, so this GrantedSet field is typically quite short.

An Example

Now that we know the format of the DynamoDB table, let’s see an example of how it’s queried. To start with, here’s a subset of the permissions we’d expect to see for a small customer:

1. Everybody in Organization 47 has permission to read all Projects.

2. Members of the Sales group can update Project 234.

3. John can create new Projects and delete all existing Projects.

4. Mary is an administrator for Organization 47, and can do anything.

Here’s the DynamoDB table, with these rules encoded using the attributes we described above. Before ready further, take a minute to study each line of this table, comparing them to our four permission rules.

Now, let’s run some of our common queries to see how the data is accessed. We won’t discuss the full Authy algorithm, but these examples should give you an idea of how it works.

Query 1: What permission does Frank have for Project 567?

To answer this query, we need to consider the different ways that Frank could be given access to Project 567.

1. He might be directly assigned permissions to Project 567 — In this example, no he isn’t.

2. He might be a member of a group that is directly assigned permissions to Project 567 — He’s in Sales, but that group is not explicitly assigned permissions to this Project.

3. He might be directly assigned permissions on the whole Organization, which implies he has those permissions for all Projects in the Organization — No, not in this example.

4. Similarly, he might be in a group that has Organization-level permissions — No, not in this case either.

5. All members of the Organization might be given permissions to Project 567 — No, not in this example.

6. All members of the Organization might have the permissions for the whole Organization — Yes, all users have CAN_READ_PROJECT set, as shown on Line 1 of the DynamoDB table.

That’s certainly a lot of things to check in order to determine that Frank has CAN_READ_PROJECT permission for Project 567. Let’s see how’d we’d do this in DynamoDB. Clearly we want to minimize the number of queries we perform.

We start by performing three DynamoDB queries in parallel:

1. We fetch the list of groups that Frank belongs to. We didn’t show the DynamoDB schema for group membership, but it’s fairly straightforward.

2. In parallel, request all the records related to Project 567, using a DynamoDB Secondary Index. This returns the list of permission assignments, regardless of whether they’re for Frank (starting with u|frank|), groups (starting with g|), or for everybody in the organization (the * wildcard value). We then use DynamoDB’s query filter to discard user records that aren’t for Frank, but we still need to return all the group records because we don’t yet know which groups Franks belongs to — not until Step 1 (above) completes.

3. In parallel, we request all records for the organization-level resource. This is where we specify permissions that apply to all Projects, not just for a single Project.

Once these three DynamoDB queries have completed, we perform an in-memory filter on the two resource lists (steps 2 and 3 above) by keeping only the records for Frank, or for any of the groups that Frank belongs to. All other records are discarded.

For all records that we didn’t throw out, we perform a bitwise OR operation to merge the individual permission sets into a single permission set, which is then returned from Authy. In this particular example, only Line 1 of our table is relevant to Frank, so the final answer is [ 2 ] (CAN_READ_PROJECT).

Note that we’re assuming here that each resource will have a limited number of rows in the table, otherwise this operation would be very intensive. This is typically true, since our smaller customers have only a few users, and our larger customers tend to place their users into a small number of groups. The merge algorithm will therefore not be too complex.

Query 2: Can Jenny, a member of the Sales team, update Project 234?

This is a similar example, but since Jenny is a member of the sales team, Line 2 of our DynamoDB table will also be used when calculating the final permission set.

• From Line 1 — Jenny gets CAN_READ_PROJECT [2].

• From Line 2 — Jenny gets CAN_UPDATE_PROJECT [4].

The bitwise OR of [4] and [2] is [6], so Jenny has permission set [CAN_READ_PROJECT, CAN_UPDATE_PROJECT], so yes, she can update this Project.

Query 3: What can John, not a member of Sales, do on Project 234?

For John, there are two lines in the DynamoDB table that provide permissions:

• Line 1 — John gets CAN_READ_PROJECT [2] on all Projects.

• Line 3 — John gets CAN_CREATE_PROJECT and CAN_DELETE_PROJECT [9] on all Projects.

Therefore, the bitwise OR is [2] and [9] is [11], so John can do [CAN_READ_PROJECT, CAN_CREATE_PROJECT, CAN_DELETE_PROJECT]. If you’re observant, you’ll realize that CAN_CREATE_PROJECT can only be assigned at the organization level, since you can never assign create permissions directly on a resource that doesn’t yet exist.

Query 4: Can Mary delete Project 135?

Yes, Mary is an administrator, and according to line 4 of the DynamoDB table, she has all the permission bits set. However, our algorithm must still perform all the other queries (including finding Mary’s groups). This is because the DynamoDB queries are performed in parallel so are already in progress (or may have completed) by the time we learn that Mary is an administrator.

Summary

So that’s how Authy works! This blog post has covered the high-level requirements, and the solution using DynamoDB. We saw the optimized DynamoDB schema for storing permission assignments, and learned how to query the database to answer permission-related questions about our SaaS platform’s authorization rules.

Although we’ve covered a lot of material, we’ve barely scratched the surface on all that Authy is capable of doing. We’re constantly adding new features, and enabling new ways for our customers to control access to their domain objects.

If you like what you’ve read, and you’d like to learn about Authy, then come join us at Galvanize!

This blog post discusses how JavaScript is compiled differently when entered into the REPL or via the eval command, versus being in a function body.

This is the fifth part in a series that dives into how the V8 JavaScript Engine computes the expression 1 + 1. This seems like a simple task, but utilizes a large portion of the JavaScript run-time environment, and so far has required four blog posts to describe. There will likely be four more to complete the full story, so kudos to anyone who reads them all!

Since I started writing this blog series, I’ve learn a lot of the unusual JavaScript quirks you wouldn’t normally think about. If you’re curious about the entire blog series, here’s what we’ve seen so far:

• Part 1 — How the 1 + 1 string is stored in the JavaScript heap.

• Part 2 — How byte codes are cached to avoid unnecessary compilation.

• Part 3 — How the string 1 + 1 is scanned into lexical tokens.

• Part 4 — Parsing the Expression into an Abstract Syntax Tree

Let’s now see how V8 rewrites certain statements if they’re entered into the REPL or passed into the eval command.

Entering Statements into the REPL

Here’s a simple example. What’s the result of the following statement, when entered into the V8 JavaScript REPL (such as Node.js, or Chrome console)?

if (3 > 5) {
  10
} else {
  20
}

Alternatively, what’s the expected value when passed to eval?

eval("if (3 > 5) { 10 } else { 20 }")

It should be no surprise that 20 is returned in both cases. How about this similar example, using a function definition and call?

function f() {
  if (3 > 5) {
    10
  } else {
    20
  }
}

f()

You might think it also returns 20, but that’s not the case. This function-based approach instead returns undefined. If you think about it, the constants 10 and 20 are effectively dead code since they’re executed without side-effects (not assigned to a variable), and they’re not explicitly returned, so the result will immediately be discarded.

Unlike some languages, JavaScript does not return the value of the then or else code paths as the value returned by the if statement itself. That is, the following code will fail:

> const max = if (a > b) { a } else { b }
              ^^
Uncaught SyntaxError: Unexpected token 'if'

Additionally, JavaScript does not support the implicit return of the last expression in a statement:

> function f(a, b) { a + b }
> f(10, 20)
undefined

Although many people think of JavaScript as a great language for functional programming, these examples demonstrate some weaknesses in that vision. As an aside, using the ternary ? : operator solves the first limitation, and lambda/arrow functions solve the second case, so at least some aspects of JavaScript allow functional style.

Back to our problem — why did our first example return 20, while the second example (the same code inside a function) gave us undefined? Let’s investigate.

Rewriting the Immediate Code

Our example worked because the JavaScript REPL, as well as the built-in eval feature, both modify the standard JavaScript semantics to be more functional in nature. Let’s face it, the REPL wouldn’t be useful if you couldn’t evaluate simple expressions. Consider what would happen if side-effect-free expressions were basically discarded:

> 1 + 1
undefined

> eval("1 + 1")
undefined

To make this practical, V8 rewrites JavaScript code (as long as it’s not inside a function) to actually return the value. Here’s our first example again, but with the code rewritten to include the necessary side-effects.

function f() {
  let result = undefined
  if (3 > 5) {
    result = 10
  } else {
    result = 20
  }
  return result
}

f()

Note the use of the explicit result variable, with a single return statement at the end of the function. You might feel that adding return 10 and return 20 statements would be more efficient, but consider the following example:

if (3 > 5) {
  10
} else {
  20
  25
}

The answer should be 25, but rewriting to use return 20; return 25; would incorrectly return 20.

Similarly, an upfront result = undefined statement might be necessary if there’s a path where no valid expression will be seen:

function f() {
  let result = undefined
  if (3 > 5) {
    result = 10
  }
  return result
}

f()

In this case, the lack of an else clause results in undefined being returned if the false path is taken.

How V8 Rewrites the Abstract Syntax Tree

As we saw in previous blog posts, V8 encapsulates all standalone code (such as 1 + 1) into an implicit function (with no parameters and no properties). This allows V8 to use the FunctionLiteral AST node as the basic unit of compilation in a consistent way across the V8 engine.

However, simply inserting the standalone code into a function body causes the computed values to be lost, and undefined to be returned. To avoid this, V8 rewrites the AST using the Rewriter::Rewrite() method (see src/parsing/rewriter.cc) to assign expressions to a temporary result variable, then automatically inserts a return statement at the end of the function body.

In terms of the actual AST changes required for our earlier if example, Rewrite() takes the following high-level steps:

1. Allocate a new temporary variable to contain the function’s result. This will be named .result, with a period at the start of the name so it won’t conflict with any user-defined variable names.

2. For each child of the IfStatement AST node (the true path and the false path), search backward through the list of child statements and insert a new Assignment AST node for the last statement that creates a value. We’ll learn more about this shortly.

3. At the end of the main FunctionLiteral's list of child statements, add a new ReturnStatement node to return the value of the temporary variable (.result).

As you’d expect, the end result will be:

function f() {
  let .result = undefined
  if (3 > 5) {
    .result = 10
  } else {
    .result = 20
  }
  return .result
}

Let’s examine this process in more detail.

Creating the Temporary Variable

The rewriter code starts in the Rewriter::Rewrite() method, which calls RewriteBody() with the list of statements (aka “body”) that appears as the child of the FunctionLiteral AST node.

Other than a few sanity checks, the most important step here is to create the temporary variable, .result, using the following code:

Variable* result = 
  scope->AsDeclarationScope()->
    NewTemporary(info->ast_value_factory()->dot_result_string());

Once created, the temporary variable and the function’s body of statements are passed into a new instance of the Processor class, which proceeds to rewrite the AST to include the necessary variable assignments. This is all done by calling Processor::Process().

Visiting the AST

The Processor class is a subclass of AstVisitor which provides common “tree walking” functionality for traversing the AST in various ways. For example, AstVisitor<AstPrinter> is used when pretty-printing the AST (see the --print-ast option), and AstVisitor<BytecodeGenerator> is used when generating executable byte codes. In our case, AstVisitor<Processor> is used to traverse the AST to add the .result = statements.

AstVisitor and its subclasses all have a Visit() method that takes an AstNode * as input. This method determines the type of that AstNode, then dispatches to the appropriate tree-walker method. For example, if the AST node has type ExpressionStatement, then the VisitExpressionStatement() method is called. Likewise, seeing an IfStatement AST node results in the VisitIfStatement() method being called.

Inside each of the tree-walker methods, the code does whatever is necessary to process the information in that node. This likely involves a recursive call to Visit() to walk through the child AST nodes. For example, VisitIfStatement() calls Visit() on each of its children, to visit the true statement body, and then the else statement body, if it exists.

As you can see from eye-balling the src/parsing/rewriter.cc file, there are numerous VisitX() methods implemented, with each one handling a different type of AST node. Not all AST nodes have a corresponding VisitX() method though, but anything required for rewriting our AST will be present.

Walking Backwards Through Blocks

One interesting part of the rewriting algorithm is that it intelligently considers when to add the .result = assignment. In the following contrived example, we can see that only 10 and 25 need to be assigned to the .result variable, whereas assigning 20 would be pointless.

if (3 > 5) {
  10
} else {
  20
  25
}

Our AST traversal code therefore walks backwards through the list of statements, only assigning the final expression to the .result variable. However, this assumes that the block is not “breakable” in the sense that the flow of control could leave the block earlier than the last statement.

For example, here’s another contrived example:

while (true) { 
  10
  break
  20
}

In this case, if we had stopped after setting .result = 20, we’d miss the fact that the code exits the loop before getting to that point, so .result = 10 must also be assigned.

Inside the Processor::Process() method, the algorithm walks backward through the list of statements in the FunctionLiteral's body. It calls Visit() on each statement, then replaces the statement with the rewritten version (in the case where .result = was added). If rewriting wasn’t necessary, the corresponding VisitX() method simply sets replacement_ to be the original statement, unchanged.

Here’s the code:

void Processor::Process(ZonePtrList<Statement>* statements) {
  for (int i = statements->length() - 1; 
            i >= 0 && (breakable_ || !is_set_); --i) {
    Visit(statements->at(i));
    statements->Set(i, replacement_);
  }
}

Note the use of is_set_ and breakable_ variables in the loop. These ensure the loop terminates as soon as one of the statements (starting at the end of the list) is rewritten, unless the block of statements could potentially contain a break statement.

Assigning Values to the Temporary Variable

Let’s now see how each individual statement is rewritten. First, note that constants like 10 or 20 have type ExpressionStatement when they appear in the FunctionLiteral's statement body.

The following VisitExpressionStatement() method is invoked whenever an ExpressionStatement node is seen:

void Processor::VisitExpressionStatement(ExpressionStatement* node){
  // Rewrite : <x>; -> .result = <x>;
  if (!is_set_) {
    node->set_expression(SetResult(node->expression()));
    is_set_ = true;
  }
  replacement_ = node;
}

Note that the code only has an effect if is_set_ is false, indicating that no value has yet been assigned to .result in the current statement body (working backwards from the last statement, to the first). In this case, SetResult() is called to insert a new Assignment AST node into the AST.

// Returns ".result = value"
Expression* SetResult(Expression* value) {
  result_assigned_ = true;
  
  VariableProxy* result_proxy = 
      factory()->NewVariableProxy(result_);
  
  return factory()->NewAssignment(
      Token::ASSIGN, result_proxy, value, kNoSourcePosition);
}

The NewAssignment() method creates a new Assignment AST node, with a VariableProxy AST node as the left child (the thing being assigned to), and the original expression as the right child.

A More Complex Example: If Statements

Now for a more complex scenario, the case where an IfStatement node appears in the AST. We must handle the case where both a true branch and a false branch are included, but also consider that the false branch is optional, therefore returning undefined.

void Processor::VisitIfStatement(IfStatement* node) {
  // Rewrite both branches.
  bool set_after = is_set_;

  Visit(node->then_statement());
  node->set_then_statement(replacement_);
  bool set_in_then = is_set_;

  is_set_ = set_after;
  Visit(node->else_statement());
  node->set_else_statement(replacement_);

  replacement_ = set_in_then && is_set_ ? 
      node : AssignUndefinedBefore(node);

  is_set_ = true;
}

In this code, we see two calls to Visit(), once for the true (then) case, and once for the false (else) case. In both scenarios, they replace the true or false statement lists with the rewritten lists, assuming anything was actually rewritten. Note how the is_set_ variable is reset back to its original value before evaluating the false case.

The replacement_ variable is conditionally updated depending on whether both the true and false branches generated values. If they weren’t both used, the entire IfStatement AST node is prepended with an assignment of .result = undefined, so as to account for the path where no expression value was assigned to .result.

Finally, the is_set_ flag is always set true at the end, indicating that a value must have been set at this point in the code. Either the value came from one of the two branches, or the if statement will have returned undefined if there was no else case.

Adding the Final Return

To finish up the rewriting process, at the end of Rewriter::RewriteBody, the following code adds a ReturnStatement AST node at the end of the FunctionLiteral's statement list.

Statement* result_statement =
    processor.factory()->NewReturnStatement(result_value, pos);
body->Add(result_statement, info->zone());

Back to Our 1 + 1 Example

Now that we understand the concept of rewriting the AST, and why it’s necessary, let’s return back to our over-arching story — how JavaScript calculates 1 + 1. It was necessary to branch off to give examples using if statements, otherwise it wouldn’t be clear why rewriting the AST was necessary.

If you recall from the previous blog post, we’ve already used constant folding to simplify our 1 + 1 expression into the even simpler 2 expression. Here’s the AST we ended up with:

Notice how V8 converted the simple expression into a function definition (with no arguments or properties), implying that the FunctionLiteral class is the basic unit of compilation.

Next, the Rewriter class modifies our code to:

function() {
  let .result = 2
  return .result
}

which gives us the following AST:

This function is now ready for compilation, and eventually execution. The result can be displayed on the console, or returned from the eval command.

Next Time…

In the next blog post, we’ll start with this rewritten AST, then generate byte codes. There’s a lot of complexity in generating byte codes, so it’ll surely be another detailed blog post.

This is the fourth part in a series that dives into how the V8 JavaScript Engine computes the expression 1 + 1. This may seem like a simple task, but it utilizes a large portion of the JavaScript run-time environment. In previous blog posts, we saw:

• Part 1 — How the 1 + 1 string is stored in the JavaScript heap.

• Part 2 — How byte codes are cached to avoid unnecessary compilation.

• Part 3 — How the string 1 + 1 is scanned into lexical tokens.

Now, in Part 4 we’ll learn how 1 + 1 is parsed to validate it against the official JavaScript grammar, with the goal of creating an in-memory Abstract Syntax Tree (AST).

The ECMAScript Grammar

As you probably know, the JavaScript language is officially defined by the ECMAScript Standard, also known as ECMA-262. However, unless you’ve studied the standard you probably didn’t know that chapters 12–15 spend roughly 200 pages (out of 860) defining the syntax of the language using a detailed Context Free Grammar.

• Chapter 12 — Provides the grammar for all the possible expressions in the JavaScript language, including an extensive list of mathematical and logical operators.

• Chapter 13 — Provides the grammar for statements (such as if, then, and while) and declarations (such as let and const).

• Chapter 14 — Provides the grammar for function and class definitions.

• Chapter 15 — Provides the grammar for top-level scripts and modules.

If you study these chapters, you’ll see the entire definition of the grammatical syntax of the language, describing the valid ordering of tokens in a JavaScript program. However, there’s nothing in these chapters to describe the actual meaning of a valid program. Instead, the semantics of the language is described in the later chapters of the specification (and not discussed in this blog post either).

The Grammar for 1 + 1

Let’s take a quick journey through JavaScript’s syntactical grammar to see how 1 + 1 will be parsed, at least on the theoretical level (we’ll later see how it’s done in V8). The grammar starts with the Script symbol, initiating a long chain of grammar rules used to derive a valid JavaScript program.

For the sake of illustration, we’ll only show the sequence of rules relevant when parsing 1 + 1. As with any context free grammar, the symbols on the right hand side are either terminals, representing tokens in our language (such as 1 or +), or are non-terminals, which can be recursively replaced using other rules.

• Script ::= ScriptBody

• ScriptBody ::= StatementList

• StatementList ::= StatementListItem | StatementList StatementListItem

• StatementListItem ::= Statement | Declaration

• Statement ::= ExpressionStatement

• ExpressionStatement ::= Expression ;

• Expression ::= AssignmentExpression | Expression , AssignmentExpression

• AssignmentExpression ::= ConditionalExpression

• ConditionalExpression ::= ShortCircuitExpression

• ShortCircuitExpression ::= LogicalORExpression

• LogicalOrExpression ::= LogicalANDExpression

• LogicalANDExpression ::= BitwiseORExpression

• BitwiseORExpression ::= BitwiseXORExpression

• BitwiseXORExpression ::= BitwiseANDExpression

• BitwiseANDExpression ::= EqualityExpression

• EqualityExpression ::= RelationalExpression

• RelationalExpression ::= ShiftExpression

• ShiftExpression ::= AdditiveExpression

• AdditiveExpression ::= AdditiveExpression + MultiplicativeExpression

• MultiplicativeExpression ::= ExponentiationExpression

• ExponentiationExpression ::= UnaryExpression

• UnaryExpression ::= UpdateExpression

• UpdateExpression ::= LeftHandSideExpression

• LeftHandSideExpression ::= NewExpression

• NewExpression ::= MemberExpression

• MemberExpression ::= PrimaryExpression

• PrimaryExpression ::= Literal

• Literal ::= NumericLiteral

• NumericLiteral ::= 1 (this part was abbreviated)

That’s a long chain of rules for a simple expression, but it’s even longer for realistic programs. We’ve only seen the relevant rules for our 1 + 1 example. You’re strongly encouraged to click on the hyperlinks (above), and you’ll see a larger set of rules, including operators such as &&, &, |, ||, *, or +.

The Abstract Syntax Tree (AST)

One of the goals of parsing the input tokens is to create an Abstract Syntax Tree. This provides the compiler with an in-memory representation of the program, which is much easier to manipulate than a one-dimensional sequence of tokens. The AST is optimally designed so that tree-traversal algorithms can validate the input program, optimize the program structure, and eventually generate byte codes.

The AST is an n-ary tree, with each node representing some portion of the input program. Nodes have various properties describing the program in more detail (for example, variable names or integer values), and usually have child nodes to reflect the nesting structure of the program.

In V8, all of the AST nodes are defined as C++ classes in src/ast/ast.h, with AstNode being the parent class for all other node types. Each class contains internal fields for decorating the node (with variable names, or literal values), as well as pointers to child AST nodes.

The following diagram shows a portion of the class hierarchy for AST nodes. Note that Statement is a sub-class of AstNode, with each of its sub-classes representing a different type of JavaScript statement. As we’ll see later, we’ll be using ExpressionStatement in our 1 + 1 example.

Here’s another portion of the class hierarchy, focusing on the Expression class, which is also a sub-class of AstNode. Each of the sub-classes represents a different type of JavaScript expression. We’ll be using the Literal class in our example.

The AST For 1 + 1

For our particular 1 + 1 example, the AST is quite simple, with only three nodes involved: FunctionLiteral, ExpressionStatement, and Literal.

At the top of the AST is the FunctionLiteral node. Our expression doesn’t look like a function, but V8 has wrapped the expression this way to make byte code compilation more feasible. In our case, the function has zero formal parameters, with zero JavaScript object properties expected.

At the second level is an ExpressionStatement node, which is the single item in a list of Statement nodes referenced by the FunctionLiteral's body field. If this was a more typical function, we’d expect to see multiple Statement nodes in that list.

Finally, the ExpressionStatement's expression_ field refers to the one-and-only expression node. In our case, the 1 + 1 expression will be “folded” into the single small integer value of 2 (we’ll see this shortly).

To help understand how each of these AST nodes is represented in memory, here’s a heavily abbreviated (and annotated) version of the Literal C++ class, which is capable of storing literal values of any type (numbers, strings, booleans etc).

class Literal final : public Expression {

public:

  // All the possible literal types
  enum Type {
    kSmi,
    kHeapNumber,
    kBigInt,
    kString,
    kBoolean,
    kUndefined,
    kNull,
    kTheHole,
  };

  // Return the type of this Literal
  Type type() const { ... }

  // Methods for testing the Literal type
  bool IsNumber() const { ... }
  bool IsString() const { ... }

  // Methods for fetching the Literal value in various ways
  AstRawString* AsRawPropertyName() { ... }
  Smi AsSmiLiteral() { ... }
  double AsNumber() { ... }
  AstBigInt AsBigInt() { ... }
  AstRawString* AsRawString() { ... }
  bool ToBooleanIsTrue();
  bool ToBooleanIsFalse() { ... }
  bool ToUint32(uint32_t* value) const;

private:

  // C++ union for storing the value.
  union {
    const AstRawString* string_;
    int smi_;
    double number_;
    AstBigInt bigint_;
    bool boolean_;
  };
};

The basic structure of Literal includes a type-tag to distinguish the various possible values, followed by a range of accessor functions for fetching the value itself. Finally, the private section of the class allows for storage of each type of value. For the full source code, see src/ast/ast.h.

Zone Memory Allocation

One interesting fact we’ve glossed over in our discussion of AST nodes is exactly where in memory they’re stored. We already know about the JavaScript heap (from Part 1 of this series), but that’s not where the AST nodes are stored. Instead, the concept of a Zone comes into play.

The Zone class provides very fast allocation of small blocks of memory, such as AST nodes. However, rather than using a garbage collection algorithm, or explicit delete calls to deallocate the memory, the entire zone is discarded at once. This makes sense when think of an AST being created incrementally, being held in memory for a period of time, and then being discarded when the AST is no longer required. There’s no concept of partially deallocating an AST.

To allocate blocks of memory from the Zone, the AstNodeFactory class (using the “factory” pattern) provides convenience methods for creating different types of AST node. For example, here’s part of the ExpressionFromLiteral() method that recognizes the next scanner token as a Token::SMI, fetches the integer value from that token, then creates a new Literal node with that integer value.

...
case Token::SMI: {
  uint32_t value = scanner()->smi_value();
  return factory()->NewSmiLiteral(value, pos);
}
...

Now that we’ve seen the ECMAScript grammar, and we’ve learned how the AST is constructed, we have enough knowledge to step through the process of parsing 1 + 1. As we learned in Part 3, this input will be streamed from the scanner to the parser as a sequence of Token::SMI (value 1), Token::ADD, and Token::SMI (value 1).

Recursive Descent Parsing of 1 + 1

To construct an AST, the V8 JavaScript Engine uses the Recursive Descent parsing technique. This is one of the easiest parsing algorithms to understand, as it simply uses recursive method calls to emulate the chain of grammar rules. That is, each non-terminal in our syntactical grammar is mapped to a C++ method. When called, the method looks ahead at the next scanner token (or tokens) to decide how to proceed, recursively calling other C++ methods as it descends through the grammar.

Let’s see how this is done in our 1 + 1 example.

Recursively Parsing Downwards

We’ll start our journey from within the CompileTopLevel() method, which calls upon ParseProgram() to convert the sequence of scanner tokens into an AST. To achieve this goal, the method initializes the scanner, then recursively invokes DoParseProgram() (we’ll see a lot of recursive calls in this discussion!).

...
scanner_.Initialize();
FunctionLiteral* result = DoParseProgram(isolate, info);
...

If we think back to the ECMAScript grammar we saw earlier, the DoParseProgram() method is equivalent to the ScriptBody rule. In particular, it includes code for parsing a StatementList, which is a sequence of zero or more StatementListItem items:

...
while (peek() != end_token) {
  StatementT stat = ParseStatementListItem();
  if (impl()->IsNull(stat)) return;
  if (stat->IsEmptyStatement()) continue;
  body->Add(stat);
}
...

At the end of the DoParseProgram() method, there’s code for creating a new FunctionLiteral AST node, which incorporates our list of Statement nodes. This new node should look familiar, as it’s the first of three nodes we’re expecting to see in our final AST.

result = factory()->NewScriptOrEvalFunctionLiteral(... body ...);

To continue recursively, the ParseStatementListItem() method is called. This method, like many others we’re about to see, uses a consistent pattern of checking the next scanner token, then determining which additional method to call recursively. For example, if Token::CLASS is the next token in the input stream, we consume that token and recursively call ParseClassDeclaration().

...
switch (peek()) {
  case Token::FUNCTION:
    return ParseHoistableDeclaration(...);

  case Token::CLASS:
    Consume(Token::CLASS);
    return ParseClassDeclaration(...);

  case Token::VAR:
  case Token::CONST:
    return ParseVariableStatement(...);

  case Token::LET:
    return ...

  case Token::ASYNC:
    return ...

  default:
    break;
}

/* none of the tokens match, continue along rule chain */
return ParseStatement(...)

In our simple 1 + 1 example, none of these look-ahead tokens are relevant, so we continue to the default case of calling ParseStatement(). In fact, this pattern continues for a while with more recursive calls:

• ParseExpressionOrLabelledStatement()

• ParseExpressionCoverGrammar()

• ParseAssignmentExpressionCoverGrammar()

• ParseConditionalExpression()

• ParseLogicalExpression()

This list should look remarkably similar to the ECMAScript grammar we saw earlier. Perhaps the only discrepancy is that some rules are very similar, such as ExpressionStatement and LabelledStatement, so they’re merged into a single ParseExpressionOrLabelledStatement() method capable of handling both.

Once we reach ParseBinaryExpression(), the pattern changes slightly because many different grammar rules, from LogicalOrExpression down to ExponentiationExpression, are merged into a single C++ method. As we’ll discuss shortly, this is handled by passing in a precedence parameter, and using precedence rules to group sub-expressions together as appropriate.

Finally, we continue down the grammar rule chain by calling more C++ methods:

• ParseUnaryExpression()

• ParsePostfixExpression()

• ParseLeftHandSideExpression()

• ParseMemberExpression()

• ParsePrimaryExpression()

Parsing the Literals

Once we reach the ParsePrimaryExpression() method, the recursion ends. We no longer have any more rules to apply, and we’re now ready to identify 1 as a literal value. Here’s the relevant part of the code:

...
if (Token::IsLiteral(token)) {
  return impl()->ExpressionFromLiteral(Next(), beg_pos);
}
...

Within the ExpressionFromLiteral() method, there’s code for fetching the token’s SMI value (1), and then using the AST factory to create a new Literal AST node, as we saw earlier.

...
case Token::SMI: {
  uint32_t value = scanner()->smi_value();
  return factory()->NewSmiLiteral(value, pos);
}
...

The return value from this code has type Expression *, which will now be passed back up the C++ call stack.

Parsing of Binary Expressions With Precedence

As we’ve now reached the bottom of the call stack, and we’re returning from the long list of recursive methods, it’s not just a simple matter of immediately returning. Each method must decide whether it’s appropriate to consume more input tokens, or whether its job is complete and returning to the caller is the correct approach.

To see why this matters, consider the expression: 1 + 2 * 3. If we’ve already seen 1 + 2, the question to be asked is whether 1 + 2 is a complete expression, or whether there’s more to evaluate. Clearly it should evaluate 2 * 3 first, before adding 1, so we need the consume the remaining tokens to form the binary expression 2 * 3 before returning to produce the binary-expression 1 + (2 * 3).

To state this another way, consider the ECMAScript rule we saw earlier:

AdditiveExpression ::= AdditiveExpression + MultiplicativeExpression

Given our example, we need to fully evaluate MultiplicativeExpression, before “returning” to the previous method to evaluate the AdditiveExpression.

Inside V8, this is all done with the ParseBinaryExpression() method, which expects the operator precedence as an argument. When parsing of the expression has completed, the precedence value of the next token is checked to see if it’s higher. If so, parsing continues at the same level in the grammar, in this case by calling ParseBinaryContinuation().

Here’s the relevant code:

    ...
    x = ParseUnaryExpression();
  }

  int prec1 = Token::Precedence(peek(), accept_IN_);
  if (prec1 >= prec) {
    return ParseBinaryContinuation(x, prec, prec1);
  }
  return x;
}

In our 1 + 1 example, ParseBinaryExpression() was called with prec equal to 6, and since the precedence of + is 12, we decide that prec1 > prec and the expression should continue at the same level in the grammar. We therefore call ParseBinaryContinuation() rather than returning up the stack.

Constant Folding

Another thing that ParseBinaryContinuation() does is attempt to “fold” constants, simplifying the expression at compile time. We already have a Literal for the first 1, the operator +, and a second Literal for the second 1. We then call ShortcutNumericLiteralBinaryExpression() with all of these values to see if they can be folded into a single Literal.

Here’s the relevant code:

...
if ((*x)->IsNumberLiteral() && y->IsNumberLiteral()) {
  double x_val = (*x)->AsLiteral()->AsNumber();
  double y_val = y->AsLiteral()->AsNumber();

  switch (op) {
    case Token::ADD:
      *x = factory()->NewNumberLiteral(x_val + y_val, pos);
      return true;
    ...

To summarize, if both of the expressions are literal numbers, then we add them together and replace them with a single Literal containing the sum.

Finishing Up

We’ve now seen the entire expression, with no further input tokens remaining, except for the final Token::EOS marking the end of the input stream. As each recursive C++ method returns, it checks the next token but only sees Token::EOS, causing it to return immediately.

Although this is fairly routine by now, there’s one interesting case worth mentioning. In the ParseExpressionOrLabelledStatement() method, the Literal AST node, with value of 2 is wrapped in an ExpressionStatement AST node:

return factory()->NewExpressionStatement(expr, pos);

Finally, as we saw earlier, a FunctionalLiteral AST node is created, wrapping the ExpressionStatement. This leads us back to the AST diagram we’ve been expecting to see:

Next Time…

We’ve now seen a large part of the process involved in parsing 1 + 1, including allocation of data on the JavaScript heap, caching of byte codes, scanning of tokens, and now parsing against the ECMAScript grammar. In the next blog post, will continue by examining how 1 + 1 (actually, now 2) is restructured into a function, which will later be converted to byte codes.

This is the third part of a multi-part series describing how the V8 JavaScript Engine calculates 1 + 1. If you haven’t read the previous posts in this series, you might like to start with Part 1 (storing the source code string in the JavaScript heap), and Part 2 (checking if the byte codes are already cached). However, since this blog post is fairly independent from the previous two, you can likely understand it in isolation.

In this part of our story of how V8 calculates 1 + 1 , we’ll learn how the input characters are scanned into tokens, which are then used as input to JavaScript’s parser. This concept will be familiar to anyone who has read an introductory compiler text book.

The Scanning Process

With our 1 + 1 example, the output from the scanner we’re expecting to see is the following sequence of tokens:

Token::SMI (value 1)
Token::ADD
Token::SMI (value 1)

Where Token::SMI is a special variant of Token::NUMBER representing small integer values, and Token::ADD unsurprisingly represents addition. Note also, the white space characters between 1, +, and 1 have been ignored, since they don’t provide any further value.

Here’s the overall flow of V8, as it scans and parses the input stream:

The first step is for the v8::internal::Utf16CharacterStream class to read the individual characters from the JavaScript heap (as seen in Part 1, the string is stored as a SeqOneByteString object). Next, the v8::internal:Scanner class converts sequences of characters into tokens (of type v8::internal::Token). Finally, the v8::internal::Parser class (which we’ll examine in a later blog post) uses these tokens to validate the input stream and build an in-memory Abstract Syntax Tree (AST).

What’s important to understand is that all these activities happen in a streaming fashion. It all starts when the Parser requests the next token, which causes the Scanner to request the next character (or characters) from the Utf16CharacterStream, which in turn reads the input string from the JavaScript heap. As with any stream-based solution, the temporary storage space requirement is minimal, as tokens are not scanned until they’re actually required downstream.

Let’s look at this process in more detail.

The v8::internal::Utf16CharacterStream Class

The Utf16CharacterStream class is responsible for reading Unicode characters from the input stream, then providing them one-by-one to the Scanner class. This seems like a trivial exercise, but as we’ll see there are interesting edge-cases to consider. First, the scanner might need to look ahead into the input stream before deciding on what the current token should be. Second, the scanner doesn’t know (or care) where the characters come from, or how they’re stored in memory.

Utf16CharacterStream Methods

Let’s look at some of the Utf16CharacterStream methods, to understand how this class is used.

• stream->Advance() — This method returns the next Unicode character in the input, fully removing that character from the input stream. This is the standard behaviour you expect when reading characters in a sequence.

• stream->Peek() — This method returns the next Unicode character in the stream, but without actually consuming that character. Therefore, it’ll still be available when Peek() or Advanced() is next called. This is useful for looking ahead into the input, without committing to consume the character until you know for sure it’s part of the current token.

• stream->AdvanceUntil(func) — Continually read (aka “use up”) characters until the func function returns true. This is useful for consuming characters until a certain point is reached, such as the end of line.

• stream->Back() — Essentially the opposite of Advance(). Returns the character back to the input stream so it’s available for future Peek() or Advance() calls. This is useful when the scanner tried to read ahead, yet decided the next character wasn’t actually part of the current token.

There are several more methods available, but these are the most important. As we’ll see when we examine some of the methods in the Scanner class, the ability to read ahead, yet also push back characters is vital for correctly scanning input tokens.

Utf16CharacterStream is Abstract

A second interesting discussion is how the characters are retrieved from memory. It turns out that Utf16CharacterStream is an abstract class, with a range of different implementations available, each focusing on a specific storage layout of the source string. In our case, the 1 + 1 string was stored on JavaScript’s heap using one byte to store each character. Other options include reading from two-byte strings, as well as from strings that are stored externally from the JavaScript heap.

Selection of the appropriate Utf16CharacterStream sub-class is performed within the ParseProgram method (see src/parsing/parsing.cc) . ParseProgram does many different things, but as far as scanning is concerned the most relevant line of code is:

std::unique_ptr<Utf16CharacterStream> stream(
    ScannerStream::For(isolate, source));

Using our 1 + 1 example, theScannerStream::For method examines the source string and determines it has type SeqOneByteString (one-byte string, stored in the JavaScript heap). The code then returns an instance of the BufferedCharacterStream class, which is the particular sub-class of Utf16CharacterStream capable of reading SeqOneByteString objects.

The most interesting part of the BufferedCharacterStream sub-class is the ReadBlock() method. This method is called upon by higher-level methods such as Peek() or Advance() to fetch the next block of characters from the input stream, however it may be stored.

Let’s now move forward in the scanner pipeline to learn how tokens are represented in V8.

The v8::internal::Token Class

Similar to other compilers, the v8::internal::Token class provides an enumeration of all token values recognized by the scanner. The definition of these tokens (see src/parsing/token.h) is managed by a clever C++ macro (TOKEN_LIST) containing a list of all tokens, combined with a second macro (T) which extracts the name portion:

Here’s the definition of the token enumeration, using C++ macros (warning: it’s not very easy to read):

#define T(name, string, precedence) name,  
   enum Value : uint8_t { TOKEN_LIST(T, T) NUM_TOKENS };
#undef T

And here is the definition of TOKEN_LIST, abbreviated from code in src/parsing/token.h.

#define TOKEN_LIST(T, K) \
    T(TEMPLATE_SPAN, nullptr, 0) \
    T(TEMPLATE_TAIL, nullptr, 0) \
    T(PERIOD, ".", 0) \
    T(LBRACK, "[", 0) \
    T(QUESTION_PERIOD, "?.", 0) \
    T(LPAREN, "(", 0) \
    T(RPAREN, ")", 0) \
    T(RBRACK, "]", 0) \
    ...

When macros are expanded, this becomes a more readable enumeration.

enum Value : uint8_t {
   TEMPLATE_SPAN,
   TEMPLATE_TAIL,
   PERIOD,
   LBRACK,
   QUESTION_PERIOD,
   LPAREN,
   RPAREN,
   ...
   ADD,
   ...
   SMI,
   ...
   WHITESPACE,
   UNINITIALIZED,
   REGEXP_LITERAL,
   NUM_TOKENS
}

As we’ll see later, token values are referenced in the Scanner and Parser classes using the syntax Token::SMI or Token::ADD.

The v8::internal::Scanner Class

Let’s now dive into the internals of the Scanner class, responsible for reading characters from the input, and generating tokens for the output. We’ll see examples of scanning operators (such as +), as well as scanning numbers (such as 1).

Scanner Methods

First, here are some interesting methods from the Scanner class. They are somewhat similar to the methods on the Utf16CharacterStream class, but operate on whole tokens, rather than individual characters.

• scanner->Next() —Return the next token from the input stream, and advanced the input pointer. Naturally, this calls the stream->Advance() method to fetch multiple characters from the Utf16CharacterStream, but will only return a single token value. In our 1 + 1 example, each token is only one character long, but normally this isn’t the case.

• scanner->peek() — Peek ahead to see what the next token will be (beyond what Next() returns) without advancing the input. This is used by the parser to check the upcoming tokens to decide whether or not the current parser rule is matched by the input.

• scanner->PeekAhead()— Peek even further ahead, necessary for parsing some of JavaScript’s more complicated syntax.

• scanner->location() — Return the location of the current token. This provides the start and end character positions within the source string.

• scanner->smi_value() — Returns the Smi (small integer) value of the current token, if any. In our example, this will return the integer 1 for both of our Token::SMI tokens.

And as you might expect, there are many other Scanner methods, mostly focused on error handling, but also for fetching the token’s associated value. As a sneak peek, here’s a small section of the Parser code using these methods:

...

if (peek() == Token::PERIOD && PeekAhead() == Token::PRIVATE_NAME) {
    Consume(Token::PERIOD);
    Consume(Token::PRIVATE_NAME);
    ...
}
...

We’ll learn more about the Parser class in the next blog post, but this code snippet gives you a sense of how Parser calls upon Scanner to return the upcoming token values.

The TokenDesc Structure

Although we’ve already discussed the token enumerated values, allowing us to write Token::SMI (for the number 1) or Token::ADD (for the + symbol) that’s only part of the what’s required for the scanner to represent a token. In addition, the scanner cares about the token’s location, the literal characters, any possible error cases, and of course, the actual numeric value of the token.

To store all this extra information, the scanner uses a TokenDesc structure:

struct TokenDesc {
  Location location = {0, 0};
  LiteralBuffer literal_chars;
  LiteralBuffer raw_literal_chars;
  Token::Value token = Token::UNINITIALIZED;
  MessageTemplate invalid_template_escape_message = 
      MessageTemplate::kNone;
  Location invalid_template_escape_location;
  uint32_t smi_value_ = 0;
  bool after_line_terminator = false;
}

The fields are:

• location — The numeric start and end positions within the source string. For example, our first Token::SMI is at position 0, and our Token::ADD is at position 2.

• literal_chars — These are the actual characters that make up the token, whether it be a number, a string, an identifier, or something else. This is important, since knowing that total_cost is a Token::IDENTIFIER is only part of the story. In addition, we also need the identifier’s name (total_cost) to distinguish it from other Token::IDENTIFIER values.

• raw_literal_chars — The is similar to literal_chars but is used for template literals. In this case, we don’t want escape sequences (e.g. \064) to be replaced by the corresponding character (e.g. 4), but instead require that the raw literal characters be passed to the template function.

• token — The token’s enumerated value, as before.

• invalid_template_escape_message / invalid_template_escape_location — If there’s an error discovered while scanning a token, these fields store the error code and location.

• smi_value_ — In the case of Token::SMI, this field stores the number’s actual integer value. This is returned by scanner->smi_value().

• after_line_terminator — Indicates whether the token appears as the first token of a new line. This is useful for automatically inserting semicolons.

Now we understand all the building blocks, let’s continue along the scanner pipeline, to see how characters are actually converted into token values.

Example: Scanning for Operator Tokens

At this point, we’re ready to trace through the operation of scanning 1 + 1. Since scanning the operator is easier than scanning the numbers, let’s start by seeing what scanner->Next() does when the upcoming input character is the + sign.

The bulk of the scanning mechanism is in the private ScanSingleToken() method. The scanning starts with a very simple lookup into the one_char_tokens[128] array, which contains a direct mapping from the first 128 Unicode characters (aka ASCII characters) to a corresponding “guess” of what the token might be. Here’s the GetOneCharToken() method, used to populate the one_char_tokens[128] array:

constexpr Token::Value GetOneCharToken(char c) {
   return
      c == '(' ? Token::LPAREN :
      c == ')' ? Token::RPAREN :
      c == '{' ? Token::LBRACE :
      c == '}' ? Token::RBRACE :
      c == '[' ? Token::LBRACK :
      c == ']' ? Token::RBRACK :
      c == '?' ? Token::CONDITIONAL :
      c == ':' ? Token::COLON :
      ...
      c == '+' ? Token::ADD :
      ...
}

In our example, the character + is mapped to Token::ADD. However, this is only a guess. What if the actual input was ++ or +=, which are both legal tokens in JavaScript? To handle this, the code looks ahead to see if the following character is also a + (in which case Token::INC is returned), or perhaps an = (returning Token::ASSIGN_ADD). If neither case is true, the original Token::ADD is returned.

Here’s the code in question:

case Token::ADD:
  // + ++ +=
  Advance();
  if (c0_ == '+') return Select(Token::INC);
  if (c0_ == '=') return Select(Token::ASSIGN_ADD);
  return Token::ADD;

Note that the Advance() method reads the next character into local variable c0_, and Select is short-hand for consuming that character, and returning the specified token.

Let’s now see the more complicated case of scanning numbers.

Example: Scanning for Number Tokens

When scanning the string 1, the process starts the same way as before. We look up the character in the one_char_tokens[128] array, which provides Token::NUMBER as the initial guess. The code then immediately calls the ScanNumber method to look more deeply at the characters appearing in the input stream.

case Token::NUMBER:
    return ScanNumber(false);

Here’s a breakdown of how ScanNumber works:

• Line 752 — A decision is made about whether the number starts with a decimal point (the . character). If this is true, the number contains only a fractional portion (such as .123), so further scanning is delegated to the ScanDecimalDigits() method. In our 1 + 1 example, we don’t take this code path.

• Line 762—Next, we check whether the first character in the number is a 0. If so, we check the following character for x, o, or b, delegating to ScanHexDigits(), ScanOctalDigits(), or ScanBinaryDigits() respectively. However, if the next character was actually an octal digit (from 0 to 7), then ScanImplicitOctalDigits() is called to handle cases such as 077 (as opposed to the more explicit 0o77). Finally, a number like 088 (out of range for an octal) is treated as a regular decimal 88.

• Line 797 — Since our input string (1 + 1) does not start with a 0 digit, we consider the number to be a decimal (base 10), possibly containing underscores (e.g. 1_000)

• Line 803 — Given that most number literals are small, we take our chances and scan the number as a Smi (small integer), delegating the work to ScanDecimalAsSmi(), returning the value as a C++ uint64_t.

• Line 807 — A Smi must be small enough to fit into 31-bits. If this is possible, we set the smi_value_ field of the token to the integer’s value, then return Token::SMI. This is the path taken in our 1 + 1 example.

• Line 819 — If the number wasn’t a Smi, continue to parse the decimal number, calling ScanDecimalDigits() to do so. Additionally, we check for a trailing decimal point (the . character), followed by another decimal number (the fractional part). Note that this code simply validates the number is well-formed, rather than extracting the actual value itself (as we did in the Smi case).

• Line 833 — This is where we handle the BigInt scenario, where the number has a trailing n. For example, 12345678n.

• Line 848 — Finally, we handle the exponent case, where the number has a trailing e following by the exponent. For example: 123e5. This is not relevant in our 1 + 1 case, since a Smi can’t have an exponent.

And then ends the scanning process for operators and numbers in JavaScript, due to the complexity of multi-character operators, different bases (hex, octal, binary), underscores used as digit separators, fractional portions, BigInts, and exponents, the whole process is quite complicated.

Next Time…

In the next blog post, we’ll continue our story of calculating 1 + 1. Now that we have a sequence of tokens (Token::SMI, Token::ADD, and TOKEN:SMI) we can see how the Parser class validates the input against the JavaScript language definition. Finally, we’ll see how an AST (Abstract Syntax Tree) is created as an in-memory representation of our program.

This is the second part of a multi-part series describing how the V8 JavaScript Engine calculates 1 + 1. It’s a very simple expression with an obvious answer, but it still requires the full mechanism of scanning and parsing the input string, generating and executing byte codes, then displaying the result, all while maintaining data on the JavaScript heap.

If you haven’t read it, it’s important you start with Part 1, although it’s also recommended you come with a passion for compiler technology.

Last Time…

Last time we saw our example client program, including how it calls C++ methods in the standard V8 libraries. Our program stores the literal string of 1 + 1 in the JavaScript heap (as a SeqOneByteString object), then compiles the expression to byte code, executes that byte code, then displays the result on the console.

// Create a string containing the JavaScript source code.
Local<String> source = String::NewFromUtf8Literal(isolate, "1 + 1");

// Compile the source code.
Local<Script> script = 
    Script::Compile(context, source).ToLocalChecked();

// Run the script to get the result.
Local<Value> result = script->Run(context).ToLocalChecked();

// Convert the result to Number and print it.
Local<Number> number = Local<Number>::Cast(result);
printf("%f\n", number->Value());

In the first blog post we traced the full String::NewFromUtf8Literal() method, so this time we’ll continue with the Script::Compile() method:

Local<Script> script = 
    Script::Compile(context, source).ToLocalChecked();

Script::Compile() is responsible for a large number of activities:

1. Checking the Compilation Cache to see if the same script had already been compiled before. This saves us from repeatedly generating byte codes for commonly used scripts.

2. Scanning the input string into Tokens. As we’ll see, 1 + 1 is converted to a sequence of token values: Token::SMI (small integer), Token::ADD, and then a second Token::SMI.

3. Parsing the tokens into an Abstract Syntax Tree (AST), providing an in-memory view of the program.

4. Generating the corresponding V8 byte codes, while performing some amount of optimization.

The return value from Script::Compile() is a Local<Script> handle, referring to byte codes to be executed by the V8 virtual machine. For now though, we’ll focus exclusively on the first step above. That is, checking if the compiled code is already available in a cache.

It shouldn’t come as a surprise that almost all JavaScript code is downloaded multiple times, either in the same browser session, or in different sessions over a period time. To avoid recompiling source code that hasn’t changed, V8 provides two purpose-built cache mechanisms. The first is the per-Isolate cache, storing compiled byte codes directly in V8’s local memory. The second approach allows embedder applications (such as Chromium or NodeJS) to save their own copy of the compiled byte codes, most likely in a disk-based format. Let’s look at each approach.

Approach 1 — The Per-Isolate Cache

The per-Isolate cache is built into V8, and is enabled by default. In V8 terminology, an Isolate is an instance of a JavaScript virtual machine, complete with its own heap memory. When V8 is embedded into applications, such as a browser, it’s common to use different V8 Isolates as a means of separating (aka “isolating”) one JavaScript run-time environment from another. Perhaps the best example is browser tabs, where code running inside one tab must not impact the code in other tabs.

When a script is submitted to an Isolate for compilation, the source code string (such as 1 + 1) is used as a key for an in-memory hash table. If that exact source code had been compiled before, a SharedFunctionInfo object, containing the script’s byte codes, is read from the cache and returned to the caller. However, if there’s a cache miss, the script must be compiled from scratch, with the generated byte codes inserted into the cache for next time.

The per-Isolate cache (in the CompilationCache class, see src/codegen/compilation-cache.h) is not just a simple hash table, but has a number of features catering for different types of script. For example, the LookupScript() and PutScript() methods cache “normal” JavaScript source code, delegating their work to the CompilationCacheScript class. In contrast, the LookupEval() and PutEval() methods manage the cache of JavaScript strings passed into the eval() function, delegating their work to the CompilationCacheEval class. Likewise, there are sub-caches for regular expressions (regexes) and other code objects.

In addition, each sub-cache in the the per-Isolate cache has multiple generations, allowing older cached items to be aged out over time if they haven’t been used recently. There has clearly been a lot of thought and optimization put into the design of this in-memory cache system.

Here’s an example of how the compilation cache is laid out in memory, showing the hierarchical hash tables:

To see the per-Isolate cache in action, enter 1 + 1 into the d8 interpreter multiple times:

$ ./out/x64.debug/d8 --print-bytecode
V8 version 8.8.0 (candidate)
d8> 1 + 1
... lots of output given, including byte codes ...
2
d8> 1 + 1
2
d8> 1 + 1
2

As expected, large amounts of compilation output is generated the first time (thanks to the --print-bytecode flag), but no byte codes are generated the second (or third) times. In contrast, if you were to specify the --no-compilation-cache command-line flag, you’ll instead see the code being recompiled every time.

Approach 2 — Caching Byte Codes in the Embedder

There are several limitations of the per-Isolate cache mentioned above. In particular, an in-memory cache will not survive when the application restarts (such as shutting down your browser). Additionally, the cache is not shared between different instances of V8, implying that a web page loaded in one browser tab does not share the cache with other browser tabs.

To solve these issues, a second type of cache is available. As discussed in Code caching, the application can request that V8 provide a serialized version of the compiled code, which is saved in the application’s own cache (such as the Chromium browser cache). This serialized data is passed back to the application using the GetCacheData() method of the Source object (see include/v8.h) then saved in the application’s own cache.

When the application attempts to compile the same script again, such as when a web page downloads the same .js file multiple times, the browser passes the CachedData back to V8 to avoid regenerating the byte codes.

The clear advantage is that code can be cached for long periods of time, even if the application restarts. However, the downside is that byte codes must be serialized (see the CodeSerializer class) from V8’s in-memory format to a sequence of bytes more suited for on-disk storage. At a later point in time, this serialized data must be deserialized again before it can be executed. All of this requires extra time, slightly negating the value of caching byte codes in the first place.

Because of this extra overhead, V8 only serializes the data the second time it’s compiled, ensuring it’s not just a one-time script that will never be seen again. Also, V8 defers that serialization work until after the code has been executed, ensuring the serialization does not diminish the user’s experience.

Tracing the Code — Making the Cache Decision

To see how these caching techniques fit into our big picture of computing the 1 + 1 expression, let’s walk through the full code path. As mentioned earlier, we start by calling upon V8’s Script::Compile() method with 1 + 1 as an input parameter. Although this method initiates the entire compilation process, we’ll only look at how the caching mechanisms are involved.

Local<Script> script = 
    Script::Compile(context, source).ToLocalChecked();

As we saw in the first blog post, this calls into V8’s API layer (see src/api/api.cc) to validate the input arguments, add a few more important values (such as the pointer to the Isolate object), as well as translate between external Local handles and their corresponding V8 internal objects.

Before too long, we reach the Compiler::GetSharedFunctionInfoForScript() method, which is where the caching decisions are made (see src/codegen/compiler.cc). Here are the basic steps that are followed:

1. Line 2647 — One of the parameters for GetSharedFunctionInfoForScript() is compile_options, specifying how the embedder cache should be used. If the caller passes kConsumeCodeCache as the value for compile_option, V8 is asked to consider using the serialized byte codes that were saved in the embedder’s cache (available in the cached_data parameter). In our case though, this defaulted to kNoCompileOptions, indicating that no serialized data is available.

2. Line 2655 — For tracking purposes, we record the number of bytes loaded and compiled for this isolate. There are 5 bytes in 1 + 1.

3. Line 2659 — We must take JavaScript’s language mode into account, since it impacts code generation and therefore the byte codes that are cached. The options are kSloppy and kStrict, representing the traditional JavaScript syntax, versus the newer strict mode.

4. Line 2676 — Regardless of whether the embedder provided a cached_data parameter, we check whether the source code is already cached in the per-Isolate cache. This dives into the CompilationCache::LookupScript() method, which delegates to the CompilationCacheScript::LookUp() method in the “script” sub-cache. Eventually, that code performs a lookup in the multi-generational hash table. Checking this cache (even if we were passed cached_data by the embedder) is super fast, given that the byte codes are already in V8’s memory.

5. Line 2686 — Given that our 1 + 1 script had not previously been compiled and cached in V8’s memory, we now consider using cached_data from the embedder. In our example though, we weren’t given any cached_data by our embedder (our simple example program), so neither of the caches provide a hit. However, if cached_data had been provided, we’d need to deserialize it into the in-memory format, and then insert it into V8’s per-Isolate cache.

6. Line 2727 — Given that neither of our caches contained the pre-compiled byte codes for 1 + 1, we now proceed to compile the source code. This is done by the CompileScriptOnMainThread() method. As we’ll see in the next blog post, this is where all the complexity of scanning, parsing, and code generation takes place.

7. Line 2736 — If the compilation was successful, the SharedFunctionInfo object (containing the generated byte codes) is inserted into the per-Isolate cache, ready for the next time that 1 + 1 is evaluated.

So, that’s an overview of the V8 code caching mechanism. If you’re interested, there are several really great articles and presentations from the V8 team on how caching works, including the very comprehensive Code caching for JavaScript developers and BlinkOn 9: Caching (more) JavaScript code in Chrome.

Next Time…

In Part 3 of this blog post series, we’ll continue by tracing further along into the Script::Compile() method. That is, we’ll learn more about how V8’s lexical scanner reads a sequence of input characters (in our case, 1 + 1) and forms them into tokens to use as input into the parsing process.

Yes, obviously the answer is that 1 + 1 = 2, but how does the V8 JavaScript Engine compute this answer?

Stepping back for a moment… one of my favourite interview questions is:

“When I type a URL into a web browser, and then press enter, what are all the things that happen to load the web page? Go into as much detail as you like”.

It’s a great question, because it shows the depth and breadth of a person’s understanding, allowing them to illustrate which parts of the process are the most interesting for them.

This is the first in a series of blog posts that will look at everything V8 does when 1 + 1 is entered. To start with, we’ll focus on how V8 stores the 1 + 1 string in its heap memory. It sounds simple, but it’s an entire blog post on its own!

The Client Application

To compute 1 + 1, the most likely approach you’d take is to fire up NodeJS, or the Chrome developer console, and simply enter 1 + 1. That will work, but in order to show the V8 internals, I’ve decided to modify hello-world.cc, one of the standard sample applications in the V8 source code.

I took the original code, which printed “Hello World”, and replaced it with the expression 1 + 1:

// Create a string containing the JavaScript source code.
Local<String> source = String::NewFromUtf8Literal(isolate, "1 + 1");

// Compile the source code.
Local<Script> script = 
    Script::Compile(context, source).ToLocalChecked();

// Run the script to get the result.
Local<Value> result = script->Run(context).ToLocalChecked();

// Convert the result to Number and print it.
Local<Number> number = Local<Number>::Cast(result);
printf("%f\n", number->Value());

Take a quick look at this code to get an idea of what it does. The lines of C++ might be fairly cryptic, but the comments will help. For this blog post we’ll focus exclusively on the first statement, allocating a new 1 + 1 string in the V8 heap:

Local<String> source = String::NewFromUtf8Literal(isolate, "1 + 1");

To understand this code, let’s start with the high-level sequence of V8 modules involved. In this diagram, the flow of execution is from left-to-right, with the return value being passed back from right-to-left to be inserted into the source variable.

• Application — This represents the client of V8. In our example, it’s the hello-world.cc program, although more realistically it could be the entire Chrome Browser, the NodeJS run-time system, or any other piece of software that embeds the V8 JavaScript Engine.

• V8 External API — This is the client-facing API providing access to the functionality of V8. Although it’s implemented in C++, the API is shaped around various JavaScript concepts, such as numbers, strings, arrays, functions, and objects, allowing them to be created and manipulated in various ways.

• Heap Factory — Internal to the V8 Engine (not exposed via the API) is a “factory” for creating various data objects on the heap. Quite surprisingly, the set of factory methods available is very different to what the external API provides, so a lot of translation is done within the API layer.

• New Space — V8’s heap is very complicated, but newly-allocated objects are usually stored in the New Space, often known as the Young Generation. We won’t cover the detail here, but the new space is managed using Cheney’s Algorithm, a well-known algorithm for performing garbage collection.

Let’s now go into more detail on this flow, focusing on:

• How the API layer decides what type of string to create, and where it should be stored within the heap.

• What the string’s internal memory layout will be. This depends on the range of characters appearing in the string.

• How the storage is allocated from the heap. In our example, 20 bytes are required.

• Finally, how the pointer to the string is returned to the application, with the goal of garbage collecting in the future.

Determining How and Where to Store the String

As mentioned above, there’s a fair amount of translation that must happen between the client application and the heap factory, where the object is actually created. The bulk of this work is performed in src/api/api.cc.

Let’s start with the client application’s call:

String::NewFromUtf8Literal(isolate, "1 + 1");

The first argument is for an “Isolate” which is V8’s main internal data structure representing the state of the run-time system, isolated from other possible V8 instances. To understand this, think about having multiple browser windows open, where each window has a completely separate instance of V8 running, each with its own isolated heap. We won’t talk about the isolate argument much, other than noting that a very large number of API calls expect this parameter.

The String::NewFromUtf8Literal method (see src/api/api.cc) starts with basic string length checking, but also decides how to store the string in memory. Given that we only provided two arguments to the call, the third type argument defaults to NewStringType::kNormal, indicating the string should be allocated as a regular object on the heap. The alternative would have been to pass NewStringType::kInternalized indicating that de-duplication of the string is desired. This feature is very useful to avoid storing multiple copies of the same constant string.

The next step is to call the NewString method (see src/api/api.cc) which invokes factory->NewStringFromUtf8(string). Note that string here has been mapped into an internal Vector data structure, instead of a regular C++ string, because the heap factory has quite a different set of methods than the external API. This difference will become more apparent later when the return value is passed back to the client application.

Inside NewStringFromUtf8 (see src/heap/factory.cc), a decision is made on the optimal storage format for the string. Naturally, UTF-8 is a convenient format for storing a wide range of Unicode characters, but when only basic ASCII characters are used (such as1 + 1) V8 stores the string in “one byte” format. To make this decision, the string’s characters are passed into Utf8Decoder decoder(utf8_data) (declared in src/strings/unicode-decoder.h) .

Now that we’ve decided to allocate a one-byte string, using the normal (not internalized) approach, the next step is to invokeNewRawOneByteString (see src/heap/factory-base.cc), where the heap memory is allocated, and the string’s content is written into that memory.

The String’s In-Memory Structure

Inside V8, our 1 + 1 string is represented as an instance of the v8::internal::SeqOneByteString class (see src/objects/string.h). If you’re like most object-oriented developers, you’d expect that SeqOneByteString would have a number of public methods, as well as several private members, such as an array of characters or an integer storing the string’s length. However, that’s not the case! Instead, all internal object classes are actually just pointers to the heap where that data is stored.

As you can see from the code comment in src/objects/objects.h, there are roughly 150 internal classes that have the common parent class of v8::internal::Object. Each of these classes consists solely of a single 8-byte value (on a 64-bit machine) referring to the object’s heap location.

Keeping this in mind, here’s what our string looks like in memory:

There are many interesting parts here:

The SeqOneByteString Object

As mentioned, this is not a fully-fledged string class, but is instead a pointer to the on-heap location of the string’s actual content. On a 64-bit machine, this “pointer” will be an 8-byte unsigned long, with the type alias of Address. Note that the data on the heap (on the right side of the diagram) is not actually a real C++ object, so there’s no point in treating this Address as if it was a pointer to something strongly-typed (such as String *)

But, you might be wondering why the additional level of indirection exists in the first place. Why not simply access the heap block directly? This approach makes sense when you consider that garbage collection can result in objects being moved around the heap. It’s important that data can move, without the client application getting confused.

To clarify, in Generational Garbage Collection, objects are first allocated in the Young Generation (New Space), and if they survive long enough, they’ll be moved to the Old Generation (Old Space). To make this work, the garbage collector copies the heap block to the new heap space, then updates the Address value to point to the new memory location. Given that the SeqOneByteString object itself is still at exactly the same memory address as before, the client software won’t notice the change.

Compressed Pointer To Map (Bytes 0–3 of Heap Block)

JavaScript is a dynamically-typed language, which means that variables don’t have types, yet the values stored in those variables do have types. The “map” is V8’s way of associating each object in the heap with a description of the object’s data type. After all, if the object wasn’t tagged with its type, the heap block becomes a meaningless sequence of bytes.

We won’t go into much detail about the map for our 1 + 1 string, other than mentioning that maps are also a type of heap object, stored in the Read Only Space. Maps (also known as Shapes or Hidden Classes) can become very complex, although our constant string uses a pre-defined map by calling read_only_roots().one_byte_string_map() (see src/heap/factory-base.cc).

Interestingly, although this map field is a pointer to another heap object, it cleverly uses Pointer Compression to store a 64-bit pointer value in a 32-bit field.

Object Hash Value (Bytes 4–7 of Heap Block)

Every object has an internal hash value, but in this example it defaults to kEmptyHashField (value of 3) to indicate the hash is not yet computed.

String Length (Bytes 8–11 of Heap Block)

This is the number of bytes in the string (5).

The Characters and the Padding (Bytes 12–19 of Heap Block)

As you’d expect, the five single-byte characters are stored next. Additionally, to ensure that future heap objects are aligned based on the CPU’s architecture requirements, an additional three bytes of padding are added (aligning the object to a 4-byte boundary).

Allocating Memory From the Heap

We briefly mentioned that the factory class allocates a block of memory from the heap (20 bytes in our case), then fills that block with the object’s data. One remaining question is how that 20 bytes is allocated.

In Cheney’s algorithm for garbage collection, the Young Generation (New Space) is divided into two semi-spaces. To allocate a block of memory in the heap, the allocator determines if there are enough free bytes between the limit of the current semi-space, and the current top of that semi-space. If there’s enough room, the algorithm returns the address of the next block, then increments the top pointer by the requested number of bytes.

This basic case is shown here, showing the before and after states of the current semi-space:

If the current semi-space were to run out of free memory (top and limit get too close), then the collection portion of Cheney’s algorithm starts. Once collection is complete, all the live objects will have been copied to the beginning of the second semi-space, and all dead objects (remaining in the first semi-space) will be discarded. No matter what, a semi-space is guaranteed to have all its used space at the bottom, and all its free space at the top, so it’ll always look like the above diagram.

In our case though, there’s plenty of free memory in the current semi-space, so we carve off 20 bytes, then increase the top pointer. There’s no need for garbage collection, and the second semi-space isn’t involved. In the V8 code, there are numerous special cases to consider, but the final allocation of 20 bytes is handled by the NewSpace::AllocateFastUnaligned method in src/heap/new-spaces-inl.h.

Returning a Handle

Now that we have a pointer to a heap block, fully populated with the string’s content (including length, hash, and map) the pointer must be returned to the client application. If you recall, the client invoked this line of code:

Local<String> source = String::NewFromUtf8Literal(isolate, "1 + 1");

But, what exactly is the type of source, and what does Local<String> actually mean? There are two key observations here:

Translating Internal to External Classes

First, it’s interesting to recall that V8 stored our string object using the v8::internal::SeqOneByteString class, which is simply a pointer to the data on the heap. However, the client application expects the data to be of type v8::String, which is part of the V8 API.

What may surprise you is that v8::internal::SeqOneByteString (a subclass of v8::internal::String) is in a completely different class hierarchy than v8::String. In fact, all of the internal classes are defined in the src/objects directory using the v8::internal namespace, whereas the external classes are defined in include/v8.h using the v8 namespace.

Revisiting the NewFromUtf8Literal method we discussed earlier (see src/api/api.cc), the very last step before returning the object pointer to the client application is to cast the result from a v8::internal::String to a v8::String.

return Utils::ToLocal(handle_result);

This conversion magic is done using macros defined in src/api/api-inl.h.

Managing the “Roots” for Garbage Collection

Second, let’s discuss what Local<String> means (which incidentally is an abbreviation for v8::Local<v8::String>). The Local concept is how we deal with garbage collection of the string object when it’s no longer needed.

As any JavaScript developer will know, objects are garbage collected when there are no remaining references to them. The collection algorithm starts at the “roots”, then traverses the entire heap to find all reachable objects. A root is a non-heap reference, such as a global variable, or a stack-based local variable that’s still in scope. If these variables are assigned new values, or if they were to go out of scope (their enclosing function ends), the data they once pointed to is potentially now garbage.

In the case of thehello-world.cc program, we also have pointers on the C++ stack that can refer to heap objects. These have no corresponding JavaScript variable name, since they only exist in the context of the C++ application (such as hello-world.cc, or Chrome, or NodeJS). For example:

Local<String> source = ...

In this case, source is a reference to a heap object, although now with an additional level of indirection. This diagram will explain:

On the left side is the C++ stack, which grows from top to bottom as the program executes, and the right side is the heap memory we saw earlier. As the client application executes, it pushes a HandleScope object onto the local C++ stack (see src/samples/hello-world.cc). Next, the return value from calling String::NewFromUtf8Literal() is stored on the C++ stack as a Local<String> object.

It looks like we’ve add yet another level of indirection, but there are benefits of doing this:

• Finding roots is easier — The HandleScope object is a place to store “Handles” (aka pointers) to heap objects. As you recall, this is exactly what our SeqOneByteString object was, an 8-byte pointer to the underlying heap data. When garbage collection is initiated, V8 quickly scans the HandleScope object to find all the root pointers. It can then update those pointers if the underlying heap data is moved.

• Local pointers are easy to manage — In contrast to HandleScope which is quite large, the Local<String> object is an 8-byte value on the C++ stack, which can be used in the same context as any other 8-byte value, such as pointers or integers. In particular, it can be stored in CPU registers, be passed to functions, or provided as return values. What’s notable is that the garbage collector is not required to locate or update these values when garbage collection occurs.

• Eliminating scopes is easy — Finally, when the C++ function in the client application finishes, the HandleScope and Local objects on the C++ stack are removed, but only after their C++ object destructors have been called. These destructors remove all the handles from the garbage collector’s list of roots. They’re no longer in scope, so the underlying heap objects may have become garbage.

To conclude the story, the source variable, referring to our 1 + 1 string is now ready to be passed to the next line in our client application:

Local<Script> script = 
    Script::Compile(context, source).ToLocalChecked();

Next Time…

There was clearly a lot of work to allocate the 1 + 1 string on the heap. Hopefully it illustrated some parts of V8’s internal architecture, as well as how data is represented in different parts of the system. In future blog posts, I’ll look more into how our simple expression is parsed and executed, which will expose a lot more about how V8 operates.

In Part 2 of this blog post series, I’ll dig into how the Compilation Cache works, to avoid compiling code more than necessary.

Given how widespread the V8 JavaScript Engine has become, being a major part of Google Chrome, Microsoft Edge, and NodeJS, it’s obviously important to test it carefully. In this blog post, I’ll summarize the different test suites included with the V8 source code.

If you’re following along at home, you’ll find these test suites in the v8/test directory within the V8 source code repository. Each subdirectory within v8/test is considered a test suite if it contains a testcfg.py file (not all of them do), although I excluded a few suites that don’t seem to do much. Each suite can be invoked with the ./tools/run-tests.py command.

% ls -1 test/*/testcfg.py

test/benchmarks/testcfg.py
test/cctest/testcfg.py
test/debugger/testcfg.py
...
test/test262/testcfg.py
test/unittests/testcfg.py
test/wasm-api-tests/testcfg.py
test/wasm-js/testcfg.py
test/wasm-spec-tests/testcfg.py
test/webkit/testcfg.py

% ./tools/run-tests.py --outdir=out/x64.release benchmarks

Build found: /Users/peter_smith/CompilerProjects/v8/out/x64.release
>>> Autodetected:
pointer_compression
>>> Running tests for x64.release
>>> Running with test processors
[00:06|% 100|+  55|-   0]: Done                                               
>>> 55 base tests produced 55 (100%) non-filtered tests
>>> 55 tests ran

We’ll do a quick tour of all 15 test suites in the v8/test directory:

• benchmarks — Standard performance tuning benchmarks.

• test262 — Conformance tests against the ECMAScript specification.

• mjsunit — Unit tests written in JavaScript.

• cctest/ unittests— C++ unit tests for internal V8 classes.

• fuzzer — Input fuzzer tests providing invalid input, possibly crashing V8.

• intl — Tests for Internationalization features of ECMAScript.

• message — Validates error messages produced by invalid JavaScript code.

• webkit — Test cases borrowed from the WebKit JavaScript Engine.

• mozilla —Test cases borrowed from the Mozilla JavaScript Engine.

• wasm-js — Validation of WebAssembly, using the JavaScript API.

• wasm-api-tests — Validation of WebAssembly, using the C++ API.

• wasm-spec-tests — Conformance to the WebAssembly specification.

• inspector — Validates the V8 inspector interface (for debugging)

• debugger — Validates the built-in debugger command.

If you refer to my previous blog post on building V8 from source code, you’ll know that run-tests.py is invoked by the gm.py build script. All of the test suites depend on binary executables first being compiled. Many suites use the d8 executable (a simple JavaScript command shell) for executing JavaScript programs and validating the results. However, other test suites such as the code-level unit tests, require a special purpose test driver.

Let’s dig into the detail…

Test Suite: benchmarks

• Run time: 32 seconds (single threaded with run-tests.py -j 1 on a 2015 MacBook Pro)

• Test binary: d8

This first test suite is focused on three performance-tuning benchmarks. The goal of each benchmark is to provide a comparison between different JavaScript engines when faced with typical code scenarios, such as processing JSON input, decompressing data, or rendering graphics. The competing JavaScript engines (such as V8, JavaScriptCore, or SpiderMonkey) are evaluated side-by-side to show how quickly they can compile and evaluate each benchmark. As a result, a lot of time has been spent on optimizing V8 to out-perform the competing JavaScript engines.

Unfortunately, experience shows that relying too much on specific benchmarks leads to over-fitting of the optimizations, with too much emphasis placed on the exact benchmark code. More recently, effort has been put into optimizing for real-world scenarios that are more representative of a web browser’s overall workload. For example, by observing the loading time for common applications such as Facebook, or Google Maps, optimizations will be more applicable to everyday use.

Inside the v8/test/benchmarks directory, there is code for three important benchmarks, each having their own unique origin:

• The “SunSpider” Benchmark — Originally created by Apple in 2007, as part of their WebKit project, the SunSpider benchmark focuses on intensive algorithms such as cryptography, string manipulation, and ray tracing. According to their website, this benchmark is no longer supported (as of 2015) and has been replaced by the JetStream benchmark.

• The “Kraken” Benchmark — Created as part of the Mozilla project in 2010, the Kraken benchmark also focuses on complex algorithms that were extracted from real-world workloads (but are not the full workload itself). Kraken still appears to be maintained, and can even be executed in your browser.

• The “Octane” Benchmark — First released by Google in 2012, and then retired in 2017, the Octane benchmark similarly focuses on computationally complex algorithms. It can also be executed inside a browser.

Let’s now look at how V8 is tested for conformance against the ECMAScript specification.

Test Suite: test262

• Run time: 37 minutes (also single threaded, on a 2015 MacBook Pro)

• Test binary: d8

The official name for the JavaScript language is ECMAScript, where JavaScript is more of a marketing name. The ECMA262 standard provides an exact specification of the language and standard libraries. All JavaScript engines are required to conform to this standard, with all major browser vendors being involved in the ECMA262 committee (known as TC39). Obviously, loading a web page into Chrome must have the same effect as loading it into Firefox, Safari, or Edge, so conforming to this specification is vital.

To help with this conformance, a test suite known as Test262 has been created. This is a browser-agnostic test suite maintained by supporters of the ECMA262 standard. Test262 contains test cases to validate the ECMAScript language and libraries, the Internationalization API, and the JSON Data Interchange Format. Although the maintainers claim there’s always room for improvement, Test262 does an excellent job of validating conformance to ECMA262.

At the language level, every aspect of the specification is covered, including grammar definition, expressions, statements, modules, and pretty much everything else. To be specific, Test262 contains 43665 individual JavaScript test files, resulting in 74677 test cases that are run through the d8 interpreter. On my machine, executing these test cases took 37 minutes, using a single CPU core.

As an example, the following test case validates the new Optional Chain feature in JavaScript. In the header comment, a reference is made to the exact part of the ECMAScript specification, showing how optional chains can appear within loops (in this case, within a for-in statement):

/*---
esid: prod-OptionalExpression
description: >
  optional chain in test portion of do while statement
info: |
  IterationStatement
    for (LeftHandSideExpression in Expression) Statement
features: [optional-chaining]
---*/

const obj = {
    inner: {
        a: 1,
        b: 2
    }
};

let str = '';
for (const key in obj?.inner) {
    str += key;
}

assert.sameValue('ab', str);

When this .js file is passed into V8 (specifically the d8 executable), the code snippet executes, with the final assert.sameValue validating whether the behaviour was correct or not.

Test Suite: mjsunit

• Run time: 3 minutes 37 seconds

• Test binary: d8

The mjsunit suite is similar to Test262, although was specifically written for V8 rather than being browser-agnostic. There are 5068 test cases implemented in .js or .mjs files, taking several minutes to execute.

As an example, the function-arguments-duplicate.js file validates V8’s behaviour when the same parameter name is used twice in the same function.

function f(a, a) {
  assertEquals(2, a);
  assertEquals(1, arguments[0]);
  assertEquals(2, arguments[1]);
  assertEquals(2, arguments.length);
  %HeapObjectVerify(arguments);
}

f(1, 2);

The familiar assertEquals function is available (as are many other matchers), and this particular code shows the %HeapObjectVerify function which is built-in to V8, but not very well documented.

Test Suite: cctest

• Run time: 5 minutes 40 seconds

• Test binary: cctest

This test suite contains almost 7000 unit tests, spread across 246 C++ files. These are code-level unit tests directly invoking methods within the V8 core. As such, a special cctest executable is first compiled, which is in contrast to other test suites relying on the d8 executable to parse and validate JavaScript code.

Each C++ file has one or more test methods, each conforming to theTEST(testName) signature. Methods call the V8 internal classes, then use macros such as CHECK, CHECK_EQ, orCHECK_GE to validate the results. For example, in test-heap.cc:

TEST(InitialObjects) {
   LocalContext env;
   HandleScope scope(CcTest::i_isolate());
   Handle<Context> context = v8::Utils::OpenHandle(*env);

   // Initial ArrayIterator prototype.
   CHECK_EQ(
      context->initial_array_iterator_prototype(),
      *v8::Utils::OpenHandle(*CompileRun("[][Symbol.iterator]
      ().__proto__")));

   ...
   
   // Initial Object prototype.
   CHECK_EQ(context->initial_object_prototype(),
      *v8::Utils::OpenHandle(*CompileRun("Object.prototype")));
}

If a test fails, a stack trace is displayed, making it easy to debug the problem.

The run-tests.py script allows invocation of individual test cases. For example: ./tools/run-tests.py cctest/test-code-pages/* runs the seven test methods in test-code-pages.cc, whereas ./tools/run-tests.py cctest/test-code-pages/OptimizedCodeWithCodePages invokes only that single test case.

These unit tests are clearly designed with developers in mind. Each test provides concise examples of how to call the V8 APIs, as well as the internal methods and data structures. I’ve found these particular test cases to be invaluable for learning the V8 internals. I’m sure I’ll be writing more about them in the future.

Test Suite: unittests

• Run time: 2 minutes 58 seconds

• Test binary: unittests

The unittests suite is very similar to the cctest suite, providing 3763 test cases spread across 237 different C++ source files. It’s actually not clear if there’s any fundamental difference between the two suites, although perhaps the distinction is purely historical.

Test Suite: fuzzer

• Run time: unknown

• Test binary: many (see below)

The fuzzer test suite allows for fuzz testing of the input passed into V8. These tests randomly modify valid JavaScript programs, surgically generating invalid inputs in the hopes of crashing the V8 engine. This doesn’t just cause a JavaScript-level exception, but could instead cause corruption in the actual C++ code, possibly allowing for security breaches.

For example, to identify a potential bug in V8’s expression evaluation, the fuzzer modifies only that part of the code, yet provides valid input for the remainder of the program. Starting with valid code:

function f(a, b) {
  console.log(a + b)
}

the fuzzer creates an erroneous program:

function f(a, b) {
  console.log(a + %)
}

All portions of this code before, and after, the a + % expression must be valid, otherwise V8 simply rejects the program before reaching the expression evaluation code. For more detail, see how fuzzing is done with Chromium.

This fuzzer test suite generates several different executable programs, to support the range of different input-types that can be fuzzed.

v8_simple_json_fuzzer
v8_simple_multi_return_fuzzer
v8_simple_parser_fuzzer
v8_simple_regexp_builtins_fuzzer
v8_simple_regexp_fuzzer
v8_simple_wasm_async_fuzzer
v8_simple_wasm_code_fuzzer
v8_simple_wasm_compile_fuzzer
v8_simple_wasm_fuzzer

Each of these executables has a main program (a C++ file), taking the fuzzed input and passing it into V8 using the necessary test fixtures, such as providing JSON as input, a regular expression, or WASM code.

int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {

   ... code for calling V8 functions that might crash ...

   return 0;
}

If the C++ function returns 0, the program ran (or was rejected) correctly, but if the fuzz-attack was successful, the V8 engine would have already crashed.

Clearly this type of testing could take a very long time to execute, especially with all the possible ways of mutating the input. Therefore tests are performed on a large test cluster, running for an extended period of time (many hours or days).

Test Suite: intl

• Run time: 9 seconds

• Test binary: d8

This test suite, consisting of 218 JavaScript source files, performs validation of the internationalization features of V8. For example, there are test cases for time and date formats, time zone manipulation, character set collation (sort orders), as well as numeric data formats. Each test case uses JavaScript functions such as assertEquals or assertFalse to validate their results.

Test Suite: message

• Run time: 8 seconds

• Test binary: d8

This test suite provides validation of error messages. Each test case has a single .js (or .mjs) file, and a corresponding .out file sharing the same base file name. For example, here’s the content of arrow-formal-parameters.js, which contains invalid JavaScript code.

(b, a, a, d) => a

and the corresponding arrow-formal-parameters.out file specifies the expected error message when arrow-formal-parameters.js is passed through the d8 interpreter.

*%(basename)s:5: SyntaxError: Duplicate parameter name not allowed in this context
(b, a, a, d) => a
       ^
SyntaxError: Duplicate parameter name not allowed in this context

If the actual output doesn’t match the expected output, the test case is considered a failure. A very simple, yet very effective test suite.

Test Suite: webkit

• Run time: 20 seconds

• Test binary: d8

This test suite is borrowed from the WebKit project, the basis of the Safari Browser. It consists of 543 JavaScript files (with .js suffix), each paired with a corresponding -expected.txt file. Each JavaScript file is passed through the d8 executable, with the actual console output being captured and compared against the expected output.

Test Suite: mozilla

• Run time: unknown

• Test binary: d8

This test suite appears to be a clone of the regression testing for the Mozilla JavaScript tests. According to the repository commits, the snapshot taken from Mozilla is at least five years old, possibly even ten years old.

This repository contains 3481 individual .js files, as well as some .java files! After running the test suite, run-tests.py reported that 1921 test were executed, although it also showed a number of test failures, with a total completion of 0%. I suspect this test suite isn’t actively maintained.

Test Suite: wasm-js

• Run time: 16 seconds

• Test binary: d8

This suite validates the standard WebAssembly object, used for accessing the WebAssembly functionality within V8. There are 94 JavaScript source files, each exercising the WebAssembly object in some way.

Test Suite: wasm-api-tests

• Run time: 1 second

• Test binary: wasm_api_tests

Similar to the previous test cases, these validate the WebAssembly functionality within V8. However, rather than expressing the tests in JavaScript (using the WebAssembly object), they directly call V8’s C++ API. There are 17 such test cases in this suite.

Test Suite: wasm-spec-tests

• Run time: 22 seconds

• Test binary: d8

This third WebAssembly-related test suite provides 190 different JavaScript source files, and the same number of matching .wast files (a human-readable WebAssembly format). Here’s an example of this format:

(module
  (memory 1)
  (data (i32.const 0) "abcdefghijklmnopqrstuvwxyz")
  (func (export "8u_good1") (param $i i32) (result i32)
    (i32.load8_u offset=0 (local.get $i))             ;; 97 'a'
  )
  ...
)

Presumably, these test cases are derived from the WebAssembly specification.

Test Suite: inspector

• Run time: 12 seconds

• Test binary: inspector-test

This suite validates the Inspector Protocol, used by external debuggers (such as Chrome DevTools) to inspect and control the state of the JavaScript engine. There are 282 individual test cases (with .js file extension) paired up with the same number of -expected.txt files. The JavaScript file is executed, and the expected output is compared with the actual behaviour.

For example, here’s the content of the scoped-variables.js test case, showing how snippets of code can be injected into a running V8 engine:

InspectorTest.log('Evaluating \'let a = 42;\'');
var {result:{result}} = await Protocol.Runtime.evaluate({  
    expression:'let a = 42;'});
InspectorTest.logMessage(result);

InspectorTest.log('Evaluating \'a\'');
var {result:{result}} = await Protocol.Runtime.evaluate({
    expression:'a'});
InspectorTest.logMessage(result);

The output of this test run is compared against the expected output in scope-variables-expected.txt :

Evaluating 'let a = 42;'
{
    type : undefined
}
Evaluating 'a'
{
    description : 42
    type : number
    value : 42
}

As you’d expect, if the output doesn’t match, the Inspector Protocol has a failure.

Test Suite: debugger

• Run time: 18 seconds

• Test binary: d8

This final test suite validates the built-in debugger. All 316 JavaScript test files invoke the debugger command (or a similar feature) to halt the program execution. The script then uses the standard debugger features to introspect the state of the program, ensuring that breakpoint debugging works as expected.

Summary

That’s it! A total of 15 different test suites for validating various aspects of the V8 JavaScript engine. Some of the test suites are written in JavaScript, whereas others are written directly in C++. Some of the test suites were written by the V8 maintainers, whereas others were from third-parties. In all, these test suites are a major reason why V8 is such a high-quality and performant product.

This first blog post is an overview of how V8 is compiled. As you can see from the V8 source code repository, the V8 Engine is mostly written in C++, requiring source code to be compiled into executable files. This should be no surprise, given that V8’s primary purpose is fast compilation and execution of JavaScript programs.

I’ll be discussing three main topics related to compiling the V8 executables:

• The gm.py wrapper script, providing a convenient approach to compile V8 from source, and for invoking the test suites.

• The GN meta-build system (invoked by gm.py) taking an easy-to-read description of the software components, then auto-generating a machine-readable build description suitable for the Ninja build tool.

• Finally, the Ninja build tool uses that same machine-readable build description to analyze inter-file dependencies and invoke the relevant compilers.

The earlier diagram (at the top of this blog post) illustrates the overall flow of tool invocation, and shows which files are read, generated, and invoked.

Let’s examine each step in detail. If you’re new to this type of compilation process, I’ll put in a shameless plug for this book on Software Build Systems. It’s been almost ten years since I wrote that book, but the underlying concepts remain the same.

The gm.py Script

The first time you compile V8, you should use the recommended gm.py script to fully compile all the object files, libraries, and executables.

$ ./tools/dev/gm.py x64.release.check

This is described in the V8 documentation as a convenience script because it’s a one-step solution for all the steps you need to get started. It takes about 20 minutes to run to completion (on my MacBook). Here’s what it’s doing:

1. Creating and Configuring the Build Output Directories

Using the best practice that object and executable files should be stored separately from the source code, the gm.py script creates the v8/out/x64.release directory. In this example, we’ve asked for V8 to be compiled for the x64.release target (Intel x86 64-bit for release images), although if you also want to compile for different targets (such as x64.debug or arm64.debug), separate directories would be created for those.

This step also generates the args.gn file in the v8/out/x64.release directory, specifying the build options for this configuration (most notable are the is_debug and target_cpu options).

is_component_build = false
is_debug = false
target_cpu = "x64"
use_goma = false
goma_dir = "None"
v8_enable_backtrace = true
v8_enable_disassembler = true
v8_enable_object_print = true
v8_enable_verify_heap = true

2. Auto-Generating Ninja files from the GN Build Specification

The next step in the build process is for gm.py to invoke the GN tool to translate the human-readable BUILD.gn file into lower-level files for the Ninja tool (with .ninja suffix).

The BUILD.gn file contains easy-to-read directives specifying the content of each build target. In the following example, the d8 executable is constructed from a small number of C++ source files, linked together with additional libraries that contain the core JavaScript engine.

v8_executable("d8") {
  sources = [
    "src/d8/async-hooks-wrapper.cc",
    "src/d8/async-hooks-wrapper.h",
    "src/d8/d8-console.cc",
    ...
    "src/d8/d8.cc",
    "src/d8/d8.h",
  ]

  ...

  deps = [
    ":v8",
    ":v8_libbase",
    ":v8_libplatform",
    ...
  ]
}

Later in this blog post, there’ll be more detail about this file format. For now, let’s look at what happens when the gn gen command generates all the .ninja files from the hand-written BUILD.gn file.

$ gn gen out/x64.release

This results in a collection of roughly 100 .ninja files in the out/x64.release directory. Each .ninja file corresponds to one of the build targets described in the BUILD.gn file.

./toolchain.ninja
./build.ninja
./obj/d8.ninja
./obj/v8_libbase.ninja
./obj/v8_simple_wasm_compile_fuzzer.ninja
./obj/v8_libplatform.ninja
./obj/v8_simple_multi_return_fuzzer.ninja
...
./obj/test/unittests/cppgc_unittests_sources.ninja
./obj/test/unittests/unittests_sources.ninja
./obj/test/wasm-api-tests/wasm_api_tests.ninja
./obj/test/common_test_headers.ninja
./obj/test/cctest/generate-bytecode-expectations.ninja
./obj/test/cctest/cctest_sources.ninja
./obj/test/cctest/cctest_headers.ninja
./obj/test/cctest/cctest.ninja

As an example, here’s the content of the d8.ninja file. At first glance, this output is quite similar to an old-style Makefile — that is, not very readable!.

defines = -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=1160 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIB
CPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -DCR_LIBCXX_REVISION=375504 ...

include_dirs = -I../.. -Igen -I../.. -I../../include -Igen -I../../include -Igen/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../include
cflags = -fno-strict-aliasing -fstack-protector -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -fcomplete-member-pointers -arch x86_64 -Wno-builtin-macro-redefined ...

build obj/d8/async-hooks-wrapper.o: cxx ../../src/d8/async-hooks-wrapper.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-console.o: cxx ../../src/d8/d8-console.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-js.o: cxx ../../src/d8/d8-js.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-platforms.o: cxx ../../src/d8/d8-platforms.cc || obj/d8.inputdeps.stamp
build obj/d8/d8.o: cxx ../../src/d8/d8.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-posix.o: cxx ../../src/d8/d8-posix.cc || obj/d8.inputdeps.stamp
...

Now that we have all the .ninja files, we can start to compile the source code.

3. Using the Ninja Build Tool to Compile the Objects and Executables

The next step in the build process is for gm.py to invoke the C++ compiler (amongst other tools). This is done by invoking the autoninja command, which itself is a wrapper for the ninja command.

$ autoninja -C out/x64.release d8

This command reads the relevant .ninja files, determines which object files are missing (or out of date), then invokes the C++ compiler to create them. This process is familiar to anyone who has used the Make build tool (or similar).

After roughly 20 minutes (on my MacBook), we end up with a fully populated build tree of roughly 2700 files, including auto-generated source files ( .cc and .h suffix), object files (.o suffix), library files (.a suffix), and a small number of executable files:

out/x64.release/obj
out/x64.release/obj/v8_libbase/time.o
out/x64.release/obj/v8_libbase/semaphore.o
out/x64.release/obj/v8_libbase/platform-macos.o
out/x64.release/obj/v8_libbase/condition-variable.o
out/x64.release/obj/v8_libbase/ieee754.o
out/x64.release/obj/v8_libbase/file-utils.o
...
out/x64.release/obj/v8_compiler/effect-control-linearizer.o
out/x64.release/obj/v8_compiler/js-native-context-specialization.o
out/x64.release/obj/v8_compiler/store-store-elimination.o
out/x64.release/obj/v8_compiler/code-assembler.o
...
out/x64.release/gen/torque-generated/src/wasm/wasm-objects-tq-csa.h
out/x64.release/gen/torque-generated/src/wasm/wasm-objects-tq-csa.cc
out/x64.release/gen/torque-generated/src/objects/map-tq-csa.h
out/x64.release/gen/torque-generated/src/objects/code-tq-csa.h
...
out/x64.release/obj/libv8_libplatform.a
out/x64.release/obj/libwee8.a
out/x64.release/obj/third_party/zlib/libchrome_zlib.a
out/x64.release/obj/third_party/icu/libicui18n.a
out/x64.release/obj/third_party/icu/libicuuc.a
out/x64.release/obj/libv8_libbase.a
...
out/x64.release/obj/d8/d8.o
out/x64.release/obj/d8/d8-posix.o
out/x64.release/obj/d8/d8-console.o
out/x64.release/obj/d8/async-hooks-wrapper.o
out/x64.release/obj/d8/d8-js.o
out/x64.release/obj/d8/d8-platforms.o
out/x64.release/d8
...

Everything is now compiled, so I can run the d8 program to execute some JavaScript code:

$ ./out/x64.release/d8 
V8 version 8.6.0 (candidate)
d8> console.log(2 + 2);
4
undefined
d8>

Looking at this example output, you might think that d8 is actually the same thing as NodeJS (and the node command), but it’s actually a simple wrapper around the core V8 libraries. It doesn’t add any of the additional functionality that NodeJS provides, but instead just supports the core JavaScript language. It’s this core library that’s linked into NodeJS, the Chrome browser, and any other software that needs to compile JavaScript.

4. Executing the Unit Tests

The final step of the gm.py wrapper script is to execute the unit tests. This is done using the run-tests.py script.

$ ./tools/run-tests.py --outdir=out/x64.release \
       debugger intl mjsunit cctest message unittests

I plan to talk about V8 testing in another blog post, so I won’t give more detail here. Let’s instead dig deeper into both the GN build tool, and the Ninja build tool.

The GN Meta Build Tool

The GN Build Tool is classified as a “meta build” tool in that it doesn’t actually invoke the C++ compiler directly, but instead converts a human-readable build description into a lower-level format suitable for the Ninja build tool. This concept was popularized by CMake, which (amongst other things) is capable of auto-generating a tree ofMakefile files, to be used by the Make tool

GN Command Line Options

We’ve already seen how GN is used (with the gn gen option) to generate all the .ninja files, but what else can it do? Here are some interesting examples:

First, we can list all the possible build targets for V8:

$ gn ls out/x64.release

//:bytecode_builtins_list_generator
//:cppgc
//:cppgc_base
//:cppgc_for_testing
//:cppgc_for_v8_embedders
//:cppgc_standalone
//:d8
//:fuzzer_support
//:gen-regexp-special-case
//:generate_bytecode_builtins_list
//:gn_all
//:json_fuzzer
//:lib_wasm_fuzzer_common
...

Next, we can show all the compilation flags, input files, and dependent libraries for one of these targets:

$ gn desc out/x64.release //:d8

type: executable
toolchain: //build/toolchain/mac:clang_x64

...

sources
  //src/d8/async-hooks-wrapper.cc
  //src/d8/async-hooks-wrapper.h
  //src/d8/d8-console.cc
  //src/d8/d8-console.h
  //src/d8/d8-js.cc
  //src/d8/d8-platforms.cc
  //src/d8/d8-platforms.h
  //src/d8/d8.cc
  //src/d8/d8.h
  //src/d8/d8-posix.cc
  ...

cflags
  -fno-strict-aliasing
  -fstack-protector
  -fcolor-diagnostics
  -fmerge-all-constants
  ...

defines
  _LIBCPP_HAS_NO_ALIGNED_ALLOCATION
  CR_XCODE_VERSION=1160
  CR_CLANG_REVISION="llvmorg-12-init-1771-g1bd7046e-3"
  __STDC_CONSTANT_MACROS
  __STDC_FORMAT_MACROS
  ...

Direct dependencies
  //:v8
  //:v8_dump_build_config
  //:v8_libbase
  //:v8_libplatform
  //:v8_tracing
  //build/config:executable_deps
  //build/win:default_exe_manifest

Finally, the reverse operation is to show which targets will be built from a specific source file.

$ gn refs out/x64.release //src/d8/d8-platforms.cc

//:d8
//tools/gcmole:v8_run_gcmole

Note that none of these commands actually tell you which targets are currently out of date. As we’ll see later, that’s the responsibility of the Ninja tool.

Understanding BUILD.gn for the “d8” Target

The commands shown above are very useful, but how does GN know about the targets and their dependencies? Let’s spend some time looking at the highlights of the v8/BUILD.gn file. If you want more information on the BUILD.gn syntax, an excellent introductory presentation is also available.

We’ll be looking at how the d8 executable is constructed. This following is the code starting at line 4744 of my copy of BUILD.gn (it’s a long file!). I’ve added the “section” comments to make the code easier to refer to.

# Section 1 - The v8_executable Template
v8_executable("d8") {

  # Section 2 - Defining the Sources
  sources = [
    "src/d8/async-hooks-wrapper.cc",
    "src/d8/async-hooks-wrapper.h",
    "src/d8/d8-console.cc",
    "src/d8/d8-console.h",
    "src/d8/d8-js.cc",
    "src/d8/d8-platforms.cc",
    "src/d8/d8-platforms.h",
    "src/d8/d8.cc",
    "src/d8/d8.h",
  ]

  # Section 3 - Optional Sources
  if (v8_fuzzilli) {
    sources += [
      "src/d8/cov.cc",
      "src/d8/cov.h",
    ]
  }

  # Section 4 - Compilation Configuration
  configs = [
    ":internal_config_base",
    ":v8_tracing_config",
  ]

  # Section 5 - Additional Dependencies
  deps = [
    ":v8",
    ":v8_libbase",
    ":v8_libplatform",
    ":v8_tracing",
    "//build/win:default_exe_manifest",
  ]

  ...
}

Let’s learn some of the main concepts of GN by walking through this example.

Section 1 — The v8_executable Template:

Out of the box, GN provides the executable command for describing how to construct an executable program. For V8, we actually use the v8_executable “template” (a GN concept) that wraps the basic executable command, providing some additional functionality for compiling V8 executables. This template is defined by including import("gni/v8.gni") at the top of the BUILD.gn file. The v8.gni file itself contains this snippet of code:

...

template("v8_executable") {
   executable(target_name) {
     ...
   }
   ...
}

...

This file also contains similar templates for v8_static_library, v8_shared_library, and v8_source_set that build upon the corresponding GN standard commands. In addition, the main BUILD.gn file also contains some template definitions, making the build description more concise by abstracting away the complexity.

Section 2 — Defining the Sources:

To specify the C++ source files to be included in the d8 executable, we define a variable that contains a list of file paths. The GN tool supports a simple programming language, including the concept of variables and values, as well as lists of values.

  sources = [
    "src/d8/async-hooks-wrapper.cc",
    "src/d8/async-hooks-wrapper.h",
    ...
  ]

Note that unlike many build tools, we’re only required to list the file paths. We don’t need to construct file name pattern matching, or specify dependencies between files. The mechanism for doing that is hidden from you in the auto-generated .ninja files.

Section 3 — Optional Sources:

There are numerous build variants for V8, supporting a wide range of host platforms, target CPUs, optimization choices, JavaScript language-level selection, and additional feature libraries. To support all these variants, GN provides an if statement for us to test variables and conditionally modify the list of sources (using sources +=)

  if (v8_fuzzilli) {
    sources += [
      "src/d8/cov.cc",
      "src/d8/cov.h",
    ]
  }

In this particular example, we’re adding support for the Fuzzilli fuzzing tool which requires additional code-coverage functionality.

Section 4 — Compilation Configuration:

To specify additional compilation flags for the d8 target, we make reference to a couple of “configs”:

configs = [
  ":internal_config_base",
  ":v8_tracing_config",
]

Here’s the definition of internal_config_base that appears earlier in the BUILD.gn file.

config("internal_config_base") {
  visibility = [ ":*" ]

  configs = [ ":v8_tracing_config" ]

  include_dirs = [
    ".",
    "include",
    "$target_gen_dir",
  ]
}

A “config” is a way to package together include paths, C++ symbol definitions, compiler flags, and additional libraries. These configs can obviously become quite complex, especially with support for multiple host platforms. But luckily, build targets simply need to reference the config by name, rather than worrying about all of those details.

Section 5 — Additional Dependencies:

Finally, to specify additional source files, or libraries to be linked into the d8 executable, we define the deps variable. Each entry in the list specifies a V8 build target, which itself provides a static/shared library, or a set of source files to include.

deps = [
    ":v8",
    ":v8_libbase",
    ":v8_libplatform",
    ":v8_tracing",
    "//build/win:default_exe_manifest",
  ]

That’s it! A relatively simple way of specifying how to construct the d8 executable, without burdening the developer with the complexities of compilation flags, dependencies, and file pattern matching. There are plenty of other GN commands/directives that we haven’t discussed, but the GN documentation shows them all.

The Ninja Build Tool

The last step in the V8 build process (with the exception of running tests) is to invoke the Ninja Build Tool to generate the object files, libraries, and executables. Given that users aren’t expected to look at the auto-generated .ninja files, there’s no need to look at further examples. However, it’s interesting to learn more about invoking Ninja, and the various command-line options available.

Speed is Everything

One of the interesting selling points of Ninja is its raw speed. Given my extensive history of using build tools like Make, I was very curious about what makes Ninja so responsive. When dealing with hundreds (or thousands) of source files, a lot of build tools will “pause” for 20–30 seconds as they determine which files are out of date. With Ninja, incremental builds seem to start instantly.

Here are some interesting facts about Ninja:

• First, the build description files (with .ninja suffix) are very simplistic. There is no complicated language to be parsed, and no advanced features requiring time to execute. For this reason, the documentation describes the syntax as “machine code”. The .ninja files are also very compact, often with minimal white space. Keeping them small and simple makes them fast to read into memory, and to parse.

• Second, implicit dependencies are stored in a single cache file, the .ninja_deps file (a 2MB binary file on my computer). In a Make-based environment, it’s common to have a unique .d file corresponding to each .cc file to store the list of C++ header files depended-on by the main .cc file. As a result, the build tool parses a very large number of files each time an incremental build is invoked. However, for Ninja, reading the one-and-only .ninja_deps file is extremely fast!

In the case of V8, the ninja tool starts reading build.ninja, which then imports the toolchain.ninja file. It’s this second file that recursively imports all the other .ninja files in the out/x64.release directory (shown earlier). Despite having roughly 100 .ninja files, reading and processing them is very fast.

Ninja Command-Line Options

To finish off, let’s show some of the commonly-used Ninja commands. To list all the available built targets, use the targets command:

$ ninja -t targets

build.ninja: gn
obj/test/common_test_headers.inputdeps.stamp: stamp
obj/test/unittests/unittests.inputdeps.stamp: stamp
cppgc: phony
cppgc_base: phony
cppgc_for_testing: phony
fuzzer_support: phony
generate_bytecode_builtins_list: phony
...
:d8: phony
:fuzzer_support: phony
:gen-regexp-special-case: phony
:generate_bytecode_builtins_list: phony
:gn_all: phony

Naturally, these build targets are similar to what was shown in the BUILD.gn file, and reported by the gn ls command.

To compile a specific target, just mention it on the command line, optionally with the -v flag if you want to see the underlying C++ compiler invocations.

$ ninja -v d8

[1/1506] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF ...
[2/1506] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF ...

...

[1506/1506] ...

To show all of the compilation commands required to build a target, without actually invoking the compiler, use the commands option:

$ ninja -t commands d8

...

Finally, to show where a particular file is used (that is, which libraries or executables depend on it), use the query command:

$ ninja -t query ./obj/v8_libbase/mutex.o

obj/v8_libbase/mutex.o:
  input: cxx
    ../../src/base/platform/mutex.cc
  outputs:
    obj/libv8_libbase.a
    obj/libwee8.a

These are the basics, but for more advanced options, see the Ninja documentation for further detail.

Summary

The V8 JavaScript engine has an excellent build system, comprised of a top-level convenience script (gm.py), which invokes the GN meta-build tool to generate lower-level build description files to be executed by the Ninja build tool. This combination of tools allows developers to work with the human-readable BUILD.gn file format, while allowing for a fast execution of the build steps, especially for incremental build.

Like many scalable SaaS applications, the HighBond platform from Galvanize uses an event-driven architecture. When a state change is made within one service (such as updating a record in the user module), other services are notified via descriptive events posted to a message queue. This allows for loose-coupling of services, leading to better scalability.

In this blog post, we’ll describe our approach for ensuring that events are sent reliably, even in the presence of failure. At Galvanize, we use Ruby on Rails as the main framework for application development, in conjunction with PostgreSQL (via Active Record), and Amazon’s SQS message queue service.

Here’s the basic workflow

1. An HTTP request comes into the web application server (Ruby on Rails).

2. The application makes a transactional state change to its local database (PostgreSQL).

3. A notification event is transmitted to other services, via a message queue (Amazon SQS).

4. An HTTP response is provided back to the user’s browser.

Although we use Rails, PostgreSQL, and SQS, the concepts are quite general and apply to other web development frameworks, database servers, and message queues.

The Problem

Although it appears simple to generate an SQS message whenever there’s a state change in a service, it’s much harder to ensure it’s done reliably, especially in the face of failure. Here’s the most important rule:

An event should be sent to the message queue if, and only if, the state change has successfully been committed to the originating database.

That is, we must avoid having the application change its local database, without the message queue receiving notification of that change. Conversely, the event notification must not be sent if the expected state change didn’t actually happen. Either of these scenarios cause the SaaS application’s services to provide conflicting views of data.

Why is this challenging? Here are three common reasons for failure:

1. The database transaction, within the source application, may fail and be rolled back.

2. The message queue may be temporarily unavailable, most likely due to network glitches.

3. Our application may fail, due to coding issues, or if the underlying operating system becomes unavailable.

Given these potential failures, we came up with several solutions for ensuring atomicity — that is, the data must change consistently throughout the entire system, or none of the changes should occur at all. The system should not allow partial changes to be visible.

Attempt 1 — Send to the Queue From Within the Database Transaction

The first approach we tried, which doesn’t actually work correctly, is to treat the message queue event as part of the database transaction. For example, in Rails we’d use the following syntax:

user.transaction do

  user.update!(salary: user.salary + 100)

  queue.send(user.name, user.salary)

  ... other updates ...

end

In this case, we update the user’s salary in the PostgreSQL database, then send a notification to the message queue indicating the user’s new salary. Once the end statement is reached, the entire database change is committed atomically, which is the expected behaviour from PostgreSQL.

This happy path sequence can be visualized in this way:

Here’s what happens if the transaction fails — that is, if the ... other updates ... cause an error.

However, this solution fails because the event was already sent to the message queue, and other services believe the change took place, even though it didn’t. Not what we wanted!

Attempt 2 — Send to Queue After Transaction Commits

To work around this failed-transaction problem, our second approach sends the event only when the transaction is known to be successful. Our code example is now:

user.transaction do

  user.update!(salary: user.salary + 100)

  ... other updates ...

end

... if we get here, transaction was successful ...
queue.send(user.name, user.salary)

If this code is successful (no rollback exception was thrown), we proceed to send the event on the message queue. There are several ways to do this, but in our case we attached an after_commit hook on the Active Record model. This hook is triggered only when the change is actually committed to the database, but not if it was rolled back.

This sequence can be visualized as:

As you’d expect, the failure case is now handled correctly. If the transaction is rolled back, no update is made to the database and no event is sent on the message queue.

However, this solution isn’t perfect either. What if the database transaction is committed, but for some reason we’re unable to send the event afterwards?

There are two main reasons this might happen:

1. The application might fail immediately after committing the database change, but before sending the event. This could happen due to badly-written code, or if the underlying OS fails. Although this seems unlikely, it’s sure to eventually happen, especially on a busy system.

2. A more common problem is when the message queue is unreachable for a period of time, possibly due to network glitches. In this case, the application should repeatedly try to resend the event, until it succeeds.

In the first failure case, the application server restarts and continues processing HTTP requests. However, at that point in time we’ve already lost track of the state change, with no way to know the event hadn’t been sent. In the second failure case, how long should we retry for? Web page requests are supposed to be fast, and the browser is expecting a quick response, so we can’t retry indefinitely.

Attempt 3 — Send Events in a Delayed Job

The main problem with our previous approach is that events were not durable (stored on disk) if they hadn’t yet reached the message queue. Therefore, any programming logic or system failure made us completely lose track of the event.

We need a solution that guarantees the events are durable whenever the database transaction commits. Luckily, the Rails delayed job mechanism provides this ability.

user.transaction do

  user.update!(salary: user.salary + 100)

  user.delay.send_event()

  ... other updates ...

end

In Rails, addingdelayto your method call implicitly inserts a snippet of code into a special delayed_jobdatabase table, acting as queue of work to be completed at a future time. A separate Rails server repeatedly pulls code from this table, executing it in the background, usually after the original HTTP request has completed.

This solutions looks similar to our previous cases, but with the addition of a delayed job mechanism:

As it turns out, this approach solves all the problems we’ve discussed so far:

1. By creating a delayed job inside a Rails transaction, the delayed job is only scheduled if the database transaction commits. In Rails, delayed jobs are queued for execution by inserting a record into a PostgreSQL table. If the transaction fails, the job is never committed to the table, so the job is never executed.

2. Delayed jobs are durable, so even if the application crashes and restarts, the delayed job is still in the database, and the event is transmitted reliably.

3. If the message queue is unavailable for a period of time, the delayed job can re-execute until it’s successful. Given that it’s running in the background, it will not impact the performance of the end user’s HTTP request. Obviously there must be a limit to the number of retries (it can’t wait forever), but most intermittent failures can be resolved this way.

Naturally, if the database transaction is rolled back, the delayed job won’t be scheduled and the event is never sent:

Is this solution good enough? Yes, it can certainly work, and parts of our system are using this in production. However, we’ve also created a more optimal solution.

Final Solution — Queue the Message Inside the Database, Send it Later

Our final solution is very similar to using delayed jobs, but is more efficient as the system scales. The key problem with delayed jobs is that scheduling a unique job for each event is quite expensive. Ideally we’d like to group multiple events, sending them to the queue as a single batch — a feature provided by SQS to eliminate network round-trips.

Our final solution involves the use of an outbox database table, which is similar to delayed_jobs, but is exclusively for persisting outgoing events. As you’d expect, this table is updated as part of the same database transaction, so changes are rolled back if the main database transaction fails.

Next, instead of using a generic delayed job server that can execute arbitrary code, we have a dedicated outbox_flush loop, focused on grouping events together and posting them to SQS in a batch. By batching into groups of 10 events at a time, the cost of posting to SQS is reduced from 20ms per event (sometimes as high as 50ms), down to an average of 2–5ms per event. In case of failure, the events are always durable, with the flush loop retrying until the events are sent successfully.

Here’s the new architecture:

As before, we have a similar flow of data. We transmit message queue events if, and only if, the main database transaction completes.

To allow this mechanism to be used consistently across our platform, we’ve wrapped this mechanism into a proprietary Ruby gem for reuse within Galvanize’s products.

Conclusion

We’ve shown how to improve scalability and reliability for an event-driven application by combining database transactions with events to be sent on a message queue. There are many reasons for failure, but by integrating events into the main database transaction, making them atomic and durable, we can be confident they will not be lost. We also have the ability to improve our message queue performance by batching together smaller events.

Of course, although these solutions address the problem from the source application’s perspective, there are still plenty of opportunities for failure that we haven’t discussed. Most notably, if the destination service (or services) fail to process the event correctly, we’ll still have a problem with inconsistent data. However, that’s another topic for a different blog post.

I recently had the good fortune of presenting at the Vancouver Amazon Web Services User Group. This monthly event, organized by Onica, is a great opportunity to network with like-minded people in the community, and to discuss AWS-related topics.

In my presentation, I provided an introduction to the Apache Spark analytics framework, and gave a quick demo of using Amazon EMR (Elastic Map Reduce) to perform a few basic queries. Here’s a summary of what was discussed.

Apache Spark — Unified Analytics Engine

Apache Spark has rapidly become a mainstream solution for big data analytics. Numerous organizations take advantage of Spark — processing terabytes of data with the goal of discovering new insights they wouldn’t otherwise have. This includes processing of financial data, analyzing web click streams, and monitoring and reacting to data from IoT sensors.

There are many ways to perform analytics with Spark. When Spark is used in a batch-processing environment, input data is placed into cheap storage (such as Amazon S3). At a later time, a Spark cluster reads the data, performs complex analytics (sometimes taking minutes or hours), then writes the final result to the output. In addition to this traditional batch-processing model, Spark also supports machine learning, real-time streaming analytics, and graph-based analytics.

What makes Spark so powerful is the ability to divide and conquer. Multiple worker nodes are created, with the analytics computation being distributed amongst them. The following diagram illustrates a Spark cluster with four worker nodes (EC2 instances). Input data is stored in S3 files, and then partitioned and shared amongst the workers. The result of the analytic computation can later be written back to another S3 bucket.

In addition to Apache Spark being a well-supported open source framework, with an active user community, AWS makes it trivial to create and manage Spark clusters as part of their EMR (Elastic Map Reduce) offering. More on that later.

Spark is Different from a Relational Database

Although Spark is often used to analyze tables of “rectangular” data (with rows and columns), and it also supports the familiar SQL language, it would be incorrect to refer to Spark as a relational database. In fact, there are numerous key differences between how Spark manipulates data, versus how the same task is performed in a relational database.

To help understand the benefits provided by Spark, let’s discuss these differences.

Programming Languages

Most relational database systems support the SQL language for querying data. In addition, many of these systems also support the concept of stored procedures, allowing user-defined code to execute inside the database. Although stored procedures provide immense value, they’re written in the database’s specific programming language, and are limited to the run-time environment provided by the database.

In the case of Spark, the SQL language is partially supported, but that’s only the starting point. Spark runs on a JVM (Java Virtual Machine) and therefore analytics code can be written in any JVM-based language, such as Java or Scala, providing compatibility with decades of existing code libraries. Additionally, the Python language is fully supported, allowing access to the great libraries and utilities that data scientists know and love.

Scalability

Relational databases can utilize multiple CPU cores, providing excellent vertical scalability. However, many of the advanced features (such as concurrency, locking, and failure recovery) are easier to support if those CPU cores are tightly coupled within a single server host. That is, all the CPUs must share the same memory space and therefore be inside the same physical host.

In the case of Spark, support for distributed computation is of primary importance, allowing a Spark cluster to horizontally scale up to much larger data sets (running on 100s or 1000s of hosts). Of course, the distributed (multi-server) nature of Spark means that concurrency, locking, and failure recovery must be handled very differently than with a centralized database.

Data Storage Formats

Because of the tightly-coupled nature of a relational database, the server has complete control over how data is stored on disk. The operations for querying, inserting, and updating data rows are optimized to use data structures such as B-Trees and WALs. The database user (a human) likely knows nothing about how these data structures work, and will never examine the underlying data files. The complexity of the database is therefore hidden.

In a Spark environment, the data formats are fully under the control of users. Data is read from disk in a generic format, such as CSV, JSON, or Parquet, and the final output is written back to disk in a similar user-selected format.

Read/Write Versus Read-Only

As a result of Spark allowing arbitrary user-chosen disk formats, all reading of input, and writing of output, happens in a user-directed way. Spark doesn’t have control of how data is placed on disk, and therefore isn’t able to insert new data rows, or update individual fields, as you’d often do in a relational database.

Instead, Spark reads the data from the input file into main memory (as much as will fit at one time), then performs the analytic computation. Once the final result is complete, the output is fully written back to disk. The key point is that Spark is not suited for transactional operations where small in-place updates are made to existing data.

Resilience

In a relational database, it’s common to use a master-slave arrangement to recover from failures. The slave server functions in a passive state, simply tracking all the changes made to the master’s data. However, if the master server fails, the slave is promoted to become the new master, with very little downtime.

Spark uses a very different approach — rather than having a hot-backup for each of the worker nodes, any failure results in the failed worker’s computation being repeated again from the beginning (or the latest check point). More specifically, Spark tracks the data’s lineage, so it knows how to regenerate the computation by replaying the same analytic tasks on a different server.

With 1000s of worker nodes, there’s a good chance that one of them will fail and its work must be replayed. Note however, it would be significantly more expensive to have 1000 slaves nodes acting as hot-backups for the 1000 primary worker nodes!

Always-On or On-Demand?

Relational databases run on a 24/7 basis. As new data arrives, or existing data is updated, the server is always up-and-running, and available to receive and store the updates. If you have a large database with lots of CPU power and lots of RAM, the infrastructure costs start to add up.

In a Spark environment, it’s common to collect data (in CSV or JSON format) and immediately place it into cheap storage (such as Amazon S3). If nothing else is done with the data at that point in time, there’s no need for Spark workers to be available. All you pay for is the low monthly cost of data in S3.

However, when it’s time to perform some analytics (for example, at the end of the month, or the fiscal year), we fire up a large Spark cluster with lots of worker nodes. Only at that time is the data read into the cluster, and the intense computation is performed. Once the work is complete, the Spark cluster is shut down to save the infrastructure cost.

A Practical Example

As mentioned earlier, Apache Spark is an open source package, freely available for download. However, there’s still plenty of effort required to configure the worker nodes and install the software. Luckily for us, Amazon EMR makes this trivial, allowing creation of a Spark cluster in a matter of minutes.

Starting the Cluster

The following screenshot shows how little must be done to start up the Spark cluster we showed earlier, with five EC2 instances (one master and four workers) of size m4.2xlarge:

Enter this information onto the single page shown above, then press the “Create cluster” button. In less than five minutes you’ll have a Spark cluster up and running!

Starting an Interactive Session

The next step is to initiate an interactive session using Apache Zeppelin, which is also installed as part of EMR. This configuration is slightly complicated, involving the creation of an SSH tunnel, then running a browser plugin to access the Zeppelin interface via that tunnel. Luckily, EMR provides excellent online help to get this working.

Once connected, you’ll see an interactive console in your browser:

Loading Data into Spark

The next step is to tell Spark how to load the data from S3. In this example, we’re loading a CSV-formatted file that we’d previously stored in an S3 bucket. We’ve chosen to use the Scala programming language, but other languages will be similar:

Querying the Data

We can now perform a query on the data. In this case, we’re computing the year portion of the timestamp values from column _c5. We then count how many records are associated with each of the years, and will display the result for the first ten years.

Summary

The presentation I gave to the Vancouver Amazon Web Services User Group was well-received, with a lot of people showing interest in Spark, EMR, and the general concepts of distributed computation.

Spark has quite a different architecture from a typical relational database, and it’s worth understanding those differences when deciding how to perform your analytics computation. In particular, Spark is well-suited for performing large-scale analytics on read-only data. However, it’s not the correct solution if you need transactional updates made to existing data.

Finally, Amazon’s EMR service makes it really easy (and cheap) to perform large-scale analytics. Why not try it for yourself?

In this article, we’ll discuss the idea of JSON Streaming — that is, how do we process streams of JSON data that are extremely large, or potentially infinite in length. In such cases, the JSON messages are too large to be held entirely in a single computer’s RAM, and must instead be processed incrementally as the data is being read from, or written to, external locations.

More specifically, in this article we’ll talk about streaming JSON data that has a non-trivial structure.

The Problem — Not all JSON is Predictable

The concept of JSON Streaming isn’t new, and numerous methods are documented on Wikipedia. However, most streaming techniques assume the elements of the JSON stream are predictable, and don’t depend on each other.

For example, a JSON stream that reports data from weather stations may consist of a sequence of JSON objects, separated by newline characters. The application reads each line as a separate record, without any need to load the entire data set into RAM. Using this model, we can process GB or TB of JSON data while only using KB of RAM!

{ "stationID": 1234, "temperature": 65, "wind": 12 }
{ "stationID": 1362, "temperature": 20, "wind": 23 }
...
{ "stationID": 1362, "temperature": 19, "wind": 24 }

However, what if the JSON contained multiple sections, with the first section providing meta-data necessary to understand the later sections? We no longer have a repeating pattern, but instead must store and update information in an internal database, as we progress through the JSON stream. In the later sections of the stream, we can refer back to that database to interpret the newly-arriving values.

For example, what if our weather data includes detail of each weather station:

{
  "stations": [
    { "stationID": 1234, "city": "Seattle", "units": "imperial" },
    { "stationID": 1362, "city": "Vancouver", "units": "metric" },
    ...
  ],
  "reports": [
    { "stationID": 1234, "temperature": 65, "wind": 12 },
    { "stationID": 1362, "temperature": 20, "wind": 23 },
    ...
    { "stationID": 1362, "temperature": 19, "wind": 24 }
  ]
}

In this example, we must first read the stations array to determine whether each weather station reports in metric or imperial units. When we later process the reports array, the values for temperature and wind will be scaled appropriately.

The concern here is that the JSON input is no longer trivial or repeating. Instead, some elements of the JSON object depend on values provided in previous parts of the same object. Our example is fairly simple, but imagine a more complicated JSON object structure with more dependencies between them. The basic JSON streaming approaches mentioned in Wikipedia are simply not going to help.

Before we discuss solutions, it’s worth mentioning an important assumption. Experts will note that JSON objects are an unordered collection of key/value pairs. For our purposes, however, we need to assume the stations key appears earlier in the stream than the reports key. The software generating the JSON stream must abide by this rule.

Let’s look at the architecture of how this can be solved.

The Big Picture

The following diagram illustrates our overall solution for reading a stream of JSON, while maintaining derived information as we progress through the input. The two main components we should focus on are the Tokenizer and the State Machine.

In this model, the input is a sequence of text characters (streamed from a file, or from a network connection), which is tokenized into the basic building blocks of a JSON object (such as StartOfObject or StringValue — more on these later). We then use a state machine to traverse the JSON object’s structure and pull out the interesting values. As we progress through the JSON object (by transitioning between states), we update the database accordingly. Finally, the transformed data is sent to the output.

Let’s look at those steps in more detail.

The Tokenizer

This component of our pipeline reads a continuous stream of characters from the input. Its job is to group the input characters into meaningful atomic tokens. To illustrate, let’s revisit our earlier example:

{
  "stations": [
    { "stationID": 1234, "city": "Seattle", "units": "imperial" },
    { "stationID": 1362, "city": "Vancouver", "units": "metric" },
    ...
  ],
  ...
}

In this example, the Tokenizer outputs the following stream of tokens:

StartOfObject
FieldName("stations")
StartOfArray
StartOfObject
FieldName("stationID")
NumberValue(1234)
FieldName("city")
StringValue("Seattle")
FieldName("units")
StringValue("imperial")
EndOfObject
...
EndOfArray
...
EndOfObject

If you read carefully through the stream of input characters, you’ll see a one-to-one mapping with the tokens sent to the output. We record the type of each token (such as FieldName), along with an optional data value (such as units).

It’s important to remember that this stream of tokens could be infinitely long, simply because the stream of input characters might be infinitely long. In reality, any JSON object that’s too large to fit into RAM is a candidate for this approach.

Streaming software generally reads input characters in small batches (for example, 4KB-8KB at a time). Although you might intuitively feel that streamed data should be processed one character at a time, that would be highly inefficient — we instead read a full disk block, or read a full network packet each time. When we run out of characters, we ask for the next block. Our memory footprint is therefore proportional to the size of an input block (such as 4KB), rather than the size of the entire JSON object.

Because of the way the token stream is created, we can also be confident the JSON object is syntactically well-formed. That is, all the open and close braces match, and the keys and values are paired correctly. However, we don’t yet know whether the JSON object is semantically correct. For example, we must still confirm that the "stations"key exists and it refers to a JSON array.

This is where the State Machine comes into action.

The State Machine

The purpose of a state machine is to remember which part of the JSON object we’re currently processing. In our weather station example, we start by scanning through the "stations" section while collecting meta-data about the location and measurement units of each station. Once we reach the end of the array, we then switch to a different state for processing the content of the reports array.

The following diagram shows a (partial) state machine for scanning through the stream of tokens, transitioning from one state to another based on the token’s type.

As with all state machines, we begin at the start state (top left) and progress from one state to the next, as we consume tokens from the input stream. If a particular state doesn’t have a transition for the next token in the input, the JSON object is considered invalid (we won’t discuss this situation). Given our example token stream, you should be able to trace through the state machine and imagine all the tokens being successfully consumed.

As discussed earlier, the state machine also has the ability to record the weather station information, for later use in the same stream. When we transition from one state to another, and that transition is annotated with an action box, the state machine performs the provided action. This allows us to update our internal database with the weather station details.

Note that the “Record Field Name” and “Record Field Value” boxes are fairly simple and merely save the values into local RAM. However, the “Validate and Save Record” box has the task of ensuring that all required fields (stationId, city, and units) were correctly provided, and they have meaningful values. The entire record can then be written to the database, or some other persistent storage.

Although we don’t show the second part of the state machine (where the reports section is consumed), the approach is generally the same. In our particular example, we’re not planning to store the output from reports in a database, but will instead send it downstream to some other consumer, or will perhaps discard the data after computing a running average. Therefore, the key difference in the state machine is that we only retrieve previous information from the database, not store it.

The Final Output

In order to do something useful, the state machine must contain an action to generate output. In our weather station example, we‘ll generate a stream of comma-separated values (CSV) data showing the equivalent information, but always using metric units (degrees celsius, and kilometres per hour).

Here’s the corresponding output:

1234,18.3,19.3
1362,20,23
...
1362,19,24

The actual data in the output, and the format you choose, is entirely your decision. The important fact is that we’ve processed a very large amount of JSON data on the input, without requiring that we load the entire JSON object into RAM at one time.

But, Isn’t This All Too Complicated?

Probably by now you’re wondering whether there’s a simpler solution. It feels like a lot of work to tokenize the input, then build a state machine, so why should we go to such extremes? Let’s discuss some design considerations:

1. Does Your JSON Really Need to Have Dependencies?

Our whole discussion has focused on using information from one part of the JSON message to interpret data from later parts of that same message. If you have a choice, simply avoid merging the information into the same stream in the first place. This makes parsing the data much easier.

If you don’t have a choice, read on…

2. Does the JSON Stream Have Predictable Structure?

If you do a Google search for “JSON Streaming” and throw in the name of your favourite programming language, you’ll find a bunch of libraries that address the problem in their own unique way.

Some of the advanced libraries support the JSON Path concept. That is, given a stream of JSON, and one or more path expressions, the library will extract all the JSON elements matching the specified paths. For example, we can extract all the weather station data by listening to the following two paths:

$.stations[*]   // on match, record the station details.
$.reports[*]    // on match, normalize the data and output to CSV.

Note that $ is the object root, and [*] means all elements in the array.

In our example, we need a library that can listen to multiple JSON paths for the same stream, performing different actions depending on which path was matched. As an example, for JVM-based languages (Java, Scala, etc), you could try JsonSurfer.

3. What If the JSON Object Has Dynamic Structure?

Where our state machine becomes worth the investment is when the JSON schema is defined dynamically. For example, the following JSON message specifies the names and types of the data that will appear later in the stream.

{
  "types": [
    "name": "string",
    "birth": "date",
    "children": "array[string]"
  ],
  "data": [
    { "name": "Fred", "birth": "1966/02/03", "children": [...] },
    { "name": "Mary", "birth": "1976/10/13", "children": [...] },
    ...
  ]
}

It wouldn’t be possible to construct a suitable JSON Path if we hadn’t already read the types element. Sure, we could use [*] to extract each row from data, but we’ll still need additional logic to traverse and validate the hierarchy of each sub-object, even if it’s now entirely in RAM.

Of course, building a state machine to accept a dynamically-structured JSON message isn’t easy, but that’s a topic for another day.

4. What If It’s Not JSON?

Finally, although we’ve focused extensively on JSON, the general approach of tokenizing characters, and then passing them through a state machine is just a good concept to be aware of. Many types of streaming data can be processed using this technique — it doesn’t need to be JSON. In fact, this is the exact approach used by the parsing function contained within most programming language compilers.

Summary

The use of state machines provides greater flexibility than most naive JSON streaming solutions. Those solutions can provide a stream of stations information, or reports information, but they can’t mix the two together. In our case, the structure of the JSON object can vary as we progress through the stream, with different actions being taken in each section.

Although our example was fairly simple, there are very few limits to the complexity of the JSON object we could handle, or the relationships between the various components. The only requirement is that data appears in the necessary order within the stream — that is, you can’t make use of data that hasn’t yet appeared.

Finally, this technique is fairly advanced, and you should consider carefully whether you actually need the full power of a state machine. Depending on your particular use-case, a simpler solution might be possible.

Processing of data via streams is something you hear a lot about these days (especially alongside Big Data). However, this is one of the few books that clearly explains why you should stream data around your distributed system, rather than using centralized data stores. This includes large-scale SaaS applications that process large amounts of data — think of your favourite social media application such as Facebook or LinkedIn.

Throughout the book, a selection of practical use-cases are discussed, first with the traditional non-streaming solution, followed by a detailed explanation of how streaming-based solutions do a better job. These design trade-offs are what makes the book worth reading from cover to cover. They’ll make you think twice before building an application with a centralized database!

Although this book is sponsored by Confluent, the creators of open source Apache Kafka, at no point did it feel like a sales pitch. Instead, this book is about concepts and design patterns that apply to any data streaming solutions (including Amazon Kinesis or Akka Streams). Don’t expect to see much source code though — that’s not what this book is about.

Key Lessons

Here are the key takeaways that make this book worth reading:

There’s No Single Way to Access Data

For me, the most important message of this book is that it’s unwise to store all of your application’s data in a single database, using a single schema. Instead, different parts of the application should store the same data in different ways, depending on their access patterns.

For example, if you press the “Like” button on a social media post, this may look like a single update to the application’s state, but actually results in multiple things happening — notifications are sent via email, statistics databases are updated to track the number of likes per article, and machine learning algorithms are triggered to recommend similar articles.

It’s unreasonable for all parts of the software application to query the same database for all these different purposes, so instead, we send the “like” event on an event stream for all those different services to listen to. They can then store and process the data however is best for them.

Logging to Sequential Files Is Very Scalable

A second key point is that logging data changes to a sequential log stream is more efficient (considering disk access time) than adding, updating, or splitting a B-Tree index, as used inside relational databases. If your application doesn’t actually need random read/write access to data, then a sequential stream of changes is potentially more useful.

Also, log streams can benefit from partitioning/sharding to reach higher levels of scale, they provide a global ordering in which the events actually happened, and can provide the ability to replay events from the past (until you decide to purge old data).

Integrating Streams into Your Existing Application

If you’re working on an application that currently uses a centralized database, and you’d instead like to start streaming your data, consider the Change Data Capture approach. By listening to incremental changes (inserts or updates) occurring inside your database, it’s possible to generate a stream of changes, in the order they occurred. We then get all the benefits of streaming data changes around the system, without modifying the original application.

Similarity to Unix Pipelines

Towards the end of the book, there’s a great comparison drawn with the Unix shell pipelines that many of us have used for decades. Standard tools such as grep, awk, sort, and uniq resemble micro-services that perform a single function, and the pipe operator | joins them together by passing streams of character data.

Summary

I recommend that anybody interested in software architecture should understand the concepts from this book. It’s free to download, and at 170 pages you can easily skip through sections you’re already familiar with.

I’m looking forward to reading Martin Kleppmann’s latest book, titled Designing Data-Intensive Applications.

You may be asking why you’d need to find the average of date values, especially since an average (or arithmetic mean) is typically an operation on numbers. For example, the average of January 7th, 1987, June 23rd, 1994, and December 10th, 1992, gives you the central date value of June 24th, 1991, but how is that useful?

One such case is anomaly detection, where it’s useful to identify date values falling outside the range of what’s considered “normal”. That is, if we determine the average of a series dates, as well as the standard deviation of those dates, we can then identify the outliers that fall beyond two standard deviations from the average.

Identifying the outliers gives us an opportunity to remove invalid data from our data set, or perhaps to start investigating the root cause of the anomaly. In this scenario, computing the average date from a large data set is important.

In this blog post, we’ll look at two different approaches to computing an average date using the Spark framework. We’ll also discuss some accuracy and performance implications.

The Problem

If Spark (particularly SparkSQL) already supported this functionally, we’d be able to compute the average of a series of dates by reading them into a Spark DataFrame (a table with rows and columns), then invoking the avg function on the appropriate column.

In this example, we’ll read the dates from the single-column CSV filedates.csv:

2017-01-02
2017-03-04
2017-05-06
2017-08-01
...
2017-10-12

To read this file into a Spark DataFrame, ensuring that the single column of data is interpreted as a Date-typed value, we use the following code:

import org.apache.spark.sql.types._

// The single column ("datecol") must be interpreted as a Date
val schema = StructType(
  Seq(
    StructField("datecol", DateType)
  )
)

// Define a new DataFrame, based off the content of the CSV file
val df = spark.read.schema(schema).csv("dates.csv")

// Compute the average of the datecol column
df.agg(avg('datecol)))

Unfortunately, this simple solution fails with the following error message:

org.apache.spark.sql.AnalysisException: cannot resolve ‘avg(`datecol`)’ due to data type mismatch: function average requires numeric types, not DateType;

It’s clear that Spark’s built-in avg function isn’t designed to support DateType columns, so a workaround is required.

Approach 1 — Convert to Number, and Back Again

Given that the avg function is intended to operate on numeric values, our first approach is to translate all the Date values into corresponding Int values, representing the number of days since a fixed point in time. We then perform the avg operation, and convert the result back to aDate value.

Unix-based systems use January 1st, 1970 as the point at which everything started (known as the “Epoch”), so we’ll do the same by converting all Date values to the number of days since the Epoch. For example, January 2nd, 2017 equates to 17168, and March 4th, 1812 equates to -57646.

This numeric conversion is possible using SparkSQL’s datediff function.

import java.sql.Date

// Dates are first converted to number of days since this date
val baseDate = lit(Date.valueOf("1970-01-01"))

// Compute a DataFrame containing the average number of days
val avgDayDataFrame = df.agg(
  avg(
    datediff('datecol, baseDate)
  )
)

This approach works well, but gives an Int as a return value, rather than a Date.

scala> avgDayDataFrame.show

+-----------------------------------------+
|avg(datediff(datecol, DATE '1970-01-01'))|
+-----------------------------------------+
|                                  17303.8|
+-----------------------------------------+

As it turns out, there’s no native SparkSQL function that does the opposite of datediff, to obtain a Date value from our numeric average. The date_add function looked like it might work, but instead requires a constant Int number of days, rather than taking the Int result from a DataFrame.

Our solution is to extract the average value from the first column of the first row of the DataFrame, and then explicitly create a new Date object (which expects the number of milliseconds since the Epoch). Note that we use Long arithmetic here, to avoid exceeding the 2³¹ limit of Int values.

// Trigger query (by running collect), and extract a native Long
// from the 0th column and 0th row of the DataFrame.
val avgDay = avgDayDataFrame.collect()(0).getDouble(0).toLong

// Now convert the number of days back to a Date type. The Date
// constructor requires the number of milliseconds since 1970-01-01.
val avgDate = new Date(avgDay * 24 * 60 * 60 * 1000)

This works, but isn’t very elegant, particularly since the final conversion to Date is done outside the context of Spark DataFrames. We therefore can’t do additional DataFrame processing in the same Spark query. The solution is to encapsulate those last few lines in a Spark UDF (User Defined Function).

import org.apache.spark.sql.expressions.UserDefinedFunction

// define a function that takes an Int, and returns the Date
val daysToDate: Int => Date = { days =>
  new Date(days * 24 * 60 * 60 * 1000)
}

// convert this to a UDF-based function that uses Spark's Column
// data type as the input and output.
val daysToDateUDF: UserDefinedFunction = udf(daysToDate)

A Spark UDF is essentially a function that accepts a Spark SQL Column-typed value as input, and returns a Column-typed value as output. This allows the function to be used entirely within a Spark query.

The new Spark query, returning a Date value is now:

val avgDayDataFrame = df.agg(
  daysToDateUDF(
    avg(
      datediff('datecol, baseDate)
    )
  )
)
val avgDay = avgDayDataFrame.collect()(0).getDate(0)

This gives us exactly what we need. Note that Spark UDFs are often reported to be inefficient, particularly because the Spark SQL optimizer is unable to understand them, and therefore unable to optimize them. In our case, we’re only running the UDF once per DataFrame (not once per row), so the performance impact should be minimal.

Approach 2 — User Defined Aggregate Functions

An alternative approach is to define a User Defined Aggregation Function (UDAF). Whereas a regular UDF acts on a single table cell, a UDAF operates on a full column to produce a single aggregated value.

Here’s how an avgdate function would be used in a Spark query:

val avgdate = new AvgDateUDF

val avgDayDataFrame = df.agg(avgdate('datecol))
val avgDay = avgDayDataFrame.collect()(0).getDate(0)

This syntax is much more readable than the previous example, given that you’re calling the avgdate function on a Date column, and getting back a resultingDate value.

To define a Spark UDAF, we must extend the UserDefinedAggregateFunction class and override the class members. The key members to be overridden are:

• inputSchema — Defines the type of values that UDAF can operate on (that is, Date values).

• bufferSchema — Defines the intermediate counters used during the aggregation. In this example, we track the count of the number of date values we’ve seen, as well as the runningtotal of the dates.

• dataType — Defines the type of the output data, in this case DateType.

• initialize() — A method setting the counters to their initial values.

• update() — A method called to add each new Date value to our intermediate counter values. Note the special handling for null field values.

• merge() — Given that Spark is a distributed analytics framework, this method joins together the counters from different Spark partitions that were potentially executed on different compute nodes.

• evaluate() — Converts the intermediate counters into a final Date value. This is done by simply dividing the total by the count, and then converting to a Date type.

class AvgDateUDF extends UserDefinedAggregateFunction {

  val BaseDate = Date.valueOf("1970-01-01")

  // each value being aggregated has this type
  override def inputSchema: StructType =
    StructType(StructField("dateValue", DateType) :: Nil)

  // intermediate values used during aggregation
  override def bufferSchema: StructType = StructType(
    StructField("count", LongType) ::
    StructField("total", LongType) :: Nil
  )

  // output type of the aggregation
  override def dataType: DataType = DateType

  // This aggregation always returns a consistent output, 
  // given a consistent input
  override def deterministic: Boolean = true

  // Initialize our internal counters.
  override 
  def initialize(buffer: MutableAggregationBuffer): Unit = {
    buffer(0) = 0L
    buffer(1) = 0L
  }

  // Update our counters with a new data value.
  override 
  def update(buffer: MutableAggregationBuffer, input: Row): Unit = {    
    val thisDate = input.getAs[Date](0)
    if (thisDate != null) {
      buffer(0) = buffer.getAs[Long](0) + 1
      buffer(1) = buffer.getAs[Long](1) +    
          thisDate.toLocalDate.toEpochDay
    }
  }

  // merge counters from two different Spark partitions
  override 
  def merge(buff1: MutableAggregationBuffer, buff2: Row): Unit = {
    buff1(0) = buff1.getAs[Long](0) + buff2.getAs[Long](0)
    buff1(1) = buff1.getAs[Long](1) + buff2.getAs[Long](1)
  }

  // Return the final value, as a Date
  override def evaluate(buffer: Row): Any = {
    val avgDays = buffer.getAs[Long](1) / buffer.getAs[Long](0)
    java.sql.Date.valueOf(LocalDate.ofEpochDay(avgDays))
  }
}

Note that for performance reasons, we access the intermediate counter variables as buffer(0) and buffer(1), rather than using their symbolic "count" and "total" names.

Arithmetic Overflow

Even if you’re not a Scala expert, you can hopefully get the gist of the previous code. That is, initialize a counter to 0, and a sum to 0, and then for every new date value, add the number of days (since the base date) to the sum, and increment the counter. Finally, divide the total sum by the count of items seen.

One limitation of this approach is Arithmetic Overflow. That is, the total variable has type Long, implying it has a maximum value of 2⁶³-1 (or 9,223,372,036,854,775,807). That’s a pretty large number, but it’s still possible to overflow that data type and have it wrap around to zero. If that was to happen, we’d get a totally incorrect result.

In reality though, this is unlikely to happen with the Long data types (it would definitely be a problem with Int). Given that we’ll likely be dealing with recent dates (that is, near to the year 2018), most of the numbers we add to will be around 17,000 (days since 1970). We’d therefore need to find the average of 500 trillion date values before overflow would happen. It’s probably not worth worrying about this case.

In fact, looking at Spark’s avg function, it uses either Double data type which can reach 10³⁰⁸, or the BigDecimal data type, which can be arbitrarily large (depending on your RAM). Clearly this is not a problem for most Spark users.

If we wanted to be really paranoid, there are ways to avoid overflow by either dividing the data set into equal-sized data sets, and then averaging the averages. Or perhaps use an approach of iteratively refining the average. We’ll leave those solutions for another day.

Performance

Finally, let’s get a rough indication of the performance of these different approaches. It’s interesting to measure performance, since User Defined Functions are reportedly slower than using native Spark functions which are handled better by Spark’s Catalyst Optimizer. Although our avgdate function is easier to use in queries, it might just be slower.

In our tests, we used Amazon EMR-5.13.0 (with Spark 2.3.0) with one master and two core nodes of type m4.2xlarge (8 CPU and 32GB of RAM). The input data set was 10M rows of randomly-generated data in CSV format, with each row having 100 columns. The test case involved computing the average date for a particular date-typed column.

For each test case, the result was computed six times, with the data from the first test run being discarded (to ignore the impact of cold caches). The reported result is the average duration (in milliseconds) of the remaining five test runs.

• Base Case (675ms — StdDev 41ms) — The standard Spark min function was used as an indication of how fast native Spark functions could read data, therefore defining a base case scenario to compare against.

• Approach 1 (947ms — StdDev 13ms) — This is our first approach of computing theavg of datediff.

• Approach 2 (1390ms — StdDev 34ms) —Our second approach, using the avgdate user defined aggregation function.

To eliminate the cost of reading the CSV file into memory (the source data resides in Amazon S3), the .cache() directive was used to pin the data into RAM. Therefore, the duration measurements are purely the time required to scan the date column and perform the averaging operation.

As you can see, our second approach takes almost 50% longer to compute the average date, compared to the first approach. It also takes twice as long as our base case of computing the minimum date value. Although some amount of code optimization is likely possible, our user-defined aggregation function is clearly not the best approach, even though it makes the code more readable.

Conclusion

Apache Spark is an excellent general-purpose framework for performing data analytics. Even though it comes with a variety of built-in analytic functions, it’s sometimes necessary to implement your own functions. Spark SQL provides a convenient mechanism for defining both cell-based UDFs and column-based UDAFs, making Spark queries easier to construct. Initial tests indicate that user-defined functions are less performant than native Spark functions.