Google, Users and Terraform

Published on January 19, 2023

• Special thanks to James McShane at SuperOrbital for their help reviewing this post and providing useful feedback.

Introduction

In this article, we will explore how SuperOrbital has been evolving its user and group management workflow, with the goal of building an end-to-end solution that helps lower the burden of user management while also helping ensure that the results match our defined requirements.

Like many small companies, SuperOrbital was started by a single person, who initially handled all of the tasks that were required to manage the company, including creating and managing users and access to the various services that we use here. As SuperOrbital has grown, manual tasks have needed to be replaced by automated processes, especially where security, accuracy, and speed were of extra importance.

Many companies struggle to make internally-public user information available via a programmatic API that can be used for non-HR¹ purposes. This data can be extremely useful and can include anything, but a few things that often prove useful include the user’s contact information, employee type, GitHub username, etc. As we started to wrestle with the need to manage users as centrally as possible, we considered how we could take the services we were already committed to using, and combine them into a new, more streamlined, and reliable process.

When we started this project, new users who joined the company had to have their user and group memberships manually created in Google Workspace, and then some Terraform code had to be modified and applied to ensure that the new employee’s primary GitHub user was given access to the SuperOrbital organization and any applicable GitHub teams.

But even more importantly, when the user leaves the company it is important that all of their access is removed and any critical data that they might have stored in Google Workspace (e.g. calendar appointments and documents) is properly backed up and/or migrated to a different user. This data migration step is especially hard to automate when the exact steps required will often differ depending on the user, their role in the company, possible legal requirements, etc.

The Toolbox

From the very beginning SuperOrbital has used Google Workspace (G Suite)² to manage the users and core technical services for our domain. This includes things like email, groups, and the like.

Because we often hire engineers with a deep and strong history of working on open-source projects, we made the decision early on to allow our engineers to continue to use their existing GitHub user names within SuperOrbital’s GitHub organization. Due to this decision, we need to maintain a mapping between these GitHub usernames and each SuperOrbital employee.

SuperOrbital, is best known for its work within the Kubernetes ecosystem. However, Terraform is another tool that we rely on extensively to manage most of our infrastructure that is not running inside a Kubernetes cluster. Terraform allows us to confidently manage our infrastructure state, by providing a useful set of features like declarative operations and drift detection while also being a production-hardened tool that includes long-term vendor support.

The Goal

Throughout this initiative, our goal has been to create a nice foundation for our user automation that rests in the sweet spot between the flexibility of a time-consuming and fragile manual process and the rigidity and complexity of a fully automated solution.

The Options

We started the investigation by digging into the capabilities of the standard Google provider for Terraform. This provider is primarily designed to integrate with the Google Compute Platform. We also quickly discovered that Hashicorp, the creators of Terraform, have written an experimental Google Workspace provider that is specifically focused on supporting the Google Workspace APIs.

Used together it looked like these two Terraform providers could potentially provide a nice bridge between the Google user management that we required and many of our downstream services, like GitHub. We already manage GitHub users and teams via Terraform, but in the current implementation, users are still manually added to GitHub and are not automatically cleaned up when their users are deleted from the Google organization.

It is worth noting that Google permissions can be difficult to set up properly, especially for service accounts.

This blog entry, entitled How to Manage Google Groups, Users, and Service Accounts in GCP using Terraform by Guillermo Musumeci, was very helpful in piecing together some of the poorly documented requirements to get all of the permissions set up properly.

In addition to looking at all the available Terraform providers that we might be able to leverage, we also considered whether simply maintaining the seed user data as a YAML file in the repo, would be a useful approach. This would mean that instead of creating the user’s directly in HCL for a given provider, we would craft a YAML file that contained a list of users and their relevant data, which would then act as a data source in Terraform for the various providers to accomplish the required user management.

As a quick example, if we decided to go this route, an individual user in the data file might look something like this:

users:
  aliyah:
    aliases: [ "ahadi" ]
    given_name: "Aliyah"
    family_name: "Hadi"
    lifecycle:
      inactive: false
    google:
      groups:
        employees:
          roles: [ "MEMBER" ]
    github:
      username: "ahadi"
      teams:
        all:
          role: "member"

The Tradeoffs

Now that we had an idea of which tools might help us accomplish the task at hand we started to experiment with a few implementations to see how each approach might work.

The GCP Provider

The official GCP³ provider for Terraform is very robust and provides strong coverage of the numerous Google Cloud APIs. However, Google Workspace and Google Cloud are really two completely different products that are intertwined, but not fully integrated, and this is one of those times that this becomes very apparent. Although this Terraform provider can be used to manage Google groups via the Cloud Identity API, there is no support for Google Workspace User Management.

The Google Workspace Provider

Our initial exploration of the experimental Google Workspace Provider for Terraform, made it clear that it could handle basic user management, however, both the underlying Google APIs and the provider left a lot to be desired.

It is pretty apparent that these Google Workspace APIs aren’t really designed with a tool like Terraform in mind, and that the provider has not yet implemented enough logic into it to compensate for some of the ways that Google transforms the data on the backend.

A general example of this is that you can define email aliases for a user so that aliyah.hadi@work-example.com can also be reached via ahadi@work-example.com. In addition, it is possible to add references to non-Google email addresses, like a personal email address such as aliyah@personal-example.com.

In Terraform, the HCL⁴ implementation of this looks something like this:

resource "googleworkspace_user" "ahadi" {
  primary_email = "aliyah.hadi@${var.domain}"

  name {
    given_name  = "Aliyah"
    family_name = "Hadi"
  }

  aliases = [ "ahadi@${var.domain}" ]

  emails {
    address = "aliyah.hadi@${var.domain}"
    type    = "work"
    primary = true
  }

  emails {
    address = "ahadi@${var.domain}"
    type    = "work"
    primary = false
  }

  emails {
    address = "aliyah@personal-example.com"
    type    = "home"
    primary = false
  }

}

Sidenote: This above HCL might also give you an idea of why we thought that the YAML representation of this data might be a better way to model the data, especially if we were going to be maintaining it semi-manually for the time being.

Given the above HCL, the provider will happily create the Google user However, one of the challenges with using the Google Workspace provider currently is that it does not compensate for the fact that the Google APIs will return all of those email addresses in a sort order that very likely differs from the order that was used in the HCL. This means that users are forced to experiment with the order of these HCL emails{} blocks to keep future runs from generating terraform plans that permanently show pending changes even though applying those changes doesn’t actually result in the underlying user object being altered in any way.

Another issue that we ran into, is that although the Workspace provider does appear to support suspending and archiving users, these processes require special user licenses, and instead, we tend to leverage data transfer processes to migrate critical user data when people leave the company. However, this workflow isn’t currently supported by the provider, and because of this, we would either need to change our current approach and purchase Archive User licenses from Google and in either case, we would need to completely disable user deletion via Terraform so that we could manage the data migration requirements before a user is automatically deleted via Terraform.

The YAML Data File

After taking a look at the providers, we also explored the differences between using Google users as the source of truth versus using a YAML file as the source of truth. The nice thing about using the YAML file is that is much more readable than the HCL representation of a user:

users:
  aliyah:
    aliases: [ "ahadi" ]
    given_name: "Aliyah"
    family_name: "Hadi"
    …
  andrei:
    aliases: [ "aagapov" ]
    given_name: "Andrei"
    family_name: "Agapov"
    …
  …

And the data can easily be read into terraform and then used to create dynamic lists of users.

locals {
  user_map = yamldecode(file("${path.module}/users.yaml"))
}

resource "googleworkspace_user" "dynamic" {
  for_each      = local.user_map.users
  primary_email = "${each.key}@${var.domain}"
  …
}

However, as we started to realize that the limitations of the experimental Google Workspace Terraform provider might still force us into manual workflows, we started to question whether pulling users into Terraform at this point in time was truly a good idea.

The Compromise

With all of this information in hand, we decided to settle on a compromise, which improved the way that we were handling things today, while still providing us with a clear direction to improving this process in the future as we grow.

Because managing Workspace users via Terraform is not very robust at the moment, there are manual processes that we will be forced to do anyhow, and since we are not currently adding and removing users all the time, we decided to keep the users themselves out of Terraform for now.

This also meant that maintaining the YAML file provided very little value, so we scratched that as well, and decided to add a custom schema to our Google users that allow us to store their GitHub username and default GitHub permission level with the rest of their user data.

So, in the end, the workflow looks something like this:

In the HCL code, we start by reading in all the Google user data.

data "googleworkspace_users" "superorbital_io_users" {}

And then we can use this data to dynamically add users to our GitHub organization.

resource "github_membership" "google_users" {
  for_each = {
    for user in tolist(data.googleworkspace_users.superorbital_io_users.users[*]) :
    user.primary_email => ({
      github_username = try(jsondecode(user.custom_schemas[0].schema_values.github_username), "")
      github_role     = try(jsondecode(user.custom_schemas[0].schema_values.github_default_role), "member")
    })
    if try(jsondecode(user.custom_schemas[0].schema_values.github_username), "") != ""
  }
  username = each.value.github_username
  role     = each.value.github_role != "maintainer" && each.value.github_role != "admin" ? "member" : "admin"
}

The for_each loop provides most of the magic in the above HCL block, by creating a dynamic list of all the Google users that have a Github username attached to their user record.

Since adding comments to the above HCL code block makes it harder to read, we have included some in the folded HCL code below if you are interested in exploring the technical details of each code line.

resource "github_membership" "google_users" {
  # Create a set of objects to iterate over when creating this set of resources.
  for_each = {
    # Iterate over the Google users and generate a list of objects.
    for user in tolist(data.googleworkspace_users.superorbital_io_users.users[*]) :
    # Create a new object with the user's primary email address as the key.
    user.primary_email => ({
      # Set "github_username" to the value in the user record or an empty string.
      github_username = try(jsondecode(user.custom_schemas[0].schema_values.github_username), "")
      # Set "github_role" to the value in the user record or the default value of "member".
      github_role     = try(jsondecode(user.custom_schemas[0].schema_values.github_default_role), "member")
    })
    # Only include the object in the results if "github_username" is not an empty string.
    if try(jsondecode(user.custom_schemas[0].schema_values.github_username), "") != ""
  }
  # Add each resulting username/role pair into our GitHub organization
  username = each.value.github_username
  # GitHub Organizations and Teams each use a different term for admin.
  # We allow either "admin" or "maintainer" to be set in the user record.
  # If you aren't an "admin" or "maintainer", then you are a "member"
  role     = each.value.github_role != "maintainer" && each.value.github_role != "admin" ? "member" : "admin"
}

Google users are manually managed within the Google Workspace UI⁵, and then Terraform leverages that data to add or remove users from any additional tools that are not currently configured to use direct SSO⁶ integration with Google.

The Conclusion

Our primary goal has been to create a workflow that will allow us to define a user, and then automatically drive the rest of the provisioning from that initial definition. Although this current process is essentially two steps (add/remove user and then trigger terraform), it remains in line with these goals and puts us in a good position to iterate on the solution as SuperOrbital grows and the providers and APIs become more robust and deeply integrated.

There is always a balance to be achieved between a set of stated goals, the available tools, and the time and risk involved in implementing and using a given solution. In our search for that middle ground, we settled on a solution that allowed us to automate much of our user provisioning while maintaining a single source of truth, and still providing the flexibility that we need to handle the sometimes complex steps required to offboard a user and all of their critical data properly.