The New York Times
The New York Times has long been called “the paper of record.” But newspapers are just one of the growing number of ways The New York Times company delivers information. The company produces videos, apps, interactive features, and podcasts—whatever it takes to get you the news you need to know.
The Times has long been a digital pioneer—they began publishing on the web in 1996, offering their first digital subscription in the early 2000s. Today their team of developers is busy keeping the company at the cutting edge of the news business, whether through highly visible efforts like mobile apps and interactive data visualizations or building the microservices and containers-based infrastructure needed to deliver one of the world’s most popular websites.
GitHub has been incredibly helpful towards The Times’s digital efforts. The Times automates as many tasks as possible. GitHub serves as the glue between many different applications and services, such as their CI/CD system and capturing meta-data for Application Catalog, a centralized source of information about all software systems in use at The Times. “To accelerate feature development, you need to focus on centralization capabilities,” says Senior Engineering Manager Shesh Patel.
For example, a developer could utilize GitHub in several ways apart from just pushing code, such as using pull requests to share progress and details about a feature, templating a checklist, and implementing PR checks to maintain high code quality standards. A pull request along with a wide array of Git events triggers a CI/CD pipeline without a developer ever needing to touch another piece of software. “Everything is configured to react around the GitHub workflow,” Patel explains. “GitHub is incredibly flexible. With webhooks or by watching pull requests, we can do just about everything we need to.”
GitHub is more than just a source code management system for us. It’s the nervous system for our entire software lifecycle process.
In addition to cutting down the number of different tools developers need to learn and interact, this approach also reduces complexity. The more complex a workflow is, the more likely it is that something will go wrong—that an error will be made or that someone will skip a step. By automating as much as possible and letting developers focus on simple, GitHub-centric workflows, they’re able to spend more time building software and less on following checklists.
This approach also applies to security. The New York Times integrates GitHub with Windows Active Directory and HashiCorp Vault to automatically synchronize access controls and share secrets. Meanwhile, The Times has been “shifting” left on security by baking security into the earliest stages of the development process. “We want security as close to the pipeline as possible without harming the developer experience,” Patel says. Once again, automation—including GitHub’s Dependabot—and simple workflows are the key. Automated security tests run when developers commit code, saving developers from the need to validate code manually.
The Times has built so much software since it joined the worldwide web in 1996 that keeping track of app information by word of mouth was not scalable. “If you wanted to partner with another team, you didn’t know how to approach them,” says Staff Software Engineer Thilak Subramanian. “We didn’t have a way to keep track of who owns an app, what technologies were in use, what resources we had available at any given time.”
The team scoured the software market for a third-party solution to solve this problem. Finding none, they decided to build their solution.
Subramanian led the creation of Application Catalog, the centralized source of information for applications, services, and software at The New York Times. The team was looking for a solution that will change the culture of maintaining application information in the organization and, at the same time, a solution that will enable data integrity and simplicity in adoption. We were looking for a solution that will be at the root of everyone’s workflow. Once again, GitHub integration was crucial. Instead of adding another tool to a developer’s workflow, teams manage their app info by adding a file at the root of their GitHub repo.
To add their application to the catalog, developers fill out a .appcatatlog.yml file with information such as the application’s name, where it’s deployed, and their contact information, and commit that file to the root of their GitHub repo. The Application Catalog listens to PR webhooks for those YAML files and uses them to populate its database after validation automatically. Developers don’t have to log in to anything to add or update their application’s listing. If they need to change or update their information, they edit the .appcatatlog.yml file. “I don’t think we’d be able to keep the information up-to-date without the GitHub workflow,” Patel says.
Application Catalog debuted at The Times’s “Maker Week” event—the company’s equivalent of a hackathon. They leveraged this event to talk about Application Catalog to aid in broader adoption. These workshops were geared primarily towards teams to add their application catalogs with a perk of gift cards through a raffle process for teams who participated. Their efforts resulted in groups onboarding 100 applications in the Application Catalog in the first 30 days, which was a massive success. But the raffle isn’t the only incentive to create and update Application Catalog YAML files, and they have registered over 200 applications up-to-date.
Through an iterative approach, they have reduced manual dependency to gather meta-data by enabling multiple automated integrations. The Application Catalog automatically registers applications with tools like PagerDuty, enabling a more unified integration with external systems and preserving the source of truth in our organization. “It’s an additional motivation for developers because registering their applications in the catalog saves them the effort of configuring other systems,” Subramanian says. “This enables a streamlined workflow.” The Application Catalog also utilizes GitHub APIs to auto-populate technical stack information such as languages and GitHub teams associated with their applications.
Developers at The Times already see the benefits of the Application Catalog. Previously, they might spend an hour figuring out who they might need to talk about an application maintained by another team. If Application Catalog saves each of The Times’s developers only an hour or two per week, that’s still an amazing amount of productivity that can go towards something else.
It’s also driving more cross-collaboration at The Times since teams can more easily find relevant projects to any challenges they’re trying to solve. The team is now working on a feature for visualizing dependency graphs across applications. “We expect teams will find that they have dependencies that aren’t maintained anymore or not known before, and will also offer a better understanding of upstream/downstream dependencies in overall business flow,” Patel says. “Those teams might then take over maintenance of those projects since they rely on them or those teams would know whom to contact during an P1 outage.”
Non-technical staff are benefiting from the Application Catalog as well. “For example, project managers can pull up lists of all the different applications in their vertical,” says Subramanian. “It can help anyone in the company get a bigger picture of software development at The Times.”
All of these experiences revolve around GitHub. “GitHub is more than just a source code management system for us,” Patel says. “It’s the nervous system for our entire software lifecycle process.”