In this post, we explain how we have combined Backstage and Vega.js to bring beautiful and insightful data visualizations into developer portals.
The combination of these two powerful open-source projects is an enabler for applying software analytics techniques, and thus better understanding what is happening in your software ecosystem.
Backstage: observe entities from different angles
Backstage, open-sourced by Spotify, is a flexible platform for building developer portals. With Backstage, development teams get a single entry point into their software ecosystem. They can easily find information about all sorts of entities that are part of the ecosystem: the components they build, the tools they use, and of course… the people they work with!
Backstage tracks all these entities in a central catalog and provides an extensible UI to explore them. Developers can observe every entity in the catalog from different perspectives. Depending on the situation, they may look at a component from a “source code” perspective, from a “CI/CD” perspective, from a “team” perspective, etc. Backstage brings all elements in a central place, make the life of developers easier.
The beauty of Backstage is that it lets portal designers craft these different perspectives with a lot of flexibility. Backstage really is an application framework: its user interface is easy to extend with new pages, new tabs, new components, etc. This is essentially done by writing React components and following some guidelines.
In the example below, one may for instance decide to add a “Telemetry” tab to provide runtime information about the component. One might also decide to add a widget in the “Overview” tab to show the number of open bugs.
The extensibility of Backstage makes it easy to bring data visualizations into the developer portal. The core platform provides some visualization components (gauges, relations graphs, etc.). But as we will see later, it is possible to bring your own visualization libraries into the framework, thanks to its plugin architecture.
Software analytics and data visualization
Software analytics has been defined as “analytics on software data for managers and software engineers with the aim of empowering software development individuals and teams to gain and share insight from their data to make better decisions.”. The “software data” is typically extracted from code repositories, issue trackers, static code analyzers, production systems, etc.
This is a field where extensive research has been done in academic and industrial settings. Research outcomes have made their way into developer tools, to support a variety of use cases: architecture understanding, defect prediction, etc.
At Avalia Systems, we apply software analytics techniques for two main use cases:
Technical due diligence projects: here, the goal is to use the information to form an opinion about a particular software organization and its assets. Software analytics allow our experts to go faster and deeper.
Continuous improvement: here, the goal is to present feedback to development teams, to make progress visible. Software analytics is about defining leading and lagging indicators, collecting related metrics and presenting insights in a compelling way.
In both use cases, the visualization of software data is an important part of the process. Visualization is effective both to explore a problem space and to explain a situation to a target audience. These two phases are very different and are best supported by different tools and techniques. Having said that, many of these tools can leverage the same underlying core libraries. Vega.js is one such library.
Vega.js and the grammar of graphics
Vega.js and Vega-Lite.js are two open-source projects that implement the concept of a “visualization grammar”. In other words, they provide a “language” to declare how a specific dataset should be represented visually. The language is used to declare how data properties should be mapped onto graphical properties.
With Vega.js, data visualization experts can describe data transformations, scales, axes, legends, etc. Vega-Lite.js provides a higher-level grammar than Vega.js, which makes it possible to write more concise specifications, at the cost of some flexibility.
The example below shows an example. On the left, we see a Vega-Lite specification, written in JSON. On the right, we see the visual interpretation of this specification. Here are the key points of the specification:
We declare that our dataset is made of 9 items with 3 attributes: “category”, “group”, and “value”.
We declare that every item in the dataset should be represented with a “bar” mark.
We declare that every bar is positioned on the horizontal and vertical axes, based on the “category/group”, respectively on the “value” attributes.
Finally, we declare that the color of every bar should be mapped onto the “group” attribute of the corresponding data item.
There is a learning curve to become familiar with this approach, but after that it becomes a very effective way to design new visualizations. The fact that specifications are written in JSON files, rather than in Javascript, makes them portable across tools.
Building a Vega plugin for Backstage
At Avalia Systems, we have been fans of Vega.js (and of the underlying D3.js library) for a long time. We have integrated it into different software analytics products, and we use it in most of our projects.
Over the years, we have built a catalog of data visualizations that we use to answer questions for our clients, and we wanted to be able to use them within Backstage portals. To achieve this goal, we have implemented a Backstage plugin. This plugin provides components, which can be used on any Backstage page. The plugin does not implement specific visualizations. It provides a way to connect a widget with a Vega or Vega-Lite specification, which is captured in a JSON file.
This enables a very effective development process. The visualizations can be designed by Vega experts, without the need to work in the Backstage codebase. Portal designers can then easily add widgets to their pages. In the example below, we have created a dashboard page with multiple widgets, but it is also possible to integrate a single widget on a page that presents other information (e.g. the page presenting information about an entity in the catalog).
To bring software analytics into our developer portals, we can now follow this process:
Get a dataset. Vega can parse JSON, CSV, TSV, DSV, and TopoJSON files. In the screenshots above, we have used the git log command to generate a CSV file with one line per commit. Each line has the date and the timestamp of the commit. We have run it on the Backstage repo, in GitHub.
Develop the Vega or Vega-Lite specification. The specification references one or more datasets, using their URLs. We usually perform this task with a local installation of the Vega Editor. This makes it easy to work with datasets available on the local machine. Note that Vega provides data transformation primitives. For example, it is possible to compute aggregations on the raw CSV data, and to map attributes of the aggregations onto graphical properties.
Use our Vega plugin for Backstage. In the example below, we see that our <Widget> component references the Vega specification. Here, we have stored the JSON files in the Backstage server static asset folder, which is why we use a relative URL. But again, the Vega spec can be deployed anywhere, provided that they are accessible via a URL.
// Every widget has a title and a reference to a Vega Lite spec
const vegaLiteWidgets : VegaWidgetDefinition[] = [
{
title: "Top contributors",
spec: "/metrics/top-contributors-by-commits.json",
},
{
title: "Activity",
spec: "/metrics/activity-matrix.json",
},
{
title: "Cumulative count of commits",
spec: "/metrics/commits-cumulative-count.json",
},
];
// We return a horizontal grid with one widget per slot
return (
<div className="widgetGrid">
{vegaLiteWidgets.map((widget, index) => {
return (<Widget
key={index}
title={widget.title}
library={VegaLibrary.VEGA_LITE}
vegaSpecUrl={widget.spec} />)
})}
</div>
);There are two aspects that are worth mentioning about our implementation of the plugin:
We have made the widget “responsive”. Vega specifications can specify a fixed width and height, and this approach is used in most examples. A more practical approach is to let Vega generate a chart that uses the area of its container in the DOM. This means that if the window is resized, the Vega chart should be re-rendered with new dimensions. With React, this can be done with a useEffect hook (see the code snippet below). Note that we also had to fix the CSS used by Vega for this to work well.
const updateDimensions = () => {
setVegaWidth(vegaRef.current.clientWidth);
setVegaHeight(vegaRef.current.clientHeight);
}
React.useEffect(() => {
updateDimensions();
window.addEventListener("resize", updateDimensions);
return () =>
window.removeEventListener("resize", updateDimensions);
}, []);
We have provided a way to inject standard configurations (similar to themes). When multiple charts are added to Backstage pages, it is important to have consistent styles. Colors, font sizes, axis styles, etc. All these elements are defined in Vega specifications. Instead of repeating them in every specification, we can inject them. In the snippet below, the elements are coded in the useEffect() hook. In practice, we load them from external files, which makes maintenance easier.
useEffect(() => {
const fetchVegaSpec = async () => {
const response = await fetch(props.vegaSpecUrl);
const json = await response.json() as VisualizationSpec;
json.autosize = {
type: "fit",
resize: false,
contains: "padding",
};
json.padding = 10;
// configure other aspects: axes, fonts, etc
// ...
setVegaSpec(json);
}
fetchVegaSpec();
}, [props.vegaSpecUrl]);Backstage and Vega.js: a powerful combination
We are still putting the final touches on the Vega Plugin for Backstage, but we are very happy with the results so far.
Backstage and Vega are two really powerful open source frameworks that we were using independently. Bringing them together made a lot of sense for us. It should appeal to any organization deploying Backstage.
Indeed, a picture is worth a thousand words, and this is also true in the developer portals!