Interviews

The National Broadband Map Sets the Standard for Open Map Sharing

The National Broadband Map was created by the Federal Communications Commission and contractors under guidance from the National Telecommunications Information Administration (NTIA). With a seven-month timeline, this project posed many challenges. Not only did this project meet this tight deadline, it delivered an unprecedented product in terms of its performance and openness for other software developers to repurpose the tools to create their own data interface. V1 Magazine editor Matt Ball spoke with Michael Byrne, geographic information officer, of the FCC at the FOSS4G event in Denver, where he gave a keynote. The topics of discussion were steps taken to ensure performance and openness, as well as the outcomes of the work.

V1: The timeline to deliver the National Broadband Map was really accelerated for a project of this scale and scope. Can you give me an idea of the timeline of the project?

Byrne: I started the last week in Feb. 2010, and we launched the site on Feb. 17, 2011, so the project was completed within a year. When I started we hadn’t received any data, we didn’t have any infrastructure, and we hadn’t built any applications. To achieve each of these three things, we had to go through three open procurement processes for the development cycle. The time constraint on us from an application development point of view, particularly in the federal space, was like light speed from what we see on other projects. There were a lot of stars aligning that allowed that to happen.

The NTIA designed all the protocols for data collection and specifications, because this was their project. We worked very closely with NTIA, and we had a formal agreement between the two agencies. The agreement was that the FCC would provide the backbone for the map, and I was brought on specifically to lead the technical side of it, but my role is broader than that.

V1: The role of Geospatial Information Office (GIO) is relatively new in federal government. Are you the first GIO at FCC?

Byrne: I am the first GIO at the FCC, and my role is to set geospatial policy across the organization. I’m working closely with the chairman on a data innovation initiative to understand our data assets, ensure that those assets are well maintained, and that they are utilized for policy decisions. It also aims to reduce the red tape and remove the need for companies to collect and submit data to us where it is no longer necessary. It’s a strategic approach to data as an asset in the agency, and it’s clear to the chairman that geospatial is a part of that. We manage assets that have a geospatial context, with licenses that are assigned for specific geographies.

V1: With your prior role as GIO in the Office of the State Chief Information Officer in California, had you also been involved in open data initiatives?

Byrne: My experience as GIO in California was similar to open data at the FCC, where it is an initiative of the chairman that aligns with the administrations goals on open data. The governor of California was very interested in a California map for all kinds of things -- disaster relief, conservation planning, and large policy initiatives. I helped implement the map of recovery act dollars in California, the California Healthcare Atlas, and other sites that provided base visualization for large government policy. I also created the California Broadband Map, which was an effort that certainly led to my current position.

The California Broadband Map was done in 2006, and at the time I was in a very small agency that is the backbone of health information in California (the Office of Statewide Health Planning and Development (OSHPD)). They manage a full set of all patient discharge records for the whole state, and because of that effort there was a very strong business case for broadband deployment in California around telemedicine. There are 10-15 world-renowned University Hospitals in the state, and connecting those to satellite offices was an initiative that we evaluated to pull broadband to remote parts of California.

The result of that work was an executive order by Governor Schwarzenneger to create a Broadband Taskforce on which I served as staff. We developed the first broadband map of California, and I led that effort. It was a really new thing in 2006, and there was still quite a bit of uncertainty about how broadband providers would deliver and share information, with a lot of concern about proprietary information on where and how they served citizens.

We took a novel approach where a third-party vendor collected addresses from providers so that this wasn’t a state effort and didn’t fall under a records act. We then aggregated the information in a raster model and we masked who provided what service. The result was a display of broadband availability by speed tiers at a high resolution that had never been seen before.

There were some very unique patterns that emerged. There were clear disparities between rural and urban, and some speed availability issues between Northern and Southern California that hadn’t been articulated before. It was fascinating work that of course led to other things.

V1: Did you find yourself being called on by other states, even before your FCC work, to explain how you approached this mapping problem?

Byrne: I owe a lot of credit to Mark Guttman a colleague at a small economic analysis firm called Costquest that specializes in utilities work. I came across Mark at the Esri/UC in 2005, where he presented a business analysis for the state of Wyoming. Mark had developed an economic model that looked at deployment of broadband, and Wyoming hired him. At the UC he gave a very interesting talk about modeling at the address level.

In the State of California at this time we didn’t have a parcel layer available to state government, and there wasn’t an address base available for the state. The idea to develop a map based on addresses was far flung. The approach that Costquest took is that they modeled addresses within Census blocks based on the distribution of roads as a modeled approach rather than real addresses. Addresses were derived based on the spatial pattern of roads, where the greater road density means the more addresses, and these were distributed along a path that matched population. After talking with Mark at length, we developed a similar model for all of California, and used that as a base for the mapping of broadband availability by address.

I modeled roughly 12 million address points for the state. At one point I had six laptops running in parallel doing this modeling at my house. For me it was a great growth of my understanding of what we can do with GIS, because you tend to think in domain sets, and if you don’t have that domain set it isn’t possible. Because you’re dealing in the abstract all the time, as soon as you can create domain sets that are still relevant to reality, even though it is an abstraction, you can begin to really understand statistics. It was a great approach, and it allowed us to do a lot of additional things.

V1: How did you approach the national broadband mapping effort?

Byrne: The FCC didn’t handle any of the policy or modeling around how the data was collected. Our role was to receive the data in packages from all the states and territories, with 56 sets of data. Accepting the data was our beginning point, and we broke that into three primary tasks.

First, we took all datasets and integrated it into one set. The second task was to perform a data audit, diving in and understanding the data. Because we were taking data from multiple parties, we had to try and understand if we got everything, and how we might improve the process to make the data better over time. The third task was the map publication.

All three tasks required open federal procurement processes, and had to be designed from scratch. The first one we wrote was data integration, as we thought of that as the foundation. The end goal was to take 56 different data packages into one container with multiple schemas that each represent the data themes that each of the states and territories mapped. The model itself contains data at the census block level, with 8.2 million blocks in 2000 geography and more than 10 million blocks in 2010 geography.

For blocks greater than 2 square miles, NTIA had defined a different protocol. Instead of the block itself they could submit line segments for roads or individual address points. The more urban, the smaller the block is. The more rural, the larger the block. Blocks greater than 2 square miles are generally rural. We had to physically integrate what amounted to 25 million records into a common container under that three-pronged protocol (blocks, road segments and addresses).

We had to take freeform shapes of wireless availability, because the previous approach was just wireline availability. Wireless or satellite availability could be defined at any number of different spatial quality. We then merged those into one dataset.

There was also a dataset for community anchor institutions where demand aggregation might take place, such as schools, libraries, community centers, and government buildings where there might be larger pipes that might allow for residential population to have more access.

The primary policy effort that NTIA wanted to get to was understanding who had what kind of availability where. Understanding the common denominator of availability is really the core question. We can look at a map and see availability from our output, but unless we understand the number of people and the demographics of people you don’t see the full spectrum of availability or how to change policies.

A core problem that we had to solve technically, was how to get at percentage of availability. At the block level it’s fairly straightforward. In road segment and address point, it’s not easy. We had this data in rural fragmented areas, and it was very difficult to conceptualize. We went back to our roots in California, with the notion of address modeling, and performed a similar function.

Without a national address or parcel database there was no way that we could understand physically where in these rural areas people are at that kind of resolution. Once we had the address point model, we intersected it with the availability data that came from states. We then overlaid address points and line segments of availability to get counts of people that have availability in those areas. We had to write up the whole data integration process, put it into a Request for Quotation (RFQ), post that for 30 days, review bids, and award the contract. We awarded that data integration contract in July of 2010.

The data audit or data review was something that hadn’t been done before, and we didn’t know exactly how to get it done. We couldn’t hire a team to survey. The “Can you hear me now?” is a great analogy, but you can’t physically survey the entire nation or even a 1% sample, it’s just logistically not possible.

The approach we took was to write an RFQ that assembled some set of data that is used as a surrogate for what current availability might be. There are commercial offerings of this data, including government offering of FCC’s Form 477. We do 100% sampling of everything in the data collection against these third party data sets. We learned very interesting things from that process, and the process has helped us to refine the requirements and give feedback to each of the states. Classic national mapping doesn’t take 100% sampling, it usually takes less than 1% sampling. We wrote and awarded that contract in August of 2010.

The third was the map publication contract. Beginning in May of last year we developed a prototype, and in fact the agreement that we had with NTIA required that we produce a draft of the National Broadband Map by September. I started in February, and we needed to have an Internet-published draft by September. We began the process to develop a prototype, and that prototype is where really focused on the REST architecture.

Congress in the statute to NTIA said that the National Broadband Map should be searchable and interactive. We really focused on the notion of search, with a core concept of the map function as the ability to look at my address and see who provides me service and if I have an opportunity to change.

In the prototype development we said gave ourselves six weeks of effort, and gave a very limited budget and scope, and we did an Agile development cycle, we met for scrums every day at 10 a.m. The development team showed constant build and constant improvement. That prototype development led us specifically to writing an RFQ for the production map that you see today. We took the prototype and wrote the requirement, working very diligently with NTIA to flush out all of those functional and technical requirements in the RFQ.

The three contracts, the development of the scopes and the working prototype, and the development of data, were all run concurrently from April to July of 2010, which was a remarkable challenge.

V1: So, that left from July to February to have a working site, with just seven months to pull it all together?

Byrne: Actually it was less than that. All the contracts were let out in July, with 30 days to respond, plus 30 days of post-response to evaluate and award. The data integration and the map publication were both made on Sept. 30th. Both starting on Oct. 1, with less than five months from contract start to initial publication. In federal government terms, this is light speed.

From a technical and procurement management perspective, there is a ton of work that has to happen, including coordination with counsel, coordination with the procurement office, coordination with the CIO and with technical teams internally. It was a crazy amount of work to make sure all of these things happened, and it all happened independently from the huge amount of work that NTIA had to do to manage all state partners. All of things that we set up were dependent on all of NTIA’s work to build all the data, which was honestly a lot more work.

V1: At what point of the process did you wrestle with the commercial off-the-shelf (COTS) vs. open source solution?

Byrne: I know it’s a little corny, but I was doing fraction reduction math homework with my son and it was really a lightbulb moment, the idea of making something that is very complicated into the least common denominator involved making search by address that element. Making sure that this was part of the RESTful architecture, meant that whatever we wanted to do with the rest of architecture would be scalable. If we hadn’t done that, the amount of change that we knew was going to happen through the requirements analysis and gathering through our multiple partners would have necessitated a waterfall software development cycle, and that would not have been within the timeline that we had. We had to do something fast and that we could change, so it had to be REST.

At the FCC we had multiple infrastructure including SQL Server, Oracle, Sybase, and some publication of KML, GeoServer and the Esri stack as well, with our entire infrastructure Java on Unix. We had multiple environments already established because of all the other work we had already done on other projects. In the prototype development we built these sets of services off of Oracle directly, with RESTful sets of architectures hitting Oracle as Java code. To incorporate the Esri ArcServer stack at that point means that there is an added layer in the software stack, which reduces time and takes longer to do certain things, as opposed to just hitting Oracle or PostgreSQL directly.

The REST just returns an answer to the query, and then we have to draw the map too. In the prototype we used the ArcServer to push out services of availability. The complicating factor came when we were dealing with 8.2 million blocks, and also 25 million individual records of availability, and close to 2,000 different individual providers. We knew there would be a requirement of individual provider footprints, and we knew that would be a challenge. On-the-fly development of individual provider polygons over the scale of the data we were talking, we were quite sure noone had done before.

In the prototype we worked with the state of New York, and tried to get to the drawing of individual providers and we were not successful. We knew we could draw a handful of providers, but the draw speed that we were experiencing would not support the kind of hit rate we knew we would get, and the consumer need for a faster return. Our return target was less than one second. The COTS solution in the prototype helped us articulate the kind of solution that would work later.

The way that we approached the RFQ was to not specify a software, but only functional requirements and open standards. We described the current stack that FCC operates under, including all the database and GIS infrastructure that we had, and all the functional technical requirements. The first was that it be RESTful, and the second was that it be built on Open Geospatial Consortium standards. We never specified open source, and in fact the winning bid never specified that it was an open source solution, they specified the strategy that they would use to approach the requirements.

In the first four weeks of our requirements analysis is where the OpenGeo stack came into play. With three and a half months to launch we determined the software stack that we were going to use. I was aware that open source software could deal with the kind of data volumes that we would face through my work as GIO of California, but I didn’t come in with the expectation that there would be a particular solution.

V1: How did you approach the map representation and interactivity?

Byrne: We did take what I think is a unique cartographic approach, with a map gallery of eight or nine maps. There isn’t one map for the National Broadband Map, there are eight or more different slices. The predominant design style that we took was single color maps.

We spent a lot of time looking at cartographic style and Internet mapping, and recognized that we’d have to do a lot of the map rendering on the fly. When we looked at a lot of the maps out there, we realized that a lot of it was confusing to people that aren’t professional geographers, particularly when you have opaque overlays or design that has six to ten different colors. Those are challenging to read, and they don’t demonstrate a clear story.

The first four maps in our gallery are single color maps that show availability with interactive legends. As you click the legend more of the map is turned on in the same color or turned off in the same color. We put buttons for different availability so that you can toggle them on or off. We graduate to more colors of maps for more complex understanding. I think the approach lets us reach a more consumer market rather than a  professional level that might understand more complex maps.

V1: You’ve taken great pains to make the National Broadband Map open to developers, was that a focus throughout the effort?

Byrne: From day one we thought of the developer as a user requirement. The reason we thought that is that we knew were going to make the first broadband map, and we knew that there all kinds of limits to our collective intellect. The total staff was four, with two people at NTIA and two people at FCC. The four of us could only think of so many things, and we kind of feel like (I’m speaking for myself, but I think NTIA will agree with me) the people out there in the commercial world are collectively smarter than us. If we could develop our application with open protocols to allow developers to do anything they wanted, within the limits of the data, that we would likely see a better application.

We componentized everything, with every page having the source API, not just illustrated, but illustrated for the call of that page. The team we used calculated the number of different combinations that the map returns, excluding search, and the rank and summarize function that takes any combination of broadband availability and any combination of geography comes to 37 billion combinations that the map returns. We built 35 different APIs, and each page consumes these and each is exposed to developers.

We didn’t necessarily think that what we built for the National Broadband Map would be the end result. We figured that with exposed APIs that the big boys like Google, Zillow, 4Square, Yelp, etc. could do something faster and better and reach a market that we hadn’t thought of, because there is data as an asset that can be queried. We realized that from the fcc.gov/developer where we saw that new applications reached new markets, and that’s beautiful.

Within the first couple of weeks we saw four or five different examples of people using our APIs and developing things. We haven’t pulled an anchor developer, but I think it’s not far from happening.

V1: While there have been application contests, and there have been the open availability of data, but both the exposure of what you’ve built and the ability to build application that might just consume a portion of what you’ve built or to rebuild it is unique, right?

Byrne: We think it’s fresh and new, and a lot of what we did I really owe to the software team that we hired. They are very innovative and smart. We defined core requirements, but they were really innovative on how those requirements were turned into what was built.

Under the Agile process they did things like instantaneous builds. They had their own cloud environment for development, and they would write software instantly that would compile on the fly. It was much faster software development where we saw constant improvement.

The cloud was an advantage, because FCC and NTIA don’t share office space, and they could see how it was coming along, and were very active along the entire phase.

V1: What has come of this in terms of the exposure of this approach, and the awareness of what is possible among other government agencies?

Byrne: I think we demonstrated time to market that isn’t common in federal government, and that has definitely raised some eyebrows. The skepticism of the use of open source for production work has been squashed, we had a hit rate that was off the charts and it stood up.

Prior to being part of the FCC, I was a member of the National Geospatial Advisory Committee, which is the advisory committee to the Federal Geographic Data Committee. We have been working with that group to demonstrate the kinds of things that are available. I’m pretty sure that FGDC will have their annual report highlighting our work, and that should come out this Fall.

As an agency, the FCC has a need to consume different services, and we’ve approached the USDA to view their NAIP imagery base as a service. We currently pay Google a license fee for an imagery layer. We’ve written a one-page business case for consuming that as an OGC-compliant service. Approaching other agencies for consumption of services might really help turn the tide.

The primary goal was to make sure NTIA meets their requirement. The secondary goal is that the National Broadband Map meets a policy outcome, it’s not there as cool technology, it needs to serve some specific policy goals. For me as a geography, the opportunity to help develop a national map product has been spectacular, and the technology has been fun. I don’t want to say that we did it because technology is cool, it’s got to serve some purpose, and hopefully it will effect some policy. If it changes some geospatial thinking in federal government, then that’s an enormous win.