Mon 23 Sep

Lessons for Community Networks

I had the good fortune to run into Isaac Wilder from the Free Network Foundation and a couple folks from at the Oakland mesh meetup a couple weeks ago. Isaac and I have gone back and forth on the viability of mesh networks, so it was nice to finally put a face with the name.

We had a great discussions on their respective networks and about the cellular network we've been working on in Papua. There's obviously good work being done by both projects; I was particularly impressed by's scale (thousands of really operational nodes!). One thing that I was really glad about was the recognition by all parties that "building mesh networks" isn't really the end goal; it's about building free-as-in-freedom decentralized community networks. Guifi for example is planning on building out fiber links to augment their backhaul.

There were a few points I took away from our discussion that I think are important guidelines for people building these decentralized community networks.

  • Embrace commerce. Our network in Papua is explicitly for-profit, and one of the most remarkable aspects of it is that it has been profitable for the local operators since soon after going live (we'll have a paper out about this soon). I think this has aligned incentives between users and the local operator to ensure reliable operation is a long-term goal. Paying for service and demonstrating a clear business model I think also conveys to users that the network is for real, and there to stay. In's network, they have independent agents ("Guifi Professionals") who provide installation service for nominal fees. In my experience one of the hardest problems in keeping a network operational is maintaining adequate involvement from experienced people who can fix problems when the arise; providing income to those people is an excellent answer.

  • Don't automate everything. One of the guys from Guifi explained that they had made explicit decisions against automating certain aspects of their network operations, like firmware updates. While they make impressively extensive use of automation for many common network activities, keeping a human in the loop is a great way to avoid accidentally messing up large parts of a network. In Papua, we rely on human intervention for handling certain faults; while it's inefficient (and we do in fact want to automate this process eventually) it's given us a core set of people who are capable of basic debugging of the system and who can fix certain problems on their own, without our intervention. In short, having a human in the loop can improve resilience of the network, not to mention the positive side effect of improving peoples' understandings of how networks work.

  • It's a social problem. Isaac said something I really liked. Paraphrased, it was "a big part of our innovation is social." Obviously there is technical innovation still occurring within the free networking community, but like Isaac I agree that building large wireless networks is essentially a solved problem (in my survey of WISPs I conducted last year I spoke with dozens of operators who had thousands of subscribers, some spread across territories spanning hundreds of miles). To me, a more interesting question than what mesh routing protocol to use is what are other, more democratic, methods of building and operating communications infrastructure? The answer we've explored in Papua is to distribute control to local entrepreneurs who are more responsive to community needs than national-scale telcos; this is analogous to what rural WISPs have done worldwide. Guifi and the FNF are really pushing the limits on what's possible with building fully horizontal networks -- in the language of my recent paper, networks that are truly decentralized from a management perspective. Our friends at Rhizomatica are taking a similar approach in Oaxaca with their cellular network, though due to the fundamental nature of phone networks it's a bit harder to make them truly decentralized (phone networks' first design imperative is the ability to bill and control usage, after all). While I personally believe the "small entrepreneur" model we're using in Papua strikes the right balance between equitable operation and practicality, I'm excited to see work on other models.

Now, all this said, I still firmly believe that for blackout circumvention a la Egypt 2011 mesh networks are a poor answer and inefficient use of resources. I also think it's important for anyone working in the space to have a firm understanding of the fundamental reasons why real mesh networks don't scale so they can design around that [1]. But it's a good thing in my mind that projects like the FNF and Guifi exist because they give us practical examples of alternatives to traditional top-down models of telecommunications. That's a cause I support and I look forward to exciting developments from both of these projects.

[1] There was a comment at the mesh meetup that night that there's no such thing as scarcity of bandwidth in a mesh network since you can always add more nodes or use more frequency. This is of course not true: radios in practice have limited bandwidth, regardless of spectrum regulation, and channel contention and forwarding load impart fundamental limitations on the capacity available in a mesh. There's also the practical problem that devices have a limit on their forwarding capacity. If a builder of a community network starts with this expectation, they're setting themselves up for failure.

Wed 24 Jul

Why Mesh Networks Still Aren't the Right Answer: The Paper

A while back I wrote a blog post about why wireless mesh networks aren't a viable answer to censorship. I've been wanting to write a more rigorous followup to that post for a while, and I'm happy to say that such a paper has finally been written and will appear at the FOCI 2013 workshop. You can take a look at it here: Building Dissent Networks: Towards Effective Countermeasures against Large-Scale Communications Blackouts.

The tl;dr is that systems for dissident communication need to prioritize user safety and scalability, and mesh networks (and many other currently proposed systems) don't fit the bill.

This is joint work with my colleagues Yahel Ben-David, Giulia Fanti, Eric Brewer, and Scott Shenker.

Thu 14 Feb

TIER's micro-telco in Papua goes live

This is too cool not to share -- TIER's VillageBTS project in Papua finally went live this week. Check out the blog post Kurtis Heimerl, whose thesis work VBTS is, wrote on the TIER blog. I spent a few weeks helping deploy this back in November.

It's only been a few days, but the network is already turning a profit (at least on operating costs; capital expenditures, like the BTS itself, were paid for by TIER's grant funds). I think that's a huge demonstration of the value this system brings to the local community, and not something you usually see in academic development work. A quick glance at the logs shows that in the first four days of operation we've already handled almost 1100 SMS (about 40% were between users of our network, with the remainder between our users and the outside world).

All in all, very cool so far. I can't wait to see how this project evolves.

Fri 1 Feb

Teardown of an Ubiquiti Nanostation Loco M5

I was doing some work on a Ubiquiti Nanostation Loco M5 yesterday and wound up bricking it (protip: the 'erase all' command in uBoot also erases the bootloader itself... whoops!). Rather than getting out the JTAG I decided I'd do a full teardown of it since I hadn't seen one of this device before.

I'd long since removed the board from the case, so here's the board itself. The silver thing on top is shielding for the RF components. It's pretty easy to pop it off with a flathead screwdriver -- just jam it into the sides until it bends up.

There are a few chips on this board. The big one in the center is the 400MHz MIPS SoC, the Atheros AR7240-AH1A. The chip below that is the Winbond 25X64VFIG, a flash memory module. On the left side of the board, we have an M-Tek H16125MCG, an Ethernet transformer (I'm not sure if this is found on all ethernet devices, or if it has something to do with PoE). Directly above the transformer is a chip that's covered in stickers, the one "mystery chip" on this board (at least to me). Removing the stickers, I found three markings. On the top, it has a logo followed by "MIRA", which appears to be a custom IC manufacturer from Taiwan. There are two other markings: one next to the logo reading P3S56040ETP, and one on the bottom reading 944ANF74-G5. I found one reference to the latter, on this site, suggesting both that this is a 32MB SDRAM chip and that the Buffalo WHR-HP-G300N has a very similar bill of materials to the NSLM5!

Inside the shielded area, directly above the AR7240, is the radio chip, the Atheros AR9280-AL1A. The two chips to the left are both Skyworks SE2593A20, a dual-band WiFi frontend which does all the switching, filtering, and so forth. If you look closely towards the bottom left of the shielded area, you can see two leads running to a pair of connectors for RF pigtails (they're the golden squares with a dark circle). These are the antenna lines -- you can follow them back to the antenna port of each of the frontend chips. I guess they left those leads for testing purposes; I don't know if they're functional.

A couple other points of interest are the serial port on the lower right side of the board, just next to the Ethernet port. The reset button is on the bottom left, under the Ethernet transformer.

Flipping the board over, we see the antennas. These are two panel antennas layered on top of each other. They're attached with some plastic screws and solder to connect them to the antenna leads, along with one metal bolt which looks like it goes to ground. The lower layer of the antenna is the only one that's soldered to the board. The upper layer actually doesn't make any electrical contact with the lower one.

Up to this point, you could probably re-assemble the board. But this is a teardown, so I broke off the solder between the lower antenna and the board. It snaps in two pretty easily.

(Note my current favorite tool in the top right corner, vicegrips!)

There's nothing too exciting underneath, just the bottom of the Atheros chips.

I still have all the parts, so if you'd like better pictures or have questions about any parts I didn't discuss, let me know!

Thu 29 Nov

Syrian Internet Cut Off

Well, it's happened again: a country's been cut off the Internet. This time it's Syria.

There's good coverage of the technical details of this outage on the Renesys and CloudFlare blogs. The short version is that all Internet traffic into and out of Syria flows through a single, government-owned ISP, the Syrian Telecommunications Establishment (AS29386). This ISP withdrew their BGP routes from their providers, disabling access for all Syrian IP's (since every ISP in Syria relies on AS29386 for their international connectivity). This is effectively the same network structure that exists in Iran, and it makes it trivial to shut off an entire nation's Internet access: all traffic flows through a single point, and the government controls that point.

My TIER colleague Yahel Ben-David brought up the great point a couple months ago that it's kind of odd that method of choice for country-scale Internet blackouts seems to be withdrawing BGP routes, as opposed to just unplugging cables or powering off routers. The Syrian government tried to blame a cable cut for this blackout, but it's pretty hard to say all your BGP routes just "accidentally" got withdrawn. Withdrawing a BGP route is literally broadcast publicly around the world, so it's not particularly subtle either. I don't have a good explanation of why this is done: maybe the engineers tasked with implementing the blackout are just familiar with BGP? Maybe they don't want to disrupt their consecutive days of uptime numbers? If you're a network engineer who's implemented a national Internet blackout and has the answer, get in touch.

Next → Page 1 of 11