First Problem: Misunderstanding

I talked about moderation before but there has been another, even more serious problem with the process of moderation. People who have never actually had a balloon reported (and therefore have never seen the “Reported!” badge in their balloon tracker) sometimes think that when balloons show a “Popped!” badge, that they have been popped as a result of someone else reporting them.

Before I go any further I would like to say that this is absolutely our fault and we should have made it clearer why balloons pop. We recently did this with the v1.1.1 update of Balloons! However this has been a real problem for our users and has resulted in a significant amount of bad reviews from people assuming we are censoring their balloons no matter how benign the content.

It is crippling to iPhone developers that we can not reply to these reviews and have no way to contact the disappointed customer (are you listening, Apple?) but I am really hoping the reviews on this subject will calm down now that we have made it clearer in the app UI.

Second Problem: Why Pop at All?

People don’t particularly like that their balloons pop for any reason and want them to fly around for years gathering hundreds of responses, this would be wonderful but unfortunately the reality of it is that with more and more balloons being launched every day, unless we also pop some every day, the probability of balloons being caught and replied to at all sharply decreases.

To explain that, let’s say that balloons are being launched at a constant rate of 1,000 balloons per day and let’s ignore the geographical constraints of where balloons travel for now and say that all balloons are available for catching anywhere on the earth. At the end of day one, you would have a 1 in 1,000 chance of having your balloon caught and replied to by someone else. Day two comes and another 1,000 balloons go up, by the end of day two your chances of having your balloon caught are now 1 in 2,000. As you can see, this would very quickly lead to a disaster where it could take weeks before your balloon is caught. Combine this with the fact that on average a balloon is caught between 5 and 10 times before someone decides to reply and the odds sink even lower.

So, we have to pop balloons if we want the service to survive.

Third Problem: Which Balloons Should be Popped?

So once we agreed that it was going to be necessary to pop balloons to keep the game fun, the question came of which balloons to pop. The popping algorithm has changed many times over the last month and has become significantly more sophisticated as we have learned more about how people are using the app.

As a starting point, when the service first launched we only popped balloons that had 7 replies on them but predictably this became unmanageable very quickly as some less interesting balloons never got replied to and so never actually popped and would hang around forever being caught and abandoned by everyone they met. The population of flying balloons got significantly bigger every day and it was clear that a fixed number of tags was a fatally flawed way to pop balloons.

Over the last month we have tried various methods of choosing which balloons to pop but the method that we implemented last week seems to be working very well and so here is a rough guide on how it works. I guess this is as close to a spoiler as you are ever likely to get with Balloons! so if you would rather believe it is all magic then look away now!

Solution: Work Out how Interesting a Balloon is!

So the algorithm of which balloons to pop now works based on an automatically calculated “interestingness” rating (thanks, flickr!) of each balloon. Interestingness is in no way a science and so don’t think we are doing anything clever, how interesting your balloon is is based on the following factors:

• Distance – Balloons that cross more continents are more likely to have met a diverse, interesting set of people. If your balloon has travelled further, it is seen to be more interesting.
• Photos – Everyone loves to look at pictures on balloons. The percentage of tags which include pictures makes a balloon more interesting.
• Words – Balloons that have one word messages or even one character messages (yes, it happens) are probably less interesting than those that have a fully formed sentence. More words is more interesting (note, not more characters, more words).
• Views – We count how many times people catch every balloon and then how many times people actually reply. If the ratio of catches to replies is high, people are abandoning your balloon without replying to it more often and so it is probably less interesting.
• Time Since Last Reply – If your balloon has been in the air for a long time since someone replied to it then the last tag may be something people are finding difficult to reply to and so it loses a few points.

Each factor is weighted slightly but I am not going to the specific weightings but we have tweaked them a few times now and they are giving good results right now.

One thing to mention is that this interestingness rating is much less reliable before we have a couple of days worth of data to work with and we also never want to pop a balloon before it has been flying at least a couple of days so any balloon which is less than 2 days old is excluded from ever being popped.

So, how do we actually decide how many balloons to pop every night? We pop a known number of balloons based roughly on the number of balloons launched on the previous day with a modification factor based on whether we want the balloon population to increase or decrease on a day to day basis.

We do not just indiscriminately pop the least interesting balloons though, there is a level of randomness to it so your balloon can still be lucky and get another 24 hours to become more interesting. Generally though, this system means that less interesting balloons get popped first to give the more interesting balloons a chance to fly further.

Another long post! Comments and discussion are very welcome.

Popularity of balloon types across all balloons since launch.

Thoughts on this

• There is a predictable skew towards the balloons included with the free version of Balloons!
• The star balloons are very popular, not surprising really as they are positioned front and centre when picking a balloon.
• Is it surprising that the bright green balloon is more popular than the classic red balloon? I thought people would go for the classic.
• The animal balloons don’t seem to be very popular! Interesting because people really loved them in beta. (The cat being the most popular animal is no shock though, it is my favourite balloon type)

I am not going to try and read too much into this but I thought it might be fun to share ;)

Balloons! server response times over the last 7 days

The advantage of being at the end of a cellular data connection is that even with response times occasionally going over 1000ms! it never got bad enough to notice while using the application.

Anyway, I deployed some optimisations to the server today and am very happy with the results.

Balloons! server response times over the last 6 hours

That should keep us going for a while!

How long until porn ruins it?

With the immediate follow up being:

Or spam?

So, we are now one week in to being live on the app store and with over 11,000 balloons launched I thought I would give an update on what has happened on this subject. The good news is that so far it has not been the huge problem that people predicted. In total, about 1.2% of all balloons have been reported and we ended up popping about 0.5% of all balloons and virtually all of them have been spam rather than inappropriate content.

Why isn’t it full of porn?

I believe that it is at least partly because of the design decision to only allow photos taken with the onboard camera to be attached to a balloon rather than images from the library. This has frustrated some users but I believe it was the right decision to make. Mainly because we wanted the photos to be taken in the here and now while you were launching a balloon, but also because of this issue.

So, no problems then?

I wouldn’t go that far but it is not an unmanageable problem so far but even so it has certainly been the subject of many conversations here in Shiny HQ this week. Sure enough even before we came out of beta we had our first spam balloon which was kinda depressing with it being from a beta tester. After launch, we continued to get a few spam balloons here and there but nothing really serious.

The spammers arrive.

The first problem we had was a couple of users who were just catching every balloon they could find, taking a picture of their spam web site and adding a link. This was our first serious problem and had the potential to easily ruin the service for the other 99.9% of our lovely users. What did we do? We decided that if someone has proven that their only use of the app is going to be for abuse and are repeatedly launching balloons with spam tags then we really have no option but to permanently ban their device from accessing the service. The iPhone is an expensive device and that is the only way to launch balloons at the moment, so banning a device should be very effective. This is absolutely a last resort but we felt it was the right decision to take to make the service sustainable. The bigger problem at the time was that we unfortunately did not have the ability to ban devices at the time so, it was time to get my head down and it was hastily implemented and deployed and the offending users banned.

On the subject of banning, we decided to make the server pretend that a balloon launched or re-launched by a banned user was successfully launched but just to drop the information before it gets written to the database. The alternative was to let the server return an error message to the user which would have only resulted in support queries and we would rather not waste time on people who choose to abuse the service. With all of that said, we have had to ban only three users so far so this kind of abuse is far from widespread.

So, banning was implemented for persistent abusers and moderation is in progress for everyone else. Everything is good, right?

One bad apple spoils the bunch?

Unfortunately there was another small problem with our moderation system. Because we were doing moderation at the level of an entire balloon rather than an individual tag, people could completely ruin an entire balloon with just one inappropriate tag. This causes the original launcher of the balloon to be confused as to why their completely innocent balloon has been reported for seemingly no reason and it was just plain sad when an amazing balloon was ruined by an idiot attaching a spam link as the last tag.

So over the last couple of days, I have implemented a new moderation system on our back end that allows individual tags to be removed as well as entire balloons and so we have a much more granular choice of whether to pop or re-launch a balloon. So, if all goes well with final testing we will have a new moderation system by the end of the day. At this point, we will re-moderate all of the flagged balloons so that we can re-launch those balloons that were ruined by a single bad tag.

So banning is implemented, moderation is now finely grained. That’s it now, right?

False positives.

Not quite! We also have some balloons being reported for seemingly no reason with no spam or inappropriate content to be found. Why? Well sometimes users are not personally happy with the content that they submitted (either location, text or photo) at the time when they launched a balloon and so they report their own balloon, simply to take it out of the sky. This is absolutely reasonable but not something we thought would happen. The problem is that it is incredibly hard for us to tell a balloon that has been accidentally reported from one that has been reported for something like this.

The solution to this one is not quite as simple as the others unfortunately and to really solve it in a satisfactory way we need to give people the ability to write us a little comment when they report a balloon. We are going to try and push out a new version of the client ASAP that asks for a reason when reporting a balloon allowing people to explain the problem with a balloon and solve this one too.

Summary

This ended up being a long post so here is the TLDR.

• The decision to not allow users to attach photos from their library seems to be sound.
• Spam is the biggest moderation problem so far and we have received almost no inappropriate content at all.
• Farming the moderation out to the users by letting them flag balloons for moderation is working well.
• The effort required to moderate the system at the moment is very manageable (a few minutes a day).
• Banning users is a last resort, but we do occasionally have to do it.
• We should have thought more about how our moderation system would work before we launched.
• We are going to be pushing out a maintenance release of the balloons client that asks for a reason for reporting a balloon.

Obviously it needs a back end web service to move all of the balloons around the skies and host all of the attached messages. The back end is (predictably) written in Ruby on Rails and we just finished moving the hosting for it to new Brightbox[2] VPS’s ready for the launch. When we do go live, server reliability and the ability to scale is pretty important, after all we are hoping it will be successful and for every copy (including the free app) downloaded, our servers get hit a little harder.

Now I don’t have a huge amount of experience building high availability sites but a few years ago in a different life (working with .net & Windows[1]) I was working on a project which needed to be extremely reliable and also was expecting a large user load on day one. This meant building a reliable database cluster and load balanced web servers. With help we did it, and it worked, but it was incredibly expensive and complicated. It took months of planning and weeks of work to build and deploy and left me with a healthy fear of doing anything like it again.

So it was a very pleasant surprise this week that when I started looking at load balancing for the Balloons! servers and found out that not only was it really cheap (£19/mo) but that it only took about 10 minutes to set up. Combine that with the use of the Brightbox MySQL cluster rather than a locally hosted DB server and with a few simple tweaks to our Capistrano deployment scripts I was able to have a good level of reliability for our back end for a trivial amount of extra cost (I halved the size of our VPS, but doubled the number of machines so the cost and capacity was roughly the same) and effort.

So yesterday Brightbox had some pretty major downtime with a SAN failing in what seems to be a fairly epic way. Luckily it didn’t affect any of my servers but if it had, Balloons! would not have been unavailable for more than a few seconds because we chose to load balance and Brightbox (very sensibly) offered to put our virtual servers on different host machines and also different SAN’s.

A quick look at twitter showed that people are naturally pretty upset about such a large amount of downtime and some people were talking about moving hosting providers. That is of course their prerogative but moving to another hosting provider is no protection against disasters like this because stuff like this happens to every hosting provider from time to time and as I understand it, it was not due to anything other than bad luck with several disks failing at the same time.

The point I am trying to make is that if your site is important enough for you to lose money/reputation from it being unavailable for more than a few minutes then building an amount of reliability in is your responsibility as no single server is ever going to be 100% reliable on its own, no matter how good the hosting provider. The cost and complexity of doing it these days should make it an easy decision to make.

Now let’s just hope we sell enough copies Balloons! to really put it to the test :)

1. This is not a rant about Windows and I am sure it is much easier to load balance stuff and build a SQL Server cluster than it was then. I would imagine the cost is still a few bands up from how we have done it for Balloons! though.
2. In the interest of disclosure, most of the team at Brightbox are friends of mine and so take this with the appropriate amount of bias although it was not written with any intentionally.

Last week one of the teams raised a question about how to create a rolling score board and we worked through a couple of ideas with some sample code but did not implement it fully. The idea stuck in my mind all week though and so in a little free time this morning I felt inspired to knock something together around it and this is what dropped out of Xcode at the end of it.

It has been implemented as a simple CALayer subclass that takes care of all of the management of the internal layers and animations and exposes a very simple interface to set the score value.

The images must be named 0.png through 9.png and included in the resources bundle but as long as you do that it should cope with any sized images as long as they are all the same size. The code can be found on github.

UPDATE: Thanks to John McKerrell for the suggestion that to give a little more realism, digits should transition in from the bottom coming from a lower number and in from the top from a higher number. This needed a bit of a hack around the 0/9 boundary but I think the outcome is pretty good. This is how it looks.

This was also a nice break from writing code on large projects, I have been really slacking on writing any code that isn’t going into a shipping codebase recently and so this was a very refreshing little break from the grind.

A friend recommended that I try out Live Poker by Zynga this weekend, the game is a really nice implementation of online Texas Hold-Em Poker, complete with friend codes so you can play with friends etc, very nice but not worthy of a blog post in itself.

What is worthy of the post is the business model that they are using, when you search for Liver Poker by Zynga on the iTunes Store this you get 6 hits ranging from free to £35. As far as I can tell you get the same exact game, the price just determines how much (play) money you start with and how much extra you get for free each week.

What a great idea…

Back at BarCamp Leeds many months ago I gave my first iPhone talk and I decided to attempt to write a native iPhone app in 20 minutes. I really wanted to talk about iPhone development but I also wanted to let people know that it is not hard to get started so I really needed to be able to do “New Project” at the start of the session and with the exception of some copy/paste for the XML fetching code to show the end to end process of building a simple app. Anyway the session went pretty well and I have given it a few other places now like NSManchester, GeekUp Liverpool and NWMUG.

Anyway I never posted the source for it as I said I would many times and yesterday @hereinthehive asked for it, so here it is.

Bear in mind this is the result of a 20 minute coding demo primarily designed to demonstrate the absolute basics of a table view application to people who had never been exposed to Objective C or iPhone development so the structure of the application is intentionally simplified and even the default cell optimisation code has been removed to keep the example simple.

With that said, have at it.

Build an iPhone app in 20 minutes – Source Code (This code is released under a MIT license).

Download: iPhone Screen Design Template.

If you have a UIScrollView containing controls which need to respond to swipe/drag touches themselves rather than scroll the container then just set delayContentTouches to NO. Once this is done the default behaviour is then to immediately pass touches into any UIControl contained within the scroll view instead of scrolling.