Dave Verwer’s Blog

We have a special event this month at NSManchester as Nokia are flying over from Finland to come and talk to us about developing for Nokia platforms along with information on how to make some money on it with the Ovi Store!

Nokia and the Ovi Store, Nokia Finland.
8th February 2010
This is a fantastic opportunity to get to talk personally with Nokia staff who are flying in from Finland especially for this event. Come along and get an insight into doing business on the Nokia platform.

• Discover the growing business opportunity with the rapidly growing Ovi Store from Nokia. Now the most international of all the apps stores, putting your apps in front of more consumers on more devices in more territories than any other.
• Find out how we support you through Forum Nokia at a local and Global level to maximise your opportunity.
• Learn about the building blocks to Nokia Ovi Store success such as Qt and WRT.

The meeting will be held at 6:30pm (talks start at 7pm) on Monday 8th February at MadLab (map). If you would like to attend, please register for a free ticket!

They say to bring the new year in with change and so that is what we are doing by changing the agenda completely for the January NSManchester meeting! :)

Our January meeting is now not going to be the Nokia meeting that we originally had planned but weather permitting we are still going ahead with the meeting, just with different content!

So, we will still be meeting at our new location of MadLab (map) on the 11th January at 6:30pm. What will we be talking about? Let’s see!

New Year’s Resolutions for Freelancers and Cocoa Developers, Matt West
Matt will be sharing his thoughts about what he is going to try to do more of, or less of this year! He will cover learning in general, having to make difficult business decisions, and observations on using social media for promotion!

Launching Balloons!, Dave Verwer
Dave will be talking about the steps leading up to the release of Balloons! and Balloons! Lite. Covering the steps that he took to prepare for the release, the successes and the lessons learned. He will also be talking about the plan to continue the marketing and PR for the app now that it is released.

The meeting will be held at 6:30pm (talks start at 7pm) on Monday 11th January at MadLab (map). If you would like to attend, please sign up!

If you had previously signed up for this meeting, you do not need to sign up again. If you no longer wish to attend, drop me a line and I will remove you from the list. The Nokia meeting is now going to happen on February 8th but will be announced separately with a much more detailed description!

So I read Andy’s list of his albums of 2000 – 2009 and while it is a great list I basically thought that I could do better! It was also an excuse to spend an evening listening to awesome music and making a Spotify playlist.

So, my top 10 albums of 2000 – 2009:

• Antony & The Johnsons - I Am a Bird Now.
• Arcade Fire – Funeral (not on Spotify).
• Arctic Monkeys - Whatever People Say I Am, That’s What I’m Not.
• Daft Punk - Discovery.
• Dan Le Sac & Scroobius Pip - Angles.
• Kings of Leon – Youth and Young Manhood.
• Maximo Park - Our Earthly Pleasures.
• Regina Spektor - Soviet Kitsch.
• St. Thomas - Hey Harmony.
• The Streets - Original Pirate Material.

I have seen more than half of these bands live this decade too, but that figure should probably be higher ;)

If you do one, track back to me and I will check it out.

One of the biggest problems we have had to deal with since Balloons! went live on the app store is the issues around how and why Balloons pop so I thought I would write up some of the experiences and explain some of the reasoning behind popping balloons!

First Problem: Misunderstanding

I talked about moderation before but there has been another, even more serious problem with the process of moderation. People who have never actually had a balloon reported (and therefore have never seen the “Reported!” badge in their balloon tracker) sometimes think that when balloons show a “Popped!” badge, that they have been popped as a result of someone else reporting them.

Before I go any further I would like to say that this is absolutely our fault and we should have made it clearer why balloons pop. We recently did this with the v1.1.1 update of Balloons! However this has been a real problem for our users and has resulted in a significant amount of bad reviews from people assuming we are censoring their balloons no matter how benign the content.

It is crippling to iPhone developers that we can not reply to these reviews and have no way to contact the disappointed customer (are you listening, Apple?) but I am really hoping the reviews on this subject will calm down now that we have made it clearer in the app UI.

Second Problem: Why Pop at All?

People don’t particularly like that their balloons pop for any reason and want them to fly around for years gathering hundreds of responses, this would be wonderful but unfortunately the reality of it is that with more and more balloons being launched every day, unless we also pop some every day, the probability of balloons being caught and replied to at all sharply decreases.

To explain that, let’s say that balloons are being launched at a constant rate of 1,000 balloons per day and let’s ignore the geographical constraints of where balloons travel for now and say that all balloons are available for catching anywhere on the earth. At the end of day one, you would have a 1 in 1,000 chance of having your balloon caught and replied to by someone else. Day two comes and another 1,000 balloons go up, by the end of day two your chances of having your balloon caught are now 1 in 2,000. As you can see, this would very quickly lead to a disaster where it could take weeks before your balloon is caught. Combine this with the fact that on average a balloon is caught between 5 and 10 times before someone decides to reply and the odds sink even lower.

So, we have to pop balloons if we want the service to survive.

Third Problem: Which Balloons Should be Popped?

So once we agreed that it was going to be necessary to pop balloons to keep the game fun, the question came of which balloons to pop. The popping algorithm has changed many times over the last month and has become significantly more sophisticated as we have learned more about how people are using the app.

As a starting point, when the service first launched we only popped balloons that had 7 replies on them but predictably this became unmanageable very quickly as some less interesting balloons never got replied to and so never actually popped and would hang around forever being caught and abandoned by everyone they met. The population of flying balloons got significantly bigger every day and it was clear that a fixed number of tags was a fatally flawed way to pop balloons.

Over the last month we have tried various methods of choosing which balloons to pop but the method that we implemented last week seems to be working very well and so here is a rough guide on how it works. I guess this is as close to a spoiler as you are ever likely to get with Balloons! so if you would rather believe it is all magic then look away now!

Solution: Work Out how Interesting a Balloon is!

So the algorithm of which balloons to pop now works based on an automatically calculated “interestingness” rating (thanks, flickr!) of each balloon. Interestingness is in no way a science and so don’t think we are doing anything clever, how interesting your balloon is is based on the following factors:

• Distance – Balloons that cross more continents are more likely to have met a diverse, interesting set of people. If your balloon has travelled further, it is seen to be more interesting.
• Photos – Everyone loves to look at pictures on balloons. The percentage of tags which include pictures makes a balloon more interesting.
• Words – Balloons that have one word messages or even one character messages (yes, it happens) are probably less interesting than those that have a fully formed sentence. More words is more interesting (note, not more characters, more words).
• Views – We count how many times people catch every balloon and then how many times people actually reply. If the ratio of catches to replies is high, people are abandoning your balloon without replying to it more often and so it is probably less interesting.
• Time Since Last Reply – If your balloon has been in the air for a long time since someone replied to it then the last tag may be something people are finding difficult to reply to and so it loses a few points.

Each factor is weighted slightly but I am not going to the specific weightings but we have tweaked them a few times now and they are giving good results right now.

One thing to mention is that this interestingness rating is much less reliable before we have a couple of days worth of data to work with and we also never want to pop a balloon before it has been flying at least a couple of days so any balloon which is less than 2 days old is excluded from ever being popped.

So, how do we actually decide how many balloons to pop every night? We pop a known number of balloons based roughly on the number of balloons launched on the previous day with a modification factor based on whether we want the balloon population to increase or decrease on a day to day basis.

We do not just indiscriminately pop the least interesting balloons though, there is a level of randomness to it so your balloon can still be lucky and get another 24 hours to become more interesting. Generally though, this system means that less interesting balloons get popped first to give the more interesting balloons a chance to fly further.

Another long post! Comments and discussion are very welcome.

They say that a picture is worth a thousand words, so I will keep this brief and get to the picture!

Popularity of balloon types across all balloons since launch.

Thoughts on this

• There is a predictable skew towards the balloons included with the free version of Balloons!
• The star balloons are very popular, not surprising really as they are positioned front and centre when picking a balloon.
• Is it surprising that the bright green balloon is more popular than the classic red balloon? I thought people would go for the classic.
• The animal balloons don’t seem to be very popular! Interesting because people really loved them in beta. (The cat being the most popular animal is no shock though, it is my favourite balloon type)

I am not going to try and read too much into this but I thought it might be fun to share ;)

I am happy to be able to announce the next meeting of NSManchester will be on the 7th December at BBC Manchester!  We are going to keep the same format from the last meeting of having two shorter talks and then a discussion/Q&A/show and tell session afterwards.

So, what do we have planned? Two awesome talks! That’s what!

Recent Advances in WebKit, Andy Shearer
Andy will be talking about how WebKit has advanced in the past few years.  Notably with mobile, HTML5 and 3D (CSS and WebGL).  I’ll demo a few of the more exciting features, and explain why such developments could radically change what we think is possible with web technologies.

Augmented Reality on the iPhone, Sam Easterby-Smith
Sam will tell the story of how he ended up creating an OpenGL based augmented reality solution for the iPhone. The inspiration, the intense weekend of remembering how to do trigonometry, the silly hacks to make it work (and how 3.1 did and didn’t help). He will be soliciting input on how the toolkit can potentially be taken forward (and possibly open-sourced) as well as how it compares with other offerings. Furthermore this will be a chance to discuss AR technologies, their use and potential impact on mobile applications in general.

The meeting will be held at 7pm on Monday 7th December at BBC Manchester, Oxford Road. If you would like to attend, please sign up!

If you would like to attend the meeting, it is important to sign up so that we can let the venue know who is coming. The BBC building is not a public building so if you don’t sign up you won’t be able to get in!

It didn’t get bad enough to cause a noticeable slowdown to users but as usage of Balloons! has ramped up this week the servers were gradually slowing down:

Balloons! server response times over the last 7 days

The advantage of being at the end of a cellular data connection is that even with response times occasionally going over 1000ms! it never got bad enough to notice while using the application.

Anyway, I deployed some optimisations to the server today and am very happy with the results.

Balloons! server response times over the last 6 hours

That should keep us going for a while!

During the development of Balloons! I would tell people about the idea and they would occasionally come back with “But what about abuse? Isn’t it going to be filled up with porn/spam/etc… within a few days?”. We also got the similar concerns when reviews started appearing, in fact the first comment on the TUAW review was:

How long until porn ruins it?

With the immediate follow up being:

Or spam?

So, we are now one week in to being live on the app store and with over 11,000 balloons launched I thought I would give an update on what has happened on this subject. The good news is that so far it has not been the huge problem that people predicted. In total, about 1.2% of all balloons have been reported and we ended up popping about 0.5% of all balloons and virtually all of them have been spam rather than inappropriate content.

Why isn’t it full of porn?

I believe that it is at least partly because of the design decision to only allow photos taken with the onboard camera to be attached to a balloon rather than images from the library. This has frustrated some users but I believe it was the right decision to make. Mainly because we wanted the photos to be taken in the here and now while you were launching a balloon, but also because of this issue.

So, no problems then?

I wouldn’t go that far but it is not an unmanageable problem so far but even so it has certainly been the subject of many conversations here in Shiny HQ this week. Sure enough even before we came out of beta we had our first spam balloon which was kinda depressing with it being from a beta tester. After launch, we continued to get a few spam balloons here and there but nothing really serious.

The spammers arrive.

The first problem we had was a couple of users who were just catching every balloon they could find, taking a picture of their spam web site and adding a link. This was our first serious problem and had the potential to easily ruin the service for the other 99.9% of our lovely users. What did we do? We decided that if someone has proven that their only use of the app is going to be for abuse and are repeatedly launching balloons with spam tags then we really have no option but to permanently ban their device from accessing the service. The iPhone is an expensive device and that is the only way to launch balloons at the moment, so banning a device should be very effective. This is absolutely a last resort but we felt it was the right decision to take to make the service sustainable. The bigger problem at the time was that we unfortunately did not have the ability to ban devices at the time so, it was time to get my head down and it was hastily implemented and deployed and the offending users banned.

On the subject of banning, we decided to make the server pretend that a balloon launched or re-launched by a banned user was successfully launched but just to drop the information before it gets written to the database. The alternative was to let the server return an error message to the user which would have only resulted in support queries and we would rather not waste time on people who choose to abuse the service. With all of that said, we have had to ban only three users so far so this kind of abuse is far from widespread.

So, banning was implemented for persistent abusers and moderation is in progress for everyone else. Everything is good, right?

One bad apple spoils the bunch?

Unfortunately there was another small problem with our moderation system. Because we were doing moderation at the level of an entire balloon rather than an individual tag, people could completely ruin an entire balloon with just one inappropriate tag. This causes the original launcher of the balloon to be confused as to why their completely innocent balloon has been reported for seemingly no reason and it was just plain sad when an amazing balloon was ruined by an idiot attaching a spam link as the last tag.

So over the last couple of days, I have implemented a new moderation system on our back end that allows individual tags to be removed as well as entire balloons and so we have a much more granular choice of whether to pop or re-launch a balloon. So, if all goes well with final testing we will have a new moderation system by the end of the day. At this point, we will re-moderate all of the flagged balloons so that we can re-launch those balloons that were ruined by a single bad tag.

So banning is implemented, moderation is now finely grained. That’s it now, right?

False positives.

Not quite! We also have some balloons being reported for seemingly no reason with no spam or inappropriate content to be found. Why? Well sometimes users are not personally happy with the content that they submitted (either location, text or photo) at the time when they launched a balloon and so they report their own balloon, simply to take it out of the sky. This is absolutely reasonable but not something we thought would happen. The problem is that it is incredibly hard for us to tell a balloon that has been accidentally reported from one that has been reported for something like this.

The solution to this one is not quite as simple as the others unfortunately and to really solve it in a satisfactory way we need to give people the ability to write us a little comment when they report a balloon. We are going to try and push out a new version of the client ASAP that asks for a reason when reporting a balloon allowing people to explain the problem with a balloon and solve this one too.

Summary

This ended up being a long post so here is the TLDR.

• The decision to not allow users to attach photos from their library seems to be sound.
• Spam is the biggest moderation problem so far and we have received almost no inappropriate content at all.
• Farming the moderation out to the users by letting them flag balloons for moderation is working well.
• The effort required to moderate the system at the moment is very manageable (a few minutes a day).
• Banning users is a last resort, but we do occasionally have to do it.
• We should have thought more about how our moderation system would work before we launched.
• We are going to be pushing out a maintenance release of the balloons client that asks for a reason for reporting a balloon.

So I have spent the last few months building our new iPhone app, Balloons!

Obviously it needs a back end web service to move all of the balloons around the skies and host all of the attached messages. The back end is (predictably) written in Ruby on Rails and we just finished moving the hosting for it to new Brightbox[2] VPS’s ready for the launch. When we do go live, server reliability and the ability to scale is pretty important, after all we are hoping it will be successful and for every copy (including the free app) downloaded, our servers get hit a little harder.

Now I don’t have a huge amount of experience building high availability sites but a few years ago in a different life (working with .net & Windows[1]) I was working on a project which needed to be extremely reliable and also was expecting a large user load on day one. This meant building a reliable database cluster and load balanced web servers. With help we did it, and it worked, but it was incredibly expensive and complicated. It took months of planning and weeks of work to build and deploy and left me with a healthy fear of doing anything like it again.

So it was a very pleasant surprise this week that when I started looking at load balancing for the Balloons! servers and found out that not only was it really cheap (£19/mo) but that it only took about 10 minutes to set up. Combine that with the use of the Brightbox MySQL cluster rather than a locally hosted DB server and with a few simple tweaks to our Capistrano deployment scripts I was able to have a good level of reliability for our back end for a trivial amount of extra cost (I halved the size of our VPS, but doubled the number of machines so the cost and capacity was roughly the same) and effort.

So yesterday Brightbox had some pretty major downtime with a SAN failing in what seems to be a fairly epic way. Luckily it didn’t affect any of my servers but if it had, Balloons! would not have been unavailable for more than a few seconds because we chose to load balance and Brightbox (very sensibly) offered to put our virtual servers on different host machines and also different SAN’s.

A quick look at twitter showed that people are naturally pretty upset about such a large amount of downtime and some people were talking about moving hosting providers. That is of course their prerogative but moving to another hosting provider is no protection against disasters like this because stuff like this happens to every hosting provider from time to time and as I understand it, it was not due to anything other than bad luck with several disks failing at the same time.

The point I am trying to make is that if your site is important enough for you to lose money/reputation from it being unavailable for more than a few minutes then building an amount of reliability in is your responsibility as no single server is ever going to be 100% reliable on its own, no matter how good the hosting provider. The cost and complexity of doing it these days should make it an easy decision to make.

Now let’s just hope we sell enough copies Balloons! to really put it to the test :)

1. This is not a rant about Windows and I am sure it is much easier to load balance stuff and build a SQL Server cluster than it was then. I would imagine the cost is still a few bands up from how we have done it for Balloons! though.
2. In the interest of disclosure, most of the team at Brightbox are friends of mine and so take this with the appropriate amount of bias although it was not written with any intentionally.

UPDATE: There was a mix up with room dates and the meeting is now on Monday 12th October, not the 14th.

I am happy to announce that we will be having another meeting of NSManchester on Monday 12th October at the BBC building in Manchester.

This month we are going to slightly change the format and have 2 shorter talks (15-20 minutes) rather than one long talk. The talks will be followed by the chance to start a discussion on any subject related to iPhone or Mac development or if you want to show something off, we would love to see it! I have no idea how this will work but given the chatty group we have, I am sure we will not be sitting in silence!

So, what do we have planned?

Mac and iPhone Accessibility, Martin Pilkington
Making your application accessible should be as important as making it not crash. This talk will give a brief over view of VoiceOver, the Accessibility Tools and some tips on how to make your Mac or iPhone application much more accessible with just 15 minutes work.

Making a Simple Core Animation Component, Dave Verwer
Learn how to build a simple Core Animation based component in 15 minutes. Dip into some of the techniques that were needed to build a simple dial style score board component using CALayer objects.

We also have two copies of Acorn and VoodooPad to give away at this meeting thanks to the generosity and general awesomeness of Gus over at Flying Meat Software. Thanks very much Gus!

The meeting will be held at 7pm on Monday 12th October and again the BBC are kindly hosting us at BBC Manchester, Oxford Road. If you would like to attend, please sign up!

If you would like to attend the meeting, it is important to sign up so that we can let the venue know who is coming. The BBC building is not a public building so if you don’t sign up you won’t be able to get in!