Introducing MeetupQuest!
For the last several years, Meetup has given a few graduate and undergrad students the opportunity to spend their summers working in the tech world as interns at Meetup HQ. They sit with the rest of the dev team, work on the code base, make improvements, fix bugs, and generally have the same opportunities and responsibilities of any other programmer here (and, yes, we pay them). This summer, our interns were Kathleen Lister, a rising senior at Brown University who focused on backend tasks, and Jillian Munson, a graduate student at Stony Brook University who was working on the front end.
In addition to a variety of smaller projects that Kathleen and Jillian finished this summer, they took it upon themselves to build an entirely new area of the site, which they call MeetupQuest. We’re pretty excited about it, and we’re going to let them walk you through the details of the creation process.
-Duncan Ariey, QA Engineer
Kathleen:
Life at Meetup HQ
MeetupHQ is a place with a mission. Many companies have taglines, goals, and “mission statements”. At Meetup, it’s a mantra. Everyone here is very excited about the prospect of having “a Meetup Everywhere about Most Everything”, and actually believes that Meetup has the potential to make the world a better place. This is what I have been lucky enough to be a part of as an intern at Meetup.
This is the main goal around which everything here is focused, but there are many consistent stumbling blocks in the way. One of these major hurdles is convincing someone sitting curled up with their computer to make the leap and venture into the scary outdoors to meetup with strangers. This fear of the unknown, set against the security and comfort of the same old routine, is a very hard mental block to overcome. A second, smaller stumbling block is that of actually letting people know about Meetups that might interest them. In our three months at Meetup my fellow intern and I spent our due time fixing bugs and the like, but we also got a chance to create a fun product to help overcome these stumbling blocks.
The Intern Project
Our plan was to come up with something that would help provide the extra motivation, the extra kick in the bum needed to get people off of their lonely couches and into the warm arms of new friends at a Meetup. What sort of incentive could we provide? We don’t have a ton of cash to bribe people with, or cars to give away in raffles. Instead, we went with a familiar reward system: points and a leader board. Levels and achievements. Bragging rights. What do all of these things add up to? An RPG. And thus “MeetupQuest: An RPG for the Real World” was born.
We came up with character classes that matched the main categories of Meetup Groups (pets/parenting became “Domestic Diva”, tech/hobbies/games became “Nerd”, etc), and got going. The idea was to create “quests” of randomly chosen Meetups for people to go to and incentivise attendance with points and levels. This random selection of Meetups is intended to expose people to Meetup Groups that they otherwise would not have been aware of. The game itself provides a context to help people overcome the hurdle of actually making it out the door. The game acts as an excuse: a reason to be talking to the strangers.
And so this game evolved. After many design meetings, arguments, and compromises, we arrived at this basic format: you begin by generating a quest. You can choose beginner, medium, or difficult (which simply refer to how tightly packed the Meetups will be), and a character class to which the Meetups should relate. A quest is then generated using these parameters (I’ll go into this later), and the you can choose to either accept the proposed quest or generate a different one. Once you have accepted the quest, you are prompted to rsvp for each Meetup (+3 points), and then attend each Meetup and ask the organizer of the group to confirm you as attended (+10 points). After you return, if you upload a photo you get +3 bonus points. These points go into whichever category each Meetup belongs to, and you level up in a category every 50 points. If you successfully attend all Meetups on a quest, you get +20 bonus points, which are applied to your total score. Our plan is to eventually provide incentives for leveling up (added abilities etc), but for now it’s just bragging rights.
Quest Generation
Implementing this was a huge undertaking. Jill will tell you about the frontend challenges, but as I was a backend engineering intern, I had many unique challenges of my own. I’ll cover a few highlights here, but try not to go into too many gory details. The first of these was figuring out how to generate a quest. I’m not going to go into the details here (if you’re interested, leave a comment!), but the main challenge was achieving a balance between randomness and having a fairly even distribution. There was also the problem of figuring out which Meetups to filter out, and which to allow. I eventually settled on filtering out Meetups with hidden locations (they probably don’t want random people showing up unannounced, right?), Meetups from private groups (same), and Meetups that last more than 7 hours (if the idea is to go to a lot of Meetups, we don’t want to be sending you on a week long cruise). The problem with this is that it still lets some bad eggs through. Some Meetups don’t put an end time, so there are still plenty of week long cruises in the mix that claim to last only 3 hours (the default Meetup duration). Some Meetups are kind of spammy and are promoting $50 tango classes. Some don’t specify a location, and default to their home city of NYC when in fact they are in Italy. The problem is that there is no way to tell the good from the bad on the backend, and I didn’t want to leave out perfectly good Meetups while trying to catch the bad. My solution was to leave this up to the member. I built out a feature which allows a member to get a new quest with only one Meetup swapped out. You give me a questId and the id of a Meetup to get rid of, and I create a new quest with all the Meetups but the marked one, and one of the gaps filled in with a brand new Meetup. (I actually use my original quest generation algorithm with length=1 to get this new Meetup. Pretty efficient, right?) From the member’s perspective, this would look like x-ing out an Meetup that was too expensive, too spammy, too long, or in Italy, and having it replaced by a new (hopefully better) one. This allows the member to tweak their quest if some of these bad eggs get through, or if a time doesn’t work for them, or a Meetup costs too much money.
Databases
I also needed to think about what tables I needed to create in the database. This is a delicate balance between not throwing away useful info, having everything you need easily and quickly accessible, and also not duplicating stuff already stored in the vast existing database. I also had to think ahead and anticipate all of my needs as the project unfolded, something that I almost did, although I’ve had to make two schema changes so far. I wound up with 4 tables: quest, member_quest, quest_action, and quest_event. Two of these were just mappings (quest -> events and member -> quests). The quest table stores info about quests, while the quest_action table stores a history of relevant actions taken by each member, used for calculating scores and tracking progress. Four tables is not too bad.
Scoring
Now I had to start worrying about scores. I decided not to store them in the database to give our scoring system more flexibility, so I had to calculate them on the fly. This was simple enough for an individual’s scores – I just calculated them based on a history of actions that had been recorded, but when it came to highscores it got a bit more complicated. I didn’t want to recalculate EVERYONE’s scores every time someone viewed the highscores page. So, I had to cache them somehow, but also make sure that the person viewing the page had their scores up to date in the list, if they made the cut. I solved this by storing top scores in each category in redis, and updating this each time individual scores were calculated in order to show each member completely up to date information about themselves, and almost up to date info about everyone else. The rest of the high scores are recalculated on a nightly basis (I’ll get into this later).
1000 Little Problems
In addition to these big problems, I also had a thousand little problems to deal with. For instance, how can each quest have a snazzy, consistent name if we aren’t storing names in the database? How do we recycle old quests without showing people the same stuff over and over? How should we divide the Meetups up by category? To solve the naming problem, I decided to generate the names on the fly. I came up with a list of nouns and adjectives for each category to be put into the format “The Quest Of The <adjective> <noun>”. I started choosing them at random based on a random number generated using the quest_id as a seed. This means that although they are regenerated each time, the name will remain the same for each individual quest as long as the list of words don’t change, because the same index will be chosen “randomly” each time. I solved the recycling quests problem by reusing quests that match the inputs in the request, and are close enough in dates, but caching quests to exclude from this in memcache, so that in a single session the member does get some pre-generated quests, but never the same one twice, and new quests begin being generated once the old ones run out.
Nightly Job
Now, I had a problem. Because we wanted to have a full quest object, complete with a name (quest_id) and a list of events, on the front end after quest generation, each newly generated quest was actually being stored in the database. This is fine, except that many of the quests are never used, and this creates a lot of clutter. I solved this problem by writing a nightly job to delete quests from the database that are not associated with any member. In this same nightly job I also recalculate all of the high scores (I told you I would get back to this!), mark past quests as past, and figure out whether or not anyone has flaked from an event (if they rsvp’d and have not been confirmed) for scoring purposes.
Thinking Ahead
Throughout all of this I had to learn to think about communicating with the front end. I built several internal objects on the back end strictly for storing data in a way that was accessible on the front end, and that fit that use case well. Also, because of the different pace at which the frontend and backend move, I had to constantly be thinking ahead on features. I had to start thinking about how to build a backend that anticipates the frontend needs. For instance, as the current manner of taking attendance on the site is somewhat lacking, we had talked about building our own attendance tool for organizers to verify that people actually attend the Meetups they are signed up for. When I had a minute, I built out this internal attendance tool on the backend, and attempted to anticipate the format of the data that would make it easiest to build out the front end later. I’m not sure if I succeeded, but thinking in this way is something that I have had to learn to do while working on this project. The attendance tool should be launching hot on the heels of MeetupQuest, to improve the member experience.
Design Decisions
These are just a few of the things that I had to think about, but I am really running out of space, so the rest will have to wait. Aside from these engineering challenges, the other main challenge was making design decisions. This was apparent all along, but really hit us when we went into usability testing. Here it became clear that things that seemed perfectly simple and logical to us didn’t make sense at all to the outside world. For instance, on the quest generation page many of the members who we tested with were very unclear on what was going on. They weren’t sure of what they were seeing, or how to use the quest generation tool, or whether or not they were already signed up for this quest. Some weren’t even clear on what the list of Meetups was a list of, thinking perhaps these would be online events rather than events in the real world. Using this feedback, we have iterated and improved some, but we still have a long way to go.
In Conclusion
Throughout this process, I have learned an awful lot. This internship gave me the chance to work with the Meetup code base, and feel my way around to begin to understand how everything works. The process of building MeetupQuest also taught me very valuable lessons in scalability, teamwork, the role of a back end engineer, planning ahead, and making design decisions, in addition to exposing me to several new technologies.
Jillian:
Previous intern projects I had been a part of were mainly focused on enhancing existing platform features or working on things that interacted solely with the back-end. On this project, I was the sole person responsible for creating a whole new set of pages and user interactions for MeetupQuest using technologies that I had only basic familiarity with from messing around on side projects or small pieces of larger school assignments. The challenge of building the front-end for MeetupQuest was overwhelming in the best case.
Building a simple app for taking attendance and tracking points sounds easy easy enough, but the funny thing I learned about front-end development is that unlike traditional stand-alone applications, your job is to send some data to any number of potential environments, screen resolutions, etc. and hope that everything not only works but looks nice as well (I’m looking at you, IE8 users). Oh, and everything should feel responsive to the end user (because making liberal use of loading graphics is lame).
From Server to Client and Back Again
One of the first things we had to get right before working on MeetupQuest was the interaction between the client and server. This is probably one of the easiest things to get right when you’re building an application from scratch, but when you’re adding on to an existing codebase you want to make sure you’re not adding a line of code in a file that literally breaks the entire thing.
Meetup uses mod_rewrite to convert URL patterns into paths pointing to the appropriate service. Once we created a custom rule using regular expressions that successfully converted meetup.com/quest into the actual path to the application, we had to modify the Struts dispatcher to point the request to the right Servlet, pull objects from the database, convert those to Java objects, and pass those along in a redirect to the appropriate JSP. Luckily for us, the problem of converting raw DB objects into POJOs had been solved a long time ago at Meetup.
We had our ‘hello world’ jsp and relative path that was able to render some back-end data in a response on the client-side. We still had a long way to go.
Javascript, jQuery, and Managing Scope with the Module Pattern
In order to take the idea from static HTML page to full-blown application, we needed event handlers and state. I had some basic familiarity with jQuery, in that I could select DOM elements and write AJAX callbacks but nothing too sophisticated. The first iteration of Javascript files for MeetupQuest were inefficient at best and impossible to maintain at worst. All of the event handlers were declared each time the page was loaded, duplicate selectors everywhere, etc. For the initial prototype that we presented, this would have to suffice but about halfway through the summer I decided to refactor all of the code before making any more progress.
One of the cool things I learned about was the jQuery event delegator function .on(). Instead of adding .click() to elements once they were added to the DOM and testing for them on page ready, I could declare all of my event handlers as $(document).on( event, element, function(){}) on the initial page load instead of doing so on $(document).ready(). The .on() function waits until these elements are added to the DOM and then registers the event handler.
Another major change I made was to rewrite the entire thing as a module. There are some great articles on the module pattern and how it helps you manage variable scope, but the existing codebase was my greatest resource in this regard. The main site already has a global Meetup object initialized, so adding a module that extended the existing prototype was incredibly easy.
The module ended up being a better idea than I had originally anticipated. Among other things, it allowed me to maintain private variables that were accessible to each of my private callback functions but inaccessible outside of the module. This was insanely useful for many things, such as storing cross-request query parameters.
Complex Interactions using JSPs and Custom Tags
A long time ago, JSP libraries probably contained all of the functionality that you could ever want for generating dynamic HTML server-side. At Meetup, I have seen things done with JSP custom tags that I seriously never even imagined possible. I won’t go into all of them (just know that there’s an entire tag library dedicated to populating Mustache templates with Python scripts – I’m just going to assume that they work something like Django templates that render HTML within JSPs and leave it at that ), but it was cool to see just how extensible the JSP tag library is and how it could really reduce code redundancy as well as help you avoid using hideous scriptlets all over the place to do more complex data manipulation.
For MeetupQuest, I made liberal use of the <web:ajax> tag, which allows you to return an HTML fragment to the client by placing the content between <web: ajax> tags and giving it a unique key. By making a request of the form /?__fragment=(key), the server would return you the content as HTML, which was then super easy to render client-side in a success callback. The major benefit of this was that I could render different views based on some complex state (is the user an organizer? Does the user own this quest? etc.) without having to write different JSPs for each one.
Code Review and Usability Testing
It would be one thing if we just had to make MeetupQuest in isolation with no thought to potential end-users outside of Meetup HQ. Half of the learning experience was having to code for a production environment, which meant supporting all of the major browsers and getting user feedback on our project. Towards the end of the summer we were able to experience the ups and downs of code review, a peer-review tool that allows other Meetup devs to look through your code and make comments. The project had to pass this step in order to make it into QA testing, but it was actually less stressful and more just really helpful as well as a great opportunity to identify potential problem areas before QA did.
While the project was in code review, we were also getting feedback from usability testing. It was incredibly entertaining and educational to see how real users of the Meetup platform reacted to our project. I think one user was actually afraid that we were planning to replace the main site feature with our Quest Generating tool (Brenna did an awesome job of reassuring them that this was definitely not the case). This step was also incredibly helpful because it really forced me to re-evaluate design decisions from a usability perspective. Just because something is logical does not mean it is necessarily intuitive (this is something that is going to pain a lot of CS majors, but it is the absolute truth for the majority of users).
As we get ready to unleash MeetupQuest on the world and I look back on all of those weird bugs and technical challenges that we faced (“why is this working in IE8 but not IE9 compatibility mode?!”), I’m also trying to take stock of all of the things I learned along the way but in the end the only thing I feel confident in saying is that I learned how to learn from the incredibly talented team of people that supported us the whole way through this crazy ambitious project.
7 Notes/ Hide
- gifts-for-men-uk liked this
- personalized-gifts-pro liked this
- lonelybob liked this
- dangerouslyclosetothesurface liked this
- makingmeetup posted this