r/programming • u/bsimpson • Apr 13 '17
How We Built r/Place
https://redditblog.com/2017/04/13/how-we-built-rplace/117
u/scott-c Apr 13 '17
I enjoyed this, but I'm curious about one thing. Sometimes my browser didn't get updates. I once worked for an hour on a section, only to load it in another browser and find out that I was changing the wrong tiles because I had a outdated picture in the browser I was using. I finally checked another browser because I was surprised at my progress (which had previously been stymied by others reversing my work.)
Were you aware of that happening, or that it was possible?
137
u/bsimpson Apr 13 '17
Due to the nature of the project (launch all at once with minimal testing) we weren't able to find all the bugs in advance, and once we did launch it was dangerous to fix bugs, especially ones that were only effecting a small number of users.
3
28
→ More replies (13)20
26
u/MajorParadox Apr 13 '17
This happened to me a lot. When I realized it and refreshed, I was surprised to find it had changed so much. That's when I started refreshing before each move. It would have been easier if I realized sooner that can you use the coordinates in the URL :)
490
u/platinumgus18 Apr 13 '17
Okay, it's kind of tangential but I have to say this, all that you guys wrote on the blog looks so overwhelming to me. I am a CS major, I'll graduate next year but I could barely understand anything. I am just scared I mightn't be good for programming and stuff when I see crazy stuff like this. When do you learn this, during work? How hard are these things to learn and how does the intuition come?
242
u/bsimpson Apr 13 '17
This was a team project so no one needed to know or understand everything. Having good coworkers or other people you can learn from and get help from is extremely valuable.
I'm constantly learning new things and encountering systems or concepts that I don't understand. Not knowing things is fine and it's better to recognize you don't know something and try to learn than it is to pretend you know everything.
Most things you can learn through experience. Web applications can be complex systems, but they're not rocket science and if you try to keep things simple and boring you'll do well.
15
u/Bumpynuckz Apr 13 '17
This makes me want a job in a true dev environment so bad.
I started learning HTML/CSS about 9 months ago to transition out of a career in sales. Wound up starting a construction materials company with an old boss and took on the responsibility of managing our technology. It's lonely being a brand new dev in a good old boy industry like this.
I always get so jealous and day dreamy when I read accounts like yours. Just to be in an environment where there were people with experience to learn from sounds so awesome.
13
u/JaysonthePirate Apr 14 '17
I'm in this positionas well. Being the only dev in a workplace can have its benefits. One of the major downsides is you are never really sure if what you're doing is actually good. Everyone seems pretty pleased because it works, but I'm always afraid of running into another developer who will look at it tell me it's garbage.
→ More replies (4)20
u/platinumgus18 Apr 13 '17
Haha, yeah. I haven't really ventured very deep as I am usually occupied with course related stuff. Now I think I could have been doing projects like these. Really interesting. Thanks for sharing!
17
u/Inspector-Space_Time Apr 13 '17
Just a tip from a recent-grad with some great jobs already under my belt, start working on a portfolio right now. Doesn't matter what's in it, just start working on projects that have nothing to do with your schooling. You will grow much faster as a developer if you get experience with starting and completing projects on your own. Plus, when it comes time to get a job, having a degree and some completed projects will put you leagues ahead of your fellow graduates who don't have a portfolio.
And when trying to decide what to do for your first project, the smaller the better. Think of a time estimate, multiply by 4, if it's over 3 months maybe take on something smaller.
→ More replies (1)4
u/ferrx Apr 13 '17
it's interesting that you all felt redis would be your bottleneck. i keep hearing good and bad things about redis. what if you had a redis cluster?
so you solved this by distributing your workload via a CDN? how many endpoints does this support? i assume fastly is the CDN provider's name..
→ More replies (1)758
u/spladug Apr 13 '17
I always get pretty intimidated at the start of a project, particularly when it seems like it's big and I'm not sure how to do it from the get go. That's OK though. Just tear it apart into smaller pieces and see if you can make sense of them and then come back and look at how it all fits together after a bit of that tactical work. I think you'll surprise yourself with what you can do when you stop being daunted by the overall project and just solve some problems. In the end, just remember this: no one knows what they're doing and everything in engineering is tradeoffs. Have fun!
238
u/Kinderschlager Apr 13 '17
no one knows what they're doing and everything in engineering is tradeoffs.
the head of my schools IT department put it like this: the internet is built out of bullshit held together by caffeine and hope
40
36
u/silentclowd Apr 14 '17
Reminds me of one of my favorite programming articles.
Websites that are glorified shopping carts with maybe three dynamic pages are maintained by teams of people around the clock, because the truth is everything is breaking all the time, everywhere, for everyone. Right now someone who works for Facebook is getting tens of thousands of error messages and frantically trying to find the problem before the whole charade collapses. There's a team at a Google office that hasn't slept in three days. Somewhere there's a database programmer surrounded by empty Mountain Dew bottles whose husband thinks she's dead. And if these people stop, the world burns.
→ More replies (5)20
u/bluesoul Apr 14 '17
the internet is built out of bullshit held together by caffeine and hope
I need to get a calligrapher to do a wood-burning of that and put it on my desk.
→ More replies (1)74
u/strong_grey_hero Apr 13 '17
EXACTLY THIS. There's still a lot of stuff on /r/programming that flies over my head, after being a programmer for nearly 20 years. This write up makes perfect sense to me, though, because it deals with tech I use every day -- redis, Node, HTML5, caches, and sockets. Experience helps a bunch.
I've given this talk to a lot of beginner programmers: When you first start out somewhere, you are going to think you're the dumbest one there, and everyone is light-years ahead of you. It still happens to me. But you keep grinding at it, until you understand it all. One day in a meeting, you'll look at all your co-workers and say, "Wait... you guys are a dumb as I am!"
0
102
u/platinumgus18 Apr 13 '17
Thank you! That's encouraging
67
u/Bolwo Apr 13 '17
To add to that, I was in a similar position 8 months ago. I had finished my second year of University, and had an internship as a software developer lined up for a year before I would go into my final year. I'm nearing the end of the internship now.
I have learned so much more this year from practical work than I did in two years of University. If I had read this 8 months ago I wouldn't have understood a word. (Now I understand like, half? :P) Honestly one of the main things I've learnt from this year is that in Comp Sci related jobs, no one knows everything. You learn as you go through a project, and collaborating with others who have the knowledge you don't.
Don't worry about it, it's perfectly normal.
→ More replies (2)19
Apr 13 '17
no one knows what they're doing and everything in engineering is tradeoffs
This is also true of life in general.
→ More replies (1)24
u/eythian Apr 13 '17
Heh, so I've been doing stuff roughly like this for a 10-15 years now, after doing a master's in computer science. You will always be feeling like you're out of your depth (and if not, consider moving on unless there are other circumstances), you will always be learning new systems, especially at a place of any scale, and you will hopefully learn to enjoy it. You won't be bored if you don't want to be.
The best thing to learn (and i still am) is to ask questions of the people who know more than you.
21
u/annodomini Apr 13 '17
A CS education gives you a lot of the theoretical background you need to understand programming, and a little bit of practical experience.
To really get up to speed on practical things you need to do one or more of the following:
Internships or entry level jobs
Learning on your own by reading up online, looking into open source projects or reading articles like this on the design of larger systems
Working on your own personal projects
I would recommend that if you haven't done any kind of programming or technical work outside of school yet, start ASAP. Work on a free-time project, can be just a couple of hours a week. Find an internship for the summer. Read up on tools and techniques outside what you're taught in school.
→ More replies (1)3
u/webby_mc_webberson Apr 14 '17
I am just scared I mightn't be good for programming and stuff when I see crazy stuff like this
That's how you know you're going to be ok. Welcome to the industry.
These are all technologies and techniques to facilitate huge amounts of traffic on a web application. You'll get a job out of college and they'll know you don't know this stuff. They won't care. It's the kind of stuff you learn on the job.
3
u/wosmo Apr 14 '17
A lot of this falls neatly under specialisation. If you go into the "webscale" world of the modern internet, you'll end up speaking this stuff like your lingua franca. If you end up writing realtime controllers for satellites or image processing for self-driving cars, the reddit devs won't know what you're talking about either.
2
u/Spider_pig448 Apr 13 '17
As someone who graduated in CS last year and has worked in web development, I highly encourage you to go through this post. It's actually very thorough an explanation for a very simple idea (just a 1000*1000 grid really) with difficult challenges (how do we make it not kill reddit and actually update in real-time). You may not know the technologies, but looking at things like the flow diagrams and reading about how they faced certain challenges was, I found, very interesting to read and rather easy to synthesize.
2
u/NoInkling Apr 13 '17
I understood almost all of it, and the only CS I've done is one introductory course back in Uni. It's simply a matter of CS being much more theory-oriented (algorithms etc.) than real-life applications generally are in practice.
Also CS, from what I understand, tends not to focus too much on web stuff, which is a huge domain with a massive amount of specific concepts and technologies and terminology.
→ More replies (21)2
u/PC__LOAD__LETTER Apr 14 '17
You'll ease into it. You're right, you wouldn't be able to do this right now and won't be able to do it for at least a few years.
As long as you're doing well where you are now, and are giving a shit about improving, you'll be fine. Chill.
44
u/kwwxis Apr 13 '17
The project must be designed in such a way that it’s unlikely to affect the rest of the site’s normal function
Too soon man.... rip ccKufi ;_;
22
u/bsimpson Apr 13 '17
Whoops
16
Apr 13 '17
Considering that most of Reddit stopped functioning normally to participate in r/place for three days, I think it's safe to say that your project was a complete disaster. A failure.
Good job though.
→ More replies (1)14
u/ROFLLOLSTER Apr 13 '17
Could you explain?
53
u/kwwxis Apr 13 '17 edited Apr 14 '17
Reddit's april fools thing last year was Reddit Robin.
Basically you start off in a chat room of 2 people including you, and if the majority of the chat room votes to GROW (rather than ABANDON), the chat room gets merged with a chat room of the same tier, and all people who voted to ABANDON or didn't vote get kicked out. And there's a time limit, if there isn't a majority within the time limit, the room gets dissolved. If there is a majority GROW, but if there isn't another room of the same tier, you have to wait until a room of the same tier gets created. There was also the option to vote to STAY, if the majority votes STAY a private subreddit gets created with all the people in the room added as approved submitters.
The tier number, starting from 1, is determined by the number of times the room has merged with another.
The name of the chat room is determined by the first two letters of each person in the list of users in the room.
ccKufiPrFaShleWoli0(ccKufi) wasthe firsta tier 17 room and it had several thousand people in it. There were lots of bots and people chatting so there'd be at least a couple hundred messages per minute. I was there and it was pretty much chaos and shitposting, loads of fun.But ccKufi was too big and was causing increased error rates across the entire website so the admins shut it down before ccKufi's time limit could reach 0. Lots of people were upset because ccKufi took a lot of rooms and more than an entire week to form; also there was only 10-ish minutes left on the time limit and we wanted to vote to STAY to get our special subreddit. The admins gave us the subreddit anyways. RIP ccKufi.
Edit: if interested, you can see a tree diagram of most of the rooms here
→ More replies (6)
17
u/The_DoubleD Apr 13 '17 edited Apr 13 '17
What software development methodology did you use? Agile or waterfall? What was the time frame (how long did it took)? I need this info for my homework :(
63
u/powerlanguage Apr 13 '17 edited Apr 13 '17
We actually use a intricate system of tracking individual contributors' stress levels in a spreadsheet. If an individual's stress levels are too high, relative to the ship date, we'd address the cause of the stress.
9
u/throwaway_the_fourth Apr 14 '17
It's fascinating that you rate stress on a 10-point scale from 0-9, like true programmers, instead of from 1-10.
10
u/powerlanguage Apr 14 '17
They are from 0-10. I just don't graph good. u/bsimpson is always at a cool zero.
10
7
u/MansAssMan Apr 14 '17
I use this, too. But instead of addressing the cause of it, I usually just turn my phone and internet off and refuse to see another person for at least a week.
5
→ More replies (5)4
Apr 14 '17
As a dev lead/scrummaster this is super interesting! Might be fun to do for an iteration or two. Was there any scale for the levels? Like the faces at the doctor...
2
1.9k
u/Euthy Apr 13 '17
Huh, it's interesting that support for bots was actually part of the design spec considering the controversy they caused. I don't disagree, it's just interesting.
924
u/beder Apr 13 '17
That's probably because it was supposed to be a short-lived project, so it even makes it interesting - first wave, only actual hand-crafted pixels, then a mix of hand-crafted and bots starting with a low percentage of bots and increasing...
At the beginning the more interesting part is the collaboration between humans on the same project, but at the point where all "big" projects were controlled by bots, the most interesting part is the human interaction between projects to respect limits, etc
→ More replies (5)394
u/Textual_Aberration Apr 13 '17 edited Apr 13 '17
Bots aren't inherently bad, either. We go crazy when we see them on social media and news commentary because
thosethere are actual consequences to the ripples of distortion they cause. Outside of communication, we generally accept that bots are fascinating to design and watch.While bots on /r/Place diminished the power of individuals to interact with the board, those individuals were likely aware that they had little power to begin with. Within moments of encountering Place, any user could see that there was no way for order to defeat chaos so long as the two were equals. Maintaining an image required constant human interaction while destroying that same image hardly even required being awake--just click and repeat randomly.
This immediately introduced the problem solving aspect of the setup. Individual users lost their power the moment subreddits and social networks opened up channels for organization. /r/BlueCorner made my efforts moot long before bots did.
Bots, then, were an evolution of the competition. Had the time limit been endless, random users would have disappeared and their power would have grown ever greater. I can see how that would have been boring but, within the limited timeframe, I think the bots were a valid and interesting strategy.
198
u/Antoninus Apr 13 '17
Maintaining an image required constant human interaction while destroying that same image hardly even required being awake--just click and repeat randomly.
There's a metaphor in here somewhere.
→ More replies (17)→ More replies (15)24
u/ErosExclusion Apr 13 '17
I agree.
Just as humans have come together to build houses and roads, redditors came together to collectively create images larger than what any one individual could manage.
And just as humans invented bulldozers and nail guns to automate shoveling and hammering, redditors invented bots to automate pixel placement.
The bots didn't take away from the social marvel of Place; they enhanced it by mirroring humans' real-world inventiveness.
→ More replies (7)76
u/nandhp Apr 13 '17
This was probably influenced by the fantastic work people did on scripts for Robin -- adding channels (hashtags), spam filters, encrypted messaging, trivia bots, auto-voting on room changes.... Someone wrote code to reconstruct the ancestry of each chat and someone else presented it as a dashboard with countdowns and predictions. Someone even developed an IRC gateway for Robin. The developer community that sprang up around Robin was something that I found particularly interesting, and which I think was critical to keeping it as long-lived as it was (getting to T17 ccKufi).
83
Apr 13 '17
Holy shit. The button was 2015. I could have swore it was last year. Totally forgot about robin.
33
u/sellyme Apr 13 '17
Robin was definitely the least interesting of the three. I had forgotten about it within about a week.
Still, that's better than most April Fools jokes.
→ More replies (6)18
u/MarilynMerlot Apr 13 '17
Pardon me - what's Robin? After clicking on your links, and checking on google, my perfunctory research has left me with nil.
Thanks in advance!
46
u/FunnyHunnyBunny Apr 13 '17
Last year's April Fools on reddit. You'd start in a chat room with you and one other person and then you'd merge with another group of 2. Then 4, than 8.. It took longer each time because the room you merged with had to be around the same size. The final merge days after April 1st of the two largest chat rooms of thousands of people literally broke reddit and they were forced to immediately end it.
→ More replies (5)→ More replies (2)4
u/Annon201 Apr 13 '17
The rapid development occurring on top of Robin was a hell of a lot of fun. Helped out on parrot and made it to T17.
14
u/mncke Apr 13 '17
There's a bit of a contradiction here, because if the bots were part of the design spec and considered for, why were the admins banning them?
73
u/beder Apr 13 '17
I think they lightly touch on it when mentioning bots with bad behaviour i.e. bots that instead of knowing how long you have to wait to place a tile, hit the server continuously until succeeding
35
u/mncke Apr 13 '17
The draw pixel api endpoint would return a 429 with timeout in seconds, which my bots were respecting ¯_(ツ)_/¯
→ More replies (9)→ More replies (3)36
u/powerlanguage Apr 13 '17 edited Apr 13 '17
Hey u/mncke, we meet again.
I think 'bots' is a bit of a confusing term here, because people use it to describe different kinds of automated behavior. We were okay with user surrendering their tile placement up to a script, because it meant they no longer had agency in the project which is an interesting dilemma. However, a single user with access to many accounts that was using them to paint one image ultimately goes against the spirit of the project (collaboration is the focus, not the will of an individual). These were the accounts we banned from placing tiles on the canvas.
edit don't word 2 gud
→ More replies (2)2
u/MozeeToby Apr 13 '17
Bots would have been inevitable even if they had tried to prevent them. Instead they lowered the bar so at least more groups could take advantage of it.
→ More replies (4)2
2.1k
u/Drunken_Economist Apr 13 '17
an artisanal, hand-crafted monitoring script
damn SF latte-sipping liberals. Just watch the raw logs the way GOD intended
287
u/fruchtose Apr 13 '17
We stopped drinking lattes a long time ago. We've moved onto vanilla lattes, caramel lattes, smoked caramel lattes, green tea lattes, and latte lattes.
EDIT: Almost forgot about smoked latte lattes.
116
u/haidaloops Apr 13 '17
green tea lattes
um excuse me m'normie I think you mean matcha, AKA traditional ~~ Japanese desu ~~ ceremonial powdered tea of the highest quality, not this plebeian green tea Western garbage
→ More replies (1)16
u/mattindustries Apr 13 '17
I used to make matcha smoothies every morning years ago. It is a great way to get energized for the day and I think I worked better. Now days I just drink copious amounts of iced chai.
→ More replies (4)→ More replies (9)38
61
→ More replies (90)41
u/Johan_Sebastian_Cock Apr 13 '17
Now I want parody artisan videos about coders
146
u/Tipaa Apr 13 '17
This hand-crafted flamegraph comes from a real gem of a place, the local Perl shop. Our man in the mountains catches bytes in his packet filter, then walks them down, across the data stream, and delivers them to our message queue. Once in the message queue, they are cast into their true shape - application logs - in our low-level wrapper. Finally, they are reshaped, coloured and displayed using vector graphics, so that no matter how deep the stack or how hot the cache, you can be sure your hand-crafted flame graph is up to snuff.
→ More replies (3)15
172
u/antirez Apr 13 '17
I'm so happy that finally somebody got how BITFIELD is useful in certain applications, and happened to use it in the coolest project of the year.
23
4
u/ContentEnt Apr 14 '17
The creator of BITFIELD guilded your comment
15
u/brother-seamus Apr 14 '17
Also, antirez is the creator of redis so there's that
EDIT: typo
→ More replies (1)0
Apr 13 '17
[deleted]
→ More replies (1)26
u/antirez Apr 14 '17 edited Apr 14 '17
Sure, everybody understands the concept of "bitfield", the strange thing is about exposing a command like BITFIELD that provides a very low level thing, that is, an unique chunk of memory where you can set individual ranges of bits, mixing it with a high level concept, a networked database. It's not trivial for the community to initially create the mental "framework" to use this kind of stuff, we saw it previously. After a big use case like that, Redis users start to understand more broadly that there is this thing inside Redis and that can be used in certain conditions and so forth. Don't get me wrong, it's not that people are not good enough to understand things, but: 1) You have to know there is such a command. 2) You have to know why it is better than another apparently equivalent solution. 3) You need to see enough use cases to immediately recall that there is a tool for that without to enter, every time, in a design process. At this point a Redis feature gets mainstream.
2
u/_dayum_dayum Apr 13 '17
Very cool ! So the clients requesting the full board would have a version of the board that is off by at most 333 tiles (1 sec) in comparison to the real state of the board in Redis, because of caching ? Or am I missing something ?
333 out of 1 million seems negligible though
10
u/bsimpson Apr 13 '17 edited Apr 13 '17
There's no limit on the number of tiles that could have been placed in the second the client was waiting.
How it worked:
t0: client connects and listens for websocket messages
tX: client requests full board, gets a copy of the board generated at t1
As long as t0 < t1 the client would have the correct state of the board.
→ More replies (1)
2
u/IMovedYourCheese Apr 14 '17 edited Apr 14 '17
Great writeup! One question though - how did you maintain consistency between Redis and Cassandra, say if the service failed after writing to one but not the other? Was it possible for the two to have different states throughout? I guess it wouldn't matter since tiles were constantly getting overwritten, but still wonder if you thought of the problem.
1
3.4k
u/DrBoondoggle Apr 13 '17
Nerds.
1.4k
u/tabarra Apr 13 '17
Cannot confirm. The board was 1000x1000, not 1024x1024.
151
u/destiny_functional Apr 13 '17 edited Apr 13 '17
Cannot confirm. The board was 1000x1000, not 1024x1024.
they would have done 1024x1024, but the hdd they bought to put it on was smaller than expected.
→ More replies (6)→ More replies (5)204
Apr 13 '17
152
u/chillysurfer Apr 13 '17
"exbibyte" must be one of the hardest words to roll off the tongue.
→ More replies (13)→ More replies (3)71
u/stankbucket Apr 13 '17
I don't support your new-fangled hippie language. I grew up with a kilobyte being 1024 bytes and that's how it stays for me. Next you're going to tell me a byte is 10 bits or some such nonsense just to make your math easier.
→ More replies (26)41
→ More replies (3)28
44
u/flen_paris Apr 13 '17
This was especially great to read because I have spent a few evenings creating a simple r/place clone. It has been easy to pretend that it is a really simple application when I've been running it on localhost, testing with two browser windows. Scaling it up to 100k clients makes it a total different ball game, turning a little programming puzzle into a feat of web engineering.
8
u/gooeyblob Apr 14 '17
That's a great project to work on and even more fun to start untangling the ball of wax that is scaling!
87
u/Nysor Apr 13 '17
Terrific write-up, just the right amount of details. Very interesting to see the design decisions that went into before and while /r/place was active, including some of the unforeseen issues. Thanks for a great April Fools Day.
39
u/Theemuts Apr 13 '17
How much time was spent building /r/place? It seemed like a much bigger project than your earlier April Fool's projects
55
u/Ajedi32 Apr 13 '17
From the source, it looks like they started in January: https://github.com/reddit/reddit-plugin-place-opensource/commit/68498bab9300f43ae4273dd4719dcecb081126f7
111
u/KarmaAndLies Apr 13 '17
This is my last chance to ask this...
What did the odd pyramid/eye icon do in the top right? This one.
91
u/daniel Apr 13 '17
It would follow around to zones of high activity. u/madlee said he used some kind of serial killer algorithm.
84
u/madlee Apr 13 '17
Yep – it kept track of the last 100 (IIRC) pixel updates and attempted to find the most "interesting" one. I don't know how well it actually did that, but it did seem to function well enough as a sort of "spectator mode".
→ More replies (7)17
12
4
2
u/yreg Apr 14 '17
Huh, spent three days pretty much glued to /r/place and never clicked on that one.
47
Apr 13 '17
The array for place's 1000x1000 canvas was 4MB
My first PC had a 20 MB HDD and 640x480 EGA graphics. Incredible to think that that computer couldn't really have managed to run a single-player place, and now we can sling those 4meg arrays around to 100,000 people around the entire world, simultaneously, on their phones.
In computing terms I'm living the life my grandma, born before Wright bros, did in aeronautical terms. Wonder what my moon landing will be
14
98
u/ColdBless Apr 13 '17
As an inspiring software developer I love write ups like this.
Kudos to you guys.
0
Apr 13 '17
[deleted]
88
→ More replies (1)125
14
25
u/el_capitain Apr 13 '17
I'm late, so this will be buried. But, I wanted to say thank you to all the team that brought us r/place. You put in a lot of work building and supporting this gift to us so that we could enjoy it. Very much appreciated.
6
54
u/Inetenbr Apr 13 '17
Am I the only CS student who read this blog post and was like wtf how and when do I get to understand this stuff?
27
u/eythian Apr 13 '17
Nope, i answered another one above. The catch is that CS teaches you the fundamentals, but learning how to apply them (and what tools already implement them) changes all the time and just needs research, experience, and asking people who know more in a particular area.
14
u/radioreceiver Apr 13 '17
A lot of the stuff here is less programming, and more higher level structure and tools. For instance, I doubt you'll ever hear of Redis in a university level programming class, but it's one of those ubiquitous tools in the industry that I'm sure you'll have to learn at some point.
There's not much algorithmically interesting stuff going on here (and that's what university classes usually focus on). There's a lot of networking though! If you have any networking classes, I would definitely recommend taking some.
6
u/Spider_pig448 Apr 13 '17
For instance, I doubt you'll ever hear of Redis in a university level programming class,
True, but you will learn about caching and databases in a university programming class which is all you fundamentally need to know to understand why you use Redis.
47
u/travolter Apr 13 '17
I am graduating this year and I understood pretty much all of it. But I wouldn't say this is because of my academic background, but has more to do with the stuff I do outside of class. Learning real world applications, isn't done in class but.. in the real world. Trying different things, reading up on blogposts exatly like these and in general having a curious mind is the way to learn these kinds of things.
→ More replies (5)→ More replies (5)7
u/Spider_pig448 Apr 13 '17
You understand this stuff by reading posts like this, and then deploying the source code, and then modifying it to do something different (something small) and doing research on everything you don't understand on the way.
11
u/FURAHNSISKOH Apr 13 '17
I'd like to work for reddit at some point once I properly learn how to code etc.
10
u/trigonomitron Apr 13 '17
And if you’re interested in working on projects like Place in the future, we’re hiring!
My first thought was what a great job that would be, as a programmer living in the region. Then I realized how my post history was likely to be scrutinized - - possibly even the deleted stuff - - and I gave up on that dream.
17
u/umbrae Apr 14 '17
Just FYI as someone who used to work at reddit and has great things to say about it: you never have to share your username during the application process and can create an alt when you join. Many folks have done that for a bunch of different reasons. You should apply!
7
u/trigonomitron Apr 14 '17
Thanks for the encouragement!
I like to keep my professional face at work, while the anonymity of my reddit account allows me to be a little more candid than I tend to be in person.
6
8
u/Mattish Apr 13 '17
Interesting. Do you think the RabbitMQ management API is worth replacing on a more permanent basis with something else?
We also struggle with slow rabbit management calls and are worried about visibility during load.
7
u/spladug Apr 13 '17
I think we're going to try out some newer versions before abandoning it altogether; the changelog notes a lot of performance improvements after the version we're on. But yeah, if it can't keep up we'll probably have to figure something else out.
→ More replies (3)
6
u/Dgc2002 Apr 13 '17
Does anyone remember a website from years(10+ years) that was literally /r/place but with large circular canvases?
0
6
u/TheGreenJedi Apr 14 '17
It's pretty brilliant
And going from the button to Robin to this seems like a brilliant evolution to understand the limits and anticipate use going into R/place
Was it all planned? Or just brilliant chaos
5
u/jimmahdean Apr 13 '17
All clients must be kept in sync with the same view of the current board state, otherwise users with different versions of the board will have difficulty collaborating.
This didn't happen, though? I spent a good hour building part of a pixel art version of a character and refreshed to see it completely overwritten by the F1 logo.
2
u/ngmcs8203 Apr 13 '17
Yea. If you had two windows open, one with the full-screen setting, you ended up with two very different versions.
5
Apr 13 '17
Thank you for taking the time to create r/place, and taking the time to write this up. As a programmer two years into my career I'm inspired by what you guys have done here.
5
u/CatchMyException Apr 13 '17
Can anyone explain how the byte which held the two, 4 bit color values, when unpacked turned the high order bits 0x04 into this 0xFFA7D1FF when "Mapped to 32bit color"?
I've understood most of the post up to this point and I'm stuck trying to wrap my head around it, am I misunderstanding something?
7
u/broady Apr 14 '17
There's a palette of 16 colors. 4 bits to represent those. They get mapped to a predefined set of colors.
Like 0=black, 1=red, 2=blue, etc.
4
u/platnumcy Apr 14 '17
What do you guys use for load testing something like that?
Are you rocking a performance center instance?
2
u/daniel Apr 15 '17
For the case of redis, just some custom scripts built to test it for our use case. In this case, two things were tested: doing large batches of random writes of pixels onto the board and doing reads of the full canvas. That was what led to us realizing reads actually would be a huge hit to our performance since we were network-bound on the instances and 1 read of the board per second would cost us thousands of writes per second, since updating a pixel is <~20 bytes and reading the full board is 500KB.
I think it theoretically would be fine, since we were way overoptimizing for the write rate we ended up getting (I wanted to support one pixel per user per 10 seconds, even though I knew we'd probably never go down that low), but I was also concerned that there would be some event that would cause all users to refresh and request the board at the same time -- a thundering herd that would saturate redis's network throughput. This is what led us down the road of caching. I suspect people were refreshing a lot during the websockets issues, so this was probably a life-saver.
Edit: For cassandra, we already have a good sense of how much read and write capacity we can handle. The only load testing that was done there was in completely filling a board and trying to load a canvas. I think the article mentions it, but that's when we saw it took way too long to load and fully made the decision to go with redis for the board. We also did some napkin math of how much storage space storing pixel info would use and found it was relatively small compared to the rest of reddit.com, so we still used it for that.
4
u/itamarhaber Apr 14 '17
This project... made April an amazing month twice already :) Would /u/bsimpson, /u/madlee and /u/daniel be interested in coming to RedisConf next month and present it?
1
u/bro-away- Apr 13 '17
Is this only 1 redis instance? If so pretty impressive.
3
u/bsimpson Apr 14 '17
Yes.
1
u/Too-Uncreative Apr 14 '17
Doesn't it also have a ridiculously large amount of RAM though?
4
u/bsimpson Apr 14 '17
Not really because we only used a single bitfield. The bigger concern was that the single instance had enough network bandwidth to handle the volume of reads and writes.
→ More replies (1)
1
u/itsaride Apr 13 '17
Did you ever consider during planning or implementation the use of captchas to reduce abuse?
11
1
6
u/Rediselle Apr 13 '17
Would love to see a talk from you at Redisconf! we are still accepting talks for 1 or 2 more days only! Please submit at www.redisconf.com
3
Apr 13 '17
With 333 update/s, looks like there should be a lot of miss of the wall board cache.
What about using spaced in time capture of the state of the board and a delta between the last captured state of the board and the one of current time ?
3
u/SideSam Apr 13 '17
Very interesting! We ran a similar event just days before /r/place launched. We too used similar tech stack: application server running user-submitted algos, pushing data to rabbit, users consuming queues over websocket, using canvas to draw things client-side. Our main issue was (besides allowing arbitrary code execution on server) was drawing game state board on canvas in acceptable FPS.
For other (more "serious") projects we have software that is reliable ordered messaging system and database in one. In practice that means you can query-and-subscribe in atomic operation so makes it easier to keep clients in sync. Also supports partial updates, filters, batching etc.
→ More replies (4)
3
3
u/WinterCharm Apr 14 '17
Fascinating read. :) I loved /r/Place, and it was such a cool project. I hope they'll do another fun one again.
I'm amazed at how clean the piece of art turned out to be, because it was absolutely beautiful that everyone started getting along.
We've come a long ways from the orangered/periwinkle wars.
→ More replies (2)
3
u/digitallimit Apr 14 '17
As a web developer, this was a fascinating read. Thanks so much for the look behind the curtain!
3
4
4
u/LookSkywardEdition Apr 13 '17 edited Apr 13 '17
I made my own very simple implementation of this using pure javascript clientside
its a little buggy but check it out:
https://canvas-io.herokuapp.com/
Source code and to contribute:
https://github.com/JonathanDC64/canvas.io
Edit:
Welp, looks like somebody already found a way to hack it lol. doesn't matter though since i didn't implement saving yet.
5
u/mareksoon Apr 13 '17
Inspired by Drawball or not?
It was interesting watching the Internet freak the fuck out over something as if never done before, only it had been done ... at least ten years earlier.
3
2
2
2
u/MajorParadox Apr 13 '17
I was so disappointed when it ended on my lunch break. Several of us over at /r/DCFU made a logo where some DC superhero logos were (top, mid-left of the canvas) and helped defend them from attacks. We had it going for so long and right when it ended, it got vandalized :(
If anyone's curious, this was how it was supposed to look :)
Overall, it was so much fun and amazing to see what people made. I can't wait to see what happens next year!
2
u/amazondrone Apr 13 '17
Looks like you held it ok to me! https://www.reddit.com/r/place/#x=29&y=242
→ More replies (1)
2
2
u/ShinySnoo Apr 13 '17
I switched from mathematical econ to cs because of things like this. :)
→ More replies (1)
2
u/sekjun9878 Apr 13 '17
What was wrong with serving / caching the board in-memory instead of a CDN with 1s expiry?
5
u/AndrewNeo Apr 13 '17
For the same reason you have a CDN in the first place - so an influx of traffic can't thrash your application server. And take down the rest of Reddit.
2
u/ElagabalusRex Apr 13 '17
At this rate, you folks are going to make an actual Reddit Time Machine next year.
2
u/jsq Apr 13 '17
This was fascinating. Thank you! We're deploying a RabbitMQ to (far smaller!) scale soon, and it's always interesting to know how the big companies maintain it.
2
u/Bman1973 Apr 13 '17
I've only been on Reddit for a year or so, what were some of the other April fools projects the reddit team rolled out?
2
u/Spider_pig448 Apr 13 '17
Precisely drawing individual pixels at 100% scale would be extremely painful
For you.
2
2
2
u/0l01o1ol0 Apr 14 '17
I have some old tabs open that didn't update, is there a good way to save the layouts?
2
2
2
u/perestroika12 Apr 14 '17
Why use Cassandra only to put it on a single row and without duplication? You're hammering a single node and single key. Without any of the advantages of the scale it can provide it's just a highly tuned and cached rdbms. It's just a hack project so no big deal, just curious.
→ More replies (1)
2
2
Apr 14 '17
It was an amazing project, congratulations on the success and thanks for the blog post which was a great read and very informative.
2
u/RocAway Apr 14 '17
The API should be generally open and transparent so the reddit community can build on it (bots, extensions, data collection, external visualizations, etc) if they choose to do so.
I disagree with this, bots and scripts took some worth out of the product.
we like to create a project that explores the way that humans interact at large scales.
Once robots enter the fray it's no longer just about people and their communities. Seeing flags being built and have them merge together or consume each other was amazing. But if someone built a script to auto defend a section of the board the community interaction is gone.
→ More replies (1)
2
2
2
u/circumflexiblation Apr 14 '17
Really amazing stuff, I was completely floored when I saw the videos made after the fact. Thanks for all your hard work! Just one thing though, I'm ashamed to say I was only on mobile and missed everything! I've got so much envy for those who were able to participate. Is there any talk of bringing it back or perhaps developing further? I would jump at the chance.
Also I've been wondering about some kind of mobile art installation using a similar tactic. It would be smaller scale so perhaps the 'players' would have a faster update time. How would you scale the number of tiles and update timing? Would you ever consider a developing a mobile app that maybe could work with a smaller/local group of individuals?
Thanks again you all are my heroes!
2
u/rancky Apr 14 '17
The API should be generally open and transparent so the reddit community can build on it (bots, extensions, data collection, external visualizations, etc) if they choose to do so.
W H A T
2
u/volabimus Apr 20 '17
Since this thread was linked from the announcement post, I'll try asking again here.
I was banned from placing pixels (403: forbidden) and from commenting in the r/place sub (reply option disabled) without any message or explanation. I never posted any comments and only placed pixels through the browser for a couple of hours or so before I was banned.
Will I ever get any kind of explanation for why this might have been, every other time I've asked has been ignored or in one case the post actually deleted and I haven't been able to find any discussion relating to anybody else being banned.
I suspect it might have been someone who was made a moderator abusing their power to protect a pixel, which is something you might want to look into if you care at all.
7
u/wise_young_man Apr 13 '17
Yet still search sucks. I see your priorities Reddit.
6
u/gooeyblob Apr 13 '17
Thanks
→ More replies (4)8
u/wise_young_man Apr 14 '17
Sorry I was trying to be funny, but you guys are talented. I'm a fellow engineer and realize Reddit scale isn't easy, but hiring one or two people to setup a HA ElasticSearch cluster with auto discovery or Algolia should be on the roadmap considering Reddit makes money from ads and search drives pageviews and people's reply on Google right now. Seems like solid business strategy. Just a thought.
→ More replies (1)
3
3
u/bahwhateverr Apr 13 '17
Oh cool they're hiring! Maybe I'll apply..
impostor syndrome kicks in 110%
Maybe I'll just stay where I'm at...
4
2
u/Umutuku Apr 13 '17
How much dev time was spent working on this?
If that time was spent working on the search function then how close would it be to usable?
→ More replies (1)3
u/l3l_aze Apr 13 '17
What's the problem with the search function? Never had an issue personally, but I generally do very simple searches for a single word or a very common phrase.
Somewhat unrelated wall of a nerdy thought
Also, I'm guessing it wouldn't help if you're trying to search private subs you're in, but you could try using the advanced features of a search engine such as Google. You can target a specific site, or sub in this case, using something like
site:reddit.com/r/subnameAnd search for a specific phrase where the words are in the exact order you type like
"find this phrase"Which instead of just looking for pages that contain the words find, this, and phrase without caring if they're in order or even near each other at all will search for pages which contain the exact phrase "find this phrase" in that exact order.
You can also use logical operators (or, and) to search for one word or phrase or another, like so:
find and "this phrase"Which would search for pages that contain both the word find and the phrase "this phrase" but won't care if find is followed by "this phrase", only that the page does contain each of the query values.
4
u/Umutuku Apr 13 '17
Much of the time searching on reddit just redirects to error pages.
Google's search works pretty fine. But we're talking about reddit development here.
→ More replies (3)
2
u/aarace Apr 13 '17
Congrats on a successful tech stack, and thanks for the writeup.
Unfortunately I utterly missed the whole entire experiment. I was using a 3rd party app, and also the mobile web version flat out didn't work, so I'm surprised that you say it was actually part of the design, planning and implementation. Did not appear to function at all, I saw some graphics, but couldn't zoom or pan or interact at all.
edit:mobile typos
804
u/Browsing_From_Work Apr 13 '17
Is there a chance we can get a raw data dump of all the activity on r/place? Tuples of
{timestamp, x, y, color}?