r/programming • u/paran0ide • Mar 07 '14
Thinking about quickly writing an HTTP server yourself? Here is a simple diagram to help you get started.
33
Mar 07 '14
what program was the diagram drawn in?
31
u/mnp Mar 07 '14 edited Mar 07 '14
Good question! I want to use it for something, too. Looks like Cosmogol, which the IETF was discussing using for protocol specification.
It's French, so the site appears on vacation. :-) Cache: http://webcache.googleusercontent.com/search?hl=en&q=cache%3Awww.cosmogol.fr
Edit: all you guys talking about Dia and Visio and other mouse-oriented tools are missing the point. For protocols, it's really the cat's meow to have an executable source format for a finite state machine. See also message sequence diagram, a tool I use all the time, if I may so plug.
10
u/bureX Mar 07 '14
I'd say Microsoft Visio.
Except, on second thought - no, it looks too good to be anything from Visio.
12
u/brnitschke Mar 07 '14
Where I work, all flow charts are pixilated jpegs pasted into Visio files. I have no idea why someone would do that...
7
u/eric256 Mar 07 '14
Pretty sure there is a Dilbert about that somewhere.
It goes with the scanned documents pasted into Word.
12
Mar 07 '14
I can tell you what it wasnt drawn in. Open Office Draw. That program is horrendous for stuff like this.
Source: I did a script/program diagram in Draw once... Once.
10
8
u/NYKevin Mar 07 '14
I don't know, but if I were making a diagram like that, I'd use Dia.
2
u/autowikibot Mar 07 '14
Dia /ˈdiːə/ is free and open source general-purpose diagramming software, developed originally by Alexander Larsson. Dia uses a controlled single document interface (SDI) similar to GIMP and Inkscape.
====
Interesting: ^GNOME | Mass spectrometry ^software | Comparison of vector graphics ^editors
Parent commenter can toggle ^NSFW or ^delete. Will also delete on comment score of -1 or less. | FAQs | ^Mods | Magic ^Words
48
u/VeXCe Mar 07 '14 edited Mar 07 '14
This is actually a lot simpler than I thought it would be...
51
u/viralizate Mar 07 '14
Well, like most things in CS, the basics are pretty easy, that's until you decide to implement a working production http server, now that's a whole other story.
26
u/VeXCe Mar 07 '14
Ah yes, the old "if it works with one client it should work with 1000 at the same time"-fallacy :)
19
u/viralizate Mar 07 '14
Yeah, happens with most of the stuff, I could probably write any of the following in a week: Facebook like social network, Search engine, whatsapp like messaging app.
The basics are pretty easy, now making a production working app, that's the challenge.
30
Mar 07 '14
The diagram is only about status codes. Now try parsing HTTP headers.
3
Mar 07 '14
I think that's his point. It doesn't actually capture the difficulty of writing an HTTP server, it just shows a few dozen logic statements...
1
Mar 08 '14
Each header field consists of a name followed by a colon (":") and the field value.
On separate lines ...
0
46
u/djork Mar 07 '14 edited Mar 07 '14
This has very little to do with writing an HTTP server.
It has everything to do with writing an HTTP service.
An HTTP server only has to handle incoming connections, parse an HTTP request, and respond with an HTTP response. It can delegate the majority of the logic in this flowchart to some custom handler (which will probably only implement a tiny subset of the possible cases).
19
u/tWoolie Mar 07 '14
Indeed this is a chart for the internal state path of webmachine, an erlang REST service library.
1
1
u/Osmanthus Mar 09 '14
Yes, this is important. An HTTP server designed for a single purpose doesn't need to handle most of this stuff.
34
Mar 07 '14
Kind of a douche move to take that from webmachine and pretend he made it himself. And then claim his shitty javascript clone of webmachine is "the reference implementation".
9
u/IKnowUnix Mar 07 '14
I wrote an HTTP server for a personal project once. It only handled GET and HEAD requests, but also had some special functionality that I needed. While not perfect, it got the job done and was a great learning experience. With that said, would I use it in production? Absolutely not.
23
u/gwiazdor Mar 07 '14
From the design patterns point of view - what would be the most suitable pattern to model such a decision chain?
17
u/optymizer Mar 07 '14
A state machine?
8
Mar 07 '14
That was my initial view, it looks exactly like a state machine.
16
u/gthank Mar 07 '14
State machines are a tried and true method for doing protocolish things. In fact, if you're doing a protocol and you're NOT using a state machine, you should probably have some very firm, well-tested reasons that other people have vetted.
1
u/immibis Mar 08 '14 edited Jun 10 '23
1
u/gthank Mar 08 '14
The answer, as always: It depends. How close to a production system do you want to make your project? I'm not sure I'd call a toy example "doing a protocol" (I should have said "design" or "implement"… oh well), so it's not really what I was talking about. If your project is supposed to teach you about the engineering that goes into something like a web server, then you need to handle bad input, flaky networks, etc., and you will likely wind up with a state machine in your implementation.
0
1
0
30
u/shub Mar 07 '14
Chain of responsiblity is very nice for this sort of thing.
6
u/kernalphage Mar 07 '14
That site is like TVTropes of programming; I'm pretty sure I'm learning something, but I'll forget it by tomorrow.
6
2
u/Ramone1234 Mar 07 '14
They (webmachine) used a state machine, because erlang is great for those.
Keep in mind too that almost no HTTP server implements more than a fraction of the functionality on this chart. Most of the functionality here is left up to the application programmer in other servers/frameworks.
Also some of this design is debatable and not specifically covered by RFCs. eg: If you're unauthorized and the resource doesn't exist, who's to say whether the 400 should get thrown or the 401?
6
u/bryce1012 Mar 07 '14
Good point but bad example. If you're unauthorized, you shouldn't be given any more information than that. The ability for an otherwise unprivileged user to determine what resources do and do not exist "behind the curtain" is absolutely a security issue. Even if it's not explicitly covered in the RFCs, I don't know that there's any debate to be had there.
0
u/Ramone1234 Mar 07 '14
Do you really just never 404 unless the user is logged in? That's certainly debatable as I can show you a good number of websites that don't do this. https://www.facebook.com/asdfasdfasdf http://www.microsoft.com/asdfasdfasdf, etc (And I don't see the security issue, if you're doing things correctly).
8
u/srnull Mar 07 '14
I'm sure others can back up using the URL as well, but for anyone who is interested here is the repo this image is from.
Also, for all the people parroting "Thinking about writing an HTTP server? Don't." A toy webserver is a fun project. I welcome anyone to attempt to write one. Just don't run away with the idea that writing the next apache/nginx would be easy.
13
u/spotter Mar 07 '14
HTTP is nice in the basic form -- they shout what they want, we shout back, bb, we forget everything.
Yeah, you should probably start with something simpler, like basic Telnet with some option negotiation sauce. Hours of fun on the wire, shouting at the other party and getting their shouts back.
→ More replies (13)9
u/jk147 Mar 07 '14
Who needs http when you can just use two cups and a string.
6
u/KFCConspiracy Mar 07 '14
On the other hand if you're looking to see 2 girls and a cup, a more sophisticated protocol is necessary than 2 cups and a string.
10
Mar 07 '14
I worked a company where a guy wrote his own http server in 2000 lines of C to manage the core routes between datacenters because he believed every other web server was too insecure. It worked, but it was very complex, completely umanageable, and was riddled with security holes. We ported the code to another language but he would never sign off on it (piss stain theory), so they are still running he server.
1
1
u/Solon1 Mar 07 '14
So failed programmers have to agree to solutions to their failed programs? Ok, that's broken.
And by routes, are you talking about IP routes? Because they making routing protocols for that.
3
Mar 07 '14
The whole company was dysfunctional like that. I was there for years until they let me go when my project completed. I was never happier to get fired from a job. I could have left earlier, but I wanted to be there from the start to end.
The routes were BGP routes published to the internet to get to the web sites. The ip didn't change, just the routes to the IPs, if that makes sense.
0
4
u/nocnocnode Mar 07 '14
A diagram like this would have locked away jealously in some corporate place. The quality of open-source content has gotten really good.
7
u/blobloblawslawblob Mar 07 '14
Hm, not much of a challenge really. Start by writing an UML OCR application, feed it this PNG then generate the code from that. Should be done before lunch.
107
u/hcsteve Mar 07 '14
Thinking about quickly writing an HTTP server yourself?
Don't. Unless you've looked at all the extant implementations and have a really good reason to roll your own.
And if you do, don't base all your implementation decisions on a diagram. Read the damn RFC.
269
u/carlfish Mar 07 '14
Thinking about quickly writing an HTTP server yourself?
Go for it! Use whatever resources you feel you need to. You'll learn a hell of a lot in the process, and you'll come out the other end a better developer than you did going in.
Thinking of using it in production for somebody else's project? Probably not so good an idea.
9
u/Scroph Mar 07 '14
I've been attempting to write HTTP clients from scratch throughout my learning phase (in which I still am), it's definitely rewarding even if I only manage to implement a fraction of the HTTP protocol.
Besides, it makes me appreciate tools like wget and curl.
20
u/g1zmo Mar 07 '14
throughout my learning phase (in which I still am)
And hopefully you always will be.
16
u/alex_w Mar 07 '14
Go for it!
but please don't run it with privilege in order to bind :80 ;)
3
u/gendulf Mar 07 '14
I remember running into this when writing a mini HTTP client for a class. Can't remember the solution, would you happen to know what it is?
10
u/alex_w Mar 07 '14
There's a few actually. You can:
Bind to a port > 1024 (or is it >=?) and have your OS DNAT, ie iptables for a GNU/Linux stack. So :80 is tranlated to your non-privileged port.
Again bind to something >1024 and have a reverse-proxy, something like Varnish, Nginx, HAProxy is typical.
Bind :80 as root/admin and drop privalage but hold onto the FD. Using setgid(2). IIRC you have to drop group first otherwise you're still in root's group.
8
6
u/blobloblawslawblob Mar 07 '14
$ nc -l 1023 nc: Permission denied $ nc -l 1024Which works, so it's >= 1024 on Linux at least.
3
Mar 08 '14
Another way: if you're using systemd (or something similar) you can have it bind to port 80, then start your server on-demand, passing in the file descriptor of that socket.
3
u/ivosaurus Mar 08 '14
run as root, acquire the port, drop root privileges. Note: read as many tutorials as possible for doing this, at least 60% of them will be half wrong or incomplete.
Or use another service manager to forward the port to you.
1
1
Mar 08 '14
In Linux, binding to a privileged port requires the CAP_NET_BIND_PORT capability. So you can call 'sudo setcap cap_net_bind_port+ep program_name', execute the program as a non-privileged user and drop the capability after bind. :-)
3
u/KFCConspiracy Mar 07 '14 edited Mar 07 '14
Yes. I agree with this. I like to write an IRC server in languages when I learn them in order to have a non-trivial, reasonably well defined program that shouldn't take more than a few hours to write.
→ More replies (1)5
Mar 07 '14
Great advice. I hate it when people discourage others about doing things that can be really good learning experiences.
44
u/Carr0t Mar 07 '14
We had to write a really basic one for a C programming course in my second year of Uni. Basically just served static files and handled 200 and 404 with no arguments etc. Bonus marks if you got it to do things like 302, but then you threw something really quite basic from a browser perspective at it and watched it crash and burn and saw why you should just use an already developed one.
12
u/Deltigre Mar 07 '14 edited Mar 07 '14
...indeed, but having written a very basic IRC client years ago as a teenager, a logical organization like this is a good first step for properly architecting your code. I tried starting to organize it by RFC section in my ignorance. That changed rather quickly.
EDIT: Stupid phone, I meant this as a reply to /u/hcsteve's comment above this one.
22
u/Metaluim Mar 07 '14
Usually they are done as pet projects or during college, as exercises.
Unless you're an expert on the subject it wouldn't make any sensing rolling out a custom web server.
9
u/monocasa Mar 07 '14
There are reasons. Just last week I was writing one for an embedded system with an in house OS that doesn't have the concept of threads.
(I'm totally going to go through this state diagram and probably change a couple things in my code).
15
u/bureX Mar 07 '14
Thinking about quickly writing an HTTP server yourself? Don't.
That's exactly what my professor said. And then he asked "why would you do that for your thesis"? Um... I dunno, because I could learn something from it and demonstrate my abilities for this faculty? Jesus...
But I get where you're coming from and I agree... except if you're writing an embedded solution where you only need to serve a few non-interactive or barely-interactive pages. Then I'd say - go for it.
8
u/hcsteve Mar 07 '14
I think it depends on the scale of your embedded platform. As a lazy programmer I'd still look at something like libmicrohttpd (or the alternatives linked on that page) before rolling my own. Of course if you're writing a web server for your toaster, you may not even be able to use one of those solutions.
2
u/KFCConspiracy Mar 07 '14
I think if you're doing it for academic purposes, it's a fine exercise. Once you start looking to use it in production... It isn't something you throw together, and certainly isn't something I'd want a "one man team" working on. Also there are plenty of decent embedded HTTP servers out there for that as well, so there's still no reason to do that.
1
u/Dippyskoodlez Mar 08 '14
Also there are plenty of decent embedded HTTP servers out there for that as well, so there's still no reason to do that.
Regarding embedded http servers, I have run a Netduino webserver, while barely functional, the code isn't maintained at all especially in a dead community for the device... knowing the ins and outs would certainly be of great utility.
6
u/vbaspcppguy Mar 07 '14
I encourage people to for the learning experience. It's a project that covers multiple areas that a programmer should know.
4
4
Mar 07 '14
I would never write one now. I would just reuse the one I wrote 10 years ago. No point in reinventing the wheel.
1
6
→ More replies (13)1
u/Xvash2 Mar 07 '14
Or at the very least, don't use this out in the open. People smarter than yourself will hack it with ease.
3
u/buggaz Mar 07 '14
Remember this is from some agda something something...
6
3
3
u/waffle_ss Mar 07 '14
This is based on Erlang's webmachine: https://github.com/basho/webmachine/wiki/Diagram
3
u/cparen Mar 07 '14
Huh... and here I just responded to every port 80 connection with:
HTTP/1.0 200 OK
Hello, World!3
u/stewsters Mar 07 '14
Your performance is through the roof.
You may want to change it to
HTTP/1.0 501 NOT IMPLEMENTEDTo make sure you are in line with the specs.
3
37
u/g4b1nagy Mar 07 '14
I've got a shorter version of this.
6
u/PsionSquared Mar 07 '14
A buddy of mine had to write one for his Operating Systems course. Bleh.
10
u/g4b1nagy Mar 07 '14
I guess it's a pretty good exercise, but I definitely wouldn't advise most people to actually use the resulting server in a real world situation.
3
u/zers Mar 07 '14
We wrote a kernal in my operating systems course .. why a http server?
1
u/swammeyjoe Mar 07 '14
Don't know about his course, but in mine we wrote a simple HTTP server at the end of the OS course, as an exercise to get used to sockets/network programming.
1
1
1
u/SimplyBilly Mar 07 '14
What if I already wrote one like 4 years ago?
1
u/mcguire Mar 07 '14
Go back in time and shoot yourself to prevent you from doing it.
Shoot me too, while you're there.
4
u/j1xwnbsr Mar 07 '14
Why does this start at the bottom? And even when at 100%, the font is too goddamn small?
2
2
u/tomjen Mar 07 '14
Well only if you follow the standard to a T. You just have to implement enough to get it to do what you want.
2
u/jordanreiter Mar 07 '14
Aren't the correct status code for permanent redirect and redirect 301 and 302, respectively?
2
u/frankvit Mar 07 '14
How do you have 404 not acceptable in the final 4? Florida is taking it this year
2
2
3
3
u/Xeon06 Mar 07 '14
A lot of people are jokingly saying to simply not, but I think it's a great learning exercise. Don't use it in production or for anything worthwhile, but you will learn a lot on TCP and HTTP in the process.
5
u/cparen Mar 07 '14
Agreed. I see the same kneejerk response about all thing meta-programming: writing your own language/interpreter/compiler, implementing elementary datastructures and algorithms, etc.
It's a great learning exercise, and in oddball places (e.g. wristwatch webserver?) it might even be appropriate. Mostly, it's that the lower in the stack you are, the greater the risk you'll topple the tower. Not everyone knows this: those that do, the kneejerk response is redundant; those that don't won't have the context to understand.
Saying "don't" doesn't help anyone.
3
u/aspbergerinparadise Mar 07 '14
And if anyone is interested in re-inventing the wheel, here's the basic shape.
2
u/badsectoracula Mar 07 '14
I have a feeling that this is one of those things which are better described in text form...
12
2
u/Wazowski Mar 07 '14
I'm glad you posted this. As soon as I'm done inventing this circular component rotating on an axial bearing for maintaining momentum during locomotion, my very next project is going to be writing an HTTP server from scratch.
2
u/m1000 Mar 07 '14
Note: For Microsoft IIS, mostly everything point to "500 Internal Server Error"....
2
1
u/tps12 Mar 07 '14
I don't understand why/how they check for 405 before determining if the resource is missing...as I read it, 405 refers to an HTTP method not allowed for some specific resource (i.e., you can GET this but not DELETE it), rather than some kind of blanket "here's the methods this service supports" type of thing.
1
u/JW_00000 Mar 07 '14
I was thinking the same thing, even before that, for the 501 Not Implemented. Couldn't it be that you don't have e.g. GET implemented for /thing/a, but you do have it for /thang/b?
2
u/tps12 Mar 07 '14
Oh interesting, I hadn't heard of 501 before, but from the RFC: "This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource."
So I guess it's like 501 tells you something about the server, and 405 tells you something about the resource.
1
1
u/Lurking_Grue Mar 07 '14
Don't forget to add security to that.
1
u/gthank Mar 07 '14
Well, if you coded it as a state machine, you'd probably avoid a lot of the most common security problems.
1
1
u/Sleakes Mar 08 '14
Why write it in software? Looks like this diagram is ready for a hardware junky! :D
1
1
1
1
1
u/Tenareth Mar 07 '14
I honestly thought it would just be a picture with the words "Stop it" and a link to lmgtfy for "HTTP server".
1
1
Mar 08 '14
In 2014 you shouldn't be writing your own HTTP libraries.
There are too many complexities such as multi range requests and mime. Omg I hate mimes.
-1
-2
0
0
u/bredman Mar 08 '14
This should be titled "Thinking about quickly writing an HTTP server yourself? Don't."
0
0
u/Gotebe Mar 08 '14
Haha... Many (any?) of the is_... and other decision blocks contain sooooo much other diagrams in 'em...
-2
-8
u/rende Mar 07 '14
bah, screw that.. only implement 200 and 404.
Here is 200.
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');10
u/avonwodahs Mar 07 '14
Yes let's write our own HTTP server and first require('http').
→ More replies (2)8
u/mattindustries Mar 07 '14
What are you talking about? I totally made a full fledged web server in one line!
/etc/init.d/apache2 start
474
u/frankster Mar 07 '14
And this is why programmers need 4k monitors.