Michael Ngusa
JF-Expert Member
- Aug 4, 2014
- 1,637
- 463
Hi! Does any one here knows what is the list of software that makes JF live and running? Just like other sites...they all have specific technical tools and codes for making it online and functioning, down here is an example for Facebook:
But enough of that. Lets present (some of) the software that Facebook uses to provide us all with the worlds largest social network site.
[h=4]Memcached[/h]
Memcached is by now one of the most famous pieces of software on the internet. Its a distributed memory caching system which Facebook (and a ton of other sites) use as a caching layer between the web servers and MySQL servers (since database access is relatively slow). Through the years, Facebook has made a ton of optimizations to Memcached and the surrounding software (like optimizing the network stack).
Facebook runs thousands of Memcached servers with tens of terabytes of cached data at any one point in time. It is likely the worlds largest Memcached installation.
[h=4]HipHop for PHP[/h]
PHP, being a scripting language, is relatively slow when compared to code that runs natively on a server. HipHop converts PHP into C++ code which can then be compiled for better performance. This has allowed Facebook to get much more out of its web servers since Facebook relies heavily on PHP to serve content.
A small team of engineers (initially just three of them) at Facebook spent 18 months developing HipHop, and it is now live in production.
[h=4]Haystack[/h] Haystack is Facebooks high-performance photo storage/retrieval system (strictly speaking, Haystack is an object store, so it doesnt necessarily have to store photos). It has a ton of work to do; there are more than 20 billion uploaded photos on Facebook, and each one is saved in four different resolutions, resulting in more than 80 billion photos.
And its not just about being able to handle billions of photos, performance is critical. As we mentioned previously, Facebook serves around 1.2 million photos per second, a number which doesnt include images served by Facebooks CDN. Thats a staggering number.
[h=4]BigPipe[/h] BigPipe is a dynamic web page serving system that Facebook has developed. Facebook uses it to serve each web page in sections (called pagelets) for optimal performance.
For example, the chat window is retrieved separately, the news feed is retrieved separately, and so on. These pagelets can be retrieved in parallel, which is where the performance gain comes in, and it also gives users a site that works even if some part of it would be deactivated or broken.
[h=4]Cassandra[/h]
Cassandra is a distributed storage system with no single point of failure. Its one of the poster children for the NoSQL movement and has been made open source (its even become an Apache project). Facebook uses it for its Inbox search.
Other than Facebook, a number of other services use it, for example Digg. Were even considering some uses for it here at Pingdom.
[h=4]Scribe[/h] Scribe is a flexible logging system that Facebook uses for a multitude of purposes internally. Its been built to be able to handle logging at the scale of Facebook, and automatically handles new logging categories as they show up (Facebook has hundreds).
[h=4]Hadoop and Hive[/h]
Hadoop is an open source map-reduce implementation that makes it possible to perform calculations on massive amounts of data. Facebook uses this for data analysis (and as we all know, Facebook has massive amounts of data). Hive originated from within Facebook, and makes it possible to use SQL queries against Hadoop, making it easier for non-programmers to use.
Both Hadoop and Hive are open source (Apache projects) and are used by a number of big services, for example Yahoo and Twitter.
[h=4]Thrift[/h] Facebook uses several different languages for its different services. PHP is used for the front-end, Erlang is used for Chat, Java and C++ are also used in several places (and perhaps other languages as well). Thrift is an internally developed cross-language framework that ties all of these different languages together, making it possible for them to talk to each other. This has made it much easier for Facebook to keep up its cross-language development.
Facebook has made Thrift open source and support for even more languages has been added.
[h=4]Varnish[/h]
Varnish is an HTTP accelerator which can act as a load balancer and also cache content which can then be served lightning-fast.
Facebook uses Varnish to serve photos and profile pictures, handling billions of requests every day. Like almost everything Facebook uses, Varnish is open source
- Facebook still uses PHP, but it has built a compiler for it so it can be turned into native code on its web servers, thus boosting performance.
- Facebook uses Linux, but has optimized it for its own purposes (especially in terms of network throughput).
- Facebook uses MySQL, but primarily as a key-value persistent storage, moving joins and logic onto the web servers since optimizations are easier to perform there (on the other side of the Memcached layer).
But enough of that. Lets present (some of) the software that Facebook uses to provide us all with the worlds largest social network site.
[h=4]Memcached[/h]
Facebook runs thousands of Memcached servers with tens of terabytes of cached data at any one point in time. It is likely the worlds largest Memcached installation.
[h=4]HipHop for PHP[/h]
A small team of engineers (initially just three of them) at Facebook spent 18 months developing HipHop, and it is now live in production.
[h=4]Haystack[/h] Haystack is Facebooks high-performance photo storage/retrieval system (strictly speaking, Haystack is an object store, so it doesnt necessarily have to store photos). It has a ton of work to do; there are more than 20 billion uploaded photos on Facebook, and each one is saved in four different resolutions, resulting in more than 80 billion photos.
And its not just about being able to handle billions of photos, performance is critical. As we mentioned previously, Facebook serves around 1.2 million photos per second, a number which doesnt include images served by Facebooks CDN. Thats a staggering number.
[h=4]BigPipe[/h] BigPipe is a dynamic web page serving system that Facebook has developed. Facebook uses it to serve each web page in sections (called pagelets) for optimal performance.
For example, the chat window is retrieved separately, the news feed is retrieved separately, and so on. These pagelets can be retrieved in parallel, which is where the performance gain comes in, and it also gives users a site that works even if some part of it would be deactivated or broken.
[h=4]Cassandra[/h]
Other than Facebook, a number of other services use it, for example Digg. Were even considering some uses for it here at Pingdom.
[h=4]Scribe[/h] Scribe is a flexible logging system that Facebook uses for a multitude of purposes internally. Its been built to be able to handle logging at the scale of Facebook, and automatically handles new logging categories as they show up (Facebook has hundreds).
[h=4]Hadoop and Hive[/h]
Both Hadoop and Hive are open source (Apache projects) and are used by a number of big services, for example Yahoo and Twitter.
[h=4]Thrift[/h] Facebook uses several different languages for its different services. PHP is used for the front-end, Erlang is used for Chat, Java and C++ are also used in several places (and perhaps other languages as well). Thrift is an internally developed cross-language framework that ties all of these different languages together, making it possible for them to talk to each other. This has made it much easier for Facebook to keep up its cross-language development.
Facebook has made Thrift open source and support for even more languages has been added.
[h=4]Varnish[/h]
Facebook uses Varnish to serve photos and profile pictures, handling billions of requests every day. Like almost everything Facebook uses, Varnish is open source
