Will Dolphin Every Move Toward Microservices?

Just a thought. I have wondered what would the best approach in making Dolphin truly scale to handle 1M users easily. Let's be honest, when we first made ideas to scale our social networks we didn't think that some of us would strike it big and then have to handle a large load. Well, I didn't at least. needless to say I strongly have to consider moving away from using dolphin (though I really like it) to move into a refactoring process. 

What I am asking is if you were a team and had to move your dolphin instance from a monolithic app to a more Service Oriented Architecture (SOA), where would you start? What if you need to move more toward Microservices, where would you start?

So far, I have thought out the following and please help me understand if I am wrong. 

Step 1: First make sure that your dolphin instance is Scale-able. This meant ditching the LAMP Stack :(

To do this we had to moved all our stack from the traditional LAMP (of course) to a newer stack. Everyone is talking about MEAN stacks but I started with something close to the Dolphin systems. 

Where we landing was a stack that moved away from Apache as a primary webserver. There's a good book by Steve Corona call "Scaling PHP" ... he said the following:

After lots of trial and error, I’ve found what I think is a generic, scalable stack. Let’s call it LHNMPRR… nothing is going to be as catchy as LAMP!

This stands for:

Linux, HA Proxy (or NGINX LB), NGINX Web Server, MySQL (Percona), PHP-FPM, Resque (or RabbitMQ, NSQ)

Step 2: I needed to make dolphin more Distributed: API Work, and Data Files

EVERYONE DON'T HAVE TO DO THIS - Since we made a gang of custom modules in dolphin, we wanted to make sure that we can have a faster site. So we split the frontend of the site and made some changes in a custom template (will spare the details here) and we made EVERYTHING speak through and API. Basically we turned all our modules into API's. This help with our mobile apps as well. 

Data Files: We moved away from the RTMP and FFMPEG  and Local uploading of media files and used a Third party service for all video converting and storing (Like AWS CDN or Rackspace CDN) Also we pulled out all the Unstructured data (avatar images, photos, video, etc) and used Blocked storage. This was and is still a pain. 

Step 3: Queue & Cache, Cache Cache

I can't say it enough. We had to master the queue & caching process of all the little items like commenting, likes, etc. This is done by using a message queue system and redis (or memcache)

I know this is a lot and i wanted to be brief. But my question is this. to truly move into the next generation of application development can someone like me form an exit strategy and move toward decoupling my dolphin core? What should be my approach if I need my site to move more into a microservice platform so that we can go big? Will dolphin ever move into this new future on the horizon?

If you're going to make a distributed system then its pretty much not dolphin at that point, you will rewrite all/most of it.

I am guessing the site is already slow? in that case I would suggest throwing more hardware at the problem. Just setup a load balancer for several app servers and do a mysql master-slave config, make your slave provide read queries and writes goes to master. Switch your table engine from MyISAM to InnoDB, that might require testing and fixing any issues that might come up. Do some innodb tuning, like setting "innodb_buffer_pool_size" to 70-80% of memory of that dedicated mysql server.

With some work on dolphin you can make it dispatch jobs to a job queue using something like beanstalkd, where you can have farms of servers taking jobs from the queue and doing 1 thing only like converting videos or images.

Also, you might wanna use Ansible or SaltStack to manage these servers.

You said you're already using heavy caching which is good. This can be your short term scaling solution while your team rewrites the whole thing more distributed. At the end of the day, either you modify dolphin so much to fit your needs that its not even dolphin anymore or rewrite the whole thing, its the same thing and depends on your team's preference.

Yep, We have done all these things. 

First, we wrote most of our modules custom and complete independent of dolphin. They there are core features that we used in dolphin(mainly the  auth, admin panel, spy data, and and profile,) but now it is time to move away form that. We inherit dolphin and I can tell that it is a good start. however, you are right. once you really need to scale it to handle 10k request in jmeter/blazemeter, then is just could not cut it. 

We basically used dolphin to get started to get user base and then we are dark launching new and old functions to strangle it.

ONE BIG issue. we have to make the app stateless, and boonex has the local caching system. (cache, public_cache that we are trying to get rid of. We technically do not need it (we are caching remotely). Even when we turn off the caching in the admin panel it is still acting up.



Any advice of this? 

 you mean boonex got all our data base, or our database only in our register server ? could boonex access our website files?

They mean Dolphin - a lot of people use the two interchangeably sometimes. Dolphin has a local cache system, and can use File (stored on disk - slower), XCache, APC, or Memcache (all three use RAM - faster). Unless you're already distributed, I recommend APC. This is all done on your server.

All BoonEx knows about you is the license and what domain it's attached to. It also has a version check system, so maybe also the version number if they checked their logs. But that's all.

If the site has become too large to justify a monster server, then it's time to cluster it (or at least start to kick services like MySQL and Chat+ onto separate boxes): https://www.boonex.com/trac/dolphin/wiki/TutorialHowToReplicateDolphin

Maybe using memcached server would help ?

Yes, there are some caching which can't be turned off, for example language file must be created.

Also in UNA we've made to allow to disable almost all caching if needed.

I mentioned something like this a long time ago, about making dolphin more distributed and scalable like the larger social networks. If you sell a mod version of this, I'll buy it.