Developer Story | From Web to Blockchain Developer
Before entering the world of Blockchain, Consensus, Wallet and Go …, I worked as a web developer mostly using Ruby on Rails. We had 2 pills, would I take an old pill, a safe and easy, comfortable pill? Or will I take the leap?
Actually, I love to develop with Ruby, its syntax is clean the most beautiful I’ve ever used. This is the most beautiful language I‘ve ever worked with. Till now, Ruby is the language I love most. And I kind of getting familiar and comfortable with it. But that time, I was offered 2 choices, one keeps working on Rails, the other is working with a whole new field, it’s Blockchain. We had 2 pills, would I take an old pill, a safe and easy, comfortable pill? Or will I take the leap? That time I and my long time teammate even make a long and silly list and discuss that what we got and what we lost after this? We have to fight in our safe zone.
Finally, we chose to work on blockchain. But at first, we worked both. Developed Rails project while research private blockchain. We chose an open-source framework Iroha. They are famous for the easy and fast implement a private blockchain network. They have a library for lots of platforms like Android iOS, JS, Python. What we need is a private blockchain network and somehow clients can interact with the network using their mobile. Seem that Iroha is a match. Ok cool just need to run the server and implement the client-side and we done. Easy!
Sadly that was a naive thought
There are two parts that we have to care about. The server and implementation of client SDK. I tried to run the server no less than 100 times but failed for the same bug. You can’t understand how frustrating it is when doing it all over again but getting at the same error. I can’t make the client be able to connect to my private network. They have the sample Android repo but full of bugs and outdated. The logic is complicated too. Until now end of November 2018, after 5 months, that issue still hasn’t solved. Did my best to understand core Iroha code but their C++ code is overkill.
They have a telegram group and every day I post my update there. Even I can’t help them as a contributor, I mean things like open PR. But at least I could try to run and report to them what worked or not. Being active in communication is important. But no matter how active I was, no one replied, seem like I monologue all the time. The one who responsible for client SDK just quitted, he said he was just a freelancer and couldn’t help me much. So the client SDK just left outdated and unmaintained.
I was in this situation for at least 2 months. I was so depressed and frustrated. I just finished my graduate research but felt like I was doing another one. We even had a discussion with my leader about the reason why we have to apply private blockchain to this project. Cause it doesn’t have much meaning. So the blockchain stuff was very fragile at that time. Hopeless and boring, no lead. I must say that I hate it. You don’t know how happy I was when somebody gives me a lead, reply or discuss it with me.
During research about the library. I found several issues. Sometimes I doubt what they‘ve demo. Cause on my side, it just not works. Also, the docs were out of date everywhere. Although it’s blockchain I can’t deploy it running on multiple nodes. Anyway, I think Iroha team has done so much coding into it, this cannot be so silly like I can’t even work on more than one node. I don’t think they are that stupid. I think there is something we are doing wrong. I can’t expect much since it’s open-source, they not working for us, we don’t pay them and the framework was still at a blazing edge of development, they haven’t go to the stable yet.
Dark night of the soul
But anyway, every time I do anything I want to get a result, for this project, at least we should make it works. And then make something that we can proud of. We should be results-oriented rather than being busy.
During development, I found that the community is super important. I always keep updated about Iroha so that I could find another way to solve it. One day I found a guy name Yasir asking about a bug on Python SDK, seems he got progress, maybe the python SDK works.
I just think of an idea. If the mobile client can’t directly interact with Iroha, why don’t we set up a Python REST API, then the web server will forward it to Iroha using python SDK. That makes the mobile team much easier to develop since they don’t need to care about blockchain, just call to an API and done, no blockchain. As I spend so much time on Android but I can’t implement the Android SDK. I don’t think we have enough time to work on iOS SDK. So instead of directly send a request to the blockchain network using GRPC, we send to REST Web server then this server forward it to the blockchain network. Ok fine, first just make it works for now. BEEM! Found the light at the end of the alley! Whohoo!
So my teammate working on the web server python REST API while I setup blockchain network, doing the infra stuff. That time I learned a lot about Docker, Ansible, k8s. My friend was very committed. Focus on Android side, he used a lot of new features.
Back at that time, we always feel pressed about deadlines. The plan is to finish in October. So we just have to make or break it
Later I realize that there was a problem with our design.
Developers usually can’t predict where the “hot spots” of their design will be.
I did a load test for my blockchain network and hell it was so slow. Even we are using private blockchain, there is only 5 nodes to confirm transaction but it took 10 seconds to commit one transaction. This is nut. Later we realized that the problem is not Iroha, Iroha is very very fast. The bottleneck is http. If we had used mobile SDK, we would have used GRPC to directly send request to blockchain network. Instead, we used a web server and forward transactions to Iroha. Even the opening of HTTP connection takes time, not to mention the web server we use, Flask, I’m not sure flask is able to handle large load since by default it is not a production grade server.
Our system was able to serve 150–200 requests at the same time. Request after that would be timeout. Since it took more than 60 seconds to have result. Hmmm. Cool. Anyway. Release date is coming. Can’t do much. YOLO!
On the release day, I was in charge of monitoring the infra. It was the first time I setup CloudWatch and use ngxtop and stuffs like that. Our system was very fragile, not only we did not have any backup plan for blockchain network, but also the code was very dirty and doesn’t have any test. We expected to have 3000 users and luckily I have changed instance type from t3.x to md5.2x large. Really I was about to use t3.x as production server. :|
But right before the release time, we got a message from my senpai which ruined my YOLO mood. He said that Docker default has only a total of 2 RAM. And that tiny amount of RAM is shared between containers, which means that if I have 4 containers, each of them will have 50 and chance is that the process will get killed or restarted by OS. Luckily later I check using docker info it was not. He talked about many other threads and stuff that we haven’t think of. Pheww, wish I have asked somebody to review our infra. And yes developers do need to know infra. That will save your life!
Developer do need to know infra — by senpai
Surprisingly the release went very well and smoothly, with no error although we did have our last change on production. So lessons learned for you that don’t mess with the production server on release day. I almost going to delete everything and start deploying all again. Thank god that everything went smoothly!
It’s important to keep your belief and commitment. Like everything in life, it requires effort and practice. If you find yourself in a rough time. Don’t quit, keep it up, keep your belief. You are just at the bottom of the graph!
