In parts 1 and 2, we saw how relational databases need us to break up data and store it in little pieces. We specifically talked about splitting up data into different tables in part 1. In this part, we will see how we can get the different pieces of data from different tables together again and what kind of problems this merging causes. Continue reading “Say NO to SQL: Part 3”
Author: Harikrishna Natrajan
A year of freelancing: What I learnt
It was just after the Diwali of 2015 that I quit my regular job and started software freelancing. Since then, my life has taken a different turn. While at work, I have been coding, taking calls, documenting, improving, improvising and deploying projects. When not working, I am travelling, trekking, cooking, reading, watching documentaries, running, bicycling and learning a lot of new skills. It has been a wonderful year, one of the best in my life. In this post, I am listing out what I learnt while sailing through the year. Continue reading “A year of freelancing: What I learnt”
Say NO to SQL: Part 2
In part 1, we saw how relational and non-relational databases differ in structure. Tables must have a singular, ‘unbreakable ‘ piece of information per row and column in a relational database, while non-relational databases can even contain tables inside tables (called nested data).
In this part, we will see how a relational database wreaks havoc with a piece of information so fundamental, that by the end of this post, you will start wondering why relational databases have been in use for so long! We are talking about human names today. We are talking about how a term I call ‘structural rigidity’ of relational databases caused an unnecessary and unwilling ‘standardisation’ in the way we look at human names today. Continue reading “Say NO to SQL: Part 2”
Say NO to SQL: Part 1
Database systems are the mainstay of applications. Without databases, it would be extremely difficult to store data in an organised manner or extract it quickly and easily. However, over the last five years, there is a shift in the type of database systems that are being used. There is a steady increase in the number of users who are adopting NoSQL databases in favour of traditional relational databases. Why is it so? First of all, what are NoSQL databases and what are relational databases? Over the course of this ‘Say NO to SQL series’, you will have all the answers. Continue reading “Say NO to SQL: Part 1”
Why the web world is rapidly adopting NodeJS
When it comes to web server software, we hear several names. J2EE (based on Java), Ruby on Rails (based on Ruby), .NET, LAMP (based on PHP) and Django (based on Python) are some of the most popular names on the list. However, in the last 5-6 years, another name has been growing in popularity rapidly. That’s NodeJS.
Based on JavaScript, NodeJS has approached the problem of serving requests on the world wide web from a new angle and helped companies set up their servers such that they are highly responsive, but consume extremely low memory. Hardware and maintenance costs have come down rapidly for NodeJS adopters, most of which are startups in their toddling phase.
If you are new to web technology, the concept of why NodeJS is a gamechanger might be lost on you. And if you are from the world of Java or .NET, it will take some time to think of solutions in the NodeJS way. That is why for this article, I got hold of NodeJS ‘herself’ for an interview. Since a lot of my work for my clients are taken care of by NodeJS, I know ‘her’ personally and ‘she’ was happy to sit down with me for a discussion on why ‘she’ is set to take on a lion’s share of the web. Continue reading “Why the web world is rapidly adopting NodeJS”
Back to squares 0 & 1: How a computer stores different types of data
Let us begin with the meaning of the term ‘compute’. The meaning of the word means ‘to calculate’. True to its meaning, the early computers were used to break down really complex calculations and solve them in a fraction of time. With developments in processing power and the shrinkage of the size of a basic computing unit, the term ‘computing’ has built upon its humble roots and now means a lot of things. ‘Computer’s are now everywhere, from the big monsters in the back offices of NASA to the sleek mobiles in your pocket to the tiny RFID readers inside your home’s door lock and railway station turnstiles. We even have devices with fantastic brains built on artificial intelligence like face recognisers, handwriting readers and trip & timetable planners.
Despite decades of advancement, some basics remain exactly the same. The oldest and time-tested basic is the way that a computer perceives and stores data. Today we may be able to calculate the distance to the sun (number), decide whether Donald Trump or Narendra Modi will win an election or not (yes/no), read and store e-books (text data), remember and get reminders for birthdays (date), set alarms (time) and enjoy sounds, photos and videos, but at the bottom of it all, the computer sees all of these data formats in only ONE universal format: a sequence of numbers. Continue reading “Back to squares 0 & 1: How a computer stores different types of data”
Understanding OAuth: Can I play in your garden?
Despite having a 9-5 job, you need to get things done inside your locked house. Your housemaid comes to clean every day at 10 am. You need your living room tiles redesigned, but the mason is scheduled to come at 2 pm. You make the necessary arrangements to leave a copy of your house keys with your neighbour, who in turn allows the maid or mason into the house on your behalf.
Is this approach really safe? Can you simply entrust the keys to your entire house to someone? Or can you restrict access so that access is given only to those parts of the house where work needs to be done? OAuth makes such selective access possible.
The problem with giving total access
There are a few problems with giving the keys to your entire house to another person.
- First, you need a highly trustworthy neighbour who will not misuse your keys.
- Second, neither you nor your neighbour knows if the maid or the mason can be trusted being alone in the house.
- It is unreasonable to ask the neighbour to stay in your house to watch over the housemaid and the mason as long as it takes them to complete their work.
What to do instead
Here is a solution.
- You do away with the single lock and key that guards your house’s front door.
From now on, the door is simply closed with a latch to keep the dust and stray animals out. - You have broken up your home security into multiple locks & keys, one pair to guard every detail in your home. You have a lock for every room and every cabinet.
- You also set up special locks for every water tap/faucet at your home. With this lock in place, no one can open the faucets, and so cannot get running water.
- If a person is appointed to visit your home, you will give them a copy of the keys to only what they need.
The maid needs access to the cabinet where you store mops, brushes and cleaning agents. She also needs access to one faucet in your house from where she can draw water for mopping.
The mason needs access to the living room. He’ll probably bring his own tools and doesn’t need access to your toolbox. - While the maid visits every day, the mason’s job is one-time. So you will visit him later and take the living room keys back. Or he may leave the keys with your neighbour.
Such modular security is complicated to set up, but it prevents misuse. This type of security micromanages the gateway to every resource. It seems daunting, even ridiculous to achieve in the real world. But it can be done without hassle in the digital world. Using Open Authentication or OAuth.
Intro to OAuth-speak
Let us revisit our home security example, but this time with OAuth jargon. We will also draw similarities with widely used Internet services.
- Your home and the things inside are resources. You are the owner of the resources.
Similarly, you own your Facebook profile and Facebook timeline. - The maid and the mason are external apps who want to use/alter your resource. We can also call them resource clients.
Candy Crush and Instagram are external apps that use your Facebook profile. - External apps may simply use your resources or alter them.
If you give permission to your neighbour’s children to play in your garden, the children bring their own toys and use your garden as a playground. Unless they are mischievous brats, they’ll leave your lawn, flowerpots and trees alone.
The maid cleans the floor/carpet in your rooms, thus altering their state from dusty to clean.
The mason rips apart existing tiles and lays new ones, thus altering the state of the living room.
Candy Crush needs to read your name from your profile. It uses your profile picture on its leader board. But with your permission, Instagram puts photos into your Facebook timeline every time you click a picture, thus altering timeline’s state. - Permissions are specific, such that particular clients can access only certain resources. The set of permissions available to a particular client is called a scope.
Your maid gets to walk on the floors in every room. She also gets to use one faucet for running water. She gets access to the cabinets with cleaning supplies. But she cannot open the fridge and eat your pastry. The scope of her permissions supports only cleaning.
The mason gets only the key to the living room.
The children get the keys to the garden.
Candy Crush only gets to read your name and profile picture. Instagram too gets to read your name and profile picture but also gets to post to your timeline.
The OAuth flow at work
Let us see how OAuth works. OAuth has the following players.
- The resource owner is the person who owns a resource. E.g, you own your Facebook profile, albums, etc.
- The resource client is the app that wants access to one of the owner’s resources.
- The authentication server authenticates the resource owner and asks for permission to grant access to the client.
- If the owner agrees, then the authorisation server hands out a token to the client. The client uses this token for all further access to the resource. The token is the key to the resource.
- Tokens are usually time-bound. They may be single-use. They can also be short term or long term, e.g. valid for a week or a month. They can also be permanent, i.e. valid for an indefinite time until the access is revoked by the resource owner.
Here is the OAuth flow for an app (or a website) XYZ (resource client) which wants to use your Twitter account for authentication and your timeline (resources).
- On the XYZ app, you choose to read from/post to your Twitter timeline.
- XYZ approaches Twitter and asks for permission to read your profile name, Twitter handle and display picture, and also to write to your timeline. That is the scope of the permissions requested by XYZ.
- Twitter’s authentication server asks you for those rights. A dialog box is shown to ask for your permission. You are probably quite familiar with such dialog boxes by now. If you are signed out, Twitter first asks you to log in.
- Once you approve, the authentication server returns a temporary entity called a ‘grant token’, a temporary token. This is still not the key to your Twitter account.
- XYZ gives the grant token to Twitter’s authorisation server, which checks the validity of the grant with the authentication server.
- Once the authorisation server is pleased, it provides XYZ with the access token, which is the key to unlock the permitted Twitter resources.
- Whenever XYZ wants your Twitter profile or to write to your timeline, it approaches Twitter with the access token. Twitter verifies the authenticity of the token with the authorisation server and once satisfied, gives XYZ the requested resource.
Conclusion
OAuth seems complicated with many moving parts, but is quite simple. It provides tokens for selective access to resources, just like keys to various rooms/tools inside your house. By micromanaging authorisation, OAuth has made the Internet safer and given it the ability to maintain your privacy within the crowd.
Further reading
Using UML diagrams: A 2-phase method to gather software requirements
Information technology is a means to automate your day-to-day tasks. But before we build systems, we need to know why we are building them. There is no way to know this unless we first gather a system’s requirements. We need to talk to various people who have a stake in the soon to be built system.
The meetings are often voice conversations and it becomes really important to record what was spoken so that there is a permanent reference. In this article, I explain a two-phase documentation system to thoroughly record a system’s requirements using UML diagrams — storyboarding and timelines.
Note that UMLs were started in information technology and software engineering. But they can be used to document the requirements for any kind of solution building. Continue reading “Using UML diagrams: A 2-phase method to gather software requirements”
Freelancer’s Arsenal: What you need to excel
It was during the June of last year that I started doing projects independently alongside my regular software programming job. It was not a very good start since I missed deadlines, failed to communicate well with clients and was rather unprofessional in certain dealings. I quit my job in November and set up a successful software freelancing / consultancy practice over the following months. After a year, I am very happy with the path that I have taken. In this post, I would love to share with you my tips on what you need to take care of to get your software freelancing / consulting practice off the ground.
Technology
This is where you start as a freelancer. Based on the core technology knowledge that you possess, you get to hunt down different types of clients in the market. Before proceeding to the other points, please spend some time to find out what technology(ies) you are thorough with. It will really help if you narrow down to technologies that you can work totally independently on. Remember that you will be freelancing (which means programming alone) or better yet, consulting a company with your expertise to make their projects world class. And do remember that, you are not going to have seniors / experienced project managers to help you all the time. Choose your technologies wisely and honestly based on your true strengths. One or more of the categories below will help you get started in today’s software market.
Mobile apps: You should have a strong knowledge of one of Android or iOS. Having done at least 5-6 apps on either platform, either in the Play Store / App Store or as internal apps used by companies and being able to demonstrate that experience will be what launches you into the world of mobile apps. It is best if you master the latest versions of the SDKs, but are able to target at least 80% of the phones in market, a lot of which will be based on older SDKs.
Suggested Key knowledge: Android, iOS, Java, Swift, Objective C.
Web applications: While web applications are divided into two parts, viz. the client side and the server side, as a freelancer / consultant, I would recommend mastering both the facets. Knowing only one side will get you only so far, because technologies in web keep changing rapidly, in a matter of a couple of months. Changes in client side technologies will massively affect the server side and vice-versa. You should, thus, be a master of both the sides in order to make effective web applications and to be able to consult companies as an all-round web applications authority.
Suggested Key Knowledge:
Client side: HTML5, CSS3, Javascript, jQuery, Angular JS, Bootstrap, Meteor, React.
Server side: NodeJS, J2EE, MongoDB, MySQL, Javascript, Java, SQL
Standalone applications: Despite mobile and web apps which are visible to the eye, a majority of the programs running in the world of computers are those that run behind the scenes to make automation easier. Image processing tools, video converters, report generation programs and the like generally run behind the scenes. There are also different kinds of servers such as web servers, push notification servers, file transfer servers and the like which are mainly standalone programs with no UI.
Suggested Key knowledge: Python, C/C++, Java, Socket programming, SQL.
Embedded devices applications: These are the emerging apps and will run on smaller devices such as sensors, wearables and special small devices dedicated for special purposes. These programs need to be extremely robust and crash proof since they run on low-memory, low powered devices and those devices run for months or years at a stretch and may be deployed in remote locations.
Suggested Key knowledge: C/C++, Shell scripting, Raspberry Pi SDK, Linux Kernel.
In addition, Arduino has been in this space for quite long, but not with good effect in the Indian tech scenario. BeagleBone and Intel Galileo are two more fast-growing platforms in the embedded and IoT space.
In addition, one should be able to use at least one version control tool, preferably Git. Mercurial may be an added advantage. I have not found it necessary to have the knowledge of any particular project management system. The preference of companies is fragmented on this topic and you may need to learn one of the hundreds of systems out there. Trello, Asana, Basecamp and Trac are some of the commonly used ones, but you may never be able to get two companies or even two project managers from the same company to settle on what the best system is, since the views are divided. I suggest going with what your client’s company has been using all along. If there is no system in place, you can choose what you like best.
Technology Concepts
If you are a consultant or a full-stack freelancer, then merely the knowledge of technology will be inadequate. There are certain concepts which you are expected to be strong in. This knowledge will help you earn a deeper involvement with your client than just coding their solution. You can make suggestions about what to use to improve their solution’s performance or reduce their costs drastically, with very low maintenance. Here are some of the concepts for today’s world of software.
HTTP protocol: Never has it been more important to know the fundamentals of HTTP protocol than it is today. As more and more types of devices embrace and consume this protocol to exchange data, mastery over the intricacies of this protocol will help the client’s software solution either run light and smooth or struggle with every network request. Complete mastery over this protocol can help design solutions which can run on the slowest of Internet connections without the user feeling the effects of the network’s sluggishness.
Suggested Key knowledge: HTTP methods, HTTP headers, HTTP response codes, JSON, multipart data type, MIME types, persistent connections.
Cache design: There are good apps which run fast, mostly when the Internet connection is excellent. Then there are great apps which seem to defy the connectivity problem and stay responsive all the time. The magic is handled by cache. Most applications only deal with very small changes to data over time. Let us say for example, a football match between Portugal and France. If Portugal score in the 117th minute, why do we have apps & websites which fetch entire swathes of information every minute? And should the server really return the name of the two teams and the score every time the client asks for the latest info? Why not just return who scored, the moment a goal is scored, and the rest of the time nothing at all. This optimisation is handled by making caches of information, wherein the system can use cached data when nothing changes and only reach for the complex algorithms when something alters the system.
Suggested Key knowledge: Redis, Memcache, LRU caches, Hashmap
Multi-threading: Multi-tasking solutions are so key to today’s software solutions. This makes it necessary to master the concept of multi-threading in your programs. This also opens up a can of worms, where threads may step over each other’s toes if not watched carefully. The concept of managing resources between various competing threads becomes important.
Suggested Key knowledge: Threads, concurrency, critical section, mutex, semaphore, messaging between threads.
Standards: Let’s face it. With the world rapidly switching to openness, be it adopting open source software or agreeing upon standards as a best practice, you need to stay on top of what standards are being released in your technology of choice. Adopting standards in your software will make it incredibly interoperable with others’ solutions. Likewise, your role as a software consultant will help companies make the best of solutions which work harmoniously with the rest of the world – and they’ll love you for it. You should consider spending some time studying what the standards in your relevant technology are and how your favourite software in that space has adopted those standards.
Suggested Key knowledge: IETF standards, drafts and recommendations, IANA, W3C, ECMA, POSIX, open hardware standards and recommendations, open web. The IETF drafts are especially important and describe each Internet protocol in detail. This is extremely important if you plan to make your own server software.
Requirement gathering and documentation
This skill becomes super important once you become a freelancer, since no one else will do it for you. This is the ability to communicate with the users of the system and document them systematically so that you can refer to them anytime. The best thing you can do is to invest your time in learning UML and in particular two UML documents: use case diagrams with use case stories and sequence diagrams with sequences of interactions among the components of the system. The former lets you write what the user wants and how he/she will go through it during a typical work day. This is written in simple English as a list. The latter lets you as the system architect, to break down the system into components called modules / classes, so that you will begin to understand how your software solution should be laid out. It will take several iterations of sequence diagrams to come up with a good solution and that is why it is important to learn and fine tune the skill.
Your professional profile (aka LinkedIn)
I cannot emphasize the importance of your LinkedIn profile enough. My LinkedIn profile has given me so many brilliant contacts and future clients. People who were completely unknown to me suddenly become paying customers after a matter of 3-4 interactions. What you do during those interactions is purely your skill, but LinkedIn provides an incredibly amazing platform for getting people to know you and your portfolio. It is extremely important to keep your LinkedIn profile updated and highlight in detail what you did during your projects throughout your career. People do read the details since they would want to take the time to invest in the best-fit freelancer / consultant.
Your contact details
As a freelancer / consultant, it is important to be visible and reachable. If your LinkedIn profile is your visibility, then your contact details enhance your reachability. While displaying your contact details online can lead to cold calls and cold emails, it is also one of the shortest routes between yourself and your potential clients. Sure, LinkedIn has messaging, but I am yet to see someone who checks and responds to LinkedIn messages with the same time responsiveness as picking up a call or responding to an SMS or Email. What’s more, a phone conversation is an example of a ‘high-touch’ communication, meaning that it is more involved, voices and opinions can be heard and a lot of ideas can be exchanged. It is the next best thing to a face-to-face interaction.
Your own tech blog
This is something that can make you stand out of the crowd as an expert in your field. This Tech 101 blog is my way of laying out how I see the world of technology. However, there are many tech blogs and you’ll want to post stories that make your style stand out. E.g. My tech blog covers general topics in technology, but explains them in very simple de-jargonised language in the form of stories. Every one of you definitely has his/her own style and your own tech blog will bring it out to good effect.
Conclusion
So there you have it. I have attempted to outline as many points as I can think of to get started. The list is in no way exhaustive. Now I would like to invite your suggestions, experiences and concerns in the comments below.
Analysing Analytics
You may have started your dream app, launched your much wanted app or set up your e-Commerce site. But once that is done, you will want to measure how well you site / app does. This is what analytics helps achieve. Analytics have now been around for a really long time, but of late they have grown really complicated and beginners are struggling to understand what to measure and what to make of all the gathered data.
Companies like Google, Flurry, Countly and several others have been investing millions to make their analytics as accurate and insightful as possible. In this post, I shall explain to you what you need to know about analytics in a nutshell and what are the basic measurements that are minimum necessities to know how well your website / app is doing. Continue reading “Analysing Analytics”