What is Hadoop?

A brief introduction to the open source big data software library

The big data systems in use today started life in Google’s laboratories. In 2003, Ghemawat et al. published a Google File System paper (2003), and this inspired two Google employees, Doug Cutting and Mike Cafarella, to start a project. Google then published another paper called ‘MapReduce: Simplified Data Processing on  Large Clusters’ (Dean, 2008). Initially, the project was named Apache Nutch, but in January 2006, they renamed the project Hadoop (named after a toy elephant that one of the founder’s children played with at the time).

The Hadoop project initially comprised 5,000 lines of code that made up the HDFS (filesystem) and 6,000 lines were for the MapReduce algorithm. Together, these two pieces of software allowed for the management of extensive data collections physically on multiple disks and other media, and large-scale processing of data.

In April 2006, Google released version 0.1.0 of Hadoop as open-source software. It meant anyone could take this code and use it on their projects. In the same way the software that allowed the Internet to flourish (HTTP, Apache web server, PHP, etc.), Hadoop gave the world open access to robust and scalable software for big data analysis. Since those early years, Hadoop’s developers have improved the code significantly, and today it is an extensive suite of powerful tools. The base Hadoop framework has the following modules:

• Hadoop Common is the core libraries and utilities required by all the Hadoop applications. 

• Hadoop Distributed File System (HDFS) is the file system that provides storage and also makes a massive amount of bandwidth available for data processing. 

• Hadoop YARN is the part of the system that manages resources in clusters and that makes them available for user applications running on Hadoop. Finally, 

• Hadoop MapReduce is the Hadoop implementation of the algorithm that performs data processing.

Hadoop is written in Java (and some C) but many people use other languages to do data analyses. Python is popular because it has excellent libraries targeted explicitly at data scientists (such as Pandas, NumPy, SciPy, MatPlotLib). Most data scientists are doing big data analysis using Spark on top of Hadoop. Spark is an analytics engine written in Scala (which can also be programed in Java and Python), and is the evolution of MapReduce (see https://spark.apache.org). The skills required for managing a Hadoop cluster are not just programming related. Knowledge of operating systems, scripting, hardware and security are all necessary when dealing with big data systems. Also, being able to connect to application programming interface (APIs) in order to deal with streaming data allows real-time analyzes. Data scientists have skills from all these areas and have to spend a lot of their time cleaning and preparing data besides planning the algorithms for data analyzes. 

Hadoop is a powerful software that has made big data accessible to everyone. The Apache Software Foundation manage the project. It is the world’s largest software foundation and this has meant that Hadoop has evolved into one of the most powerful suites of software on the planet. There will be more features and software packages, and this will ensure that the tools for data science will evolve and become even more important for the evolving world of data science. 

A Virus in the Information Age.

How misinformation made things worse, yet good information prevailed

COVID-19 is unquestionably a challenge, because it is a physically difficult illness to deal with and there was a resulting panic like no other in recent history, especially in the media. The proliferation of online news and social media makes it hard to see the truth of what is happening, but scientists across the world have been using information systems to allow people to see the truth of what is happening despite governments and leaders trying to hold back on information.

The initial days of the virus in 2019 were dark days from an information perspective. The Chinese government was not transparent with the information released, and there were many rumors of police suppressing people who tried to talk about the spread of the disease. Soon, it became inevitable that they could not keep this information to themselves, and the effect spread globally.

The next barrier in terms of the truth was social media, where fake news was being published that caused panic and social unrest. The results of this made an already bad situation worse, with people panic buying because they had read that there would be shortages of commodities ranging from toilet roll to meat. The effect of this misinformation was to make people leave home and head to the supermarkets with thousands of other people, a result that cannot be good when a virus is so easily spread between people in proximity.

John Hopkins University has been working tirelessly to provide an information dashboard to the world that has shown the real situation, with none of the claims from politicians impeding the real truth. The Center for Systems Science and Engineering (CSSE) comprises researchers who collaborate with the Department of Civil and Systems Engineering (CaSE) at Johns Hopkins University. They claim to be “united by the goal to better understand and improve societal, health, and technological systems for everyone”*. They have been working around the clock to ensure that unbiased information is available and that it is constantly accurate and up-to-date, and they have allowed the less biased media outlets such as the BBC to publish accurate and realistic insights into the spread of the disease. While Donald Trump has been proclaiming that the “Chinese virus” was barely a threat to his nation, the public could see the progress of the virus and see that his statements were merely poor attempts to hide the truth. They have provided an academic light in these dark times that has allowed anyone, from a worried citizen to the government, with a reliable and trustworthy place to keep track of the situation. They have made the data from their studies available to scientists around the world, and through their hard work are giving us the ability so see the effects of all the efforts to contain and restrict the spread of the disease.

In an age that is defined by information, the researchers of John Hopkins University have provided a lifeline of truth. Along with the medics, police, supermarket workers and pharmacists in the world who have kept day-to-day life going for us, these scientists are also examples of people who are helping immensely. To see their efforts for yourself, visit https://coronavirus.jhu.edu/map.html

* https://systems.jhu.edu/

How worried do we need to be about encryption?

There is a lot of debate right now about the importance of encryption. I’ve seen the dark side of not using encryption on a network when a hacker highjacked my email credentials in the early days for WiFi and my PayPal account hacked (and yes, I did have some money taken from my account, but the bank covered it, thankfully!)

Since my unfortunate experience, I have always been an advocate of using encryption where necessary. Whether it be on a public WiFi (using VPN) or when backing up my data onto encrypted hard disks. However, there is an essential point to take into consideration - that encryption also hampers law enforcement efforts and having encryption everywhere could make the police’s job challenging.

In recent news articles, it has been reported (for example, https://www.reuters.com/article/us-apple-fbi-icloud-exclusive-idUSKBN1ZK1CT) that Apple has cancelled its effort to encrypt user’s backups on their servers with end-to-end encryption. Many journalists have rushed out to report that this is an example of them not taking user privacy seriously and giving in to the FBI’s demands. Much more needs to be considered for this assumption to be the only reason for the decision.

Firstly, Apple must receive a considerable number of requests from people who legitimately lose their devices, and want to recover data. If it were to use end-to-end encryption, and Apple did not know the key, then it would be impossible for them to recover data. It’s not like they’re storing the data on the servers with no encryption at all, they are encrypting everything in iCloud - just not using end-to-end encryption. This means that Apple has a key and they can unlock if you lose your password, device, or if the authorities request access. The only issue would come if the data were released and then the authority in question leaked some personal information or got hacked. Then there is the potential for misuse of data.

Secondly, if the FBI, or any other official investigation agency, need to find information, it is because someone has done something terrible. They’re not going to go demanding Apple give out user’s data for no reason. By using too much end-to-end encryption, we are letting people get away with crimes. Finding criminals is something that needs to be possible for law enforcement; otherwise, murders, rapes and other horrendous crimes will go unsolved.

Thirdly, if people are so worried about dodgy information in their backups, putting it in a public cloud is never a good idea. You can back up securely yourself and ensure that your sensitive data is safe.

It is unquestionably a controversial topic, and everyone will have a different opinion. I do not mind if the police or any other authority has access to my data if it will help in an investigation. If I were up to no good though, I’m sure my opinion would be different. While half of the Internet seems to be bothered by all this, I do not see it as a huge issue. Yes, encryption is good, but iCloud is encrypted. So, only people with good reason are going to be able to access the data. Keep using that VPN and stop worrying about your backups, unless the FBI are after you of course!

A MacBook Pro 16” review from a developer's perspective

Specs of the model reviewed:

• 2.3GHz 8-core 9th-generation Intel Core i9 processor Turbo Boost up to 4.8GHz

• AMD Radeon Pro 5500M with 4GB of GDDR6 memory

• 16GB 2666MHz DDR4 memory

• 1TB SSD storage

The saga of the Apple butterfly keyboard has seemed like an endless one. It is still something that consumers will associate with Apple for many more years to come. I had one keyboard replacement for my laptop, and it was a complete pain. Luckily the replacement was done in 24hrs, but the hassle of having to spend a working day without my computer was enough to make me realise the pain that some people must have gone through with longer waits and more failures.

Since then, I have had no further issues. However, as soon as there was an announcement that Apple had replaced the butterfly mechanism with something in between a traditional scissor and the butterfly that would not have the same issues, I was interested. Not only were repeating keys annoying and replacements inconvenient, but I was also not keen on the feel of the butterfly mechanism. There was not enough travel and the keys made far too much noise.

So, while my previous laptop still had good resale value, I took the plunge and picked up the new 16”. I chose the higher-end base model (i9) because I do a lot of data analytics and use virtual machines to test computer code. As a coder, the addition of a physical escape key was the icing on the cake, as I loathed the Touch Bar escape key, especially when working in editors like VIM and when programming. To my pleasant surprise, there is far more to the new MacBook than just a new keyboard.

I was reluctant to upgrade to the last generation of 15.4” MacBooks for a simple reason: throttling. I wanted the capabilities of an 8-core processor to be able to virtualise and to be able to run complex Spark scripts using the cores as cluster nodes. However, when I experimented with a colleague’s machine, the performance was severely limited due to thermal issues. Apple claims that the 16” has a new cooling system that has made this much less of a problem, and after using the machine for a couple of weeks and doing extensive multi-core processing, I can attest that this is true. The performance is stunning, and it is easily 2x the speed of the 4-core models that I’ve been using in the past. It makes a previously painful 10-minute data analysis that made changing scripts laborious, into something that only takes a few minutes and equates to much more productivity. It has also handled video conversions (using compressor) with ease and makes editing 4K a pleasure in Final Cut.

The most profound performance increase is in virtual machines. I do a lot of work with students running on different platforms and therefore get code that cannot be smoothly run on a Mac. Windows runs amazingly well on Parallels, and it has gone from being something I dread running because it is slow and takes ages to do anything, to a non-hassle. I have to do a lot of work with Hadoop systems and prefer to do this in a virtual machine so I can experiment without affecting my production environment. Linux also performs well, and even when under load it never slows down.

The screen is gloriously big, and allows me to have lots going on when I’m working. It is not much larger than the older model 15.4”, but it looks far better with the smaller bezels. The brightness and clarity are excellent.

The 1TB of storage also means that I have the space for virtual machines, lots of datasets and video. This move to increase storage and bring down the price is one of the best Apple have ever made, and I am glad they have listened to their users and made the minimum workable for professionals.

Less relevant to me, but still notable, are the speakers. I do not do audio and video professionally except for presentations, so the speakers are not something I use in my typical workflow. Nevertheless, they sound great, and I am happy with the improvements. I still prefer a dedicated speaker for music, but while travelling this will mean I do not need a portable speaker anymore and less luggage is always a bonus.

Are there any downsides? I cannot find much to complain about at all. The weight is substantial but worth the trade-off for the power and screen size. The fans are loud when putting the system to work, and in a silent office, this can be a little disturbing for fellow workers. I use an iPad when working in places such as cafés and I think the 16” is too big for these locations, but the predecessor (15.4”) was also the same. I also would not find a laptop this size comfortable when flying. For most people, it will be worth waiting for the 13/14” to be released next year with the new keyboard. If you are not a creative professional or coder, then that will more than suffice.

In conclusion, the 16” MacBook Pro is an excellent improvement over the previous generations. The keyboard is the star with the escape key and much improved key travel. The i9 offers incredible power, enabled through a much improved cooling system, and the extra storage means professionals no longer have to worry about carrying around external hard disks. The speakers rival portable ones and mean less luggage when travelling and will be a massive benefit for people dealing with audio on the move. In sum, this is the Apple laptop that we’ve been waiting for, and it has far more to it than just a better keyboard.

Research

It’s been a bit quiet here for a good reason. While I’d typically be actively writing here, I’ve been focussing my attention on publishing in academic journals instead.

I'm putting the finishing touches to a research paper that looks at the issues faced by organisations with Big Data, and I am currently doing qualitative research in nine multinational organisations.

This is just the start of a much lore long term research project which I am doing as a part of the Center for Consumer Driven Growth.

We aim to build a model and methodology for diagnosing organisations to help make them more customer-centric. The aim is to identify the product-centric organisations and help them transition to be customer-centric.

As a result, I will not be able to publish as much as I have been doing recently. I will be checking in here with regular updates and I am sure some of the material I write will be extra, and all that will be published here.

Thanks to all who read this blog. If you are a professor like me, good luck with the coming exam and evaluation period, it will soon be Christmas and it will all be over!

Online protection

There are a few steps everyone can take to be safe online. Well done to The Guardian for summarising the essential steps anyone can take.

This is the bare minimum you need to do if you don’t want to lose passwords and have hackers take over your accounts. Spread the word!

Welcome to my new blog

Here I’ll be publishing articles written about my teaching subjects and my life in general.

More updates very soon!