Data can be defined as a systematic and scientific storage of a particular quantity or size. It is the different values of that quantity represented together in a set. A collection of huge chunks of data thus is known as Big data. Big data is a collection of large chunks of datasets that are to be processed using complex computing techniques. It contains huge variety of information which are extracted from these chunks of data. Big data technologies are vital in providing more accurate and scientific analysis, which may lead to more concrete and precise decision-making resulting in greater operational efficiencies, cost reductions, and reduced risks for the business firms. To harness the power of big data, you would require an infrastructure that can manage and process huge volumes of structured and unstructured data in real-time and can protect data privacy and security.
Big Data is being the most spread latest technology that is being used in almost every field either may be business, education, or research. Some of the field are:
• Travel and Tourism
• Financial Institute
• Health Care Sector
• Marketing and Sales Sector
• Telecommunication Sector
• Government and Military
(Big data applications, 2020)
As Big data can help to run the business firm smoothly. An ability to process Big Data brings in multiple benefits. Businesses can utilize outside intelligence while taking decisions. Access to social data from search engines and sites like Facebook, Twitter are enabling organizations to fine tune their business strategies. Improved customer service helps to run the business. Traditional customer feedback systems are getting replaced by new systems designed with Big Data technologies. In these new systems, Big Data and natural language processing technologies are being used to read and evaluate consumer responses.
• Early identification of risk to the product/services if any
• Better operational efficiency
Big Data technologies can be used for creating a staging area or landing zone for new data before identifying what data should be moved to the data warehouse. In addition, such integration of Big Data technologies and data warehouse helps an organization to offload infrequently accessed data. (Zeng, 2013)
Big Data Use Cases
The successful implementation of big data analytics is to show how to run the business and how much business gives output. It help organization to make proper decision at right time.
The two aspects business firm needs are:
1. Keeping the scope limited
2. Finding the need of consumers
A single set of data are used in the multiple industry use cases, but a firm need to identified and prioritize the use of these use cases. (Big Data Use Cases)
1. Analysing 360-degree view of the customer
2. Product customization and identifying the customer behaviour patterns.
3. Delivering with quality product and work towards customer long term retention
4. Forecasting demand and maintain the optimum level of price
5. Optimizing product/service portfolio
6. Developing a new product/service that will be a success
7. Creating efficient marketing mix
8. Catching the opportunities for business growth
9. Improving warehouse processes
10. Optimizing delivery and distribution
11. Managing inventory efficiently
12. Optimizing manufacturing
13. Fostering preventive maintenance
14. Managing product quality
15. Strengthening quality assurance
16. Preventing fraud
Some of the real case study cases of some industry are in listed below:
Industry: Financial Technology
Use case:Big data technologies helps to analysis the real time data or streaming data. Financial institutions have access to each and every financial data ,user data and using the predictive analysis pattern financial institute prepare plan and polices to general public. Big data tends to analysis the trend of customers ,predict purchase behaviour, and alert users to fraud and also help financial institute to analyse the market for loan risk assessments.MoneySQ uses Tableau as analysing tool for big data. Before Tableau Staff of MoneySQ identified the business targets , these identified targeted customers are analysed through multiple platforms from excel and online and decision takes time . But with the implementation of Tableau with in the day the precise data output are given.
Organization: Providence St. Joseph
Use case: With the use of big data technology, these companies can now easily analyse the raw data. Especially in health care industry, patients real time data like sensor from patient-monitoring systems to predict. Fields like weather and seasonal data are used to analyse need of staff and bed. Now, Providence St. Joseph can clearly analyse the disparate sets of raw data with the help of big data analytics tools in order to improve patient care and reduce costs. Providence St.Joseph identified the general issue of waste and they analyse to reduce waste. Wasteful practices includes costly supplies, medications which overall increase the cost per patient which result in health care less affordable for patients.
(Big data examples use cases, 2011)
Critical analysis of Big data technologies
The emerging Big Data technologies is transforming its nature day by day and due to which it attracts different sorts of public business firm. It has totally changed the way we think and attract the new industry and due to we can analyse the data in proper channel.
Big Data Technology is mainly classified into two types:
1. Operational Big Data Technologies
The day to day data we generate are commonly known as Operational Big data. Online transaction, social media or other related transactions are generally data set of operational big data. Some of the real life examples are:
• Online ticket bookings like railway, air flight and all
• Amazon, Flipkart, Walmart, Snap deal are example of online shopping.
• Data from social media sites like Facebook, Instagram, what’s app and a lot more.
• The employee details of any Company.
2. Analytical Big Data Technologies
It is more complex version of Big data technologies. Analytical big data helps to identify the crucial real-time business decisions. Some of the real life examples are:
• Stock marketing
• Space mission
• Weather forecasting information.
• Medical fields where patients are constantly monitored. (Kiran, 2019)
Generally big data technologies are divided into four categories. some of them are:
• Data Storage
• Data Mining
• Data Analytics
• Data Visualization
Fig : Source ( (Kiran, 2019))
Different framework are there in order to implement in different types of framework. Depending upon the workloads large scale data processing enterprise use different types of technology. Some of them are:
1. Artificial Intelligence
As we know AI, is burning topics in today’s world. Smart machines capable of completing the various task which need human intelligence is particularly termed as Artificial Intelligence. From self-driving car to SIRI all are related aspects of AI. Approaches like Augmented machine learning and deep learning into accounts to make remarkable shift in each and every tech industry. AI applications are like drug treatment, healing patient and conducting surgery in operation.
2. NoSQL Database
Modern application are designed using NoSQL.They are actually deployed in the application where real time processing is done and big data analytics are done. It delivers a method for storing and retrieving of data. Especially unstructured data are generated from the application and store for faster performance.Some of examples are : MonoDB,Redis etc. Companies like facebook,google,twitter store huge amount of data with in a single day.
3. R Programming
The open source project is done from R programming language which is free and used for statistical computation and visualization. It is mostly used by data miners and statisticians for to design the statistical software and data analytics.
4. Data Lakes
All formats of data which are structure and unstructured data at any scale are stockpile in consolidated repository.
In data lakes ,data are proceed by accumulation and data are saved with out transforming it into structured data which at last general different sorts of data analytics graph for visualization. Data from log file , social media,click stream and IOT devices are stored in data lakes which can be proceed in real time and machine learning can be implemented for better business learning.
In the process of data accumulation, data can be saved as it is, without transforming it into structured data and executing numerous kinds of data analytics from dashboard and data visualization to big data transformation, real-time analytics, and machine learning for better business interferences. It helps organizations to know new insights regarding the business which overall help them to grow their business and engage more customer.
5. Predictive Analytics
This is subset of big data analytics; It identified the future behaviour with the help of prior data using machine learning algorithm, data mining and statistical modelling.
With the help of predictive analytics, it generates the compelling degree of precision and help us to track the right data flow and everything.
For instance: to explore the relationships among various trending parameters in any fields. Such models are designed to assess the pledge or risk delivered by a specific set of possibilities.
6. Apache Spark
Apache spark is noted as the one of the speedest big data analytics generator for big data transformation. Languages like python, R,SCALA and java are used. It has built in features for streaming, SQL ,machine learning and graph processing support.
Due to downfall or limitation of spark for data processing Hadoop was introduced which is much more faster and it reduce waiting time between interrogating and execution of data where as spark is also used in Hadoop for storage and processing.
7. In-memory Database
In this type of analytics in-memory database (IMDB) is stored in the main memory of the computer (RAM) and controlled by the in-memory database management system.But generally, conventional databases are stored on disk drives.
It is built in order to reduce the time to omit in disk.But all the data should be stored in main memory because there can be a high chance of losing the data.
The rise of decentralized database has now impacted the big huge cooperation. Coins like bitcoin, ethereum are built under this methodology. Once written data is never changed .It is highly secure and industries like banking,financing,insurance,healthcare and retailing can get benefited from it.
10. Hadoop Ecosystem
Hadoop is new apache based open source project which is more used for commercial packages. It facilitates with a variety of varied components and services name ingesting,storing,analysing and maintaining inside it. In Hadoop,HDFS is storage layer which is suitable for distributed storage and processing while data is being store.It provides streaming access to file system data. (Patranabish, 2016)
Use of Big Data tools on the dataset
Data set is a large collection of data of particular organization. Ecommerce business can get huge benefited with the help of data which are collected from the day to day operation of customers.
Big corporate firms know the real value of information generated from customers and customer choice while shopping in their online portals .Big data analytics helps the company to generate the more and more customer preferences.Data-driven through data analytics help ecommerce businesses regularly measure and improve on the list of following things :
1. Improve shopper analysis into more precise.
2. Improve customer service.
3. Personalize customer experience.
4. Provide more secure online payment processing and detecting fraud
5. Better target advertising to right customer.
(Carsten Bange, 2015)
1) Predict Trends
Big online business firms always seeks to identified the best selling product before competitors. There are few strategies in which big data can detect the predict of such product.
2) Optimise Pricing
Before the era of big data analytics pricing systems are impacted through other firms competition and depending upon the competitors and value of their particular product price are set. Due to big data innovation and analysis Optimal pricing can be setup for a particular product.
3) Forecast Demand
Forecasting the need of customer or demand of customer has always been very hard for the business firm. For instance:If an ecommerce sell shirts online, then firms need to predict in advance how many shirts of each colour will be demanded.
4) Create Personalised Stores
With identified the unique product for each and every product is very vague but with the help of big data personalisation increase in sales.
5) Optimise Customer Service
If a customer is satisfied with the product, retention is very much common but if a customer faces a problem while doing shopping then this customer will not return in that particular business firm. Big data allows businesses to optimise their customer service in more accurate and enhanced service.
6) Generate More Sales
With the help of big data, business firm can now generate more and more customer towards there site and generate huge volume of sales.
7) Offer more secure online payment processes
With secure online payment process and fraud detection and analysis due to big data algorithm, it has been even greater secure online payment process. For reference: PayPal is using big data resources to enable machine learning algorithms. These algorithms analyse billions of transactions with in single entry to identify potentially fraudulent transactions and report to the system.
Big data architecture
Big data analytics is achievable only through the systematic process of big data architecture.Large amount of data are now analysed for proper business decisions. Vital data can be extracted through the means of big data from ambiguous data. The big data architecture framework serves as a blueprint for big data Infrastructures Company and solutions, precisely defining how big data solutions will work, the components that will be used, how information will flow, and security details.
Fig : Big Data Architecture
In big data architecture it consists of four logical layers which are described below:
Big Data Architecture Layers
• Big Data Sources Layer: All the process like batch processing, real-time processing of big data sources like data warehouses, relational database, SAAS application and IOT devices are easily managed with Big data.
• Management & Storage Layer: It precisely receives data from sources which convert into particular format and store it.
• Analysis Layer: It usually extract business intelligence from the big data storage layer.
• Consumption Layer: It store these results from the big data analysis layer and presents them to the outer layer for proper presentation - also known as the business intelligence layer.
Big Data Architecture Processes
• Connecting to Data Sources: Any format of data can connect to different storage system, protocols and networks, conntectors and adapter are used to connect.
• Data Governance: It analyse the privacy and security of data starting of ingestion of data through processing,analysis,storage and deletion
• Systems Management: It is highly complex and scalable, large-scale distributed unstructured or structured dataset are monitored using the central management console of big data system.
• Protecting Quality of Service: Data qualities are precious in business and data hampering should be there .Data integrity should be maintained.
Benefits of Big Data Architecture
The huge volume of data is increase day by day. Online streaming has now increased traffic in more ways than pervious. Traffic sensors, health sensors, transaction logs are all stored. All these data are important but without getting the report of these data are useless. Following are the list of things that big data architecture can solve the real life issues:
• Reducing costs: Due to growth in latest technology like Hadoop and Cloud-based analytics it helps to save the cost to store the chunk of data
• Making faster, wise decisions. Using the different streaming component of big data architecture, decision are made in real time.
• With the facilities like predicting the future new product line can be developed (Big Data Architecture , 2019)
Conclusion and Recommendation
As big data is huge different strategies are used in order to analysis the data generated from the big data. We can, not only find the customer wish product but also predict the market demand and many more. Big data has changes the horizon of ecommerce industry getting more and more customer. If proper analytics of big data is done, Ecommerce platform will be very much beneficial from the output of Big data. Different types of Big Data analytics frameworks which focused with Big Data analytics workloads are analysed to get proper result against set of criteria. In this report we identified that all existing framework lacks a single limitation. Due to which existing frameworks does not met the criteria to fulfil the attributes of Big Data together or they require high level of expert knowledge to work with. However, most of Big Data analytics lack the expertise to use the system. Therefore, a new approach is required to deal with new and new Big Data analytics problems.
Big data examples use cases. (2011). Retrieved from Tableau: https://www.tableau.com/learn/articles/big-data-examples-use-cases
Big Data Architecture . (2019). Retrieved from Microsoft: https://docs.microsoft.com/en-us/azure/architecture/guide/architecture-styles/big-data#:~:text=A%20big%20data%20architecture%20is,big%20data%20sources%20at%20rest.
Big data applications. (2020). Retrieved from Simplilearn: https://www.simplilearn.com/tutorials/big-data-tutorial/big-data-applications
Beshah, B. M. (2016). CRITICAL ANALYSIS OF EXISTING BIG DATA ANALYTICS FRAME WORKS. Retrieved from Journalcra: https://www.journalcra.com/sites/default/files/issue-pdf/13921.pdf
Big Data Use Cases. (n.d.). Retrieved from Big data Analytics News: https://bigdataanalyticsnews.com/big-data-use-cases/
Carsten Bange, T. G. (2015). big data use cases. Retrieved from BARC : http://barc-research.com/research/big-data-use-cases-2015/
Harvey, C. (2017). Retrieved from https://www.datamation.com/big-data/big-data-use-cases.html
Kiran, R. (2019). Top big data technologies. Retrieved from Edureka: https://www.edureka.co/blog/top-big-data-technologies/
Osman, M. (n.d.). Benefits of using big data. Retrieved from Bigcommerce: https://www.bigcommerce.com/blog/ecommerce-big-data/#the-benefits-of-using-big-data-in-ecommerce
Patranabish, D. (2016). Big data impacting ecommerce industry. Retrieved from Yourstory: https://yourstory.com/2016/11/big-data-impacting-e-commerce-industry
Zeng, J. &. (2013). Looking inside the black box. Strategic Organization.