Harvard Business Review authors Thomas H. Davenport and D.J. Patil described data scientist as “The sexiest job of the 21st century”. Essentially this new profession is about generating all the information from the rapidly growing mountains of data – i.e. Big Data – that a company needs to e.g. better adapt to changing business conditions or to be able to work more efficiently. The idea isn’t exactly new – so what makes it so sexy?
A great deal. The job as data scientist is like being a gold digger who holds an extensive knowledge of how to find and extract the information nuggets from the depth of the data centres in the quickest, most efficient and most profitable way. Like a pioneer stepping foot on new ground, identifying patterns where others only see data chaos. Sometimes their work resembles that of a forensic expert, using the minutest clues to deduce causes for inadequacies in business processes. And they can look into the future with a certain degree of reliability because data scientists don’t rely on gut feeling but base their predictions on the latest developments and methods of predictive analysis. If all these capabilities together don’t make a job sexy then what does???
With this in mind, it’s no wonder that data scientists are hot property at the moment. Less so in Europe, but unbelievably desirable in USA and China. In these countries many companies are realising that rather than capitulating in the face of this exponential data flood, you can actually use it to your advantage – provided you have the relevant experts on hand. More and more start-ups are also placing data science at the heart of their business as it is predestined for the development and implementation of innovative business ideas.
It goes without saying that the job as data scientist isn’t easy. Proper all-rounders are needed in order to be able to fulfil all the tasks we’ve mentioned so far. The job as data scientist is hybrid of a mathematician, computer scientist, statistician, software developer and business process development manager. Experts are needed who can think beyond departmental borders and see the whole company picture. Last but not least, data scientists are those employees who are able to ask the right questions concerning any business shortcomings and to draw the optimal conclusions from the answers.
It’s exactly these right questions and optimal conclusions that bring a big data application to life. Because big data tools may be highly intelligent solutions matured over two decades, but just as a grand piano needs a pianist, these tools need the right users who can play them to their potential. The reverse is also true: Data scientists can only do their job properly if they work with the right instruments.
This is exactly where enterprise search solutions, such as the search appliance InSpire from Austrian provider Mindbreeze, come into play. They help to prepare the mountains of data that an overwhelming majority of companies struggle with so that data scientists can extract all the relevant information that they need for a particular task. The term “search” in this context is a real understatement – these systems go far far beyond the realms of a standard search query that you might be familiar with typing into a browser. In essence: Enterprise search transforms the unstructured world of endless masses of emails, documents, videos, photos and audio files into a structured environment where data is connected via meaningful correlations. In other words: It turns gold dust into nuggets.
For this, a range of functions are placed at data scientists’ fingertips that have been developed over many years, mainly within the global specialist community. The basis of an anterprise search solution is the indexing. A high-performance solution can handle from many millions to billions of documents – per day. This performance can be increased if needed by clustering multiple search appliances.
During the indexing process enterprise search systems extract all the information that help data scientists to do their job as best possible from the different file formats – top systems support over 500 different file formats – and from different sources across the whole company, whether internal applications, cloud applications or the internet. Before data scientists’ eyes, a single managing director calendar entry links to an email attachment that appeared to have disappeared in the support department’s archive and to an open invoice saved on a finance team member’s PC appear – all this quickly forms a more complete picture of the customer, who is clearly annoyed. The data scientist’s task is to make sure that this customer is given a call before they throw in the towel and try their luck with a competitor.
There are thousands of similar examples, and with the further development of enterprise search solutions, who can be particularly played out to their full potential in the competent hands of data scientists, the number of these applications is growing daily. The examples range from simple CRM tasks to highly complex process topics to help an airport, for example, to optimise their baggage handling so that expensive idle time for planes and frustrating waiting time for passengers at the baggage reclaim can be reduced to a minimum.
The value that the synergy of enterprise search solution and data scientist can bring to a company is clear: More efficiency, optimisation of processes and the promotion of business and revenue-related knowledge are just a few of the aspects that all companies who dare to take the necessary steps from the unstructured into the structured world would benefit from. Now if that isn’t sexy, I don’t know what is!