{"id":16342,"date":"2022-06-08T10:51:38","date_gmt":"2022-06-08T09:51:38","guid":{"rendered":"https:\/\/amplyfi.com\/?p=16342"},"modified":"2024-03-22T13:15:51","modified_gmt":"2024-03-22T12:15:51","slug":"data-discovery","status":"publish","type":"post","link":"https:\/\/amplyfi.com\/2022\/06\/08\/data-discovery\/","title":{"rendered":"Data discovery: How do we process the ocean of information on the web for better decision making?"},"content":{"rendered":"[vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”]
<\/i> <\/div>
Gordon Fraser, Engagement Manager<\/strong><\/em><\/p>\n

Gordon, AMPLYFI\u2019s Engagement Manager, is responsible for building and nurturing relationships with key clients. He has spent the past 10 years in the disruptive technology space, providing a global customer base with the tools necessary to succeed in today\u2019s competitive environment.<\/em> <\/div>

<\/div><\/div>[vc_column_text]We are experiencing a period of rapid data growth. In 2015, the amount of information available on the web hit 1 Zettabyte (ZB), in 2020 this number grew to 40ZB, and according to <\/span>Cisco\u2019s Global Web Traffic Report<\/span><\/a>, this number is set to reach 174ZB by 2025. To put this into perspective, if you attempted to download 174ZB it would take over 400 million years with the average global broadband speeds currently available.\u00a0<\/span><\/p>\n

It is also predicted that there will <\/span>be 5.3 billion total internet users by 2023<\/span><\/a>, up from 3.9 billion in 2018. That\u2019s a staggering 66 percent of the population online, continuously adding data to multiple endpoints. On top of this, with the advancement of technology like Elon Musk’s Starlink, people in even the most remote locations will be connected, further increasing these figures. It is safe to say that the amount of data available on the internet is vast, unstructured and near immeasurable, and will increase in abundance in the foreseeable future.<\/span><\/p>\n

Currently the majority of the focus is on the surface web, which accounts for less than<\/span> 5% of the total information available on the internet<\/span><\/a>. This data is already indexed by standard search engines like Google, Yahoo! and Bing which is why finding and accessing this information is relatively easy. But what about the deep web? The deep web consists of all non-indexed unstructured data primarily found behind paywalls, logins and authentication forms. This data is extremely difficult to structure and analyse, and thus if a business had the ability to do just that, it would ultimately provide them with a competitive advantage.<\/span>[\/vc_column_text][\/vc_column][\/vc_row][vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”][image_with_animation image_url=”16347″ image_size=”full” animation_type=”entrance” animation=”Fade In” animation_easing=”default” animation_movement_type=”transform_y” hover_animation=”none” alignment=”center” img_link_large=”yes” border_radius=”none” box_shadow=”none” image_loading=”lazy-load” max_width=”50%” max_width_mobile=”default”][\/vc_column][\/vc_row][vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”][vc_column_text]There are platforms on the market that successfully access and ingest this information at scale, but few of them provide reliable insight extraction models that can be used for market intelligence and clinical decision-making.\u00a0<\/span><\/p>\n

So how do we access this unstructured, deep web information in order to empower our businesses to make better, faster decisions? First, let\u2019s explore the difference between structured and unstructured data.<\/span>[\/vc_column_text][\/vc_column][\/vc_row][vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”][vc_column_text]\n

Structured vs Unstructured Data<\/b><\/h3>\n[\/vc_column_text][vc_column_text]Not all data is created equal, and is usually classified as <\/span>structured or unstructured<\/span><\/a>. This classification has an impact on how data is collected, processed and presented.\u00a0<\/span><\/p>\n

Structured data is found in models that are pre-defined i.e. templates or spreadsheets, and is highly organised and easily analysed. An example of this would be point-of-sale information structured into rows of purchase transactions in an Excel spreadsheet.\u00a0<\/span><\/p>\n

On the other hand, unstructured data is usually text-heavy and quite difficult to analyse. Social media posts containing ideas and opinions are a good example of this. Although elements like hashtags do structure some of the content, the majority remains vastly unstructured. <\/span>Other examples of unstructured data include intranets, company wikis, email folders, network drives, academic journals, literature and archives to name a few.<\/span>[\/vc_column_text][\/vc_column][\/vc_row][vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”][image_with_animation image_url=”16348″ image_size=”full” animation_type=”entrance” animation=”Fade In” animation_easing=”default” animation_movement_type=”transform_y” hover_animation=”none” alignment=”center” img_link_large=”yes” border_radius=”none” box_shadow=”none” image_loading=”lazy-load” max_width=”50%” max_width_mobile=”default”][\/vc_column][\/vc_row][vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”][vc_column_text]Whilst there is merit to analysing structured data, it is readily available to most companies through an array of applications, therefore it is not going to give you the competitive edge. If you had the capability to extract insights from an untapped resource like unstructured data however, it would result in various benefits including, but not limited to, reduced time and effort spent on analysis, higher quality insights, driving innovation, and enabling better decision making and foresight.<\/span>[\/vc_column_text][vc_column_text]\n

Why Focus on Unstructured Data?<\/b><\/h3>\n[\/vc_column_text][vc_column_text]It is estimated that <\/span>80 to 90 percent of all data<\/span><\/a> collected by companies is unstructured, and is growing faster than structured data<\/span> at roughly <\/span>55% – 65% per year<\/span><\/a>.<\/span> But, it is also extremely complex to analyse. If it could be accurately analysed and at a rapid rate, it would provide businesses with a distinct competitive advantage. There are, however, limited platforms available in today\u2019s market that can readily access and analyse this type of data at scale.\u00a0<\/span><\/p>\n

So, why is unstructured data important? The simple answer is that intelligence is hidden in the millions of documents or presentations which are difficult to access and even harder to read and analyse in large volumes. Internal proprietary data in large organisations is often siloed and inaccessible, often leading to mass duplication as employees are unaware of what exists on their network. Picture a sales team\u2019s folder in a company employing 1,000 Sales Development Managers (SDRs). If 100 of those people leave next month, how do sales managers know what potential opportunities lie within the hundreds of documents those employees produced? Unless these documents are searchable and easily accessible at scale, the data will likely lie dormant for decades leading to missed opportunities.\u00a0<\/span><\/p>\n

Similarly, external public data like news sources publish enormous amounts of content. The Associated Press publishes <\/span>over 1,200 articles per day<\/span><\/a>, which is more content than an analyst could hope to read in a week from a single source. If we wanted to analyse 1,000 news sources, say, the time it would take for humans to find, read and analyse would grow to many\u00a0 lifetimes – clearly impossible. Thus the opportunity is there for organisations to leverage machine analysis to read and analyse hundreds of thousands of documents per day, in fact, a lifetime of human reading roughly every 20 minutes, to draw insights for optimal and effective decision making.<\/span><\/p>\n

Data Analysis Tools<\/b><\/h2>\n

In a <\/span>2019 Deloitte survey<\/span><\/a>, less than 37 percent of executives in the sample believe their companies are insights-driven and the remaining 63 percent are aware of this data but do not have the infrastructure to harness its potential. Below we discuss three machine-driven technologies that can be utilised by organisations to overcome this barrier.<\/span><\/p>\n

Natural Language Processing (NLP)<\/span><\/h3>\n

Natural Language Processing, or NLP for short, is a component of Artificial Intelligence that involves machine understanding and interpretation of human language in an insightful way. By utilising NLP, one is able to structure data and produce useful insights through the use of models such as translation, sentiment analysis, entity recognition, topic modelling and segmentation, and relationship analysis.\u00a0<\/span>[\/vc_column_text][image_with_animation image_url=”16349″ image_size=”full” animation_type=”entrance” animation=”Fade In” animation_easing=”default” animation_movement_type=”transform_y” hover_animation=”none” alignment=”center” img_link_large=”yes” border_radius=”none” box_shadow=”none” image_loading=”lazy-load” max_width=”50%” max_width_mobile=”default”][vc_column_text]The above figure depicts how AI and machine learning can be used to understand language at a sentence level. The machine identifies which entity is the subject (Ford) and which sectors and topics it relates to (automotive and carbon emissions respectively). It is also able to analyse the polarity of the connection between them, indicating the presence of\u00a0 positive or negative sentiment linked to the entity in question.\u00a0<\/span><\/p>\n

Further complexity exists when the machine needs to disambiguate words in order to understand the true meaning of a sentence. An example of this is being able to differentiate the distinct sense of the word \u201cbank\u201d:<\/span><\/p>\n