Big Data and predictive analytics in oil and gas

It takes over 50 % of the time of petroleum engineers and geoscientists to collect the data, according to the report by Brule. And many companies are looking to adopt big data solutions in oil and gas to make the process more efficient. 

In fact, North America has the lion’s share in global big data in the oil industry. The market is driven by high demand for solutions in the Gulf of Mexico and the US shale fields, which will help to make better decisions and increase performance. ConocoPhillips said that the sensors the company deployed in the Eagle Ford shale basin of South Texas helped reduce the drilling time by around 50%. The North Sea is expected to account for the second-largest share in the market. Digitalization of oilfields in the region contributed to a 40% reduction in operating cost in the North Sea, during the low oil price regime. 

As it is reported by Mehta, based on the results of a survey done by General Electric and Accenture in 2018,81% of the executives named big data one of the top three priorities of oil and gas companies. And it is driven mainly by their need to improve the efficiency of oil and gas exploration. 

So how can you benefit from big data in oil and gas? What are the challenges in big data in the oil and gas industry? And how to solve them? Let’s find out.

How can you benefit from big data in the oil and gas industry?

Manage seismic data 

Drilling a deepwater oil well often costs over 100 million $. And it comes as no surprise that you need to be very precise when looking for the location.  For instance, to avoid any risks and save time and money, Shell uses fiber optic cables (created in a special partnership with Hewlett-Packard for these sensors), and the data is then transferred to its private servers, maintained by Amazon Web Services (AWS). 

Upstream analytics begins with the collection of seismic data with sensors across a potential area of interest looking for petroleum sources. Then the data is aggregated, cleaned, processed, and analyzed to choose the best location for drilling. Seismic data can further be combined with other data sets (a company’s historical data on former drilling operations, research data, etc.) to identify the amount of oil in reservoirs. In research done by Joshi et al., Big Data solutions were used to analyze the micro-seismic data sets to model the fracture propagation maps during hydraulic fracturing. The authors of the research utilized the Hadoop platform to manage and process extremely large volumes of data generated by micro-seismic tools. The success ratio was improved by detecting the potential anomalies based on the previously failed tasks.

Optimize drilling process

Current offshore drilling platforms have about 80,000 sensors, which are expected to generate 15 petabytes of data during the lifetime of the platform. The platforms are equipped with sensors that collect massive volumes of data. The analytics of the data allows ensuring that machines are working properly and are not damaged due to breakdowns or failures. It also allows for predictive maintenance of and timely replacing of the equipment, as well as reducing downtime and the number of non-fault-funds. As a result, the big data in oil allows saving time, money, and effort.

Improve reservoir engineering

Big data solutions help to collect and process data that oil and gas companies need to make reservoir production more effective. The data is collected using a number of downhole sensors (temperature sensors, acoustic sensors, pressure sensors, etc.). For example, with big data analytics, companies can develop reservoir management applications to get timely and actionable information about changes in reservoir pressure, temperature, flow, and acoustics. That allows getting more control over their operations and increasing reservoir profitability.

Improve logistics

The major problem that concerns logistics in the oil and gas industry is transporting petroleum while reducing risks as much as possible. To ensure that gas and oil are transported safely, companies use sensors and predictive maintenance. It helps to detect any faults in pipelines and tankers (fatigue cracks, stress corrosion, seismic ground movements, etc.,). That allows ensuring safe logistics of the petroleum products. 

The benefits from big data in oil and gas are significant. However, Only 36% of the oil and gas companies have invested in big data and analytics. And only 13% of them use the insights from technology as enhanced business intelligence solutions. This underlines the fact that many companies have not embedded big data and analytics completely in their systems, but are just applying a part of the technology. 

The key challenges of implementing big data in oil and gas:

1. One of the critical challenges in digital oilfields is the data transfer from the field to data processing facilities based on the type of data, amount of data, and data protocols.
2. The other issue is the frequency of data collecting and the quality of the collected data.
3. Another important challenge is the thorough understanding of the physics of the problem. Expert petroleum engineers should cooperate with data scientists to use the right Big Data tools and find solutions to the various problems in the field of petroleum engineering
4. The experts need to specialize in open-source models, cloud technologies, pervasive computing, and iterative development methodologies. For instance, Shell has about 70 people working full-time in the data analysis department along with hundreds more spread over the world participating on an ad hoc basis.

How to choose Big Data specialists for your project:  

1. Choose the location with a vast talent pool
2. Settle on a Big Data development vendor with solid expertise in Big Data engineering, Data Science, ML, BI, Cloud, DevOps, and Security.
3. Hire Big Data developers with expertise in:

• Hadoop ecosystem and Apache Spark. They allow large data storing and processing by distributing the computation on several nodes.
• Such cloud-based tools as Snowflake, EMR, Dataprots, Cloud Composer, BigQuery, Synapse Analytics, DataFactory, DataBricks
• SQL/NoSQL databases.
• Such big data tools as RedShift, Hive, Athena for querying data.
• Maintaining old MapReduce Java code and rewriting it using a more recent Spark technology.
• Scala, Python, and Java.
• Kubernetes constructs that are used to build Big Data CI/CD pipelines.
• Kafka, AWS Kinesis or Apache Pulsar for real-time big data streaming.

What you need for effective big data in oil and gas solution:

1. You need to accurately define the business problem and define KPI’s
2. Combine big data methods with physics-based data analysis, using an interdisciplinary team of computer scientists and petroleum engineers 
3. You need to have the data appropriately structured and cleaned up. Only then, you can further transform the data into insights. ETL (extracting, transforming, and loading) and further cleaning of the data account for around 80% of any big data analytics time.
4. Deliver the results as a user-friendly interface.

How N-iX can help:

• N-iX has an internal talent pool of 1,100+ specialists and a team of 80+ data analytics experts. 
• N-iX has been recognized by ISG as a Rising Star in data engineering services for the UK market and positioned in the Product Challengers Quadrant both in the data science and data Infrastructure & cloud integration services. 
• Our professionals have strong expertise in the most relevant tech stack for implementing big data, business intelligence, data science, AI/machine learning solutions.
• We have established long term partnerships and developed big data solutions for such global companies as Gogo, Lebara, and  Fortune 500 companies.
• Our big data experts can help you with big data architecture design, cloud-based big data solutions, data science software, machine learning algorithms, as well as customized data science applications and reports.
• The company complies with international regulations, including ISO 27001:2013, PCI DSS, ISO 9001:2015, GDPR, and HIPAA, so your data will always be safe.