Our experience through Data Engineering Summit 2023 in Bengaluru

AI is everywhere, but there’s less attention to the engine running behind it. Data Engineering is what makes the right data available at the right time for all these AI use cases. Happy that someone thought about it and organized this 2-day event DES2023, focused on Data Engineering in Bengaluru, where we got to hear from industry experts and peers about ongoing and evolutionary concepts in this space. 

Being a technology-focused company, TO THE NEW has always made its presence felt in such conferences around the world. This time, it was the Data Engineering Summit in Bengaluru, India on 27-28 April 2023 where I attended the event with fellow Data Engineers Nitesh Saxena and Kedhar Natekar. Finding and reaching the venue was relatively easy. 

The registration process was quick, and the conference started at around 10 AM in the morning with the first talk being on ‘Data Engineering in the new age – AI is here, where is your data?’ followed by multiple other sessions spread through 2 days. 

We attended the majority of the sessions. The summit featured 35+ sessions, 50+ speakers from the Data Engineering community, and 10+ exhibitors.

The talks were a good mix of concepts, visions, case studies, best practices, and panel discussions. The list of all the talks can be found here on the official page of the event https://des.analyticsindiamag.com/agenda/

We at TO THE NEW always like attending conferences as we get to meet new people and learn a lot from them. During the event, we were able to network with a diverse group of individuals who are pioneers in this field. They shared various use cases that they had implemented in their projects, providing us with valuable insights. Additionally, we received informative mentoring sessions from industry thought leaders, which gave us valuable guidance in the form of one-on-one sessions.

Out of all the great talks, below are the ones that we liked the most:

Day 1:

1. ETL as ‘T’ Loses Lustre, where a viewpoint was presented that transformations in ETL are fading away and instead most of them are happening at source and target with very few being part of the pipeline.
2. Data eats Mar Tech for breakfast!, which was an interesting session on how data engineering transforms marketing strategies.
3. Panel discussions around ‘The Future of Data Engineering: Trends and Predictions for the Next 5 Years’ – It was great to hear Industry stalwarts sharing their vision on the future of Data Engineering.
4. The rise of the Data Mesh Architecture: A Paradigm Shift in Data Engineering – An interesting session highlighting how data platform architectures have evolved over time and why few organizations are solving their data problems using Data Mesh architecture.
5. Building Real-Time Data Processing Systems with Apache Kafka – A case study explaining how Delhivery has used a real time data processing approach to solve their logistics use cases. 

Day 2:

1. Data Fabric or Data Mesh: Which approach best suits building a future-proofed data platform? – Another session that solidified our knowledge of these evolutionary data architectures with real-world examples.
2. Building Data Lakes and Data Warehouses: Challenges and Best Practices – A session emphasizing the applicability of Data Warehouses vs Data Lakes depending on the problem in hand.
3. Ops is the New Oil – An interesting take on how DataOps is actually bringing all data together and solving organizational problems in an automated way.
4. DataOps enabling data management – Another session emphasizing the importance of DataOps and Data Management which are sometimes forgotten by Data Engineers at the start of a project.
5. From Batch to Real-Time: Building Streaming Data Pipelines – A session walking everyone through the journey from batch to real time and when it should actually be considered.

There were some interesting quotes shared around Data Engineering by speakers, one of which we particularly liked ‘If Data is the new oil, Data Engineering is the refinery that processes it’.

Apart from technical sessions, on Day 1 we had the privilege to hear from Padma Shree awardee legendary Indian Shuttler Prakash Padukone, where he shared his perspective on how technology is changing the sports industry, the life lessons that helped him to improve consistently and how those lessons will help every IT professional.

The long two-day summit ended with a stand-up comedy performance by Abijit Ganguly, which was a perfect way to end the summit on a light and positive note.

We thoroughly enjoyed these two days of intense learning and interactions with fellow community members.

In conclusion, I’d like to leave you with this image that symbolizes the promising future of data engineering.

The current state of data engineering can accurately be depicted as a field that is rapidly advancing along with the machine learning, artificial intelligence, and data analytics technologies. As the future has arrived in the realm of data engineering, professionals in this field are constantly innovating and using new tools to assist organizations in making better decisions. With limitless potential, the future of data engineering will undoubtedly continue to transform businesses and industries in unprecedented ways.

The future truly is here, and we at To The New are committed to contributing in this space.

Finally, Thanks to Analytics India Magazine for organizing this wonderful event. Looking forward to attending and presenting in more such events in future.