In this talk, we’ll look at software architecture, team organization and the interplay between technology and humans. We’ll address some of the patterns and anti-patterns that can be observed when organizations try to evolve towards more decentralization and autonomy, and take a look at some strategies to ensure technology supports and enable the desired outcome. We’ll also mention data mesh as it’s 2023 and we have to.

At first glance, ingress is an easy concept: you route traffic from the wider world into your cluster. As you layer on SSL and load balancing, the principles stay the same and everything works with minimal thought and effort. But as your infrastructure grows, your clusters grow, the interactions get more complex, and your security requirements explode. In this session, I’ll walk you through how we designed and built an Ingress Controller and have converted our clusters to use it in production to support millions of requests. It wasn’t easy but running it as an open source effort from the start encouraged our team and customers to review, explore, and consider situations outside our original plans.

How to efficiently build and manage hundreds of Kubernetes Clusters that serve modern online analytics databases, for different customers? To add to the challenge, what if customers need to run their own clusters inside their own private clouds? We are sharing our system design that solves it.

How to provide fully managed online analytics databases like Pinot to hundreds of customers, while those Pinot clusters are running in each customer’s own private virtual cloud? The answer is by combining the power of Kubernetes with our automated scalable architecture that can fully manage a fleet of Kubernetes clusters.

When companies consider using SaaS (Software as a Service) products, they are often held back by challenges like security considerations and storage compliance regulations. Those concerns often require that the data stays within the same virtual cloud owned by the company. And it makes managed solutions very hard for companies to implement.

In StarTree we have built a modern data infrastructure based on Kubernetes so companies can keep their data inside their own infrastructure, and at the same time get the benefits of using a fully managed Apache Pinot cluster deployed in the customer’s cloud environment.

We have designed a scalable system based on Kubernetes that enables remote creation, maintenance, and monitoring of hundreds of Kubernetes clusters from different companies. This enabled us to scale quickly from a handful of deployments to over 100+ Pinot clusters in a short time span with just 10+ engineers.

In this talk, we will explore the benefits of using the Rust programming language for building custom Kubernetes operators. Rust's focus on safety, performance, and reliability makes it an ideal language for developing robust, scalable, and efficient software solutions. We will delve into the unique features of Rust that make it an attractive choice for building Kubernetes operators, including its memory safety guarantees, zero-cost abstractions, and built-in concurrency support. We will also discuss some of the challenges of using Rust in the Kubernetes ecosystem, such as interfacing with other programming languages and managing dependencies. Finally, we will demonstrate how to build a simple Kubernetes operator using Rust and explore best practices for deploying and managing it in a production environment. Attendees will come away from this talk with a deeper understanding of the advantages and trade-offs of using Rust for Kubernetes development and practical tips for getting started with building their own Rust-based operators.

In this talk, I will be sharing about Ibis, a software package that provides a more Pythonic way of interacting with multiple database engines. In my own adventures living in Zimbabwe, I’ve always encountered ibises (the bird versions) perched on top of elephants. If you’ve never seen an elephant in real life I can confirm that they are huge, complex creatures. The image of a small bird sitting on top of a large elephant serves as a metaphor for how ibis (the package) provides a less complex, more performant way for Pythonistas to interact with multiple big data engines.

I'll use the metaphor of elephants and ibises to show how this package can make a data workflow more Pythonic. The Zen of Python lets us know that simple is better than complex. The bigger and more complex your data, the more of an argument there is to use Ibis. Raw SQL can be quite difficult to maintain when your queries are very complex. For Python programmers, Ibis offers a way to write SQL in Python that allows for unit-testing, composability, and abstraction over specific query engines (e.g.BigQuery)! You can carry out joins, filters, and other operations on your data in a familiar, Pandas-like syntax. Overall, using Ibis simplifies your workflows, makes you more productive, and keeps your code readable

How would you build a database to support sustained ingestion of several hundreds of thousands of rows per second while running near real-time queries on top?

In this session, I will go over some of the technical decisions and trade-offs we applied when building QuestDB, an open-source time-series database developed mainly in JAVA, and how we can achieve over a million row writes per second on a single instance without blocking or slowing down the reads. There will be code and demos, of course.

We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read replicas, or faster batch ingestion.

Operationalizing computer vision models remains a considerable challenge for companies. In this talk, you'll learn how we leverage Boston Dynamic's SPOT robot and a suite of computer vision models to perform industrial inspections, increasing accuracy and safety. I'll show you some of the challenges we face and our process to deploy these technologies.

Tech Leading is a hard job. You need to be an architect, carer, leader, servant, software engineer, business analyst, DevOps engineer, diplomat and a lot more all at the same time… What it means to be the tech lead of a team will be explored, and its joys and responsibilities. More importantly, we will navigate the complex balance of responsibilities and the wide variety of skills needed. Real-life experiences will be shared and lots of advice on becoming the best tech lead you can be. Whether you would like to become one, you currently are doing the job or you are leading tech leads this talk might be interesting for you.

Building a multi-cloud network requires complex VPN configurations and policies. How can we enable secure communication across Kubernetes clusters without these problems? By making Skupper we have been facing these challenges and in this discussion, I want to present what problems we solve and what architectural decisions we use. Finally, a demo to show an actual use case.

In this talk, Sergey will share his team’s experience of bringing Temporal Cloud to life. He will explain architecture of the service, with the primary focus on building its Control Plane. Sergey will cover approaches and patterns used in the process and lessons learned from successes and mistakes along the way. This should be of interest to engineers building or thinking of building multi-tenanted hosted services, which has become the most sustainable way of monetizing open source server products.

Redpanda is a Kafka®-compatible streaming data platform that not only handles user writes and reads, but also needs to maintain a tremendous number of background processes—including log compaction, object store uploads, data and leadership balancing. The challenge is that these background tasks compete with the main workload for finite resources, like CPU, I/O bandwidth and memory.
Some of these background tasks share a common property: a growing backlog.
Keeping the backlog size small with minimal influence on write/read path latency requires tuning the number of shares given to the background task.

This talk walks you through our challenges with scheduling background tasks in Redpanda, and brings you behind the scenes to learn how we leveraged the PID controller (an idea borrowed from industrial applications) to manage the amount of shares given for background tasks with a growing backlog.

Have you ever wondered why 87% of machine learning models never make it to production? Who must be held responsible if a machine learning algorithm discriminates or shows bias? Are the decisions taken by these models trustworthy? In this talk, let’s unravel the answers to such complex questions!

Machine learning has had a significant impact in many areas, including medicine, entertainment, security, and education, but its use can also result in increased cognitive dependence on technology and ethical concerns such as bias. Therefore, it is crucial to address these issues by reducing the impact of human biases and creating trustworthy, reliable, and understandable machine learning systems.

The key takeaways of my talk would likely include the importance of understanding and interpreting the decision-making processes of machine learning models, as well as the need to ensure that these models are fair, accountable, and trustworthy in their predictions and actions. Additionally, the talk may highlight the challenges of building interpretable models and the importance of evaluating and testing models for bias, as well as the need for transparency and accountability in the development and deployment of machine learning systems.

This talk introduces some of the history and evolution of Peer-to-Peer (P2P) systems in the last 20 years. Although the novelty of the concept has faded, it did still provide many contributions to the design of distributed systems that are still relevant nowadays. The talk address the connections to filesystem research from the 80s, the evolution of DHTs, and its influence on the NoSQL DBs and Blockchains.

In this talk, I will explore the fascinating world of AI image generation, a field of research that can produce realistic, creative, and diverse images from text, sketches, or other inputs. I will introduce the main concepts and techniques of AI image generation and showcase some of the best AI image generators. I will also share our own work and experience in developing AI image generators for Freepik, a leading platform for free graphic resources. I hope to inspire you with the amazing capabilities and possibilities of AI image generation, and invite you to join me on this exciting journey.

Data Mesh is a new socio-technical approach to data architecture, first described by Zhamak Dehghani and popularised through a guest blog post on Martin Fowler's site (https://martinfowler.com/articles/data-monolith-to-mesh.html). Since then, community interest has grown, due to Data Mesh's ability to explain and address the frustrations that many organisations are experiencing as they try to get value from their data. The 2022 publication of Zhamak's book on Data Mesh (https://www.oreilly.com/library/view/data-mesh/9781492092384/) further provoked conversation, as have the growing number of experience reports from companies that have put Data Mesh into practice. So what's all the fuss about? On the one hand, Data Mesh is a new approach in the field of Big Data.

On the other hand, Data Mesh is an application of the lessons we have learned from domain-driven design and microservices to a data context. This talk will explain how Data Mesh relates to current thinking in software architecture and the historical development of data architecture philosophies. They will outline what benefits Data Mesh brings, what trade-offs it comes with and when organisations should and should not consider adopting it.

Data Mesh is a new socio-technical approach to data architecture, first described by Zhamak Dehghani and popularised through a guest blog post on Martin Fowler's site (https://martinfowler.com/articles/data-monolith-to-mesh.html). Since then, community interest has grown, due to Data Mesh's ability to explain and address the frustrations that many organisations are experiencing as they try to get value from their data. The 2022 publication of Zhamak's book on Data Mesh (https://www.oreilly.com/library/view/data-mesh/9781492092384/) further provoked conversation, as have the growing number of experience reports from companies that have put Data Mesh into practice. So what's all the fuss about? On the one hand, Data Mesh is a new approach in the field of Big Data.

On the other hand, Data Mesh is an application of the lessons we have learned from domain-driven design and microservices to a data context. This talk will explain how Data Mesh relates to current thinking in software architecture and the historical development of data architecture philosophies. They will outline what benefits Data Mesh brings, what trade-offs it comes with and when organisations should and should not consider adopting it.

In this talk, Sammy Landstrom will tell the story of our journey evaluating several on the fly analytics engines, to selection and implementation of the Clickhouse engine. We will share our successes, and some challenges we’ve faced, and show some examples of what a hundred billion connectivity measurements per day looks like when measuring the state of global connectivity.

In the era of continuous evolution, software products need to keep up with the rapid pace of change. The products we design and develop must evolve, adapting to support business growth and align with the latest technological changes. In this way, we ensure they can be maintained and built upon, instead of simply becoming legacy.  

In this talk, Joaquin will explore how the principles of evolutionary architecture can help us visualise software product evolution, and how these visualisations can be used to prioritise and drive platform modernisation initiatives. 

How can we ensure responsible Gaming is always present and accessible in our industry? 

Data collection done by people is a wasteful act and could result in duplicated work by different people. Gathering info for tasks, or for the ability to maintain code or infrastructure - Documentation plays a crucial part in that.

In this talk, I’ll show you a structured way to write a technical doc, without being a technical writer - So everyone could do it to their best ability. I’ll explain why you should care about these docs, and how eventually it serves your best interests (Yes, more than 1). If you want to save your time and other people’s time - Writing documentation well could have a great impact on that.

We’ll quickly define a ‘model on production’. There is a myriad of definitions people use right now. A variance wholly justified. Because the exact way we do define it depends. Among other factors on the size of the company, the number of models, the properties of the data, and so on. While we’re at it we’ll also answer some pinning questions such as: Is it ever OK to duct-tape the model deployment process? What about using shortcuts and opting for some manual work? Pithy answers. We’ll pause to take on any questions. If there will be too many to fit, we’ll provide contact information and make sure to address all enquiries after the presentation. Now practice makes perfect. Or nearer perfection at least. So we’ll briefly present two cases of solutions solving the same problem for two clients, both implemented with a vast difference. We’ll explain why. Both solutions required optimising search engine for results that translate into higher revenue. Yet, the companies are on opposite sides of the scale. One a huge, mature retailer, another a much younger and smaller online-booking business. The cost must always be justifiable. So, using these cases we’ll succinctly show how to fit a solution to match the context of the organisation. How to utilise it well. Next, quickly going through some details of the models and infrastructures, we’ll explain the reasoning behind the critical decisions as well as highlight the pros and cons of both approaches.

GitOps and CD are two terms that you can throw around in a meeting and colleagues will be impressed. "They really must know their stuff", people quietly think. It is widely recognised what improvements and general advantages in the process they bring to a DevOps flow.

In this talk we will explore some of the more advanced implementation quirks, using ArgoCD as an example:

- Release versioning and multi-environment.

- No more Helm hooks?

- Orchestrating deployments: sync waves and waves.

- Another layer of GitOps: automatic app creation.

- Sync windows.

Become a speaker in this slot taking part on the Lightning talk challenge: https://www.jonthebeach.com/lightning-talks

The past and future evolution of software development as we know it.

This talk doesn't try to "prove" anything, except that I have a long history in this industry :) 

It's an argument-by-analogy (and therefore immediately suspect, yes?) about how we got to where we are (hint: evolution) and where we are going (hint: more evolution) writing software design in the large.  The implication is that the Cloud revolution is still just beginning. If that's so, how will we be building Cloud systems 10 years from now?
This talk takes a stab at answering that question while trying to be "concurrently" interesting and entertaining.