Name: Uncorking Analytics with Apache Kafka, Apache Flink, and Apache Pinot™ Masterclass
Uploaded: 2025-08-20T19:16:07.369Z
Duration: 39 min 8 s
Description: Uncorking Analytics with Apache Kafka, Apache Flink, and Apache Pinot™ Masterclass

Transcript for "Uncorking Analytics with Apache Kafka, Apache Flink, and Apache Pinot™ Masterclass": He well, can you just let us know, I mean, when are we good to start, or should we wait for a few more minutes? Good. Then just start sharing the screen. I'm hoping my screen is visible. So, that's great. The text record. So, welcome on. Again, this is just a quick, I would say, introduction on, deal time analytics or uncocking analytics with the KFP stat, like, primarily Apache Kafka, Apache Flink, and Apache Pinot. And along with me, I'll be I'll be taking the first thirty, thirty five minutes to walk you through what does this stack look like, what were the origins, and, a cool demo, hopefully. And then towards the end, we also have Michael from Shadow Traffic. So Shadow Traffic, of course, Michael will just do justice to what it does, but it is used, for synthetic data generation, which is what I've been using, in the demo today, as well as we have been using, at starting, for a couple of our use cases as well as for internal testing and benchmarking. Having said that, I'll just quickly begin with the, first, part of the agenda, which is talking about analytics and the KFP stack in between. So to start with, we'll just quickly go, slightly back in time. So this is back, around 2010 or 2011 time frame. LinkedIn was just getting started. And it was primarily getting used, as a, let's say, static website. Right? People used to submit their resumes and just come and browse for their, jobs, etcetera, whenever they were looking for a job, so on and so forth. However, what LinkedIn wanted to do was increase the engagement of the platform. Right? And to do that, they created bunch of new features, especially some of the widely used features is whose view to your profile. Right? So whenever somebody looks at your profile, you get a notification immediately as well as you are able to then slice and dice this data or view analytics on how many people have looked at your profile in last seven days, last, one month, so on and so forth, right, and across which region, etcetera. So you get your analytics of who's viewing your profile at your fingertips. And, also, there are certain other talent reports that LinkedIn insight talent insights has. And apart from that, any number that you see on, LinkedIn to direct, either the field algorithm, that is getting computed, which has certain, metrics and aggregations. So what happens in the back end for some of these numbers, is something of this nature. Right? An aggregation query is fired to give me a count of a particular number for a particular time range and a particular user and then group by certain dimensions that are associated. Right? This is a typical OLAP or analytical query that is being fired on multiple of these pages or multiple of these features, that you see on LinkedIn today. Right? But what is unique about this? Right? The unique about the unique feature about this is primarily related to three different dimensions. One is the total number of users. Probably LinkedIn has more than 1,000,000,000 users at this point. QPS here refers to queries per second. Right? So, primarily, the amount of query that each page generates, each user generates is close to 10,000 plus at any point of time. So anybody and everybody can open up one of these pages, and then that those numbers on those pages will quickly start popping up, immediately. And then the third is, latency freshness. Right? And latency and freshness, primarily. So the latency since these are customer facing dashboards, because you as an end user customer are looking at these metrics, the latency has to match with what, the users are used to from a transactional system. Even though these are analytical queries, the response time has to be matching with what you guys are used to with transactional system, which is order of less than hundred milliseconds, p 99, and then the data freshness aspect. You typically don't want to wait for an ETL pipeline to time to put a notification immediately as well as you want the number one twenty. Hey. Sorry. We seem to have lost Jayesh. We're gonna locate him. I unfortunately cannot present his section of the presentation because they're his slides, but we will find him and, be right back. Alright. We're gonna locate Jayash. In the meantime, we're gonna rearrange the agenda. I'm gonna talk a little bit about how you test the kind of things that, Jayash was talking about. So he's gonna walk you through a real use case, and I think he's gonna spend a little bit of time expounding upon all of the different dimensions of data that you're able to do, stream processing over and joins in Windows. And a critical problem that you're going to see as he starts presenting is how do you test any of this? And when he runs his demo, you know, how do you actually fabricate any of this on the spot to do it? It's it's very unlike, having a relational database where you populate your tables with a 100 rows and off you go. There's a time dimension with streaming that makes testing these things significantly harder. And this is the work that I do with Shadow Traffic. As Jayesh mentioned, I work with StarTree. Shadow Traffic powers today's demo. I'm gonna show you just a a little bit about what it's like, and then Jayesh is gonna take back over, and basically show you it all in action. So let me, let me share my screen real quick. Here we go. I was gonna run through a couple slides here. Nothing's terribly fancy. So, yeah. The the problem as I mentioned, if you're gonna work on these kinds of, systems that Jayash is gonna talk about, your architecture is probably gonna look something like this. It's gonna be probably a multi service architecture, at least in terms of having multiple Apache Flink nodes, if not Apache Kafka, and then the lake stack like StarTree on top of it. But you're gonna need some data to actually do end to end testing. It's not like you're just testing a function at a time or a service at a time. Ideally, you would be able to use production data to do all this. I I certainly advocate for that when it's possible. Many times you can't, and and even if you can, you really can't test future scenarios like what happens if we double the many users, or what happens if there are particular traffic spikes, or you can come up with all kinds of hypotheses that can change. I, advocate for a style of testing called simulation testing. I obviously didn't invent it. It is decades old. I just think it really hasn't received the attention that it should. The idea is that you take some process and you move it as far left to your architecture as possible, and you drive through what looks like real customer traffic to be able to, test your services end to end. And when it's done particularly well, you sort of have these knobs that allow you to change the nonfunctional aspects of your data. So volume, velocity, variety, that kind of thing. So, yeah, I I don't shadow traffic. It is a simulation testing product. It is just a Docker container. You run this on your own laptop. You can run this on Fargate, EKS, wherever Docker containers run. You boot it up. You give it a JSON configuration file that describes the shape and the semantics of your data, and then off it goes. It actually connects, directly to your back end of choice. For many folks that I work with, that's Apache Kafka because this is sort of where the problem is most pressing. It also works directly with s three, Postgres, Webhooks. You know, many major systems are supported. You name it. They probably support it at this point. And you start these containers. They connect directly. You don't have to write, as I said, any code. It's very easy to scale. You can add more of these things. And the reason that this is so helpful is the API allows you to model these complex scenarios that Jayesh is gonna get into. So these are kind of the main draws that people use shadow traffic for. I would say it's it's fluid and easy to model, you know, nested realistic data. Your data, I I I don't I've not seen your stuff. I can guess that it is more complex than foo and bar. It's probably nested. It probably has lots of type unions being two type unions and options and stuff like that. And that's just the real world in data, and you need a tool that makes it easy to do those kinds of things. Relationships, if your data is multidimensional, you have different streams, different tables, there are some identifiers that join between them. So let's say, customers and orders. If your data doesn't actually have that key representation that's gonna have, a set of orders for each customer, it's very, very hard to test. Same with sequential data. If you wanted to have something like, an abandoned shopping cart, the order that the events arrived actually matters. You have to actually go and look at an item before you put it in your cart, before you maybe, like, walk away and do nothing. And there there are many such sub problems when it comes to testing these streaming architectures. These are just a few, and I I find new ones all the time. They're they're sort of fun to solve. So, yeah. Shadow traffic is a product that makes all this easy. You can either write this a p API by hand, directly, which is once you will still do that. You can start from a schema. So you can start with a an Avro schema, JSON schema, or whatever. Shadow traffic can approximate what traffic looks like for that. And, of course, because we're we're in the age of AI, you can use, Cursor Copilot, Claude, any of those tools. I ship knowledge files that make your editor really smart without using shadow traffic, and virtually all my customers end up doing this. To show you what it looks like to just a glance, this is a very simple example just meant to fit on the slide, but you'll get the point. You would take a snippet of the data that you want to, generate. You would write it out in JSON, And then what you do is you incrementally remove the concrete value. So instead of John Doe here, we would replace that and put a little marker. It's a little function site that says, hey. Let's do a string here that looks like a full name. So instead of 30, we would replace this and do a statistical distribution. And, again, these are very simple examples just to fit on slide and kind of frame it in your mind. But the hard work that I've done and what makes the product valuable is I've created a large library of these functions that allows you to create relationships across datasets or state transitions to say a happens, then b happens, and c happens. There are an ops for parallelism, chaos operations, you name it. I add more things to to it all the time, and this is really what allows it to have that that composite behavior. It's been a lot of fun to build. As I mentioned, I work with StarTree. I work with many other brands that you probably heard of. This is definitely not a toy. It's a labor of love, and it's been a lot of fun building. So, yeah, check it out. Shout traffic I o. Drop me an email. I I'd love to chat with other people in the streaming ecosystem. Drop me a line. I'd love to hear from you. But, otherwise, I'm gonna stop sharing my screen, and, I guess we'll see if Jayesh is connected and see where we're at because I I can't quite see it. Jayesh, do we have you back on the line? I'll be right back. We'll go find out. And thanks thanks, Santiago. I should also ask while while I'm waiting to find j s. Yeah. Any questions that I can answer? I could answer other questions about shadow traffic. I am not an expert on StarTree, but I I worked at Confluent for five years. I worked on the Kafka ecosystem for ten years. I'm pretty fluid in streaming and stream processing. Any questions on your mind? More than happy to answer them. Great way to use the time. And, yes. Yeah. It can it can ShoutOut traffic can do XML. So people often ask, you know, what I showed you was JSON. Roger, you know, is a column. Hey. How's it going? I think we found JS. I'll, I'll finish answering this question, and I'll then I'll bow out. Yeah. It it is a JSON API. Oh, so yeah. I just see a couple of the way that serialization actually works. So you can generate parquet, XML, whatever, and it it gives you the ability to sort of decouple those two things. But, yeah. I think we found JS, so I'll I'll tag him back in. Thanks, everybody. Database. It is basically a columnar database designed for fast analytical queries. Again, not a transactional system, but a analytical system, primarily able to ingest data from variety of sources. So So even though I mentioned it was born back at LinkedIn to ingest data from Apache Kafka, it can actually ingest data from, batch based sources as well. So it is built on the Lambda architecture. You can ingest real time data and combine that with offline data or batch data and run fast analytical queries. The use cases for this are many. Primarily, the user facing analytics that you see or the partner apps that you see for some of the applications, are what Apache Pinot is very much, suitable for. So these are some of the use cases that Apache Pinot or actually, all three of the players from the stack, the KFP stack have been used across some of these examples. Right? So for example, I'll take an example of, Stripe here because we've already talked about, LinkedIn. So you can see Stripe has close to one petabyte of data. So at any point of time, any transaction which is going through Stripe, as a payment, gateway goes via the same flow. Right? An event is generated into a Apache Kafka stream, which then moves into Apache Flink for certain preprocessing and is then eventually ingested and stored into Apache, Apache Pinot. The purpose for that is merchant dashboards among other purposes. Right? So each merchant who is using Stripe gets a dashboard of real time view of what is their ARR looking like, what is their MRR looking like, how many invoices are pending, how many transactions are in process, how many transactions failed in last ten minutes, one minute, so on and so forth. Right? So this is real time analytics for some of these merchants. Again, the beautiful part is the number of merchants for Stripe is close to 1,000,000 or more. Right? At any point of time, all of them can simultaneously log in and start looking at their dashboards. And that is why you can see 20,000 queries that are getting fired onto the system. That is Apache Pinot. And then the data size is close to one petabyte. So any transaction that you do on Stripe is actually moving through all three layers of Apache Kafka, Apache Flink, and Apache Pinot. Again, at a high level, this is what the combined architecture would look like. I've just, notified Confluent and StarTree as being the sort of service providers for some of these, let's say, services. But what you would typically end up having is on the left hand side, you would have bunch of data sources, your transactional systems, your IoT devices, your mobile applications, which are actually generating actions or events in the real world and translating that into digital format, which will then move into a data streaming layer, which will then first subsequently move to a stream processing layer. And that is where, Confluent as a service provider provides a managed offering for the sync. And all of this data can be used or stored within, StarTree cloud and used for a variety of purposes. Right? Either for dashboarding internal use cases, but primarily for your, partner dashboards or for your, end user dashboards primarily. So I'll just keep give you an example of couple of reference architectures, and these are actual real world use cases. Right? So this is one of I cannot name the company here, but this is one of the top 10 video game companies. And this is a live system which is in production. So what they do is they collect data from their client consoles. So if you are using a system like, Sony x y z, as well as using, PlayStation controls, etcetera. So there are certain metrics that are extracted or certain events that are generated from these consoles, these, game devices that are continuously pushed as a metric extractor first into a Apache Kafka system. So you have Apache Kafka as a beginning point, and these, extractors then are moved into Apache Flink for further preprocessing. Right? They have certain business logic which gets applied again again on the consoles, primarily for hiding PII data in this case, which is then moved on to a subsequent Apache Kafka topic. In this case, what they are doing is they're pushing that, enriched, metrics that are coming from these, consoles, into into StarTree cloud and then visualizing these metrics using Grafana dashboards. The purpose for visualizing these metrics is to identify anomalies, specifically if there are any, launches of new games or if there are any peak event sales events within their games. How is the game performing? What is the jitter rate looking like? Are people able to, play the game? Is there lag in terms of the service of the, game on their consoles? So on and so forth. Right? These are certain examples of metrics that, this gaming company uses this for. So any and every event that is generated on one of these gaming consoles again moves through all these three layers, in a streaming fashion pushed onto a system like, Apache Pinot and then visualized in, Grafana in this case. The order of magnitude is pretty robust in this case as well. They ingest close to one to 2,000,000, events per second from, some of these, during some of these peak sale events and moving through all these three particular, systems in this case. Another example is Uber. So Uber has this, the use case of crash analytics. Right? Since Uber is very, very widely used across multiple devices as well as multiple operating systems, so on and so forth, and they keep patching their systems very regularly. So what they want to do is they want to understand what is known as crash analytics. Right? Basically, if they are pushing a new version of the app to a particular set of data users or to a particular set of devices, how is that new app performing, and are there any bugs associated with this particular app? Right? So you can imagine if there is even a small bug and, bug within the Uber app, and it is not opening up or faulty while booking a Uber ride for you. And one to 2% of the audience, it is a very heavy revenue loss for the company. Hence, they have built this crash analytics as an application. Basically, the app itself keeps sending certain telemetry information about itself, about the health of the app within a particular device, which then moves on to a a Apache Kafka layer, which is then further processed by Apache Flink. In this case, they do certain lookups. They enrich the information that is coming in from the telemetry, from the device itself in terms of when was this app published, what is the version, what are the associated, certain PR associated with this particular version of app, so on and so forth. Right? Certain metadata about the version of the app that is deployed on this device, and that is pushed, into Apache Kafka and then, again, pushed into primarily Apache Pinot for visualization. Right? And if if there if there is any bug detector or any issue detected, for example, on a particular Android Samsung Android device using, Android version, which is slightly older, what about the percentage of users that are affected, how it can be quickly patched, so on and so forth. Right? So this is primarily used for cache analytics. Again, this use case was actually served, using Elasticsearch, earlier, and they moved from Elasticsearch to Apache Pinot, and they saw close to $2,000,000 of saving just in terms of the infrastructure cost, across all these, three systems. So this is actually a use case live in production, and there is a good case study if you want to, follow it up further, in terms of detailed reading. Now I take a couple of minutes to talk about StarTree. So, StarTree is the company, of course, where I work, and it provides what we call as managed, offering. Right? So managed offering for Apache, Pino itself. And the good thing is, this is not just available in cloud. StarTree can be actually be deployed in your own cloud as well. So this is what we call as, BYOC or bring your own cloud, which can we can basically take your AWS account or your Azure GCP account and deploy the complete managed offering within your account. Is. So the data pane remains within your account, and the control pane remains with StarTree. So this is a complete managed offering. We, of course, have a dedicated SaaS as well, just like, the confluence of, like, Databricks of the world. We we can, of course, have a dedicated cluster for you. And then finally, what we are bringing something new is BYOK as well. Right? So this is basically bring your own Kubernetes. We we already have bring your own cloud, but you can literally take any existing Kubernetes cluster that you have, either on EKS, GKE, AKS, etcetera, and deploy the complete managed, version of this particular, Apache Pinot as a service on your application. It's a right. So this is in terms of the flexibility of how you can deploy, this allows you full control to go from a dedicated SaaS towards BYOC and BYOC. And then on top of, Apache Pinot, StarTree has built a bunch of, additional applications, if I may, just to make, life easier, just to make monitoring easier, and just to make some of the scale of these operations, much, much easier. Right? Typically, to begin with the local local way of, ingestion, we'll probably look at some bits of that in the demo. So to be able to quickly create a new schema, new table, and start ingesting this data from some of these systems. Other is, a real time anomaly detection. So since Apache Apache Pinot or the KFP stack is capable of ingesting this data in real time, there is huge power in terms of detecting any anomalies and alerting on these anomalies. Right? So for example, the case of Uber crash analytics, if you're able to detect that within ten to fifteen minutes of that event happening or that series of event happening, that is highly, valuable information for, some of the use cases that we are powering. Right? And so there's a third eye app application, which is a real time anomaly detection application. And then there are certain, other performance aspects as well, right, in terms of doing upsurge. So Apache Pinot does support upsurge at, real time rather than doing any sort of, back end merges or later merges. As well as what it does support is what we call as a count cloud tiered service. You can define the, let's say, life cycle of your data. So, like, if you were to store the data for one year, a one year worth of data, six months in hot tier, six months in six months in cold tier, but let you know the main standard offering or the standard query layer. You can go ahead and very well do that. It is very simple to just tweak a couple of knobs and adjust the life cycle of data at a table level. Right? So this is some of the other applications that are built on top of the open source Apache Pinot project to make, slightly more efficiency gains, slightly better, performance, but more importantly, a very high product developer productivity and all the gains from not having to manage, system. Having said that, I I think this is a quick time for a demo. So what I've done is for the demo setup is, actually, as as I mentioned earlier, I'm using shadow traffic to generate these synthetic streams of data. So, primarily, what I'm generating is three streams here. One is click stream data. So you can imagine somebody is using a mobile application and clicking on that mobile application or browsing, and that generates a stream of event. The other is the subsequent stream, which is information about the user, and other is the subsequent stream, which is the information about the product itself. Right? So what I'm currently doing is I'm pushing all of this via shadow traffic, dynamically into a raw click stream data as a Apache Kafka topic. And then I'm passing that data onto Apache Flink to in this case, handle two things. One is mask certain data for me. So I'm masking, I think, email address in this case, as well as I'm doing a lookup join. Right? Or very simple join of if this is the user and this is the user ID in primary, click stream data and the rest of the user information, like the address is, as part of the, another stream. How do I go about joining these two streams and, moving ahead? So I'm taking, using those two pieces of information in Apache Flink or doing those two processing in Apache Flink and pushing the enriched information back to, another Apache Kafka topic, which is the enriched click stream data. Now Apache Apache Pinot has direct connection to this, to Apache Kafka as we saw earlier. So we are just hooking up to the enriched Apache Kafka, enriched enriched click stream data in this case, and then visualizing all of that data into a streamlet application just to show how some of this, works out. Right? In this case, this consumer can be any, let's say, BI tool or a microservice or in, custom dashboarding, in, TypeScript or any other, let's say, visualization too. And I I've just stopped sharing the presentation. I'll move to some of the aspects and what you try to explain that I have, done in this case. So the first and foremost is the, shadow traffic configured. Of course, Michael will go deep into looks like, but just to give you a quick snippet of what it takes to generate some of this data, it is actually very simple in terms of defining the generators. So in this case, you can see I'm defining three different types of generators here. Right? One is the user info. The next is the product info. And then sorry. And then finally, it is the click stream, raw data, right, which is then which is what I'm then pushing, as, details into a system. Right? What is the platform? What is the mission ID? What is the device type? So and so forth. So this is what it is generating the data and actually pushing it to my local, Apache Kafka topic, in this case or local Apache Kafka cluster, in this case. The next subsequent step is I've defined these, tables into Apache Flink as well. So each one of the topics that is getting generated here so just for quick context, this is what my Apache Kafka setup ends up, looking like. So you have these three topics which are continuously generating the generating this data and pushing this data into these three Apache Kafka topics, which is click stream to our product info and user info. And then on the second step, what I have done is, basically, I have this link sequence. I've created this tables for each one of these raw Apache Kafka topics, first of all. Right? That that is the base I'm starting with. This is a schema that it looks like. So this is, the click stream raw data. The next is the product info. And similarly, the third is the user info to be able to define the schema and then associate that particular table from where is it getting the data. Right? In this case, which is, let's say user info or product info created and absurd table into Apache Kafka because, the user ID might get, repeated and so the information gets absurded and then attach it to the topic, which is user info. So as newer and newer records come into this particular topic, this Apache Flink table gets populated automatically. And then finally, I've created a click stream, enriched information, which where you just, street, mosque, email, mosque, as well as certain other user informations is not present in the raw table, itself. Right? The first name, last name, email address, one and so forth. And then this is the logic where I'm just doing a basic join here to be able to join the raw clickstream data to the product info and user info table to be able to merge those two streams together at run time using a temporal join and, again, push that back into a subsequent, Apache Kafka topic. So this is what the setup currently looks like. Just to go back to what it looks like from, Apache Kafka perspective and Sling perspective, this is a what that, looks like. So for example, this is the user info table. You can see this has user ID, phone number, etcetera, etcetera. And similarly, there are, other topics available. And clickstream and then just the final topic which gets the enriched data after joining the raw streams from clickstream raw and click, clickstream raw product info and user info. What I'll do is just in the quick, in the meantime when I go towards Apache Pinot, what I'll do is I'll just quickly start up, shadow traffic in the back end so that we can actually live generate this synthetic data and see this data transfer, happening from Apache Kafka to Apache Flink onto Apache Pinot, subsequently. Right? So this might take couple of minutes for it to actually start pumping in the data and go through all of these process because I'm running this on my local. In the meantime, I'll just quickly explain the final piece of the puzzle, which is Apache Pinot itself and what does the Apache Pinot setup, look like. So in Apache Pinot, what I've set up is primarily two set a single set of table, which is the click stream enriched. As I mentioned, it does not need any sort click stream, enriched table. This is what the schema of that particular, table ends up looking like. Right? Basically, the set of all of the plat, keys that we were seeing as well as the keys associated with, the user info and the product info that we want to go ahead and look up. Right? Just to I I won't create a live table right now, but if you wanted to create a live table and live schema into p node, it is as simple as passing this, JSON. So, JSON structures onto, APIs that are available for you to do. Right? So any point and click that I do on, Apache Pinot is actually possible to be able to done via API that is already exposed when the Apache Pinot cluster comes up. Right? So this is the bunch of APIs that comes up. So if I actually wanted to add a new schema, I can actually go ahead and actually, do it probably. So let's say we add a click stream enriched table. So this is what it would end up, looking like, I hope. Yep. So you can see the schema is successfully created. Alright. Getting indexed within this particular table for further creating. So I'll just go ahead and add this. So you can see there is an API for add the table, which needs a JSON payload similar to what I was quickly showing you. Let me just clean this up. Oh, I think the demo boards are not too happy with me. I'm just, sensing in errors. Yeah. The schema name was a slight mismatch, and I just corrected the schema name to the new schema that I had created. And, yeah, the table is, created. Right? Alright. So if I go back here, go back to so this is the Apache Pinot UI, by the way. So you can see there was an existing single table. If I just go back and quickly, refresh this, you would see another table that has come up, right, which is the click stream enrichment. And you can see there are already close to two fifty. Yeah. Already 25 k or two fifty k records that are coming in into the, system. Right? If I probably keep, doing a continuous refresh of this, you would see more and more data that is coming into, Apache Pinot automatically at run time because I'm pushing some of this data. Right? I'll just go ahead and probably do a simple point query. I hope I did not break my local system. Yeah. It seems like I might have broken my local system. So the server will automatically come back up. But the idea is this is the Apache Pinot table, and this is a Apache Pinot UI for you to be able to go ahead and query the system, as is or to be able to then attach any visualization layer to, or or any sort of this, any sort of visualization layer like streamlet, tab, etcetera, to some of this. Right? Just to give you a sense of what that visualization layer might end up looking like. So this is an example of that visualization, layer that is, that that I've created as a streamlit app, hooking onto one of the Apache Pinot clusters that I have on an existing setup. Right? You can end up sorry. Am I I hope I'm still audible and fairly clear. Yeah. So yeah. This is what the the simple UI would hey. Thanks. Thanks, Michael. Thanks for your permission. So, yeah, this is just a UI layer on top of a click stream data. Right? So click stream data is typically used for doing, either event ranking, basically, how many people are viewing your products, looking at your products, adding it to the cart, so on and so forth, or doing, further analysis in terms of what the product funnel looks like. Right? Where do you see drop in the funnels, so on and so forth? And all of this should be possible at real time because the user behavior can change at any point of time. You want to tweak your page flows. You want to take tweak your offers, so on and so forth. Just probably go back to Pinot in the hope that okay. Oops, I might have in shut down my cluster for some reason with the with trying to overload the ingestion rate. But, yeah, this is what the end to end, sort of flow would typically end up looking like. And just to go back to Apache Flink once, this is a running job that you see is continuously actually taking the data from an input Apache Kafka stream, massaging those data, and then pushing it onto a subsequent Apache Kafka topic. Right? So this is what I have as a Flink running on the system as well. With this, I I've mostly concluded what I wanted to share for, today. Just want to know if you want to keep in touch with, specifically the Apache Pinot community. If you have any questions, if you have any thoughts, you can actually go ahead and join some of these community. Again, Apache Pinot is an open source project, given the via the Apache Foundation. You'll be more than happy you should be able to more than happy to go ahead and download the images, play around with the jars, and figure out some of these use cases. Right? You'll see immediate benefit in some of the analytical queries you're running in terms of the latency that you're seeing or how easy it is to hook up to any one of your existing Apache Kafka, Kinesis, GCP pops up streams and start, moving around your data. So you can just probably take a quick scan or peek at some of these QR codes and try to join the Slack community and ask questions there if there is, anything. But I'll I'll stop sharing. If there are questions, I'm more than happy to answer them right away or, we can do that towards the end of the presentation as well. Make it like beef. Elliot, make it like beef. Elliot, make it like beef. You should be good to start with, your stuff. So you filled the time when you're not disconnected. I I did my thing. I was I was gonna re ready to move into some karaoke before he gets back. But, no. I didn't think I I think it may be a question I could kinda see till I get it going. What do you think is, like, the most challenging part of running these kinds of architectures in your experience? So they have it it can get complex, really fast. So it needs a certain level of a mind shift, like, especially if the shift is coming from a batch based system to a streaming system. Just the way you want to think about some of these systems is very different because they're always not thinking of a static dataset that, okay, x has happened. I'm moving x amount of data from a to b processing, x amount of data in Apache Flink, but this is a continuous stream that is coming in. So that takes a little bit of a mindset shift in terms of, okay, how do we go about designing this one. And the second part is identifying, okay, which part is suitable for what? Not everything. You don't want to jump up doing everything or no. Everything is possible in Apache Flink. I can write any sort of code. Any sort of query is possible in Apache Kafka. Why not write a 16 pager and 16 pager query and figure out how that, looks like? And similarly for Apache Kafka as well. Right? Not everything needs to be a thing. So, again, it may be more from a solutions architect lens, but you need a slight mindset shift as well as understanding where does the power of some of these, platforms lie. Yeah. That makes sense. There there's a real question, from, I think from from Robert. Hey. Hey, Robert. I said, can you show the Apache Flink job that's doing the enrichment? Yeah. Okay. I I'll I'll so I'll I'll actually I'll be I did not go towards that, but we'll we'll have we'll have a public GitHub repo for some of these, and we'll actually go ahead and share that complete GitHub repo for you. Just to be, straightforward to the point, the current thing job is creating those three set of tables, and then there is a simple temporal join at, at the end, which is joining the raw click stream, stream, raw click stream data to the product info and the user info data. But, I I I'm slightly overdue on my end to be able to share that public repo. I have that working on my end. I just have to sort of work. How do we make that public in terms of, our starting GitHub repo and, shared access to it? I hope, that was helpful. Coming to you soon, Robert. I know where to find you. We are on LinkedIn. Yeah. Any other questions for us? Thanks, everybody, for attending. I think that may be a wrap. Hey, any anything from you or do you think I'm good? Cool. Alright. Thanks so so much for referring to Nick. Good to see you. Hey. Thank you.