DCAD helper - Databricks Certified Associate Developer for Apache Spark 3.0

© Provided by The Times of Israel Databricks Israel team. (Ariel Zandberg)

US software vendor giant Databricks opened its first office in Israel to grow its business in the country and tap into local talent.

Never miss important Israel stories - get the free Times of Israel Daily Edition

The Generation AI-themed event will feature keynotes from co-founders Ali Ghodsi, Matei Zaharia, Patrick Wendell and Reynold Xin, highlighting advancements to the Databricks Lakehouse Platform, the importance of AI governance, and the future of LLMs

Tens of thousands of attendees will hear from live virtual guest Satya Nadella (Chairman and CEO at Microsoft) and special guest speakers Eric Schmidt (Former CEO of Google) and Lin Qiao (Former Head of PyTorch; Co-founder and CEO, Fireworks)

SAN FRANCISCO, June 2, 2023 /PRNewswire/ -- Databricks, the data and AI company, revealed the full agenda and lineup of featured speakers for the upcoming Data + AI Summit, a global event for the data community. On June 26-29, tens of thousands of data leaders, open source enthusiasts, and Databricks customers and partners will come together in person in San Francisco and virtually from around the world to learn about large language models (LLMs), the lakehouse paradigm, and Databricks' latest product innovations and contributions to the open source community. Tickets are still available to join the data community in San Francisco; register here to attend the live event.

This year's Generation AI theme highlights the inflection point reached with the rise in the popularity of LLMs. The AI revolution will be a catalyst for how every company reimagines its data, product, and corporate strategies. Attendees can expect to hear from top experts, researchers, and open source contributors as they share actionable best practices and compelling insights about their data journey. Highlights include thought-provoking sessions from data leaders from pioneering companies like Adobe, American Airlines, Grammarly, JetBlue, JP Morgan Chase and Co., Michelin, Nike, Northwestern Mutual, Nvidia, VISA, the Texas Rangers, T-Mobile, and more.

Data + AI Summit will feature keynotes from Databricks co-founders — and co-creators of Apache Spark™, Delta Lake, and MLflow — Ali Ghodsi, Matei Zaharia, Patrick Wendell, and Reynold Xin. Attendees will also hear from a broad lineup of data and AI luminaries, open source pioneers, and global thought leaders, including:

• Satya Nadella, Chairman and CEO, Microsoft (live virtual guest)
• Eric Schmidt, Former CEO and Chairman of Google; Co-Founder, Schmidt Futures
• Hannes Mühleisen, Creator of DuckDB
• Harrison Chase, Creator of LangChain
• Daniela Rus, Professor of EECS and Director of the Computer Science and AI Laboratory at MIT (CSAIL)
• Dawn Song, Professor, Department of Electrical Engineering and Computer Science, UC Berkeley
• Larry Feinsmith, Head of Global Tech Strategy, Innovation, and Partnerships, JP Morgan Chase & Co.
• Wassym Bensaid, Sr. Vice President, Software Development, Rivian
• Lin Qiao, Former Head of PyTorch; Co-founder and CEO, Fireworks
• Nat Friedman, Creator of Copilot; Former CEO, GitHub

The annual event will feature compelling technical training sessions, open source community meetups, networking opportunities, and industry-specific breakout events, including the following highlights:

• 250+ breakout sessions highlighting LLMs, data sharing, data governance, and industry trends. Speakers will cover the latest innovations in Delta Sharing, Databricks Marketplace, Unity Catalog, and more in keynotes, lightning talks, and hands-on training. Attendees will also receive technical deep dives on leading open source projects and technologies like Apache Spark™, Delta Lake, MLflow, Dolly, PyTorch, dbt, Presto/Trino, and DuckDB.
• Industry-specific content tracks that dive into the power of data and AI within the Financial Services, Healthcare and Life Sciences, Media and Entertainment, Public Sector, Manufacturing, and Retail sectors. These breakout forums will highlight industry-specific data and AI use cases, customer panels, interactive demos, and opportunities to connect with peers and partners in the Industry Lounge.
• Sessions on how to build LLMs explain how Databricks makes it easy for you to develop and deploy custom LLMs. Attendees will also have the opportunity to learn more about Dolly, the first open source, instruction-following LLM, fine-tuned on a human-generated instruction data set licensed for research and commercial use.
• Training and certifications with lectures, hands-on, instructor-led courses, and certification options. Courses will cover everything in the Databricks Lakehouse Platform, from advanced data engineering to performance tuning on Apache Spark™ to scalable machine learning — all hosted by industry-leading technical experts.
• Networking events for attendees to interact and collaborate with other data pros. Events will include a Developer Lounge, Meetups, Birds of Feather meals, LLM Hackathon, partner networking events, and evening receptions. 
• Women in Data + AI celebration honoring women's remarkable contributions and achievements in this field. Gain valuable insights into the dynamic world of data and AI as our esteemed panelists share their personal stories, illuminating their paths to success and pivotal roles in shaping the industry.

Data + AI Summit will showcase over 100 sponsors and partners, including leading companies from across data and AI, such as Alation, Anomalo, Ascend.io, AWS, Capgemini, Collibra, Dataiku, Deloitte, dbt Labs, EY, Fivetran, Immuta, Infosys, Labelbox, LeapLogic, Matillion, Microsoft, Privacera, Prophecy, Qlik, Sisense, Snowplow, Tecton, ThoughtSpot, and more.

Check out the Databricks blog and the event's full agenda to learn more. Register here.

About Databricks
Databricks is a data and AI company. More than 9,000 organizations worldwide — including Comcast, Condé Nast, and over 50% of the Fortune 500 — rely on the Databricks Lakehouse Platform to unify their data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe. Founded by the original creators of Delta Lake, Apache Spark™, and MLflow, Databricks is on a mission to help data teams solve the world's toughest problems. To learn more, follow Databricks on Twitter, LinkedIn, and Facebook.

Contact: Press@databricks.com

Databricks, a vendor known for setting up data lakehouses for enterprises, today announced a partnership with database virtualization player Datometry to facilitate easy transitions from legacy data warehouses.

The company said the integration will supply teams a simple way to migrate data warehouse workloads to Databricks’ lakehouse architecture without worrying about usually pressing aspects like cost or time.

The move marks another effort from Databricks to lure more customers to its data platform and better take on competition such as data cloud platform Snowflake.

Datometry’s 4x faster migration to data lakehouse

Moving data and applications to the cloud from an on-premises setup is no easy task. Companies have to hire system integrators to rewrite the embedded SQL and configuration and make the whole thing work on the new platform. This not only takes a lot of time and capital but is also prone to error.

Founded in 2013 and backed by $34 million in venture capital, San Francisco-based Datometry bridges this gap by providing enterprises with an SaaS platform that lets data and applications written for an on-prem data warehouse run natively in the cloud. The solution continuously intercepts the workloads’ communication with the original database and translates and redirects it to the new cloud platform. It delivers everything as is, including SQL statements as well as features like stored procedures, macros and recursive queries in real time.

With this tie-in with Databricks, Datometry has joined the Ali Ghodsi-led company’s technology partner program. The move will see Datometry provide its platform as a validated integration for the Databricks lakehouse, allowing enterprises to quickly connect and pull in their data and applications from legacy on-prem platforms.

The company says it can deliver migrations four times faster and at just 20% of the cost of other approaches.

“We’re proud to be partnering with Databricks,” Mike Waas, CEO of Datometry, said. “This partnership will enable organizations to break free from the vendor lock-in of legacy databases and adopt a lakehouse architecture four times faster than with any other approach.”

Partner program drives visibility

With its technology partner program, Databricks provides relevant third-party solutions to its customers, allowing them to work seamlessly in their lakehouses. Meanwhile, these third-party solutions get a newer set of customers to target and work with.

However, in this case, it is not just Datometry getting new customers. The integration for migrating data and apps will accelerate potential customers’ journeys to Databricks’ lakehouse services. Additionally, when customers can quickly bring workloads into the lakehouse and put them to maximum use, Datometry’s revenue, which operates on a pay-as-you-go basis, will also grow.

A similar tactic has been adopted by Databricks’ competitor Snowflake. In January, it signed an agreement with Mobilize Net to acquire SnowConvert, a suite of tools that uses sophisticated automation techniques with built-analysis and matching to re-create functionally equivalent code for tables, views, stored procedures, macros and BTEQ files in the Snowflake data cloud.

New York-based Catalyst, a company mobilizing customer data for enterprise growth, today announced it has received strategic funding from Databricks Ventures, the venture capital arm of Databricks.

While the amount infused remains undisclosed, the move marks Databricks’ first investment in the growing customer intelligence category. Prior to this, the Ali Ghodsi-led data and AI company had primarily backed prominent data stack players such as dbt Labs, Matilion, and Alation. 

Catalyst said the investment would deepen the integration between its offering and Databricks’ lakehouse, enabling a better user experience for their joint customers. 

Catalyst offers customer intelligence for retention, upsell

Founded in 2016, Catalyst is an SaaS platform that aggregates customer data from multiple sources into one intuitive view and provides sales and success teams detailed insights into customer maturity, health and upsell potential.

“We help enterprises organize all of their customer data from CRMs like Salesforce, customer usage data from platforms like Databricks, Redshift and BigQuery, and any other user data (like support tickets, emails) that may live inside tools like Mixpanel, Zendesk, Jira and Gmail,” Edward Chiu, Catalyst’s CEO, told VentureBeat.

Once this data is organized, Catalyst performs analytics, powered by Databricks’ lakehouse and AI engine, to identify which customers are ready for upsell/expansion and which ones are at risk of going away. It also pairs the insights with automation capabilities to automatically take necessary actions — like sending targeted emails — for each customer at the right time.

With this investment, Catalyst is expanding its engagement with Databricks, enabling joint customers to directly integrate the data they have in their Databricks lakehouse. The company says this will simplify the user experience and enable customers to get more value from their existing investments in Catalyst and Databricks. 

As part of this, Chiu said, Catalyst and Databricks also plan to launch a product feature where AI-driven predictive intelligence will provide signals when a customer is ready to spend more money. The feature will be called Expansion Signal. 

Competitors

While companies like Gainsight and Totango operate in the same space as Catalyst, Chiu claims they are legacy solutions built on outdated architecture and not modern in their data ingestion workflows. He said Catalyst’s integration with Databricks can onboard/implement customers within weeks, as against more than six months taken by other solutions. 

This results in faster ROI, which is critical in the current economic scenario where companies are struggling to get new customers and looking to extract more revenue from existing ones. 

Notably, the CEO added that Catalyst is the only platform that proactively tells enterprises which of their playbooks are actually generating positive results in customer adoption or increased spending. 

Prior to this investment, the company had raised a total of $65 million and its last public valuation was $245 million.

The era of AI everything continues to excite. But unlike the internet era, where any company announcing a dot-com anything immediately rose in value, the AI gods appear to be more selective.

Nvidia Corp. this week beat its top-line whisper number by more than $300 million and the company’s value is rapidly approaching $1 trillion. Marvell Technology Inc. narrowly beat expectations this week but cited future bandwidth demand driven by artificial intelligence and the stock was up more than 20% on Friday. Broadcom Inc. was up nearly 10% on sympathy with the realization that connect-centricity beyond the central processing unit is what the company does really well. Meanwhile, other players such as Snowflake Inc., which also narrowly beat earnings Wednesday and touted AI as a future tailwind, got hammered as customers dial down cloud consumption.

In this Breaking Analysis, we look at the infrastructure of AI examining the action at the silicon layer specifically around Nvidia’s momentum. Since much of AI is about data, we’ll also look at the spending data on two top data platforms, Snowflake and Databricks Inc., to see what the survey data says and examine the future of real-time data and automation as a catalyst for massive productivity growth in the economy.

To do so, we have a special Breaking Analysis panel with John Furrier and David Floyer.

How Nvidia plans to own the data center with AI

Two years ago we published this research report, laying out our thesis as to how Nvidia will disrupt the $1 trillion x86 installed base.

Basically it was a roadmap of Nvidia’s plan to take a massive chunk out of Intel Corp.’s general-purpose data center dominance. We had a positive outlook on the Nvidia’s prospects specifically thanks to its software expertise and end-to-end capabilities. We noted not just the GPUs, but the tens of thousands of other components, the networking, the intelligent network interface cards and the full stack that Nvidia was building.

Here’s an excerpt from that report:

Nvidia wants to completely transform enterprise computing by making datacenters run 10X faster at 1/10th the cost. Nvidia’s CEO, Jensen Huang, is crafting a strategy to re-architect today’s on-prem data centers, public clouds and edge computing installations with a vision that leverages the company’s strong position in AI architectures. The keys to this end-to-end strategy include a clarity of vision, massive chip design skills, new Arm-based architectures that integrate memory, processors, I/O and networking; and a compelling software consumption model.

Nvidia’s results called the ‘greatest beat in the history of earnings reports’

Nvidia’s accurate results are evidence that vision appears to be coming to fruition.

John Furrier called ChatGPT the “Web browser moment.” Jensen Huang calls it the iPhone moment. Either way, Nvidia blew away its numbers with a $670 million revenue beat and cited its second-half supply is going to be significantly better… and laid out a forceful and compelling narrative that budgets are shifting away from x86 to what the company calls accelerated computing.

Nvidia’s valuation is nearly nine times that of Intel’s, and ChatGPT has been a massive catalyst for Nvidia. Here’s a summary of the conversation that followed on our panel.

AI infrastructure, semiconductors and data: The rise of parallel computing and cloud-optimized GPUs

In the conversation, Floyer and Furrier talked about major shifts occurring in the tech industry landscape. AI infrastructure, semiconductors and data are the crux of these transformations, driven significantly by the adoption of parallel computing and cloud-optimized GPUs. The following three key points emerged:

• Parallel computing has become pivotal, as demand for CPU cycles has spiked dramatically. This shift has led to an increase in simpler, more efficient processor technology, propelling companies such as Nvidia to the forefront with its GPU-led architecture.
• Tech giants such as Tesla Inc. and Apple Inc. have also aggressively invested in parallel computing, with a focus on neural networks. These companies are fundamentally re-architecting their hardware to accommodate the surge in CPU demand.
• Intel, once a the dominant player in the CPU world, has failed to keep pace with this paradigm shift. Floyer stated that the company’s future as a leader is in jeopardy.

In addition, there’s more than parallel computing at play. Other facets of the semiconductor industry are also undergoing significant changes:

• Furrier noted that Nvidia was well-placed from the inception with GPUs and capitalized on the initial crypto craze. The AI trend has been a tailwind and has now turned its focus to AI. With the crypto market cooling, the attention has shifted to cloud-optimized GPUs and AI, which Furrier believes is the next-gen hyperscale technology.
• There’s a looming competitive battle on the horizon. While Intel may retain its dominance in server technologies through its Xeon line and traditional original equipment manufacturers, the emerging markets are heading toward chip-level connectivity and cloud-optimized silicon.
• AI’s role is not just limited to chatbots such as ChatGPT. The physical layer, often overshadowed, is believed to be the next major wave. It’s akin to the open systems interconnection or OSI model, where the physical layer is addressed first, followed by other layers.
• Nvidia’s CEO believes the data center’s future lies in becoming an “AI factory,” marking a drastic shift in spending toward AI-powered or accelerated computing. While this statement is self-serving, it’s a powerful marketing metaphor that strategically positions Nvidia to take advantage of this shift.

Bottom Line: The combination of AI infrastructure, semiconductors, and data will drive the next wave of technological advancements. Companies that can successfully ride this wave will likely shape the future of the industry. The battle among industry players is set to intensify, making this a crucial space to track.

Watch three top analysts discuss the future of AI and the shift to the AI-powered data center.

How an AI alpha engineer summarizes the GPU shortage

In a candid conversation with a community member of theCUBE’s, this deep AI expert shared the following:

Lots of competition coming after Nvidia’s dominant position

So with that as a backdrop, let’s look at some of the silicon competition to Nvidia and other firms possibly getting a boost from AI. Nvidia is disrupting Intel, that’s clear, as is Arm Ltd. Advanced Micro Devices Inc. is competing head on with both companies and has done an amazing job of bring AMD back to prominence. All the cloud players are developing silicon, as is IBM Corp. Broadcom is competing for share in merchant silicon and is focused on the surrounding components including intelligent NICs, along with Marvell in connectivity.

And several other players are building semiconductor capabilities, including Apple, Tesla and Meta Platforms Inc. And finally Chinese companies are designing and manufacturing silicon chips in an effort to achieve independence. So Nvidia is far from alone in this market, but it has a big lead.

Here’s a summary of the analyst conversation on Nvidia’s success in the AI space, the importance of neural networks, the role of hyperscalers and geopolitical concerns.

First, the panel discussed Nvidia’s lead in the AI business, primarily thanks to its GPU technology and innovative CUDA software. They believe Nvidia will continue to innovate by adding more neural networks to its repertoire. Both Apple and Tesla were noted for their heavy investments in neural networks, with the former dominating consumer computing and the latter focusing on inference work for its autonomous vehicles. The conversation led to the broader picture of AI, which they see as a driving force toward automation.

Next, the hyperscalers were brought into the mix, with Amazon Web Services Inc., Google LLC, Microsoft Corp., and Alibaba Group Holding Ltd. all developing their own AI products and chips. China’s looming influence in this market is also noted. Amazon, with its deep experience in silicon and a long history with AI, was highlighted as a potential leader. AWS’ approach to generative AI and aggressive messaging were seen as pivotal to its positioning.

Third, the conversation turned to the example of AWS’ acquisition of Annapurna Labs. AWS wasn’t satisfied with Intel’s performance or price, so it began partnering with Annapurna, and ultimately bought the company. AWS then used Annapurna to design Arm-based chips in-house. The panel speculated if AWS could follow a similar path to compete with Nvidia, potentially by acquiring AI startups to innovate its offerings. But Amazon.com CEO Andy Jassy’s famous quote that “there’s no compression algorithm for experience” favors Nvidia.

The following key points are noteworthy:

• Nvidia is seen as a dominant force in the AI market, driven by its GPU technology and CUDA software.
• Apple’s and Tesla’s heavy investment in neural networks is expected to continue influencing AI hardware design.
• Hyperscalers such as AWS, Google, Microsoft and Alibaba are major players, designing their own chips or AI products.
• AWS’ acquisition of Annapurna Labs demonstrated the potential for hyperscalers to lower costs and Boost performance by bringing design in-house.
• The potential for AWS or other hyperscalers to acquire startups or innovate in-house to compete with Nvidia is possible, but in the foreseeable future they will be reliant on Nvidia.
• Geopolitical concerns, especially related to China and Taiwan’s Taiwan Semiconductor Manufacturing Corp., were raised as potential risks to watch out for as firms such as Nvidia and Apple are exposed.
• The cost per compute cycle must come as AI grows, including energy costs. Automation is the key to justifying the expense of AI.

Watch this 10-minute deep dive into the competitive landscape for silicon chip design, the role of hyperscalers, AI inferencing and the cost of AI. 

Bottom line: Nvidia has plenty of competition but their lead is substantial and in the world of semiconductors major shifts go in long cycles. 

Snowflake catches a cold

Let’s shift gears and look at Snowflake’s quarter and talk about where it fits in AI. The reason we say Snowflake catches a cold is because it narrowly beat but was very cautious about the outlook, citing more tepid consumption patterns relative to the past — and investors sold. Ironically, Snowflake’s chief financial officer was suffering from a nagging cough that plagued him throughout the conference call. Despite the selloff, Snowflake’s momentum is still strong with very low churn. The fact. however. is that customers are optimizing costs by reducing retention policies – which lowers storage costs and makes queries run faster – so less storage and compute equals lower revenue.

Snowflake’s play is to be the iPhone of data apps. Or the App Store if you will. It wants to be the best platform to build data apps — better than the hyperscalers, better than Databricks… better than anyone. And it has made some acquisitions such as Applica and now Neeva Inc., which support the envisioned outcomes of Snowpark, a developer experience announced in 2020 that use interfaces other than SQL (such as Python, Scala and others).

The following summarizes the discussion on the future of data infrastructure in relation to AI and automation, using Snowflake and Databricks as case studies.

Floyer is of the opinion that future company architectures should aim to reduce their workforces through automation. To do so, transactional data and analytic data need to be unified, and they have to share the same databases to minimize time lags and drive real-time automation. He believes that in the long run, architectures such as Snowflake may not support this model as they would require a more direct approach from data sources to applications.

Regarding Databricks, Furrier discussed how it has successfully capitalized on the big-data wave. However, John also believes that the introduction of AI will change this landscape, bringing about a shift in the infrastructure platforms. He feels databases will become invisible, automated by AI, and data storage will be controlled by developers and applications, leading to a complete reversal of the current script.

Floyer responded by expressing the importance of databases for maintaining consistency in the future, even in a more distributed form. He doesn’t believe developers will completely take over the role of data management, as databases relieve developers of many tasks. He also believes that developers will benefit from a plethora of new tools, leading to simpler orchestration of automation.

Bottom Line: There will be a major shift in the data infrastructure landscape toward distributed, developer-controlled databases and increased automation. However, there is a divergence in opinion on how much control developers will have over data management and the extent to which databases will remain an essential tool. The underlying theme is that change is inevitable, and companies will need to adapt to stay relevant.

Watch this clip of the three analysts discussing Snowflake, Databricks, developers and the future of data platforms.

Survey data confirms deceleration in Snowflake’s momentum

The chart below is based on a survey of 1,700 information technology decision makers or ITDMs comprising 264 Snowflake accounts. It shows the granularity of Snowflake’s Net Score across those 264 accounts. Net Score is a measure of spending velocity based on Enterprise Technology Research’s proprietary methodology.

The lime green bars show the percentage of new customers adding Snowflake. The forest green is the percentage that are spending 6% or more. The gray signifies flat spend. The pinkish bars show spending down 6% or more and the bright red is churn. Subtract the reds from the greens and that equals Net Score. The blue line shows Net Score and the yellow line shows the share of mentions within the data set.

The notable points are:

• Snowflake’s Net Score peaked in the Jan 2022 survey and has steadily declined since.
• It continues to be highly elevated and among the highest in the data set.
• The decline in Snowflake’s Net Score is a function of a major shift toward flat spending within the customer base. This was accentuated on the Snowflake earnings call with CEO Frank Slootman’s comment that “the CFO is in the business,” meaning the finance function is imposing caps on spending growth.
• The churn in Snowflake accounts is very small, supporting its 150%-plus net revenue retention.

Snowflake and Databricks compete to define the ‘Modern Data Platform’

Let’s now share a different view and bring Databricks into the equation and see how they stack up with Snowflake. This chart compares the data from Snowflake (N=264), Databricks (N=225) and Streamlit (N=111). It plots Net Score or spending momentum on the Y axis and account overlap/presence based on the Ns on the X axis. The squiggly lines indicate the progression over time.

Several points are notable in the data:

• The presence of of Databricks in the enterprise over the last two plus years has significantly increased and the account overlap between the two leaders sets up a looming battle.
• Snowflake’s Net Score decline has brought it in line with that of Databricks.
• Despite the macro pressures, all three platforms are above the 40% line, which is an indicator of a highly elevated spending velocity.

Let’s delve into the key points that arose during this conversation about these data points.

The discussion begins with an acknowledgment from Furrier of the validity of Snowflake’s churn data. Its churn rates are low despite the current economic headwinds causing a market slowdown. So they’re not losing customers, similar to the dynamic among the cloud players. Major shifts, such as the hype around AI right now, tend to cause a freeze in the buyer market. This leads to a “wait and see” approach, further slowing down spending in the sector. But customers are not defecting.

An important comparison arises between Snowflake and Databricks, two significant players in the market. Snowflake’s strong business model has set it apart and allowed it to lead the market in the early stages. However, Databricks has gained substantial traction through leveraging the open-source community and consistently enhancing its robust product offerings.

A salient point in the discussion revolves around data retention policies. Snowflake, for example, stores an extensive amount of data for its clients, but the question arises: Is all this stored data being utilized effectively? This was a course of conversation on the earnings call where customers are being forced to control costs and shortening retention times is a logical way to do so.

Floyer stressed that the value of data diminishes as it ages, prompting a shift toward efficiently capturing and extracting the value of data close to its source before disposing of it. Barclays analyst Raimo Lenschow asked what we thought was a salient question on the earnings call: Is the trend of shorter retention times potentially enduring and will this lead to a change in the future growth of data storage needs

• The expectation is that the increase in the efficiency of data capture and utilization will result in a larger proportion of data being ephemeral.
• Yet the amount of data generated will be so large that it will continue to grow exponentially thanks to the emergence of new industries and use cases and the impact of AI.

Furrier introduced the concept of “hyper data scalers.” These could be entities similar to cloud hyperscalers but focused on storing vast amounts of data for foundational AI models.

• This aligns with the ongoing evolution of data storage and usage models, with foundational AI models driving this change.
• There’s potential for the creation of new forms of data clouds, differing from the current offerings by companies such as Snowflake and Databricks. These would support more distributed cloud data architectures but would require new types of databases and standards.

The conversation concludes with a focus on the value of historical data for AI, particularly for pattern recognition and training AI models. There’s speculation about a new model of data infrastructure where the emphasis is less on storage and more on real-time use and domain-specific data. This could significantly alter the landscape of data handling in the future.

Bottom Line: The current traction of both Snowflake and Databricks is notable. Both companies have strong managements and seemingly loyal customer bases and they’ll both likely leverage AI effectively. The market is large enough for both to thrive in the near to mid-term with longer-term trends around real-time data and AI inferencing challenging the status quo. 

Listen and watch the three analysts discuss churn rates, real-time data, whether more data will become ephemeral and if that will negatively impact storage growth.

Uber for the enterprise: The future of data apps

The world of applications is shifting toward data apps. Today’s data silos are largely a function of data being embedded in applications that automate processes. Increasingly, we believe business logic will be infused into data and apps will be build using this new model.

The example we often use is Uber for the enterprise, where a digital twin of your businesses is created. People, places and things are digitized as independent data elements, but those data “products” are discoverable, governed and have coherence. A semantic layer enables these data elements to be completely connected and understood by the system and each other.

Using the Uber example: Riders and drivers are connected with data related to destinations, estimated times of arrival, and prices, based on demand and supply. This is done in real time without incurring significant tradeoffs among availability, latency and consistency. We believe that these types of apps will require new thinking around data architectures, standards and platforms. But more important, they will drive new levels of automation and productivity for businesses.

Here’s a bulleted summary of the closing conversation between the three analysts using the Uber example and the impact on productivity:

• Revenue per employee in Uber: Uber’s success in harnessing data from cars, drivers, streets and road conditions to optimize its business has led to significant increases in revenue per employee. From 2021 to 2022, Uber’s revenue per employee grew from $600,000 to $971,000, far surpassing the typical software company’s $225,000 to $250,000 per employee.
• Automation and AI’s impact on businesses: Uber’s automation model represents a future where businesses have to automate extensively and leverage AI to remain solvent. Companies that don’t adopt this model within a decade may face significant risks from emerging startups using AI tools and real-time data more effectively.
• Elon Musk’s impact on industry productivity: Elon Musk has had a significant impact on industries such as automotive and space with his innovative approach to productivity. By producing “software cars” instead of hardware cars, and with similar innovation in SpaceX, Musk has shown that to survive and thrive, companies must adapt to new technologies and workflows.
• Real-time data capture is key: The most valuable data for decision-making is real-time data. Companies such as Uber have excelled by making real-time decisions based on immediate data.
• AI in incumbents versus new entrants: Incumbents such as Dell Technologies Inc., IBM, Hewlett Packard Enterprise Co., Oracle, ServiceNow Inc. and Salesforce Inc. can certainly leverage AI. However, there’s a belief that a new model akin to keyword search, initially overlooked but eventually dominant, may emerge and become a game-changer in the industry.
• Predicted shift toward simplification: The panel predicts that the ongoing shift in technology represents a new paradigm. This shift is likely to supply rise to new startups, much like the advent of the web did. We believe that the next wave will focus on simplifying things and reducing the steps it takes to accomplish tasks. As with the web’s rise, the companies that can work within and enhance this nascent market stand to capture the most value. The big players will also be involved, reaping their share of the value from this shift.

Unlike previous generations of companies, particularly witnessed in demise of the East Coast minicomputer business (Apollo, DEC, DG, Prime, Wang), today’s leaders are much more paranoid about disruptive technologies. But blind spots exist and often incumbents are so focused on protecting their franchises that it leads to slower growth and lack of innovation. AI represents an opportunity for both incumbents to drive automation into existing platforms and disruptors to bring new models to industries.

Unlike the web, which was often seen as a bifurcated opportunity between bit- and atom-based businesses, AI has the potential to be even more ubiquitous.

Watch the three analysts discuss the Uber example, the impact automation has had on Uber’s revenue per employee, and the future imperative of leveraging real time data.

Keep in touch

Many thanks to Alex Myerson and Ken Shifman on production, podcasts and media workflows for Breaking Analysis. Special thanks to Kristen Martin and Cheryl Knight, who help us keep our community informed and get the word out, and to Rob Hof, our editor in chief at SiliconANGLE.

Remember we publish each week on Wikibon and SiliconANGLE. These episodes are all available as podcasts wherever you listen.

Email david.vellante@siliconangle.com, DM @dvellante on Twitter and comment on our LinkedIn posts.

Also, check out this ETR Tutorial we created, which explains the spending methodology in more detail. Note: ETR is a separate company from Wikibon and SiliconANGLE. If you would like to cite or republish any of the company’s data, or inquire about its services, please contact ETR at legal@etr.ai.

Here’s the full video analysis:

All statements made regarding companies or securities are strictly beliefs, points of view and opinions held by SiliconANGLE Media, Enterprise Technology Research, other guests on theCUBE and guest writers. Such statements are not recommendations by these individuals to buy, sell or hold any security. The content presented does not constitute investment advice and should not be used as the basis for any investment decision. You and only you are responsible for your investment decisions.

Disclosure: Many of the companies cited in Breaking Analysis are sponsors of theCUBE and/or clients of Wikibon. None of these firms or other companies have any editorial control over or advanced viewing of what’s published in Breaking Analysis.

Image: greenbutterfly/Adobe Stock

SAN JOSE, Calif., May 31, 2023 — Aporia announced today a partnership with Databricks, the data and AI company allowing organizations to unify their data, analytics, and AI. The strategic partnership will provide seamless ML Observability to customers using Databricks’ Lakehouse Platform, AI capabilities, and MLflow offerings.

With Aporia, Databricks customers can now monitor their ML models in production without duplicating any data from their Lakehouse or any other data source. Aporia’s quick deployment on Databricks enables monitoring billions of predictions without data sampling, production code changes, or hidden storage costs.

To monitor large amounts of data, organizations often need to duplicate data from their data lake to their monitoring platform. However, this often leads to highly inaccurate results with issues going unnoticed, drift and false positive alerts, and difficulties in properly monitoring for bias and fairness. Aporia’s integration with Databricks allows organizations to monitor all of their machine learning models on Databricks in just a few minutes, leveraging their existing database investment — even for use cases that require processing large volumes of predictions, such as recommendation systems, search ranking models, fraud detection models, and demand forecasting models.

“As the AI market rapidly expands, there’s a growing demand for effective tools to monitor and maintain machine learning models in production,” said Liran Hason, CEO of Aporia. “Aporia’s integration with Databricks expedites the adoption of Observability as a critical component for organizations investing in AI and Machine Learning. Our partnership allows us to provide a solution that is both centralized and cost-effective, addressing a critical need for our clients and empowering them to make data-driven decisions with confidence.”

For more information, visit https://www.aporia.com/partners/databricks.

About Aporia

Aporia is the ML Observability platform, trusted by Fortune 500 companies and industry leaders – such as Lemonade, Bosch, Munich RE, Sixt, and Armis – to monitor, visualize, and maximize the value of machine learning models in minutes. Aporia empowers data science and ML teams to confidently monitor, explain, and gain insights to Boost models in production.

Source: Aporia

LAS VEGAS, NV / ACCESSWIRE / May 31, 2023 / Tego Cyber Inc. (OTCQB:TGCB), a cybersecurity company focused on developing innovative cyber threat intelligence and autonomous correlation and hunting tools, today announced the launch of its threat intelligence platform on Databricks Marketplace (https://www.databricks.com/product/marketplace).

Databricks Marketplace is an open marketplace for exchanging data products such as datasets, notebooks, dashboards, and machine learning models. To accelerate insights, data consumers can discover, evaluate, and access more data products from third-party vendors than ever before. Providers can now commercialize new offerings and shorten sales cycles by providing value-added services on top of their data. Databricks Marketplace is powered by Delta Sharing allowing consumers to access data products without having to be on the Databricks platform. This open approach allows data providers to broaden their addressable market without forcing consumers into vendor lock-in. The addition of Tego Cyber's threat intelligence platform to the marketplace will supply Databricks users access to the latest threat intelligence data, enabling them to make more informed decisions about their security posture.

"We are very proud to participate in the inception of the Databricks Marketplace and be the first threat intelligence data provider available in their marketplace," stated Shannon Wilkinson, President & CEO of Tego Cyber. "The Databricks Marketplace will enable us to market our threat intelligence and threat correlation engine applications to a market segment that, to this point, we have not been able to access. The Tego threat intelligence feed for Databricks will allow customers using various data platforms to quickly have access to enriched threat intelligence that can empower them to have faster insights into threats within their organization without the need to ship their data elsewhere."

Lipyeow Lim, Technical Director for Cybersecurity GTM at Databricks added, "Cybersecurity is a team sport. The Tego and Databricks integration enables security teams to innovate in their fight against cyber criminals by having access to enriched IOCs. The Tego threat intelligence feed not only helps security teams to do their day-to-day job more efficiently and effectively, but also provides all the data in Databricks Lakehouse where they can experiment, create, and test their own security analytics and AI/ML models and contribute these back to the cybersecurity community at large."

Mrs. Wilkinson continued, "Using Tego's threat intelligence and threat correlation solutions allows customers to find exposures that they otherwise would not have seen. The Global Incident Response Leader for a Fortune 500 recently called to inform us that without Tego's threat correlation engine and enriched, actionable threat intelligence data, his team would not have detected the initial compromise of a well-known ransomware group in their environment. Taking the lessons learned of building at-scale and at-speed threat correlation for SIEM platforms, we are looking forward to providing the same to Databricks users where they have full ownership of their data and the ability to create true XDR capabilities irregardless of what technologies they use."

According to Melissa Knight, Tego Cyber's Chief Information Security Officer, "threat correlation is a valuable tool for businesses as it helps them to proactively monitor and detect potential security threats. By having unparalleled visibility into their security posture, companies can respond rapidly to threats and minimize the risk of a security breach. With Tego Cyber's seamless integration with Databricks, customers can leverage the power of advanced analytics, machine learning, and curated threat intelligence data to detect and respond to threats in real-time."

About Tego Cyber Inc.

Tego Cyber Inc. (OTCQB:TGCB) was founded to mitigate the disparity in the rapidly evolving cyber threat hunting, correlation, and threat intelligence market. The Company is focused on developing solutions for threat intelligence and autonomous threat hunting/correlation. Tego's curated threat intelligence feed not only contains a comprehensive list of indicators of compromise, but also provides additional context including specific details needed to counteract threats so that security teams can spend less time searching for disjointed indicators of compromise. Tego's threat correlation engine integrates with top security and datalake platforms to proactively identify threats. The Tego threat correlation engine allows security teams to find threats faster using curated data feeds, powerful and low latency searches across large disparate data sets, and user-friendly visualizations that help reduce the time to detection and response. For more information, please visit https://tegocyber.com.

Forward-Looking Statements

The statements contained in this press release, those which are not purely historical or which depend upon future events, may constitute forward-looking statements within the meaning of Section 27A of the Securities Act of 1933 and Section 21E of the Securities Exchange Act of 1934. Statements regarding the Company's expectations, hopes, beliefs, intentions or strategies regarding the future constitute forward-looking statements. Prospective investors are cautioned that any such forward-looking statements are not guarantees of future performance and involve risks and uncertainties, and that real results may differ materially from those projected in the forward-looking statements because of various factors. All forward-looking statements included in this press release are based on information available to the Company on the date hereof and the Company assumes no obligation to update any such forward-looking statement. Prospective investors should also consult the risks factors described from time to time in the Company's Reports on Forms 10-K, 10-Q and 8-K and Annual Reports to Shareholders.

Tego Contact:

Tego Cyber Inc.
8565 S Eastern Avenue, Suite 150
Las Vegas, Nevada 89123
USA
Tel: 855-939-0100 (North America)}
Tel: +1 725-726-7840 (International)
Email: info@tegocyber.com
Web: https://tegocyber.com
Facebook: facebook.com/tegocyber
LinkedIn: linkedin.com/company/tegocyber
Twitter: twitter.com/tegocyber

SOURCE: Tego Cyber Inc.