Main Menu

Recent posts

#71
Ubuntu Blog / Migrating from Apache Spark 3...
Last post by tim - May 28, 2026, 06:45 PM
Migrating from Apache Spark 3 to Spark 4

The purpose of this guide is to highlight the key differences between Apache Spark 3 and Spark 4, and provide advice on how to plan a migration. Let's get started.

The biggest changes

Let's talk about the biggest changes between Apache Spark 3.x and Spark 4.

Scala 2.12 no more

First up, there's no support for Scala 2.12 in Spark 4, so any jobs you have that are using that version will need to be recompiled against Scala 2.13. This is generally straightforward, but note that all the dependencies of your job must also be recompiled against Scala 2.13. Depending on how many dependencies your job has, this might be a large project. 

Scala 2.13 also overhauled the `collections` library with a major rewrite of its API. Given the collections library is a foundational library, there's a good chance you might also need to make code changes. Fortunately, there are free and open source tools that can help you to migrate your codebases, like Scalafix . You can add Scalafix as a dependency of scala build tool (SBT) and enable the "Collection213Upgrade" rule:

sbt> scalafixEnable
sbt> scalafixAll dependency:[email protected]:scala-collection-migrations:2.13.0
Java 17 and 21

Spark 4 discontinues support for old versions of the Java runtime below version 17. Java 17 is now the intended default, although Java 21 is also supported. If you are running Spark jobs or plugins that require an older version of Java, this might break things for you as there have been some changes – for example the Java XML and servlet libraries – that might require code to be adapted.

Promising developments

Some of the changes are quite promising. There are some great new capabilities – but also some changes to default behaviour. Let's tackle the defaults first.

New defaults

ANSI SQL mode by default: in Spark 3, invalid operations often returned NULL, but in Spark 4, they'll throw runtime exceptions (e.g., division by zero or invalid type casts) to ensure data quality. This could be a breaking change to your queries and pipelines, so you'll need to test carefully.

Structured JSON logging: in Spark 4, logs are written as structured JSON by default, which makes them easier to process using logging infrastructure like Loki. But note that this change could break existing monitoring and alerting, so again test carefully.

RocksDB: the backend for shuffle and state management in structured streaming now defaults to RocksDB, which can significantly reduce JVM heap pressure and result in fewer garbage collection stalls.

New capabilities

New "VARIANT" data type: designed for semi-structured data like JSON, the new data type delivers significantly faster queries than traditional string-based JSON parsing.

SQL pipe syntax: it's now possible to chain SQL operations using the |> operator.

Procedural SQL and scripting: Spark 4 adds support for multi-statement SQL scripts with local variables and control flow (IF/WHILE logic), which means you can likely write an entire ETL pipeline in pure SQL.

Improved genAI capabilities: full support for vector data types and optimized batch inference for LLM-driven workloads.

Migration strategy

So now we understand the big changes, let's give some thought to how to plan and execute a migration from Spark 3 to Spark 4.

Foundational changes

Before you start to change your Spark code, it's best to make sure that your underlying infrastructure can support the new requirements.

  • You'll need to migrate your runtime environment to Java 17 or Java 21. If you're using Canonical's solution for Apache Spark on Kubernetes , we've already taken care of this for you.
  • If you're using Scala, you'll need to migrate all code and internal libraries to Scala 2.13.
  • Make sure all third-party connectors and plugins (e.g. Spark-RAPIDS, Kafka) have published versions for Spark 4/Scala 2.13.
  • Replace any javax.* imports in your Java code with jakarta.* for servlet and XML compatibility.
Runtime validation

In some ways, the biggest breaking change in Spark 4 is likely going to be that ANSI SQL mode is enabled by default.

  • Set spark.sql.ansi.enabled=true in your current Spark 3 environment. This will surface any division-by-zero or numeric overflow errors that would have previously been hidden by silent NULL values.
  • Replace risky operations with "safe" alternatives. For example, you could use try_cast instead of cast to maintain the old behavior of returning NULL, in order to preserve the old behaviour.
Enabler modernization

The next stage will be to validate that your workloads continue to run smoothly with the new enablers in Spark 4.

  • If using Structured Streaming, test your jobs with RocksDB, which is now the default state store.
  • Check your CREATE TABLE statements. Without an explicit USING clause, Spark 4 will default to the provider in spark.sql.sources.default instead of Hive.
Rollout plan

Our suggestion is to start with migrating your batch workloads first, followed by streaming workloads, and move analytical/decision support workloads last. Move each workload type incrementally. Carefully validate results, inspect logs, and check for discrepancies.

Conclusion

I hope this guide helps you with your migration. As with many of these things, preparation is key. If you'd like to learn more about Canonical's Charmed solution for Apache Spark, feel free to get in touch via the contact form .

Further reading

The purpose of this guide is to highlight the key differences between Apache Spark 3 and Spark 4, and provide advice on how to plan a migration. Let's get started. The biggest changes Let's talk about the biggest changes between Apache Spark 3.x and Spark 4. Scala 2.12 no more First up, there's no support [...]


Categories: charmed_spark, data_engineering, data_science, spark, spark_big_data
Source: https://ubuntu.com//blog/migrating-from-apache-spark-3-to-spark-4 May 28, 2026, 11:03 AM
#72
Ubuntu Blog / Introducing Workshop: launch ...
Last post by tim - May 28, 2026, 06:45 PM
Introducing Workshop: launch sandboxed development environments on Ubuntu with a single command

Developers now benefit from consistency and repeatability for cutting-edge workflows, including agentic AI.


Today, Canonical announced the release of Workshop, a solution for launching development environments with a single command. These environments are configured once, and can be reproduced on different machines. This means consistent workflows across development machines and deployment pipelines, and less time managing dependencies. 

"Developers operating at the cutting edge want to focus on what they're building, not on dependencies or workstation configuration," said Jon Seager, VP of Engineering at Canonical. "Workshop enables developers to achieve that elegantly with a single YAML file that defines their environment, and pulls the exact dependencies and components they need. Workshops also serve to standardize and sandbox agentic tooling consistently across teams."

Composable and repeatable development environments 

The time window between "cutting edge" and "mainstream" continues to shrink, meaning developers don't have time to wait. However, this typically means extensive manual effort: both in configuring all the software you require, and ensuring it can harmlessly and effectively run on your machine. The problem scales when team members need to collaborate, often on different hardware.

Workshop brings composability and repeatability to developer environments. Individuals can pull SDKs such as Ollama, OpenCode, NVIDIA CUDA, and AMD ROCm by including them in their Workshop configuration files. Once an environment is specified, creating, upgrading, or winding down an environment can be done with a few keystrokes.



Workshop environments are defined in simple YAML documents, and can therefore be version controlled and shared among project contributors. Workshops can be trivially reproduced with a single command, and feature an interface system, inspired by snapd, that simplifies host resource allocation.

Agent-ready sandboxing

In the fast-paced world of agentic AI, teams need resilient confinement to ensure they can develop at speed, without risking harm to the host system. Workshop's development environments run in unprivileged system containers to minimize the attack surface available to workloads running within them. 

Repeatability applies to security, too. In Workshop, SDKs are limited to a uniform means of requesting access to resources, such as access to the desktop to display a GUI app, or access to the SSH agent from the host machine. Security doesn't depend on knowing the individual protocols of a container, and strict access controls can be enforced for AI agents. 

"Ease of use for developers shouldn't mean ease of access for AI agents. There's naturally a tension between these two 'user' groups, but Workshop resolves it through strict enforcement of access controls," said Dmitry Lyfar, Engineering Manager at Canonical. "Resource allocation remains simple and consistent across all environments to minimize human error, while non-privileged defaults effectively constrain workload capabilities."



Access your host's resources in the simplest way possible

Workshop's isolation doesn't mean you're cut off from the specific capabilities your hardware has to offer. Workshop dispenses with complex mapping scripts and filesystem paths, offering instead a standardized way to access mounts, devices, and network services from the contained environment. 

If a pre-designed SDK already exists to access a specific piece of hardware, you can choose to incorporate it into your YAML to optimize the resulting toolchain for your machine; if it doesn't, you can create a custom one just for yourself. 

Get started with Workshop

Ensure you're running LXD 6.8 or newer before installing Workshop.

sudo snap install --channel=6/stable lxd

sudo snap install --classic workshop

To learn more about managing modular workspaces, exploring the SDK store, or building SDKs, read the Workshop documentation here .

Developers now benefit from consistency and repeatability for cutting-edge workflows, including agentic AI. Today, Canonical announced the release of Workshop, a solution for launching development environments with a single command. These environments are configured once, and can be reproduced on different machines. This means consistent workflows across development machines and deployment pipelines, and less time [...]


Categories: Developer Tools, Ubuntu
Source: https://ubuntu.com//blog/introducing-workshop-sandboxed-development-environments May 27, 2026, 08:57 PM
#73
Ubuntu Blog / Run agentic workloads on Arm ...
Last post by tim - May 28, 2026, 06:45 PM
Run agentic workloads on Arm and Ubuntu

The era of prompt-and-response AI is behind us. We are now firmly in the age of agentic AI and the world needs a new class of compute built for this reality. That is why the launch of the Arm® AGI CPU  is a pivotal moment. And to harness that raw, bare-metal efficiency at scale, it requires an optimized, resilient, and securely designed operating system.

In the lead-up to Ubuntu Summit 26.04, Canonical and Arm are collaborating to certify the new Arm AGI CPU on Ubuntu 26.04 LTS (Resolute Raccoon) .

A shared history: pioneering Arm infrastructure

Canonical didn't just jump on the Arm bandwagon; we have been pioneering emerging architectures for decades. Our vision, as stated in the historic 2012 Ubuntu Server 12.04 LTS press release , was a "future in which low-energy, hyperscale servers come to dominate for many workloads." Ubuntu 12.04 LTS was the first enterprise Linux distribution to provide support for Arm-based servers, laying the early groundwork for the hyperscale era. From those early 32-bit development boards, Canonical has treated Arm as a first-class citizen, evolving Ubuntu into the dominant OS for the 64-bit server market.

As hyperscalers transitioned to cloud-native Arm processors in 2018, Ubuntu became the go-to foundation, providing day 0 support for all public cloud Arm instances. Canonical's commitment goes beyond the operating system itself. Our infrastructure suite – including MAAS, Openstack, MicroCloud, Ceph, LXD, and Canonical Kubernetes – has featured highly optimized, native Arm64 support for years. This has enabled seamless bare-metal provisioning and orchestration, ensuring that when new Arm64 silicon drops, the software ecosystem is already waiting.

The new benchmark: Arm AGI CPU

For 35 years, Arm has licensed its IP to power the world's computing. The introduction of the Arm AGI CPU expands this platform strategy, giving the ecosystem the ultimate flexibility to build on Arm – whether licensing IP, adopting Arm Neoverse® CSS, or deploying Arm-designed silicon. Arm AGI CPU is a high-performance engine built specifically to address the power and density constraints of modern AI workloads.

Hardware highlights include:

  • Massive compute density: Up to 136 Arm Neoverse V3 cores spread across a dual-chiplet design.
  • Efficiency: Manufactured on TSMC's 3nm process, delivering maximum throughput at 300W TDP.
  • Sub-100ns Memory Latency: 12 channels of DDR5 memory support, delivering 6GB/s of memory bandwidth per core to eliminate data starvation in real-time AI inference.
  • Rack-scale economics: The 1OU dual-node reference design packs 272 dedicated cores per server unit, completely reshaping data center economics.

"The shift to agentic AI demands a new class of infrastructure where performance, efficiency, and scalability are engineered together from the ground up. By collaborating with Canonical to certify Ubuntu 26.04 LTS on Arm AGI CPU, we are enabling developers and enterprises to deploy next-generation AI workloads with the performance density, power efficiency, and software readiness required for the agentic era."— Eddie Ramirez, VP of Go-to-Market, Cloud AI Business Unit, Arm

Ubuntu 26.04 LTS on Arm AGI CPU: hyper-efficient AI infrastructure

In high-stakes data centers, silicon is only as powerful as the software running on it. To extract every ounce of performance from the Arm AGI CPU, Ubuntu 26.04 LTS – backed by Canonical's full-stack infrastructure suite – provides a secure, resilient, and frictionless foundation.

  • Cutting-edge kernel 7.0 with 15-year support: Ubuntu 26.04 LTS ships with the Linux 7.0 kernel, bringing the latest upstream kernel innovations directly to enterprise environments. Backed by up to 15 years of enterprise maintenance with Ubuntu Pro, this release natively unlocks advanced Arm64 features like MPAM and nested virtualization.
  • Arm64 kernel livepatching: In the world of continuous agentic AI, downtime is unacceptable. For the first time, Canonical's Kernel Livepatch is available for Arm64 servers, allowing operators to apply critical security updates to the kernel without ever rebooting.
  • Memory safety at the core: Ubuntu 26.04 LTS integrates Rust into the system layer (such as sudo-rs and coreutils), aligning perfectly with hardware-enforced security capabilities, like MTE, built into the Neoverse V3 architecture.
  • Out-of-the-box optimization: Ubuntu certification brings rigorous, continuous validation in Canonical labs for the entire infrastructure portfolio to deliver secure, stable, and highly performant day-one deployments on certified hardware.

"The long-standing collaboration between Canonical and Arm has consistently driven  the evolution of enterprise infrastructure. Our joint efforts to certify the Arm AGI CPU on Ubuntu 26.04 LTS ensures a secure, dependable foundation to orchestrate agentic AI in data centers and cloud"— Cindy Goldberg, VP of Cloud and Silicon Partnerships at Canonical

Join Us at Ubuntu Summit 26.04

We invite you to join us at Ubuntu Summit 26.04 to see the future of infrastructure firsthand. Attend Arm's dedicated workshop on optimizing workload on Arm and Ubuntu through agentic AI and static performance analysis for application tuning.

Register for Ubuntu Summit 26.04

In the lead-up to Ubuntu Summit 26.04, Canonical and Arm are collaborating to certify the new Arm AGI CPU on Ubuntu 26.04 LTS (Resolute Raccoon). Learn what this means for developers and agentic AI.


Categories: ARM64, cloud, Silicon Partner Program, Ubuntu, Ubuntu 26.04 LTS, ubuntu certification
Source: https://ubuntu.com//blog/run-agentic-workloads-on-arm-and-ubuntu May 26, 2026, 11:00 AM
#74
Ubuntu News / Canonical’s Workshop: sandbox...
Last post by tim - May 28, 2026, 06:45 PM
Canonical's Workshop: sandboxed, reproducible dev environments

Canonical has released Workshop, a new open-source tool to create reproducible development environments with a single command. Using YAML files, the same development setup can be reproduced across different hardware and devices, reducing dependency headaches and configuration drift. Environments in Workshop are built from SDKs (packages that install languages, frameworks and tools). Most of these come from the SDK Store, which supports versioned channels similar to the Snap Store so that projects can define specific SDK versions to use. Canonical offers SDKs for Ollama, OpenCode, NVIDIA CUDA and AMD ROCm at launch, but users can create and define project-specific SDKs [...]

You're reading Canonical's Workshop: sandboxed, reproducible dev environments , a blog post from OMG! Ubuntu . Do not reproduce elsewhere without permission.


Categories: News, Canonical, dev tools, LXD, Workshop
Source: https://www.omgubuntu.co.uk/2026/05/canonical-workshop-dev-environments May 27, 2026, 04:52 PM
#75
Ubuntu News / Raspberry Pi 6 won’t arrive b...
Last post by tim - May 28, 2026, 06:45 PM
Raspberry Pi 6 won't arrive before 2028 (and won't have an NPU)

The Raspberry Pi 6 won't be released before 2028 and it won't feature an onboard NPU to handle AI compute tasks. Insight into plans for the Pi 6 were shared by three of the company's key engineers and leaders in an AMA (ask me anything) session on Reddit on 21 May, 2026. Based on past launches the gap between major Pi models (Raspberry Pi 2, 3, 4 and 5) is around 3-4 years. The Raspberry Pi 5 launched in 2023. That should put the Pi 6 on course for launch in 2026 or 2027. But Raspberry Pi co-founder and CEO [...]

You're reading Raspberry Pi 6 won't arrive before 2028 (and won't have an NPU) , a blog post from OMG! Ubuntu . Do not reproduce elsewhere without permission.


Categories: Hardware, News, Raspberry Pi, SBCs
Source: https://www.omgubuntu.co.uk/2026/05/raspberry-pi-6-2028-no-npu May 26, 2026, 07:24 PM
#76
Ubuntu News / Cinnamon desktop is getting i...
Last post by tim - May 28, 2026, 06:45 PM
Cinnamon desktop is getting its own, native screenshot tool

Linux Mint developers are building a new screenshot utility for the Cinnamon desktop, ahead of its next major release. The home-grown tool will give users more options when taking screenshots and will "accommodate the differences between CSD (Client Side Decoration) and SSD (Server Side Decoration) windows" to provide 'cleaner' looking screenshots. Currently, Cinnamon rolls with the GTK-based gnome-screenshot. That tool works fine, but it doesn't render shadows in windowed app screenshots on Cinnamon. It does, however, include pixel artefacts around the corners of windows, caused by the drop shadow bleeding through: It's not super pretty, and as someone who works [...]

You're reading Cinnamon desktop is getting its own, native screenshot tool , a blog post from OMG! Ubuntu . Do not reproduce elsewhere without permission.


Categories: News
Source: https://www.omgubuntu.co.uk/2026/05/linux-mint-new-cinnamon-screenshot-tool May 25, 2026, 02:40 AM
#77
Ubuntu News / Canonical to shutter Ubuntu P...
Last post by tim - May 24, 2026, 10:36 PM
Canonical to shutter Ubuntu Pastebin after 18 years of service

Canonical will decommission its long-running text-hosting service Ubuntu Pastebin on May 31. The company is pulling the plug as part of a broader "infrastructure modernization and migration project". Ubuntu Pastebin, which works similarly to GitHub's Gist, albeit without the revision history, has been available to the community since late 2007 It originally launched under the paste.ubuntu.com domain before adding pastebin.ubuntu.com, which is live until the end of the month. The service was partly launched to stop thedistro's official IRC support channels from being flooded with reams of terminal output by users seeking help. It was also used by community members to [...]

You're reading Canonical to shutter Ubuntu Pastebin after 18 years of service , a blog post from OMG! Ubuntu . Do not reproduce elsewhere without permission.


Categories: News, Canonical, pastebin
Source: https://www.omgubuntu.co.uk/2026/05/canonical-ubuntu-pastebin-shutdown May 24, 2026, 09:28 PM
#78
Ubuntu News / Ubuntu 26.10 daily builds now...
Last post by tim - May 24, 2026, 10:36 PM
Ubuntu 26.10 daily builds now available to download

Daily builds of Ubuntu 26.10 'Stonking Stingray' are now available for download, as development on the distro's next major release kicks in to gear. As the name suggests, new ISOs are produced from development code on a (mostly) daily basis, giving those keen to test October's release in advance the ability to do so. However, because package updates can break the ability for a bootable image to be created, it's not unusual for there to be temporary gaps between new daily builds being available. Daily builds will continue to be produced for remainder of the Ubuntu 26.10 development cycle, right [...]

You're reading Ubuntu 26.10 daily builds now available to download , a blog post from OMG! Ubuntu . Do not reproduce elsewhere without permission.


Categories: News, daily builds, Ubuntu 26.10
Source: https://www.omgubuntu.co.uk/2026/05/ubuntu-26-10-daily-builds May 24, 2026, 04:14 PM
#79
9to5Linux / Shelly 2.3.1 GUI Package Mana...
Last post by tim - May 23, 2026, 11:01 PM
Shelly 2.3.1 GUI Package Manager for Arch Linux Improves Notifications



Shelly 2.3.1 open-source graphical package manager for Arch Linux-based distributions is now available for download with various new features and improvements.

The post Shelly 2.3.1 GUI Package Manager for Arch Linux Improves Notifications  appeared first on 9to5Linux  - do not reproduce this article without permission. This RSS feed is intended for readers, not scrapers.


Categories: Apps, News, package manager, Shelly, Shelly-ALPM
Source: https://9to5linux.com/shelly-2-3-1-gui-package-manager-for-arch-linux-improves-notifications May 23, 2026, 07:49 PM
#80
9to5Linux / Linux Mint 23 Getting New Cin...
Last post by tim - May 23, 2026, 11:01 PM
Linux Mint 23 Getting New Cinnamon Screenshots Tool, Network Improvements



Linux Mint 23 is getting a new Cinnamon Screenshots tool, network improvements, Nemo enhancements, theme improvements, and more.

The post Linux Mint 23 Getting New Cinnamon Screenshots Tool, Network Improvements  appeared first on 9to5Linux  - do not reproduce this article without permission. This RSS feed is intended for readers, not scrapers.


Categories: Distros, News, Linux distribution, Linux Mint, Linux Mint 23
Source: https://9to5linux.com/linux-mint-23-getting-new-cinnamon-screenshots-tool-network-improvements May 23, 2026, 06:08 PM