Posts Tagged ‘KotlinNotebook’
[KotlinConf2024] DataFrame: Kotlin’s Dynamic Data Handling
At KotlinConf2024, Roman Belov, JetBrains’ Kotlin Moods group leader, showcased Kotlin DataFrame, a versatile library for managing flat and hierarchical data. Designed for general developers, not just data scientists, DataFrame handles CSV, JSON, and object subgraphs, enabling seamless data transformation and visualization. Roman demonstrated its integration with Kotlin Notebook for prototyping and a compiler plugin for dynamic type inference, using a KotlinConf app backend as an example. This talk highlighted how DataFrame empowers developers to build robust, interactive data pipelines.
DataFrame: A Versatile Data Structure
Kotlin DataFrame redefines data handling for Kotlin developers. Roman explained that, unlike traditional data classes, DataFrame supports dynamic column manipulation, akin to Excel tables. It can read, write, and transform data from formats like CSV or JSON, making it ideal for both analytics and general projects. For a KotlinConf app, DataFrame processed session data from a REST API, allowing developers to filter, sort, and pivot data effortlessly, providing a flexible alternative to rigid data class structures.
Prototyping with Kotlin Notebook
Kotlin Notebook, a plugin for IntelliJ IDEA Ultimate, enhances DataFrame’s prototyping capabilities. Roman demonstrated creating a scratch file to fetch session data via Ktor Client. The notebook’s auto-completion for dependencies, like Ktor or DataFrame, simplifies setup, downloading the latest versions from Maven Central. Interactive tables display hierarchical data, and each code fragment updates variable types, enabling rapid experimentation. This environment suits developers iterating on ideas, offering a low-friction way to test data transformations before production.
Dynamic Type Inference in Action
DataFrame’s compiler plugin, built for the K2 compiler, introduces on-the-fly type inference. Roman showed how it analyzes a DataFrame’s schema during execution, generating extension properties for columns. For example, accessing a title column in a sessions DataFrame feels like using a property, with auto-completion for column names and types. This eliminates manual schema definitions, streamlining data wrangling. Though experimental, the plugin cached schemas efficiently, ensuring performance, as seen when filtering multiplatform talk descriptions.
Handling Hierarchical Data
DataFrame excels with hierarchical structures, unlike flat data classes. Roman illustrated this with nested JSON from the KotlinConf API, converting categories into a DataFrame with grouped columns. Developers can navigate sub-DataFrames within cells, mirroring data class nesting. For instance, a category’s items array became a sub-DataFrame, accessible via intuitive APIs. This capability supports complex data like object subgraphs, enabling developers to transform and analyze nested structures without cumbersome manual mappings.
Building a KotlinConf Schedule
Roman walked through a practical example: creating a daily schedule for KotlinConf. Starting with session data, he converted startsAt strings to LocalDateTime, filtered out service sessions, and joined room IDs with room names from another DataFrame. Sorting by start time and pivoting by room produced a clean schedule, with nulls replaced by empty strings. The resulting HTML table, generated directly in the notebook, showcased DataFrame’s ability to transform REST API data into user-friendly outputs, all with concise, readable code.
Visualizing Data with Kandy
DataFrame integrates with Kandy, JetBrains’ visualization library, to create charts. Roman demonstrated analyzing GitHub commits from the Kotlin repository, grouping them by week to plot commit counts and average message lengths. The resulting chart revealed trends, like steady growth potentially tied to CI improvements. Kandy’s simple API, paired with DataFrame’s data manipulation, makes visualization accessible. Roman encouraged exploring Kandy’s website for examples, highlighting its role in turning raw data into actionable insights.
DataFrame in Production
Moving DataFrame to production is straightforward. Roman showed copying notebook code into IntelliJ’s EAP version, importing the generated schema to access columns as properties. The compiler plugin evolves schemas dynamically, supporting operations like adding a room column and using it immediately. This approach minimizes boilerplate, as seen when serializing a schedule to JSON. Though the plugin is experimental, its integration with K2 ensures reliability, making DataFrame a practical choice for building scalable backend systems, from APIs to data pipelines.
Links:
[KotlinConf2025] Charts, Code, and Sails: Winning a Regatta with Kotlin Notebook
In the high-stakes world of competitive sailing, where every decision can mean the difference between victory and defeat, an extraordinary tool has emerged: Kotlin Notebook. Roman Belov, a distinguished member of the JetBrains team, shared a captivating account of leveraging this innovative technology to triumph in a 24-hour regatta. The narrative transcends a simple code demonstration, illustrating how interactive programming becomes a critical asset in a dynamic, unpredictable environment like the open sea.
This journey highlights the power of Kotlin Notebook as more than just a development tool; it’s a platform for real-time problem-solving. While a seasoned developer, Roman’s most cherished hat is that of a yachtsman. He uses the notebook to translate complex nautical challenges into actionable, data-driven decisions. The essence of the task is to navigate a course, which is essentially a graph with nodes representing different locations and edges representing the path between them. However, unlike a typical graph problem, the rules of sailing introduce complex variables. The boat cannot sail directly into the wind, and its speed is heavily dependent on the angle of the wind. This means the graph is constantly changing, making traditional route-planning algorithms obsolete.
The solution required a tool that could rapidly process data, visualize outcomes, and allow for on-the-fly adjustments. This is where Kotlin Notebook excelled, providing a live, interactive environment. Roman outlined how he could use the notebook to perform crucial tasks in the middle of the race: visualizing the race course on a map, calculating the fastest path based on current wind conditions, and dynamically adjusting the route as the wind shifted. This is achieved by creating a “sailable roads” model, which evaluates every potential path on the graph at regular intervals and discards any that are impossible given the wind direction. For the remaining paths, the notebook computes the optimal boat speed and time to complete that segment, effectively modeling the race in real time.
Roman then showcased the brute-force search algorithm that was used to find the optimal path. The code, written in Kotlin, was surprisingly straightforward and demonstrated the language’s elegance and readability. The algorithm, running within the notebook, would constantly iterate through the potential paths, calculating the time to finish for each one and discarding any that were slower than the best time found so far. The visual output of the notebook, which could render the different routes directly on the map, was a game-changer. It transformed abstract data and calculations into a clear, visual representation that allowed the sailors to make quick, informed decisions.
The application of Kotlin Notebook in this unconventional scenario proves its versatility beyond traditional data science or development tasks. It demonstrated how a tool designed for rapid experimentation can be applied to complex, real-world problems. The interactive nature of the notebook allowed Roman to combine data analysis, algorithm execution, and visual feedback into a single, cohesive workflow, enabling him and his crew to stay ahead of the competition and ultimately, win the race. This story is a testament to the power of a modern programming language and an adaptable toolchain, turning a challenging maritime endeavor into an exciting display of computational prowess.
Links:
- Roman Belov on JetBrains Blog
- JetBrains website
- Charts, Code, and Sails: Winning a Regatta with Kotlin Notebook | Roman Belov