{"id":31142,"date":"2025-05-21T05:00:00","date_gmt":"2025-05-21T03:00:00","guid":{"rendered":"https:\/\/sii.pl\/blog\/?p=31142"},"modified":"2025-05-22T15:27:40","modified_gmt":"2025-05-22T13:27:40","slug":"kan-a-revolution-in-neurons-a-new-generation-of-deep-learning-networks","status":"publish","type":"post","link":"https:\/\/sii.pl\/blog\/en\/kan-a-revolution-in-neurons-a-new-generation-of-deep-learning-networks\/","title":{"rendered":"KAN: a revolution in neurons \u2013 a new generation of deep learning networks"},"content":{"rendered":"\n

Artificial intelligence has seen rapid advancements, but the core unit of neural networks \u2013 the artificial neuron \u2013 has remained largely unchanged. The newly introduced Kolmogorov-Arnold Network (KAN) challenges this norm by rethinking how activation functions are used and learned. With its innovative structure, KAN offers improved efficiency, interpretability, and performance, potentially reshaping the future of deep learning.<\/p>\n\n\n\n

Fundamentals of classical neuron operation<\/strong><\/strong><\/h2>\n\n\n\n

The role of AI has rapidly expanded in our daily lives, influencing everything from personal assistants and recommendation systems to more advanced fields like healthcare diagnostics, autonomous vehicles, and natural language processing. Numerous architectures and layer types, such as convolutional layers and embedding layers, have been developed. However, despite these groundbreaking advancements, the fundamental unit that powers these networks \u2013 the artificial neuron \u2013 has remained largely unchanged since its inception.<\/p>\n\n\n\n

The core concept of the neuron, which is inspired by the biological neuron’s in the human brain, is still defined by a relatively simple mathematical equation that sums the input values, applies weights, adds a bias, and then passes the result through an activation function<\/strong>.<\/p>\n\n\n\n

This simplicity is key to the flexibility and power of neural networks, allowing them to be scaled up into deep architectures without changing the foundational building block. In essence, while the neural network landscape has evolved rapidly, the neuron at its heart remains the same.<\/p>\n\n\n\n

Below is a mathematical representation of an artificial neuron:<\/p>\n\n\n\n

\"A
A mathematical representation of an artificial neuron<\/figcaption><\/figure>\n\n\n\n
\"The
Fig. 1 The graphical representation of an artificial neuron with n inputs (source: Wikipedia<\/a>)<\/figcaption><\/figure>\n\n\n\n

Neural networks rely on activation functions to introduce nonlinearity, allowing them to solve complex problems. While weights are adjusted during training, biases and activation functions typically remain fixed. According to the Universal Approximation Theorem, even a single hidden layer with enough neurons can approximate any continuous function.<\/p>\n\n\n\n

However, complex tasks often require deep neural networks \u2013 multi-layer perceptrons (MLPs) \u2013which stack multiple layers of neurons to handle challenges like image recognition or language translation.<\/p>\n\n\n\n

Kolmogorov-Arnold theorem as inspiration<\/strong><\/strong><\/h2>\n\n\n\n

Recently, a new type of algorithm known as the Kolmogorov-Arnold Network<\/strong> was introduced, marking a significant advancement in neural network architecture. This novel approach shows strong potential to outperform traditional multi-layer perceptrons, offering new opportunities for improved performance and efficiency in solving complex tasks. Its innovative structure could help overcome some of the limitations of current deep learning models, potentially transforming the landscape of neural network research and applications.<\/p>\n\n\n\n

This new algorithm is inspired by the Kolmogorov-Arnold Representation Theorem, which states that any complex function with many variables can be decomposed into simpler functions, each depending on a single variable. This decomposition makes it easier to tackle complicated problems by focusing on one factor at a time.<\/p>\n\n\n\n

KAN \u2013 a new neural network architecture<\/strong><\/strong><\/h2>\n\n\n\n
\"KAN
Fig. 2 KAN \u2013 simplification of the complex relationship between feature and label (source: KAN: Kolmogorov-Arnold Networks<\/a>)<\/figcaption><\/figure>\n\n\n\n

In this algorithm, this concept is applied by breaking down a complex, nonlinear machine learning problem into smaller, more manageable components. Each custom activation function simplifies the complex relationship between the feature and label. This is shown in Figure 2 above.<\/p>\n\n\n\n

By summing the results of all these simplified functions, we obtain the final prediction for the overall problem. Mathematically, this can be expressed as:<\/p>\n\n\n\n

\"the
The final prediction<\/figcaption><\/figure>\n\n\n\n

In KAN implementation function \u03c8 is an identity operation \u03c8(x)=x.<\/p>\n\n\n\n

A more intuitive representation for a neural network would be in matrix form. The matrix below illustrates how a layer with n inputs would look in the Kolmogorov-Arnold Network (KAN) algorithm.<\/p>\n\n\n\n

\"matrix\"
Matrix<\/figcaption><\/figure>\n\n\n\n

Where for every input is specified an activation function \u03d5.<\/p>\n\n\n\n

Activation functions serve as learning parameters<\/strong><\/strong><\/h2>\n\n\n\n

In comparison to a standard MLP, where activation functions are fixed at the nodes and weights are the only learnable parameters on the edges, the Kolmogorov-Arnold Network (KAN) takes a different approach.<\/p>\n\n\n\n

In KAN, the activation functions themselves are learnable and are placed on the edges, while the nodes perform a sum operation on the outputs of these activation functions. This shift allows for more flexibility in how the network models complex relationships. <\/p>\n\n\n\n

The paper illustrates this concept using the graph shown in Figure 3.<\/p>\n\n\n\n

\"Fig.
Fig. 3 KAN \u2013 activation functions and nodes (source: KAN: Kolmogorov-Arnold Networks<\/a>)<\/figcaption><\/figure>\n\n\n\n

How to make a custom activation function?<\/strong><\/strong><\/h2>\n\n\n\n

Everything about this algorithm sounds logical and intuitive, but how can we estimate an activation function for every input and apply it multiple times in each layer? The answer to this question lies in B-splines. A B-spline (or Basis spline) is a flexible curve composed of several connected segments, which allows it to create smooth and complex shapes.<\/p>\n\n\n\n

In simple terms, a B-spline is a curve defined by control points, allowing it to be shaped in various ways to approximate different functions. One of its key advantages is that B-splines can be used for numerical differentiation, making them ideal for backpropagation when training a Kolmogorov-Arnold Network (KAN).<\/p>\n\n\n\n

A notable feature of B-splines is that adjusting a single control point only affects the local portion of the curve, leaving the rest unchanged. By manipulating these control points to fit the B-spline to the desired shape, we effectively train the network. These control points act as the primary trainable parameters, forming the core idea behind the Kolmogorov-Arnold Network.<\/p>\n\n\n\n

Below, on Figure 4, is an example of curve created by few control points.(graph from Wikipedia). More detailed informationan be found on Wikipedia<\/a>.<\/p>\n\n\n\n

\"Krzywa
Fig. 4 Curve formed by several checkpoints (source: Wikipedia<\/a>)<\/figcaption><\/figure>\n\n\n\n

So, with an understanding of the backbone idea of KAN, our artificial neural network would now have a slightly different shape. In MLP, our trainable parameters are weights, but now our trainable parameters are control points in B-splines.<\/p>\n\n\n\n

A single layer of a multi-layer perceptron (MLP)<\/strong> can be described using vectors and matrices:<\/p>\n\n\n\n

\"\"
A multi-layer perceptron (MLP)<\/strong> can be described using vectors and matrices<\/figcaption><\/figure>\n\n\n\n
    \n
  1. Input Vector: The input features form a vector x <\/code><\/li>\n\n\n\n
  2. Weight Matrix: The weights connecting the inputs to the neurons are represented by a matrix <\/code><\/li>\n\n\n\n
  3. Bias Vector: Each neuron has a bias, represented as a vector b <\/code><\/li>\n\n\n\n
  4. Linear Transformation: The input undergoes a linear transformation z=Wx+b <\/code><\/li>\n\n\n\n
  5. Activation Function: An activation function is applied to z, producing the output vector a=f(z)<\/code><\/li>\n<\/ol>\n\n\n\n

    The single layer for the KAN algorithm would be slightly different.<\/p>\n\n\n\n

    \"jedna
    Jedna warstwa dla KAN<\/figcaption><\/figure>\n\n\n\n

    Where \u03d5_mn are an activation function\u2019s. In paper[1]<\/a> the activation function has next definition<\/p>\n\n\n\n

    \"an
    An activation function\u2019s<\/figcaption><\/figure>\n\n\n\n

    Comparison of KAN and MLP: performance and interpretability<\/strong><\/strong><\/h2>\n\n\n\n

    From the formulas above, you can see that MLPs have fewer trainable parameters than KAN, which means KAN require more computational power and time to train.<\/p>\n\n\n\n

    However, the innovation in this new algorithm is that we can use fewer neurons and layers by optimizing not just the weights in a neuron, but also the activation function. During training, gradient-based optimization is used to adjust the positions of spline control points \u2013 just like weights in traditional networks. The paper describes how a KAN model with fewer layers outperforms standard MLP networks.<\/p>\n\n\n\n

    In addition to the smaller size and optimized activation functions, KAN offers a major advantage in interpretability. By inspecting the learned B-splines, we can gain insight into how the model is working, unlike MLPs, which operate as a complete black box. Another strong feature of KAN is its support for continual learning \u2013 during fine-tuning, the model retains knowledge from the original task. This is due to the property of B-splines, where adjusting one control point only affects the local area of the curve.<\/p>\n\n\n\n

    Application and practical potential<\/strong><\/strong><\/h2>\n\n\n\n

    The original KAN paper [1]<\/a> presents several examples across different tasks where the KAN model achieved higher test accuracy with fewer trainable parameters compared to traditional MLP models. For example, in the task of signature classification, the KAN model achieved an accuracy of 81.6%, while the traditional MLP model achieved 78% (Table 3 from [1]<\/a>).<\/p>\n\n\n\n

    Additionally, the paper demonstrates that in tasks requiring symbolic regression and the discovery of higher-order polynomial relationships (such as fitting data to complex functions), KAN networks consistently outperformed MLPs by better capturing these relationships without the need for deep architectures or excessive training time.<\/p>\n\n\n\n

    However, publications generally conclude that while KAN is not universally superior, it can outperform classical models in specific tasks and domains. A more comprehensive comparison was conducted in the KAN or MLP: A Fairer Comparison paper [4]<\/a>, where the KAN model’s performance was evaluated across machine learning (on eight datasets), computer vision, NLP, and audio processing tasks. The study showed that standard MLP models slightly outperformed KAN models, with accuracy differences ranging from 0.2% (machine learning tasks) up to 8% (computer vision tasks).<\/p>\n\n\n\n

    Furthermore, several follow-up studies have explored adaptations of KAN in specialized domains. One of them is that the Fourier KAN outperforms MLP on the Text Classification Head Fine-tuning paper [5]<\/a>, where a modified version of KAN called Fourier KAN (FR-KAN) was proposed. In this study, FR-KAN was used as an alternative to MLP-based text classification heads and demonstrated significant improvements, achieving an average accuracy increase of 10% and an F1-score improvement of 11% across seven pre-trained transformer models and four text classification tasks.<\/p>\n\n\n\n

    Currently, there are no known production-level deployments of KAN, but the model has sparked significant interest in the research community. Numerous studies have proposed modifications and adaptations of KAN, demonstrating its potential to achieve higher performance in specific use cases. However, these works also emphasize that KAN is not a one-size-fits-all solution \u2013 it requires careful tuning and thoughtful application to deliver superior results.<\/strong><\/strong><\/p>\n\n\n\n

    So far, KAN has shown the most promise in tasks involving symbolic reasoning, time series analysis, and function approximation.<\/p>\n\n\n\n

    \"\"<\/a><\/figure>\n\n\n\n

    Conclusions and the future of KAN<\/strong><\/strong><\/h2>\n\n\n\n

    The Kolmogorov-Arnold Network (KAN) introduces a fresh perspective on long-standing principles in neural networks, particularly the activation function, which has remained mostly unchanged for years. While the basic structure of the artificial neuron remains intact, KAN enables us to finely tune the activation function, allowing the network to better model complex relationships between features and labels. This flexibility could lead to improved performance in capturing intricate data patterns.<\/p>\n\n\n\n

    However, several important questions arise:<\/p>\n\n\n\n