Introduction

We can use Neural Networks not only for predicting the class of an image, completing sentences, or classifying sentiment of a paragraph of text. We can also use Neural Networks for solving scientific problems. In this article, you will learn about using a trained Neural Network to simulate the phase space of a Harmonic Oscillator (a weighted rigid pendulum). Here, there’s a twist. We will train the Neural Network not by calculating the loss between the outputs and the ground truth data, but we will train using the derivatives. We will leverage our Physics knowledge to train the Neural Network.

Prerequisites

I expect the reader to be knowledgeable about basics of training a Neural Network from scratch- using a library like PyTorch, Jax, etc. I will use PyTorch in this project. I expect no knowledge of Hamiltonian Dynamics or college-level Physics, but knowing the basics of Physics up to High School level. Being well-versed in Differential Calculus is required. Experience with Python is helpful. I also expect the reader to have read an earlier post on simulating a spring with Hamiltonian Mechanics and plotting the phase space as well as the trajectory: Modelling a Spring System in Hamiltonian Mechanics: Using Euler’s Method for Trajectory Plotting.

Objectives

We will model our system using Hamiltonian Dynamics. And using the Python function, we will plot the phase space of the system. And then using the the derivatives, we will train a Neural Network using those derivatives. Then we will use the trained Neural Network to again simulate the system and plot the phase space of the system. And we will see how the NN is doing. If you don’t know about Hamiltonian Dynamics or phase spaces, then that’s okay. I will cover what we need.

Code

All code used in this project is available on GitHub: ritog/harmonic.

Our Physical System

We will work with a rigid pendulum.

The related variables are:

• A body of mass on a rigid rod of length
• Angle ( is straight down)
• Angular momentum,
• Gravity

Defining the Hamiltonian of Our System

1. Potential Energy(): Height is , so,

How?

2. Kinetic Energy ():

Rotational kinetic energy is . For a point mass, moment of inertia, , and angular velocity, .

So, Kinetic Energy in terms of momentum,

Remember that, in Hamiltonian mechanics, the Hamiltonian of the system is:

And the Hamiltonian equations are:

If we calculate theses, we will get:

You can trust me, or check using a pen and paper. Remember that I said the same thing in the last article as well?

Implementing

This is how we implement this in Python. Using this function, we can find the derivatives, and using these, we can find the trajectory.

Here’s a basic implementation using NumPy:

def pendulum_dynamics(t, state: List, m, l, g) -> List:
    """
    inputs:
    t: The current time, solvers expect it.
    state: A list or array containing [q, p].
    m: mass
    l: the length of the rigid pendulum
    g: gravitational acceleration
    output:
    the list [dq_dt, dp_dt]
    """
    dq_dt = state[1] / (m * np.power(l, 2))
    dp_dt = -m * g * l * np.sin(state[0])
    return [dq_dt, dp_dt]

Or, we can write a version using torch. This will make things better.

def pendulum_dynamics_tensor(t, state: torch.Tensor, m, l, g) -> torch.Tensor:
    dq_dt = (state[:, 1] / (m * torch.pow(torch.tensor(l), 2))).unsqueeze(1)
    dp_dt = (-m * g * l * torch.sin(state[:, 0])).unsqueeze(1)
    return torch.cat([dq_dt, dp_dt], dim=1)

We can use the earlier function to plot the trajectory of the pendulum.

from pendulum_nonlinear import pendulum_dynamics

# different initial states
init_a = [0.5, 0]  # small release
init_b = [3.1, 0]  # close to top
init_c = [0, 5.0]  # close to bottom with force


# params
m = 1.0    # mass
l = 1.0    # length of rod
dt = 0.05  # time-step
g = 9.8    # gravitational acceleration



def run_simulation(init_state: List):
    p_vals = []
    q_vals = []
    for i in range(1_000):
        q, p = init_state
        _, dp_dt = pendulum_dynamics(t=dt, state=[q, p], m=m, l=l, g=g)
        p = p + dp_dt * dt
        dq_dt, _ = pendulum_dynamics(t=dt, state=[q, p], m=m, l=l, g=g)
        q = q + dq_dt * dt
        init_state = [q, p]
        q_vals.append(q)
        p_vals.append(p)
    return p_vals, q_vals

a_p_vals, a_q_vals = run_simulation(init_a)
b_p_vals, b_q_vals = run_simulation(init_b)
c_p_vals, c_q_vals = run_simulation(init_c)


# Plotting
plt.plot(a_q_vals, a_p_vals, label="$q_0=0, p_0=0$")
plt.plot(b_q_vals, b_p_vals, label="$q_0=3.1, p_0=0$")
plt.plot(c_q_vals, c_p_vals, label="$q_0=0, p_0=5.0$")
plt.xlabel("$q$")
plt.ylabel("$p$")
plt.title("$p v. q$ for different inital conditions")
plt.legend()
plt.tight_layout()
plt.show()

This is the plot that the code generates:

The Blue and green loops represent the pendulum swinging back and forth. It doesn’t have enough energy to go over the top, so it stays trapped in a closed loop. The Orange loop is right on the edge! If you had just a tiny bit more energy, the pendulum would stop swinging back and start spinning 360° continuously.

One of the reasons that Hamiltonian Neural Networks are better than vanilla Neural ODEs comes from Hamiltonian Mechanics. There’s a special property called Liouville’s Theorem, which says that the total area under the curve for the total set of initial points will be preserved for the total set of end points. That is, if you think of the initial points as a blob of points, then the blob might stretch, skew, twist, or distort, but the total area will remain constant. This is how Neural Networks handle data in higher dimension, and maps training data to target. I recommend that you watch Alfredo Canziani’s video from NYU CDS: 02 – Neural nets: rotation and squashing to get great a visual grasp of this. These kind of transformations are modelled and studied extensively in Linear Algebra.

This incompressible flow of points, and fluid-like behaviour (as opposed to gas-like, as gases compress and expand) are great for Neural Networks, and NNs trained using Hamiltonian mechanics are much more robust and well-behaved than simple NODEs.

I am not writing more about this here. Maybe in a future post I write more on this.

Hamiltonian Neural Network

With our Python function, we can generate the data from our knowledge of Physics.

With Deep Learning, we solve the opposite problem- going from data to the Physics.

Standard Neural ODEs try to learn the derivatives directly from the available data.

Input: , Output:

But, Hamiltonian Neural Networks are smarter. Instead of training the NN to predict the data, we force the NN to predict the Hamiltonian of the system.

Here, is the set of trainable parameters of the neural network.

We ask the Neural Network to output the Hamiltonian of the system. And, as the Neural Network is just a chain of differentiable math operations, if we calculate the gradients of the output ($), with respect to the variables and , then, what we have are predicted time derivatives of the Hamiltonian.

Here is our Neural Network:

import torch
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"


class HNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear_block = nn.Sequential(
            nn.Linear(2, 200),
            nn.Tanh(),
            nn.Linear(200, 200),
            nn.Tanh(),
            nn.Linear(200, 1),
        )

    def forward(self, x):
        H = self.linear_block(x)
        return H

There are two things to note in the code:

• There is no activation function at the end of the final Fully Connected (FC) layer. Because we want the Neural Network to output a value that is perceived as the total energy of the system, and it’s a real-numbered value, not limited to the range of the function.
• We chose the tanh() activation function. As the function is thoroughly differentiable at every point.

We want to feed a batch of 200 pairs of and to the NN.

Training the Hamiltonian Neural Network

We want to write vectorized code. We don’t want to bottleneck the model by feeding in data through naive for loops.

For that, we can write a function to get the derivatives of the model, in batches:

import torch
from HNN import HNN

device = "cuda" if torch.cuda.is_available() else "cpu"

def get_model_time_derivatives(model, x):
    """
    Compute time derivatives [dq/dt, dp/dt] for a batch of inputs.
    x: Tensor of shape (Batch_Size, 2)
    """
    H_hat = model(x)

    # sum the energy to get a scalar,
    # the gradients separate out perfectly per row
    grads = torch.autograd.grad(H_hat.sum(), x, create_graph=True)[0]

    # grads shape: (Batch, 2) -> [dH/dq, dH/dp]

    # flipping (Symplectic Swap (Hamilton's Eqs))
    # dq/dt =  dH/dp
    # dp/dt = -dH/dq

    dH_dq = grads[:, 0].unsqueeze(1)
    dH_dp = grads[:, 1].unsqueeze(1)

    return torch.cat([dH_dp, -dH_dq], dim=1)  # - because minus

There is a nice trick with the summing up of the predicted Hamiltonians. PyTorch can only find gradients of scalars. And here we have a tensor of predicted Hamiltonians. We can just sum them up, and then find the gradient with respect to the whole batch of the inputs. And everything gets neatly stored in rows. Gradient of a sum is equal to sum of gradients.

I am not going deep into it for now. I hope that you know why this is the case.

Here’s what the training script looks like:

import torch
from tqdm import tqdm

from HNN import HNN
from hnn_model_derivs import get_model_time_derivatives
from pendulum_tensor import pendulum_dynamics_tensor

device = "cuda" if torch.cuda.is_available() else "cpu"

# params
m = 1.0    # mass
l = 1.0    # length of rod
dt = 0.05  # time-step
g = 9.8    # gravitational acceleration

mean = torch.tensor([-3.0, 3.0])
std = 0.1
init_states = (6 * torch.rand(1_000, 2) - 3).to(device).requires_grad_()

true_derivatives = pendulum_dynamics_tensor(t=dt, state=init_states, m=m, l=l, g=g)

hamiltonian_nn = HNN().to(device)
loss_func = torch.nn.MSELoss()
optimizer = torch.optim.Adam(hamiltonian_nn.parameters(), lr=1e-2)

n_epochs = 1_200

for epoch in tqdm(range(n_epochs + 1)):
    deriv_pred = get_model_time_derivatives(hamiltonian_nn, init_states)
    loss = loss_func(deriv_pred, true_derivatives)

    optimizer.zero_grad()
    loss.backward(retain_graph=True)

    optimizer.step()
    if epoch % 100 == 0:
        print(f"Epoch: {epoch}\t Loss: {loss}")

torch.save(hamiltonian_nn.state_dict(), "hamiltonian_nn_1.pth")

Note the calculation of loss- loss = loss_func(deriv_pred, true_derivatives). We calculate the loss between predicted and true derivatives. You are usually accustomed to see the loss being calculated between y_pred and y. But the parameters of the model get updated through backpropagation, as the optimizer receives them: optimizer = torch.optim.Adam(hamiltonian_nn.parameters(), lr=1e-2).

After running this script, I had a loss of - which is great.

Note that I train the model derivatives to be close to the true derivatives - unlike normal NNs - where we train the model to output values close to the ground truth.

Throughout the training, and generating data, maintaining the graph of the computation is crucial.

Plotting the Trajectory as Predicted Using the Model

Now, I will plot the trajectory as solved from the derivatives predicted by the Neural Network. Here, we are using Semi-Implicit Euler’s Method to find points in the phase space. To learn more about this, read the previously mentioned post.

# Here, we have an already trained HNN
# We use it to plot trajectory
import torch
from matplotlib import pyplot as plt

from HNN import HNN
from hnn_model_derivs import get_model_time_derivatives

device = "cuda" if torch.cuda.is_available() else "cpu"

# different initial states
init_a = torch.tensor([0.5, 0]).to(device)  # small release
init_b = torch.tensor([3.1, 0]).to(device)  # close to top
init_c = torch.tensor([0, 5.0]).to(device)  # kicked from near bottom

# params
m = 1.0  # mass
l = 1.0  # length of rod
dt = 0.05  # time-step
g = 9.8  # gravitational acceleration

hamiltonian_model = HNN().to(device)
hamiltonian_model.load_state_dict(torch.load("hamiltonian_nn_1.pth", weights_only=True))


def run_simulation_HNN(init_state: torch.Tensor):
    p_vals = []
    q_vals = []
    for i in range(1_000):
        curr_state_tensor = (
            init_state.clone().detach().unsqueeze(0).requires_grad_(True)
        )
        derivs = get_model_time_derivatives(hamiltonian_model, curr_state_tensor)

        dq_dt = derivs[0, 0]
        dp_dt = derivs[0, 1]

        q_old, p_old = init_state

        p_new = p_old + dp_dt * dt
        q_new = q_old + dq_dt * dt

        init_state = torch.tensor([q_new, p_new]).to(device)

        q_vals.append(q_new.item())
        p_vals.append(p_new.item())
    return p_vals, q_vals


a_p_vals, a_q_vals = run_simulation_HNN(init_a)
b_p_vals, b_q_vals = run_simulation_HNN(init_b)
c_p_vals, c_q_vals = run_simulation_HNN(init_c)

# Plotting
plt.plot(a_q_vals, a_p_vals, label="$q_0=0, p_0=0$")
plt.plot(b_q_vals, b_p_vals, label="$q_0=3.1, p_0=0$")
plt.plot(c_q_vals, c_p_vals, label="$q_0=0, p_0=5.0$")

plt.xlabel("$q$")
plt.ylabel("$p$")
plt.title("$p  v. q$ for different inital conditions simulated via Hamiltonian NN")
plt.legend()
plt.tight_layout()
plt.savefig("FIG5.png")

Note that, we are not using a Python function for this plot, but load the trained weights from a .pth file: hamiltonian_model.load_state_dict(torch.load("hamiltonian_nn_1.pth", weights_only=True)).

This is the plot that we get:

For the inner loops (blue and green), The network did a decent job learning the “swinging” motion! It captured the concentric nature of the phase space near the center. The outer “loop” (orange): This trajectory starts at q=3.1. In our clean phase space figure, this was a closed loop (the “eye”). In our HNN simulation, it drifts off significantly.

This is due to the fact that the point was an out-of-distribution data point for the model.

Conclusion

We have trained a Neural Network’s parameters so that the model learns the Physics from the data, by making its gradients be close to the real derivatives. We saw that we can leverage our Physics knowledge in training of Neural Networks, and leverage Physical properties like Liouville’s Theorem to train well-behaved NNs with predictable behaviour.

@ONLINE {,
    author = "Ritobrata Ghosh",
    title  = "Training a Hamiltonian Neural Network",
    month  = "jan",
    year   = "2026",
    url    = "https://ritog.github.io/posts/hamiltonian_nn"
}

Introduction

Hamiltonian Mechanics provides us with ways to deal with dynamical systems computationally. We can simulate a system of a weighted spring using Hamiltonian mechanics. We can plot trajectories using the simulation. We will use Euler’s method and Semi-Implicit Euler’s method to plot phase spaces, and compare between them. In a future post, I will describe the training of a Hamiltonian Neural Network for learning the physics of a non-linear system, and simulate it.

Code

All code used in this project, as well as code that can be used to reproduce the project can be found on GitHub: ritog/spring_hamiltonian.

Prerequisites

I expect the reader to have some programming background in any imperative language. I will use Python. I expect no knowledge of Hamiltonian Dynamics or college-level Physics, but knowing the basics of Physics up to High School level is needed. Being well-versed in Differential Calculus is required. Experience with Python is helpful.

Objectives

We will model our system using Hamiltonian Dynamics. We will first write a function to get derivatives of two variables at each step. And using the Python function, we will plot the phase space of the system. For doing that we will need to integrate them using a DE solver - we will use Euler’s method. Then we will see if that suffices. If not, we will switch to implicit Euler, and plot the same phase space. If you don’t know about Hamiltonian Dynamics or phase spaces, then that’s okay. I will cover what we need. This is a trivial example for something I want to build up to- training a Hamiltonian Neural Network. But that’s not the objective of this article.

Our System

We have a mass attached to a spring of negligible mass.

And we are concerned with only two variables of the system- the position , and the momentum (). These are the only two variables needed to construct the Hamiltonian. And if we plot versus in a 2-dimensional figure, what we get is called a phase space.

Hamiltonian of the System

The Hamiltonian of the system is the total energy of the system. It is the sum of kinetic and potential energies of that system. And it is characterized by position and momentum of a particular body.

You are probably more familiar with Newtonian Mechanics where we are more concerned with force. Newton’s Second Law of Motion, , is a second-order differential equation, , where denotes the position of the body. Computational solvers work better with first-order differential equation, or a system of first-order DEs. If you would like to read a nice article on DEs and solving them computationally with Julia, read this post: Solving First Order Differential Equations with Julia, Hacker News discussion.

With Hamiltonian system, we get what computers are better at solving. Although that’s not why it was invented or used for a long time. With Hamiltonian mechanics, we have conservation laws baked in, and it can represent evolution of the system through time as a flow, and it provides many geometrical niceties which we will not delve (I know) deep into in this article.

The Hamiltonian of our system, is the sum of kinetic and potential energies of the system. And it is always conserved.

Hamiltonian, of our system:

The Hamiltonian is a function of and only.

The above equation can be re-written as:

As . And the position is simply rewritten as .

Hamiltonian Equations

The Hamiltonian Equations are:

If you have never seen partial derivatives before- they are simple if you know derivatives. In partial derivatives, the variable you are differentiating with respect to, is treated as a variable. All other variables, and constants are treated as constants.

Let’s say there is a function, then and . Even though and are variables, when we are differentiating with respect to one of them, we are treating the other as a constant.

Now, if we calculate the Right Hand Sides of 1 and 2 above, we get,

You either trust me, or pick up a pen and paper and work through them!

Now, look at the equations above. This is the core beauty of Hamiltonian mechanics. The change in through time is dependent on and the change in is dependent on .

Writing a Program

Now we can code up the Python function that will give us and . This is where we will get the true derivatives that will later guide our training, and will also help up simulate the system.

def harmonic_dynamics(t, state: List, m, k) -> List:
    """
    inputs:
    t: The current time, solvers expect it.
    state: A list or array containing [q, p].
    m, k: The mass and spring constant.
    output:
    the list [dq_dt, dp_dt]
    """
    dq_dt = state[1] / m
    dp_dt = -k * state[0]
    return [dq_dt, dp_dt]

This is a basic function that returns a Python array. We will not deal with PyTorch arrays or SIMD hardware for now.

This function returns terms analogous to velocities, but to plot the phase space, we will need something analogous to positions. And we need to integrate.

Euler’s Method

We could use something like scipy.integrate, but let’s implement our own solver in Python. We will implement a solver using the Euler Method.

You can read formally about Euler Method in any decent book covering DEs, but, put simply, it says that- “The state in the next moment is the current state plus a small step in the direction of the derivative.”

In Math:

Coding Things Up: The Dynamics of the System

Now, we can get the “positions” in the phase space, using the code below.

from harmonic import harmonic_dynamics

# params
m = 1.0  # mass
k = 1.0  # spring constant
q0 = 0   # initial position
p0 = 1   # initial momentum
dt = 0.1 # time-step

init_state = q0, p0
q_vals = []
p_vals = []

for i in range(0, 200):
    q, p = init_state
    dq_dt, dp_dt = harmonic_dynamics(t=dt, state=[q,p], m=m, k=k)
    q = q + dq_dt * dt
    p = p + dp_dt * dt
    init_state = [q, p]
    q_vals.append(q)
    p_vals.append(p)

We are taking the following values for the system:

• mass,
• spring constant,
• initial position,
• initial momentum, (started at the center with some momentum)

And for time-stepping, we are taking .

Now if we plot both q vs. p (phase space), and q vs. t, we will get:

Semi-Implicit Euler’s Method

We see that the q vs. p plot is spiralling outward with time. This is not the expected behaviour as the total energy is supposed to be conserved. We also see that this ideal version of the spring is swinging more and more as time goes on. This is not happening due to friction or gravity. They are not even considered here! And nothing is pumping energy into the system.

This is happening due to the limits of Euler’s Method. Imagine walking on a circular track. Euler’s Method says, “take a direction in the direction of the path- the tangent. If you take a straight step along the tangent of a circle, you land slightly outside the circle. Do this 200 times, and you spiral outwards.

To fix this, we need a Symplectic integrator- one that preserves the area in phase space (and thus behaves better with energy). The simplest version is Semi-Implicit Euler.

Instead of updating both q and p based on the old state, we update one, and then use that new value to update the other.

We do this:

• Calculate dp_dt using the current q.
• Immediately update p.
• Then, calculate dq_dt using this new, updated p.
• Update q using that.

Things will be clearer in code. We keep everything else intact. But we change the loop to this:

 for i in range(0, 200):
    q, p = init_state
    _, dp_dt = harmonic_dynamics(t=dt, state=[q, p], m=m, k=k)
    p = p + dp_dt * dt
    dq_dt, _ = harmonic_dynamics(t=dt, state=[q, p], m=m, k=k)
    q = q + dq_dt * dt
    init_state = [q, p]
    q_vals.append(q)
    p_vals.append(p) 

Doing this, if we plot again, we get:

Now, it is a well-behaved phase space with no spiralling. And the position vs. time plot is also a smooth sinusoidal wave, as expected.

Conclusion

We saw that, while simulating a weighted spring in Hamiltonian mechanics, the way we choose to integrate matters a lot.

This was a linear system. In a future post, I will simulate a non-linear system, and train a Hamiltonian Neural Network to simulate the system.

@ONLINE {,
    author = "Ritobrata Ghosh",
    title  = "Modelling a Spring System in Hamiltonian Mechanics",
    month  = "jan",
    year   = "2026",
    url    = "https://ritog.github.io/posts/implicit_euler"
}

Introduction

In this tutorial, we will learn about solving Differential Equations with Julia. Differential Equations are everywhere in Science and Engineering problems. And, being able to solve Differential Equations using computers is very convenient. We will use the DifferentialEquations.jl package for solving Differential Equations.

Who is this for

This is for people who are already familiar with Differential Equations from Mathematics, and who can code, preferably in Julia. Only the very basics of Julia language is required. If you are unfamiliar with Julia, and yet you have non-trivial experience with any of Python, C/C++, Ruby, JavaScript, golang, etc., you are fine, and you can follow along.

What we will do

In this tutorial, we will only focus on using the high level API of the DifferentialEquations.jl package, and we will learn, from scratch, how to translate linear first order Differential Equations to Julia, and how to solve them using the library. We will not learn about and code up numerical methods of solving Differential Equations such as Euler’s method, Runge-Kutta methods, etc. Neither we will talk about testing for accuracy, testing convergence of our algorithms. We will simply and plainly have some DEs in our familiar mathematical notation, and then translate them to a form Julia and the library can understand, and then we call high level APIs of the DifferentialEquations.jl library to solve our DEs. That’s it.

And for this post, we are limiting ourselves to First Order Differential Equations.

What are Differential Equations: a very quick recap

Numerical Solutions of Differential Equations

There are some situations where finding the exact solution to a Differential Equation is not possible, or is impractical. This is why we try to find approximate solution. In many situations, this is good enough. When an analytical solution is impossible, numerical solutions is all we have.

First Example: Radioactive Decay

Radioactivity is the phenomena in which an unstable nucleus loses energy by radiation.

In radioactive decay, if a sample of radioactive nucleus is kept, through several kinds of radiation, the original sample reduces in mass. If there were number of nuclei originally, after a certain amount of time, there would be number of nuclei, where .

We know that the amount of decay is directly proportional to the number of nuclei present in a sample.

The sign is negative, because, the rate decreases with decreasing number of nuclei.

We will reframe this equation so that we can implement the equation in Julia, and solve it using the solver.

Julia package DifferentialEquations.jl expects functions to be in this format-

f(du, u, p, t) = p * u

Where du is the first derivative, u is the function, t is the time-span for which we want the function’s output, and p is the parameter, or a set of parameters.

From here, we can frame du to be du[1] = -lambda * u[1].

import Pkg
Pkg.add("DifferentialEquations")
Pkg.add("Plots")

using DifferentialEquations, Plots

   Resolving package versions...
  No Changes to `~/.julia/environments/v1.11/Project.toml`
  No Changes to `~/.julia/environments/v1.11/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.11/Project.toml`
  No Changes to `~/.julia/environments/v1.11/Manifest.toml`

function radio_decay!(du, u, p, t)
    lambda = -p
    du[1] = lambda * u[1]
end

radio_decay! (generic function with 1 method)

N = [60.0e18]           # initial condition

# parameter
lambda = 2.28e6         # in SI unit, s^{-1}
p = lambda

# time-span
tspan = (0.0, 5e-6)     # time-span is very small because the half-life is very small

(0.0, 5.0e-6)

prob = ODEProblem(radio_decay!, N, tspan, p)

ODEProblem with uType Vector{Float64} and tType Float64. In-place: true
Non-trivial mass matrix: false
timespan: (0.0, 5.0e-6)
u0: 1-element Vector{Float64}:
 6.0e19

sol = solve(prob)

retcode: Success
Interpolation: 3rd order Hermite
t: 17-element Vector{Float64}:
 0.0
 4.3859649122807024e-7
 6.41342034530829e-7
 9.724902283236061e-7
 1.2462761785169967e-6
 1.5654877939700708e-6
 1.8726226804630439e-6
 2.196788921416817e-6
 2.5203638024729064e-6
 2.8511887164148543e-6
 3.183703238312926e-6
 3.519781092460714e-6
 3.857529202669002e-6
 4.19723875341719e-6
 4.538177668884617e-6
 4.880273689241893e-6
 5.0e-6
u: 17-element Vector{Vector{Float64}}:
 [6.0e19]
 [2.2085932679488193e19]
 [1.3911043910467977e19]
 [6.538685467576463e18]
 [3.5026563834049787e18]
 [1.6917664046442598e18]
 [8.399157113217204e17]
 [4.011209837867301e17]
 [1.9182281652161018e17]
 [9.022998274997682e16]
 [4.227947963649966e16]
 [1.9650908828375188e16]
 [9.098778059095644e15]
 [4.194139573993325e15]
 [1.927908419090074e15]
 [8.838633947171844e14]
 [6.727182427233061e14]

plot(sol, title="Radioactive Decay", xlabel="Time", ylabel="N")

Second Example: Newton’s Law of Cooling

As we have seen in an example before, we know that something cools much faster when the difference between the temperature of that object and the surrounding temperature is comparatively higher. When the difference is lower, it takes a higher amount of time to cool down.

The problem was quantified by Isaac Newton himself. Let’s understand it.

The temperature of an object, varies with time, , and it depends on the difference of the temperature of the object and its surroundings- the ambient temperature .

where is a constant of proportionality, the value of which depends on the material.

Now we will implement this DE in Julia, and solve it using DifferentialEquations.jl library.

function newtons_cooling!(du, u, p, t)
    k, T_a = p
    du[1] = k * (u[1] - T_a)
end

newtons_cooling! (generic function with 1 method)

f(du, u, p, t) = p * u

T = [80.0]                # Temperature of object
k = -0.06                 # k, the cooling constant, negative, because cooling
T_a = 20.0                # the ambient temperature
tspan = (0.0, 300.0)      # time-period for which we want to tract the values of the function, i.e. temperature of the object
p = (k, T_a)              # defining a tuple of parameters as expected by the library

(-0.06, 20.0)

prob = ODEProblem(newtons_cooling!, T, tspan, p)

ODEProblem with uType Vector{Float64} and tType Float64. In-place: true
Non-trivial mass matrix: false
timespan: (0.0, 300.0)
u0: 1-element Vector{Float64}:
 80.0

sol = solve(prob)

retcode: Success
Interpolation: 3rd order Hermite
t: 19-element Vector{Float64}:
   0.0
   0.18593390178623953
   2.045272919648635
   6.383848011370185
  11.959325821307548
  18.73361986647917
  26.74893491369098
  35.99530039828558
  46.545554755847775
  58.51808266239741
  72.13894261696798
  87.73678833989489
 105.79306069340441
 126.99454945496298
 152.3564523878484
 183.42977158769716
 222.68158609043894
 273.9980369813687
 300.0
u: 19-element Vector{Vector{Float64}}:
 [80.0]
 [79.33435782064105]
 [73.07086984267411]
 [60.9075122329383]
 [49.276503335091164]
 [39.49832238666797]
 [32.054188527309826]
 [26.921535861341653]
 [23.675325765485024]
 [21.792012228502433]
 [20.79153270072668]
 [20.31058542718253]
 [20.10523457704901]
 [20.029607729780903]
 [20.006573373974568]
 [20.0011132329579]
 [20.00018200375497]
 [20.000082850038083]
 [20.00001776796253]

plot(sol,
    xlabel="Time, t",
    ylabel="Temperature, T",
    title="Temperature vs. Time")

Conclusion

We have seen the very basics and intuitive explanation of First Order Differential Equations, and learned about how DifferentialEquations.jl solves DEs, and how to frame DEs according to the format expected by that library.

Acknowledgements

I learned a lot about the content of this tutorial from- some amazing teachers that I had in college, and some amazing textbooks like Mathematical Methods for Physics and Engineering by Riley, Hobson, Bence and Mathematical Methods for Physicists by Arfken, Weber, and Harris. I learned about the DE solver package from many content from the creator: Chris Rackauckas, and some example videos from this YouTube channel- doggo dot jl.

Discuss

If you have read this post, and found it interesting or edifying, please let me know. I would like that very much. If you have any criticism, suggestion, or want to tell me anything, just add a comment or let me know privately. Discuss this post on the Fediverse, Hacker News, or Twitter/X.

Changelog

This is an Open Source blog. Feel free to inspect diffs in the GitHub repo.

Cite this Article

@ONLINE {,
    author = "Ritobrata Ghosh",
    title  = "Solving First Order Differential Equations with Julia",
    month  = "mar",
    year   = "2025",
    url    = "https://ritog.github.io/posts/1st-order-DE-julia/1st_order_DE_julia.html"
}

Support

I highly appreciate donations. If you liked this post, or the content in the site in general, support me via Ko-Fi, Liberapay or if you are in India, via UPI ritogh8@oksbi.

Introduction

In this Notebook, trajectories of the Logistic Map [1] is plotted with Python.

import numpy as np
import matplotlib.pyplot as plt

def logistic_function(parameter: float, initial_value: float) -> float:
    return parameter * initial_value * (1 - initial_value)

def logistic_function_trajectory(parameter: float,
                                 initial_value: float,
                                 num_iter: int) -> np.array:
    trajectory = np.zeros(num_iter)
    trajectory[0] = initial_value
    for i in range(num_iter-1):
        trajectory[i+1] = logistic_function(parameter, initial_value)
        initial_value = trajectory[i+1]
    return np.array(trajectory)

logistic_function_trajectory(parameter=2.2, initial_value=0.8, num_iter=10)

array([0.8       , 0.352     , 0.5018112 , 0.54999278, 0.54450159,
       0.54564314, 0.54541675, 0.5454621 , 0.54545303, 0.54545485])

xs = np.linspace(1, 200, 200)
ys = logistic_function_trajectory(parameter=2.2, initial_value=0.8, num_iter=200)

def plot_trajectory(xs, ys, x_label, y_label, title):
    plt.figure(figsize=(20,10))
    plt.scatter(xs, ys, marker='o', edgecolor='black', c=point_colors * 50);
    plt.xlabel(x_label);
    plt.ylabel(y_label);
    plt.title(title);
    plt.grid();

point_colors = ['cyan', 'magenta', 'yellow', 'black']

plot_trajectory(xs=xs, ys=ys, x_label='$n$', y_label='$x_n$', title='Trajectory')

Plotting vs.  for

x_n = logistic_function_trajectory(parameter=3.4, initial_value=0.2, num_iter=200)
x_n_hat = logistic_function_trajectory(parameter=3.4, initial_value=0.200001, num_iter=200)

ys = abs(np.subtract(x_n, x_n_hat))
xs = np.linspace(1, 200, 200)

plot_trajectory(xs=xs, ys=ys, x_label='$n$', y_label='$ x_n - \hat{x_n} $', title='Trajectory for $r = 3.4$')

We can thus plot the trajectories of the Logistic Map with Python and relevant libraries.

References

1. https://en.wikipedia.org/wiki/Logistic_map
2. Bradley, Liz: Nonlinear Dynamics: Mathematical and Computational Approaches via Complexity Explorer, Santa Fe Institute

Cite this Notebook:

@ONLINE {,
    author = "Ritobrata Ghosh",
    title  = "Non Linear Dynamics Trajectories with Python",
    month  = "apr",
    year   = "2023",
    url    = "https://www.kaggle.com/truthr/non-linear-dynamics-trajectories-with-python"
}

Introduction

There are many places that teach you complex number like High School, college classes, YouTube channels, and so on.

But they earlier ones often heavy on solving problems by-hand, and almost every time, they do only that.

The latter ones focus mainly on shiny visuals or advanced properties that are not intended for the uninitiated.

This is a piece written for both people who are getting introduced to Complex Numbers for the first time, and for the people who have studied it in college, and are hazy on the topic. (Why won’t they be? They had to focus on solving boring problems by hand.)

When I saw Jeremy talk about Complex Numbers on the APL study group, I knew I had to write something- I did my Junior year Math paper on Complex Number in High School!

The idea is to:

• quickly introduce Complex Numbers, and how to work with them
• talk about their origins: why need complex numbers in the first place?
• show how they extend the number line
• show how works as an operator, and briefly what operators are
• talk about a real world example, and probable future usage

I will keep it short and crisp.

Comples Numbers

Complex Numbers, as you might have seen are written like this:

Here, the whole thing in the right-hand side is the number. a is the Real part, and b is the imaginary part.

Do not get misguided by the names. Imaginary numbers aren’t imaginary at all. They make appearance, or rather, they can represent stuff in real life really well! We will soon see.

You can add and subtract complex numbers. You add (or subtract) the real and imaginary parts separately.

Let me show you.

They can be multiplied and divided with real numbers and other complex numbers. They work just as you expect them to.

Let’s not spend too much time here.

As how they behave, and more related information can quickly be found- and good ones at that- on internet.

I suggest the Khan Academy course on Complex Numbers.

Oh, I haven’t told you the value of yet.

, is defined such that,

That’s it!

Why do we Need i, and the Whole Complex Number Scenery

The story goes, when the Greeks were ridden with famines and other troubles, the Oracle of Delphi told them to double the height of the cube made in honor of Apollo.

The Greeks could not do it! How could they?

To double the size of a cube, you need to find the cube root of 2. The Greeks didn’t know how to that!

Too bad.

You are probably familiar with the story that Pythagoras, frustrated with the fact that the the size of the diagonal of a square with the side of 1 could not be measured, ordered to off someone! Just like that!

Because the square root of 2 was not measurable then. It is not a number that you write down like the others.

Human beings needed something that could be tamed- numbers that always had roots- no matter what the number was!

This is how it came to be. Complex numbers are numbers that always has roots. No matter what.

And with the advent of imaginary numbers, it was possible to calculate the square root of negative numbers, too. But that came later.

Extending the Number Line

The number line is a simple animal.

It is one-dimensional. It extends both ways of zero.

And thus, it was limiting.

Complex numbers was defined in such a way that they added one extra dimension to the number line.

It wasn’t so one-dimensional anymore.

Picture: Wikipedia

Now, numbers weren’t limited to a line, but it was a plane.

And many more operations became possible that simply weren’t before.

i as an Operator

i is not simple. Adding it or multiplying it has consequences.

If you don’t know yet, very simply, operators are stuff that act on other stuff.

(No mathematical rigor whatsoever can be expected from the previous line)

You know operators. We are talking about it in APL. We use them whenever we code.

Functions are also operators.

Think, for example, about the - operator.

This is our friendly negative sign. But add it before a positive number, and the result will be a negative number. The number will no longer remain positive.

Think about the + operator in math. It takes two numbers, not one. And the result is the sum of both of them.

Operator takes stuff in it, and spits out results.

i is also an operator.

Multiplying a number with i rotates the number by 90 degrees counter-clockwise in the complex plain.

Rotate it twice, and you get the negative of a number. Because, you know, i is the square of -1. And multiplying a number with square of i negates it, i.e. rotates the number by 180 degrees, or throws it to the dark, ehm, negative side.

This has important consequences.

Example of Use of Complex Number in Real World: Electronics

The scope of this piece is very limited.

But I will show you one example.

Alternating Current is not steady, or fixed like Direct Current or DC. AC current, the one powers our home is the only practical form of electricity that can be practically transported from one point to another over a long distance.

This form of electricity acts like a wave.

And whenever waves are involved, we have to deal with angles, frequencies, and amplitudes.

When angles become involved, we can use trigonometric functions like cosine and sine.

And complex numbers are related to trigonometric functions via the Euler’s Formula:

And, when we can use the special number, e, we can solve many problems easily because the function returns itself after differentiation and integration.

That makes a lot of the things easier.

While this is the reason for using complex numbers in electronics in the first place, they also make life easier for easier calculation and quantity manipulation.

Why Bother: Isn’t One More Dimension Enough?

You might think that complex numbers are such nuisance, and we could have done the same clever things with adding one more dimension to the real number like we do in geometry. Would not a ‘y’ suffice instead of this dimension riddled with ‘imaginary numbers’?

The answer is no.

You will not have the same behavior then.

You won’t be able to get the niceties that is afforded to you via Euler’s formula and the access it provides you to the exponential function.

I hope I was able to give you some intuition about complex numbers, and their uses.

Please let me know what you think.

Subscribe to my substack to get posts on your inbox, or subscribe via RSS.

Introduction

Natural Language Processing is a fast-advancing field. And it is also one of the fields that require a huge amount of computational resources to make important progress. And although breakthroughs are openly announced, and papers are released in free-to-access repositories such as arXiv, Open Review, Papers with Code, etc., and despite (sometimes) having the code freely available on GitHub, using those language models is not something widely accessible and easy.

Let me provide more context. BERT is a state-of-the-art encoder language model. It takes days to train the model from the ground up even when using very powerful GPUs that only a few entities have access to. In 2019, NVIDIA used 1472 NVIDIA V100 GPUs to train BERT from scratch in 53 minutes. Yes, 1,472!

One estimate puts the cost of training GPT-3, a 175 billion parameter model, for a single training run at $12 Million USD.

And such language models are released every now and then. How do you use these powerful language models for your task?

Here Hugging Face comes to the scene. They aim to solve this problem by providing pre-trained models, and simple API so that you can use them, fine-tune them, and use the API in your applications.

In this article, my goal is to introduce the Hugging Face pipeline API to accomplish very interesting tasks by utilizing powerful pre-trained models present in the models hub of Hugging Face.

To follow through this article, you need not have any prior knowledge of Natural Language Processing. I, however, assume minor prior experience in writing Python code.

In this article, I will go over, describe, and provide examples for the following tasks using Hugging Face pipeline-

• Sentiment Analysis
• Zero-Shot Classification
• Text Generation
• Mask-Filling
• Named Entity Recognition
• Question Answering
• Summarization
• Translation

Installing and Importing

! pip install transformers[sentencepiece] > /dev/null

If you have not seen a pip install with a square bracket in it before, don’t worry. It means that you are installing the transformers package with extra support added for the sentencepiece package. You can find out more about square parentheses installs in pip here.

from transformers import pipeline

We will use the utilities available in the pipeline namespace in the transformers library.

Sentiment Analysis

Before I begin going through the specific pipelines, let me tell you something beforehand that you will find yourself. Hugging Face API is very intuitive. When you want to use a pipeline, you have to instantiate an object, then you pass data to that object to get result. Very simple! You are soon to see what I mean.

classifier_sentiment = pipeline("sentiment-analysis")

That’s it. You call the pipeline() method with the task you want to accomplish as an argument. And you assign a name to it. You are done now. You can now begin to use the object as a function to achieve what you want. Let’s see an example-

See, it returns a dictionary contained in a list that has two items, label and score. The label part tells us its prediction, and the score tells us its confidence score.

As an aside, I think they are structured this way because this structure is easily compatible with .json and similar filetypes which are very common in APIs.

Let’s see another example.

See, how the classifier is aware of my emotion about pineapple pizzas? 🙃

Zero-Shot Classification

When you want to classify something using Deep Learning, in many cases you need to train it with labeled examples. This approach is known as Supervised Learning. Even when leveraging transfer learning, you need to train your model with quite a few labeled examples in the domain of your choice. Zero-shot classification is different than that. In this, you use a pre-trained model to classify a given string and some labels of your choice. The model returns you the confidence score for each model.

Let me show you how this works. First, you instantiate an object and assign it a name just like you did in the case of the Sentiment Analysis pipeline.

classifier_zero_shot = pipeline("zero-shot-classification")

Then you pass a string, along with the labels of your choice to test how well they correspond to your sentence.

See, the given sentence is about a profession, and the model tells you that. And it knows that it is much more related to education than politics.

Essentially, when you are doing Zero-Shot Classification, you are supplying a string to the pipeline, and also labels. The pipeline returns how accurate those labels are.

Text Generation

Before I heard about Deep Learning, I heard about Natural Language Processing, because I heard that you can generate text with it! This is the most exciting part of NLP to me, personally.

As the name suggests, this pipeline lets you generate text. It just needs you to supply a prompt, a text that will determine what the generated text will be. Let me show you how.

generator = pipeline("text-generation")

See what I mean! Even with zero training, it can create logically coherent text that is even interesting to read. Robots will take over writers soon 🤖!

You can control the maximum length of the output of the pipeline, and even ask for multiple possible outputs, like so-

Whoa, it’s poetic!

We can accomplish more interesting things with transfer learning. But that is a story for another day.

Mask Filling

In this pipeline, if there is a word that you hide and supply the string with the hidden word, the pipeline will predict that word. This is like magic!

unmasker = pipeline("fill-mask")

You pass the sentence with the masked word, and it will predict it. You can choose the number of predictions you want to see. You just have to pass a value to the top_k parameter.

It can predict the words quite well. Just remember to mask the word using <mask>.

Named Entity Recognition

This pipeline recognizes the proper nouns in your sentence. And it also classifies them. Let’s see an example.

ner = pipeline("ner", grouped_entities=True)

The grouped_entities parameter just ensures that the pipeline can recognize the names that have more than one word, like, you know, Chocolate Factory.

See how it recognizes a person, his profession, and location on its own, and also labels them as “PER”, “ORG”, and “LOC” (meaning location), respectively.

Question Answering

This pipeline provides you with an answer to a question you ask from the context that you provide.

question_answerer = pipeline("question-answering")

It can answer multiple questions from the same prompt.

Summarization

This is one of the most practical tasks that has existed for a long time. Deep Learning really changes the landscape of the task.

Let’s see it in action.

summarizer = pipeline("summarization")

Let’s summarize a piece of text from a blog post I recently read.

Now, that’s something! The efficacy of the summarizer really shocked me. It does this task so well.

Translation

The translation pipeline translates a piece of text from one language to another. It also lets you choose the language model of your choice.

First, let’s translate a text from English to German. I will use the Helsinki-NLP/opus-mt-en-de model for this task.

translator_de = pipeline("translation", model="Helsinki-NLP/opus-mt-en-de")

Note that you are not explicitly telling the model about the language of the input language or the output language. A language model is only trained for translating from one language to another. It cannot translate other languages. If you want to translate from German to English, you’d have to use another model from the model hub. You can find all the translation models here.

The cool thing is that Hugging Face is not limited to Romance languages or European languages in general. Let me show you by translating a piece of text from Bengali.

translator_bn = pipeline("translation", model="Helsinki-NLP/opus-mt-bn-en")

Conclusion

I have shown you several pipeline APIs and what they do and how they work.

Here are the suggested next steps-

• Experiment. Play with the pipelines by yourself, try different things with different parameters, run inference on examples of your own, test edge cases, run wild with it. I believe that is the best way to learn.
• Do Projects. Use the knowledge you gained here in simple projects. We only learn by doing.
• Hugging Face Course. Hugging Face has recently released a free course where they teach you about Transformers and their library ecosystem. It is a good next step.
• Explore. Explore the Hugging Face website, learn more about their APIs, Services, pre-trained models, and many more things it has to offer.

If you notice any error, mistake, or typo, please feel free to let me know. If something is not clear, please let me know that as well. I will do my best to correct it and will credit you here.

Let me know if you have any questions at all.

(Originally published as a Notebook on Kaggle.)

Introduction

Digital electronics is probably the most impactful and widespread technology now. It forms the basis of all of the chips- from ARM microcontrollers to processors in cellphones, all the way to uberpowerful AMD Threadrippers. In all these cases, digital electronics reign supreme.

It is very useful to learn not only about basic principles but also how these principles and the components of Digital Electronics exactly form the basis of the modern world, i.e., how do we go from basic logic gates to fully functional computers.

Elements of Computing Systems

Digital Electronics is not only important because it is the base of the modern technology that literally governs our lives, but it makes us better as programmers, and computer scientists.

Whether we are self-taught programmers or a person with a 4-year CS degree, we were never taught how we go from basic logic gates to CPUs.

A book that I have come across- The Elements of Computing Systems by By Noam Nisan and Shimon Schocken does just that. It teaches you how the basic building blocks of modern computers actually create the computers.

[Although it shows the new 2nd edition, I am reading the first one]

It promises to teach you, actively, how we can start from nothing but a NAND gate and go all the way to a full-fledged computer with an operating system, and a general-purpose programming language that runs on it, which can be used to create anything. The book is also known as nand2tetris.

I have started reading this book and working through it every Sunday, and I have almost finished a chapter.

Building Gates from Elementary Gates

People who have taken a Digital Architecture class or a Digital Electronics class will know that NAND (NOT AND) gates are called “universal gates” because some combination of them are able to create any other gates. So, when you have a NAND gate with you, you can create a general-purpose computer with it. And the book makes you do that. It makes you create many logic gates starting from AND, NOT, and OR with nothing but a pre-implemented NAND gate.

So, a NAND gate is already implemented and supplied. You first have to create AND, OR, and NOT gates. Then you go on to further create XOR gates, Multiplexors (MUX), Demultiplexors (D-MUX), and multi-bit versions of these gates and basic gates with what you have built previously, viz. AND, OR, NOT, and NAND gates.

This is a fascinating task in itself, and very intellectually stimulating.

This is where the core of the post comes in.

I will show you how to implement any Boolean function with nothing but three logic gates. This is a trick worth knowing.

Let’s start.

Implementing a Boolean Function

// Custom.hdl
// this is a part of Hashnode blog Convoluted demo

CHIP Custom {
    IN a, b, c;
    OUT out;

    PARTS:
    Not(in=a, out=nota);
    Not(in=b, out=notb);
    Not(in=c, out=notc);
    And(a=nota, b=notb, out=notanotb);
    And(a=notanotb, b=notc, out=w1);
    And(a=a, b=notb, out=anotb);
    And(a=anotb, b=notc, out=w2);
    And(a=anotb, b=c, out=w3);
    And(a=a, b=b, out=ab);
    And(a=ab, b=notc, out=w4);
    Or8Way(in[0]=w1, in[1]=w2, in[2]=w3, in[3]=w4, out=out);
}

// Custom.tst
// test file for testing custom chip
// could be found at- https://gist.github.com/ghosh-r/c4e6f5ceb1e7ea2e3ba3601c9de121be

// test file for a custom chip in Convoluted, a Hashnode blog

load Custom.hdl,
output-file Custom.out,
compare-to Custom.cmp,
output-list a%B3.1.3 b%B3.1.3 c%B3.1.3 out%B3.1.3;

set a 0,
set b 0,
set c 0,
eval,
output;

set a 0,
set b 0,
set c 1,
eval,
output;

set a 0,
set b 1,
set c 0,
eval,
output;

set a 0,
set b 1,
set c 1,
eval,
output; 

set a 1,
set b 0,
set c 0,
eval,
output;

set a 1,
set b 0,
set c 1,
eval,
output;

set a 1,
set b 1,
set c 0,
eval,
output;

set a 1,
set b 1,
set c 1,
eval,
output;

// Custom.cmp
// compare file for chip Custom.hdl
// found at- https://gist.github.com/ghosh-r/c4e6f5ceb1e7ea2e3ba3601c9de121be
// test file at- https://gist.github.com/ghosh-r/cef52b9f6ac017e00d64460b025a53fe

|   a   |   b   |   c   |  out  |
|   0   |   0   |   0   |   1   |
|   0   |   0   |   1   |   0   |
|   0   |   1   |   0   |   0   |
|   0   |   1   |   1   |   0   |
|   1   |   0   |   0   |   1   |
|   1   |   0   |   1   |   1   |
|   1   |   1   |   0   |   1   |
|   1   |   1   |   1   |   0   |

// Xor.hdl

CHIP Xor {
    IN a, b;
    OUT out;

    PARTS:
    Not(in=a, out=nota);
    Not(in=b, out=notb);
    And(a=a, b=notb, out=w1);
    And(a=nota, b=b, out=w2);
    Or(a=w1, b=w2, out=out);
}

Conclusion

I showed you how to implement any Boolean function with nothing but three elementary logic gates. However, you should keep in mind that this approach is impractical when there are more variables, and the chip you want is more complicated.

Follow the blog to get more similar posts.

Follow me on Twitter or connect with me on LinkedIn.