JavaScript Moving transition without the use of CSS

I'm using a small script to follow the cursor with a div element. This script makes the element strictly follow the cursor. What I'm trying to do is to add some kind of duration to the process of "following" the cursor. I tried CSS transitions but the animation always ended up breaking. Can somebody please help me with this?

Let's say mouse is somewhere, and then it changes position by around 100px. I want to specify the duration like if i was using CSS... But the thing is that I can not use any transitions but only some javascript magic instead...

document.body.addEventListener("mousemove", function(e) {

  var curX = e.clientX;
  var curY = e.clientY;

  document.querySelector('mouse').style.left = curX - 10 + 'px';
  document.querySelector('mouse').style.top = curY - 10 + 'px';

});

body {
  background: #333;
  height: 500px;
  width: 500px;
}
mouse {
  display: block;
  position: fixed;
  height: 20px;
  width: 20px;
  background: #fff;
  border-radius: 50%;
}

<body>
  <mouse></mouse>
</body>

I was wondering how to add a transition without using the CSS but I'm not the most advanced when it comes to JavaScript.

[edit] : I don't wanna use window.setTimeout.

[edit 2] : I wanted to use transition: 0.1s; but as I said it broke the effect when user moved the mouse too quickly.

Solution

There's a whole bunch of ways to do this, as you can see in the other answers, each with its own "feel". I'm just adding one more, where the dot approaches the cursor by a percentage of the remaining distance.

let curX = 0, curY = 0, elemX = null, elemY = null;
document.body.addEventListener("mousemove", function(e) {
  curX = e.clientX;
  curY = e.clientY;
  if (elemX === null) [ elemX, elemY ] = [ curX, curY ];
});

let amt = 0.1; // higher amount = faster tracking = quicker transition
let elem = document.querySelector('mouse');
let frame = () => {
  requestAnimationFrame(frame);
  elemX = (elemX * (1 - amt)) + (curX * amt);
  elemY = (elemY * (1 - amt)) + (curY * amt);
  elem.style.left = `${elemX}px`;
  elem.style.top = `${elemY}px`;
};
frame();

body {
  position: absolute;
  background: #333;
  left: 0; top: 0; margin: 0; padding: 0;
  height: 100%;
  width: 100%;
}
mouse {
  display: block;
  position: absolute;
  height: 20px; margin-left: -10px;
  width: 20px; margin-top: -10px;
  background: #fff;
  border-radius: 50%;
}

<body>
  <mouse></mouse>
</body>