Would there be any problem if I use an electrolytic capacitor instead of a ceramic one?
Electrolytics have much, much larger parasitic resistances. Or under more layman terms: Electrolytics are always "slower" than ceramic capacitors. However, you're using a Arduino Leonardo, which as a 1uF ceramic capacitor across VCC / 5V and GND. You "might" get lucky and the capacitor there will save you. Its very far away and might not work though.
Capacitors are basically considered the most important and numerous component on modern electronics. You should basically spam them to increase your reliability. They're a lucky-charm, many "mysterious" issues get solved with more capacitors, lol. Well, at least at the beginner levels.
Lots of 100nF capacitors (not too small, not too big) help out the majority of designs. Of course, an expert would run real power models and simulate the results. But a beginner should instead just place a 100nF capacitor next to every VCC-pin of every chip you use. Seriously.