SQLITE3 Raspberry Pi Problems and Solutions

SQLITE3 on Node-Red on Raspberry Pi

This is the November 27, 2020 update of an old, outstanding problem for Raspberry Pi owners who use and regularly upgrade Node-Red and who simultaneously use the Node-Red SQLITE node. The issue has also been known to affect the serial port and i2c on upgrading.

I KNOW I’m not the only one who has had issues with SQLITE3 upgrading in Raspberry Pi and Node-Red, if it happens to you… here is some invaluable info…

After a recent, successful upgrade to Node-Red 1.2.5 (leaving the SQLITE node slightly behind as it can be painful to update when there are node-js upgrades involved), I was feeling quite perky when along came NR 1.2.6 – which completely trashed my setup.

DaveCJ and others on the Node-Red DISCOURSE channel helped me yesterday without which I’d have been stuck on NR 1.2.5 possibly for a long time. The NR script you see here for upgrading Node-Red suffered a node-js upgrade-related issue which could cause failure. On the morning of November 27, 2020 this was FIXED.

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

These days whenever I upgrade Node-Red I use that script above ( a link to which is also included in my own “The Script”) and I subsequently (no reboot required) use the 4 lines below on my Raspberry Pi 4 installations which came to life years ago on RPI2 and Raspbian Stretch and which have been upgraded to Raspbian Buster (now called Raspberry Pi OS) and updated regularly (without starting from scratch or losing valuable data).

Once you have done the Node-RED upgrade as user PI…use these 4 lines (again as user PI)..

node-red-stop
cd /home/pi/.node-red
npm rebuild sqlite3
node-red-start

Be warned – the SQLITE rebuild will throw out MANY warnings… even in November 2020 – the improved NR script and the 4 lines above DO solve the SQLITE upgrade problem – but you MUST be in the right directory as indicated for the latter). Use the 4 lines above AFTER upgrading Node-Red and perferably before starting to run Node-Red.

I originally noted while upgrading other NR nodes – that when it came to the SQLITE update in the NR pallette manager, SQLITE failed. I stopped NR and ran the above – which usually worked but this has always been a CONSTANT pain as there is always the chance that this fix would not work. It does seem to work just fine now but ALWAYS make a backup before upgrading SQLITE in particular.

SQLITE3 mess

The above represents a TINY fraction of the warning messages you see during the SQLITE rebuild. They are mainly utterly pointless warnings but watch out for any ERRORS – they will be reasonably obvious.

That’s it, nothing else, just those 4 lines EXACTLY as above and despite all those warnings, the SQLITE upgrade should work just fine. It may take several minutes for the rebuild to complete after which you can restart Node-Red without a reboot.

But REALLY – the answer, if at all possible, despite its undoubted usefulness, is to DITCH SQLITE – this nonsense has been going on for YEARS… SQLITE in Node-Red is the ONLY Node-Red node package I know of that gives me constant grief like this.

I originally picked SQLITE due to fears that MYSQL (then Maria) might do excessive writing to SD. Today with booting and running on SSD over USB3 that is issue may vanish depending on your use-case.

Facebooktwitterpinterestlinkedin

19 thoughts on “SQLITE3 Raspberry Pi Problems and Solutions

  1. was joking 🙂
    i mean, it’s known mysql corrupt sdcard on the long run, due to wear leveling and continuous writing… but sqlite too, with it’s long compile times and failing in this, and redoing and updating and what else, starts to be no good for sd, too 🙂

  2. If you are going to use databases on a RPi then you should use the more industrial SD cards.

    The very best SD cards use SLC (Single Level Cell) technology but are hard to find and very expensive – but often come with 5 year guarantees.

    MLC (Multi-Level Cell) based SD cards are easy to get and are more expensive than TLC (Triple Level Cell) cards but have much better endurance.

    The standard SD cards are usually TLC and are what is called consumer grade. Even then, some brands seem to work more reliably than others.

    I have been using Transcend 32GB High Endurance SD cards for a while and have no issue with MySQL databases.

    If your project and/or data has real value then you should use MLC wherever possible.

    1. Do you have specific known good makes and models in mind? I’ve not heard Transcend recommended before – it is usually Sandisk and Samsung that get mentioned as winners.

      1. Samsung Evo and Evo+ 32GB cards all the way down!

        My Pi2 has now been running several years on one and my Pi3 running for over a year. Even occasional power outages and forced power terminations (e.g. pull the plug) haven’t caused any issues at all.

  3. I could easily see me reverting to Maria (MySQL) as the SQLITE issues continue with no sign of a script fix. In Node-Red when updating, I frequently have to “rebuild” Sqlite3 (several minutes) which is a pain – and the build script is full of un-useful warnings about statements falling through.

    1. what about MongoDB? It uses json as native data format, it’s nodejs based (and you already have it), has excellent nodered support (AFAIK)…

      install: https://yannickloriot.com/2016/04/install-mongodb-and-node-js-on-a-raspberry-pi/

      web gui: https://webapplog.com/mongoui/

      comparison with sqlite and mysql: https://db-engines.com/en/system/MongoDB%3BMySQL%3BSQLite

      if you want to go for mysql, here some optimizations for PI: https://central.owncloud.org/t/mysql-mariadb-optimal-settings-for-running-on-raspberrypi-3-model-b/17004

    1. Excellent feedback. I’ve yet to put a large HD on the Pi. These notes however have enabled me to install/upgrade Node-Red on Buster on the Pi3 and now P4 and it all works without having to start from scratch.

      1. i see no point in using a 500gb hd, you’ll never use it fully with a raspberry (unless you use it as a nas, which, if not an rpi4 model, is not worth because of the shared bus between usb and eth), and at that point you can just use mysql or others which will not need to get a PITA every time you update your nodes 🙂

        1. The only reason I used the 500GB drive is that I had it on hand after replacing the hard drive ib a laptop with an SSD drive.

    1. I WAS going to come back with a quick answer “that’s fiine if you know how to do it” – then realised you’ve actually provided the answer by the look of it – well done.. it defeats me why the author left in such a level of warnings when the end user running his or her script can do nothing about them.

  4. I have just spent a considerable amount of time pulling sqlite out of my systems and moved on to MariaDB due to these ongoing issues with the node red module. It was brought to a head when I wanted to move some of the flows to a HASSOS (Home Assistant OS) machine and discovered you cannot run the sqlite node on that OS because you need the make tools to compile and they are not available on HASSOS. That and the fact that the Home Assistant native log can get very large and uses sqllite by default. So I bit the bullet and upgraded and MariaDB works well, the Pi4 has no problem running the database server on top of everything else.

    1. Hi Steve

      Well,historically I moved to SQLITE and put that in my “The Script” because conventional wisdom said it was lighter on SD writes than MYSQL (hence Maria) – but but of course the goalposts have changed with SSD (and coming up in here this week, a small eMMC module to replace SD). I really think the people inolved in bringing SQLITE to Raspberry Pi (much as we all appreciate this stuff is most likely voluntary) need to rethink the compilation or make precompiled modules more readily available for SQLITE – I’ve lost more than one night’s sleep over the years with constant issues installing or upgrading SQLITE in Node-Red – almost all other nodes just update, no problem.

      I have Maria installed of course but never gotten around to modifying my queries to work with Maria/MySQL.

      I think first I’ll take a look at that comment above by Rinie Kervel then if all else fails spend the time to eliminate SQLITE.

  5. People first move over to SSD and when budget is OK change to Pi4. I made a clone of my freshly generated Debian in 3,5 minutes SSD>SSD clone on USB 3.
    I was trying to generate another fresh copy in a Pi 3 as a backup to my homeserver, but that hanged at installing Node_RED(half an hour was enough to kill the Script by pulling the power off). A SSD with USB3 of 120GB costs 23 euro. The benchmark shows also almost 300MB/s big chunk writing speed.

Leave a Reply

Your email address will not be published. Required fields are marked *

Leave the field below empty!

This site uses Akismet to reduce spam. Learn how your comment data is processed.