SQLITE3 Raspberry Pi Problems and Solutions

SQLITE3 on Node-Red on Raspberry Pi

This is the November 27, 2020 update of an old, outstanding problem for Raspberry Pi owners who use and regularly upgrade Node-Red and who simultaneously use the Node-Red SQLITE node. The issue has also been known to affect the serial port and i2c on upgrading.

I KNOW I’m not the only one who has had issues with SQLITE3 upgrading in Raspberry Pi and Node-Red, if it happens to you… here is some invaluable info…

After a recent, successful upgrade to Node-Red 1.2.5 (leaving the SQLITE node slightly behind as it can be painful to update when there are node-js upgrades involved), I was feeling quite perky when along came NR 1.2.6 – which completely trashed my setup.

DaveCJ and others on the Node-Red DISCOURSE channel helped me yesterday without which I’d have been stuck on NR 1.2.5 possibly for a long time. The NR script you see here for upgrading Node-Red suffered a node-js upgrade-related issue which could cause failure. On the morning of November 27, 2020 this was FIXED.

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

These days whenever I upgrade Node-Red I use that script above ( a link to which is also included in my own “The Script”) and I subsequently (no reboot required) use the 4 lines below on my Raspberry Pi 4 installations which came to life years ago on RPI2 and Raspbian Stretch and which have been upgraded to Raspbian Buster (now called Raspberry Pi OS) and updated regularly (without starting from scratch or losing valuable data).

Once you have done the Node-RED upgrade as user PI…use these 4 lines (again as user PI)..

cd /home/pi/.node-red
npm rebuild sqlite3

Be warned – the SQLITE rebuild will throw out MANY warnings… even in November 2020 – the improved NR script and the 4 lines above DO solve the SQLITE upgrade problem – but you MUST be in the right directory as indicated for the latter). Use the 4 lines above AFTER upgrading Node-Red and perferably before starting to run Node-Red.

I originally noted while upgrading other NR nodes – that when it came to the SQLITE update in the NR pallette manager, SQLITE failed. I stopped NR and ran the above – which usually worked but this has always been a CONSTANT pain as there is always the chance that this fix would not work. It does seem to work just fine now but ALWAYS make a backup before upgrading SQLITE in particular.

SQLITE3 mess

The above represents a TINY fraction of the warning messages you see during the SQLITE rebuild. They are mainly utterly pointless warnings but watch out for any ERRORS – they will be reasonably obvious.

That’s it, nothing else, just those 4 lines EXACTLY as above and despite all those warnings, the SQLITE upgrade should work just fine. It may take several minutes for the rebuild to complete after which you can restart Node-Red without a reboot.

Update Feb 16, 2021 – the 4-line SQLITE fix seems to be holding on Node-RED 1.2.9, NPM 6.14.11 and NodeJS 10.23.3


28 thoughts on “SQLITE3 Raspberry Pi Problems and Solutions

  1. was joking 🙂
    i mean, it’s known mysql corrupt sdcard on the long run, due to wear leveling and continuous writing… but sqlite too, with it’s long compile times and failing in this, and redoing and updating and what else, starts to be no good for sd, too 🙂

  2. If you are going to use databases on a RPi then you should use the more industrial SD cards.

    The very best SD cards use SLC (Single Level Cell) technology but are hard to find and very expensive – but often come with 5 year guarantees.

    MLC (Multi-Level Cell) based SD cards are easy to get and are more expensive than TLC (Triple Level Cell) cards but have much better endurance.

    The standard SD cards are usually TLC and are what is called consumer grade. Even then, some brands seem to work more reliably than others.

    I have been using Transcend 32GB High Endurance SD cards for a while and have no issue with MySQL databases.

    If your project and/or data has real value then you should use MLC wherever possible.

    1. Do you have specific known good makes and models in mind? I’ve not heard Transcend recommended before – it is usually Sandisk and Samsung that get mentioned as winners.

      1. Samsung Evo and Evo+ 32GB cards all the way down!

        My Pi2 has now been running several years on one and my Pi3 running for over a year. Even occasional power outages and forced power terminations (e.g. pull the plug) haven’t caused any issues at all.

  3. I could easily see me reverting to Maria (MySQL) as the SQLITE issues continue with no sign of a script fix. In Node-Red when updating, I frequently have to “rebuild” Sqlite3 (several minutes) which is a pain – and the build script is full of un-useful warnings about statements falling through.

    1. what about MongoDB? It uses json as native data format, it’s nodejs based (and you already have it), has excellent nodered support (AFAIK)…

      install: https://yannickloriot.com/2016/04/install-mongodb-and-node-js-on-a-raspberry-pi/

      web gui: https://webapplog.com/mongoui/

      comparison with sqlite and mysql: https://db-engines.com/en/system/MongoDB%3BMySQL%3BSQLite

      if you want to go for mysql, here some optimizations for PI: https://central.owncloud.org/t/mysql-mariadb-optimal-settings-for-running-on-raspberrypi-3-model-b/17004

    1. Excellent feedback. I’ve yet to put a large HD on the Pi. These notes however have enabled me to install/upgrade Node-Red on Buster on the Pi3 and now P4 and it all works without having to start from scratch.

      1. i see no point in using a 500gb hd, you’ll never use it fully with a raspberry (unless you use it as a nas, which, if not an rpi4 model, is not worth because of the shared bus between usb and eth), and at that point you can just use mysql or others which will not need to get a PITA every time you update your nodes 🙂

        1. The only reason I used the 500GB drive is that I had it on hand after replacing the hard drive ib a laptop with an SSD drive.

  4. The above represents a TINY fraction of the warning messages you see during the rebuild.
    Why not fix the compiler flags as the code runs fine?


    gcc -g -Wpedantic -Werror -Wall -Wextra \
    -Wsign-compare -fPIC -std=c89 -Wno-long-long \
    -Wno-implicit-fallthrough \
    -c -o sqlite3.o sqlite3.c

    1. I WAS going to come back with a quick answer “that’s fiine if you know how to do it” – then realised you’ve actually provided the answer by the look of it – well done.. it defeats me why the author left in such a level of warnings when the end user running his or her script can do nothing about them.

  5. I have just spent a considerable amount of time pulling sqlite out of my systems and moved on to MariaDB due to these ongoing issues with the node red module. It was brought to a head when I wanted to move some of the flows to a HASSOS (Home Assistant OS) machine and discovered you cannot run the sqlite node on that OS because you need the make tools to compile and they are not available on HASSOS. That and the fact that the Home Assistant native log can get very large and uses sqllite by default. So I bit the bullet and upgraded and MariaDB works well, the Pi4 has no problem running the database server on top of everything else.

    1. Hi Steve

      Well,historically I moved to SQLITE and put that in my “The Script” because conventional wisdom said it was lighter on SD writes than MYSQL (hence Maria) – but but of course the goalposts have changed with SSD (and coming up in here this week, a small eMMC module to replace SD). I really think the people inolved in bringing SQLITE to Raspberry Pi (much as we all appreciate this stuff is most likely voluntary) need to rethink the compilation or make precompiled modules more readily available for SQLITE – I’ve lost more than one night’s sleep over the years with constant issues installing or upgrading SQLITE in Node-Red – almost all other nodes just update, no problem.

      I have Maria installed of course but never gotten around to modifying my queries to work with Maria/MySQL.

      I think first I’ll take a look at that comment above by Rinie Kervel then if all else fails spend the time to eliminate SQLITE.

  6. People first move over to SSD and when budget is OK change to Pi4. I made a clone of my freshly generated Debian in 3,5 minutes SSD>SSD clone on USB 3.
    I was trying to generate another fresh copy in a Pi 3 as a backup to my homeserver, but that hanged at installing Node_RED(half an hour was enough to kill the Script by pulling the power off). A SSD with USB3 of 120GB costs 23 euro. The benchmark shows also almost 300MB/s big chunk writing speed.

  7. I’m trying to get follow along with a Node-Red flow with GPS data that uses sqlite and was advised to use phpliteadmin as it is “easy to use and set up”. Well, I’ve spent the every evening this past week trying to the phpliteadmin to work and I have not clue. Anyone have pointers or can send me in the right direction? I’ve not been able to find much in a search. I get to the webpage and then it asked to great a database, but then I get the error that sqlitDatabase not installed.

    RPi 4, Node-Red 1.2.5 and installed the sqlite/phpliteadmin with Synaptic.

    1. It’s such a long time since I installed SQLITE from scratch – I use “the script” to install NR, MQTT, SQLITE and the various NR nodes on a fresh Raspberry Pi. The only issues I’ve had, have been upgrades ro SQLITE node – but it looks like the upgrade to the NR install/upgrade script has fixed that. So – before thinking about NR, you need SQLITE itself in there. Never used Synaptec so can’t comment on that. phpliteadmin will create databases but only if the actual SQLITE itself is installed on the Pi. Possibly look at “the script” for pointers?

    2. extracting from the script, let’s say for debian 10: you need apache+sqlite:
      sudo apt-get -y install apache2 libapache2-mod-php7.3 sqlite3 php-sqlite3 php-xml php-mbstring

      then install phpliteadmin:
      cd /var/www/html
      sudo mkdir phpliteadmin
      cd phpliteadmin
      sudo wget –no-check-certificate https://www.phpliteadmin.org/phpliteadmin-dev.zip
      sudo unzip phpliteadmin-dev.zip
      sudo mv phpliteadmin.php index.php
      sudo mv phpliteadmin.config.sample.php phpliteadmin.config.php
      sudo rm *.zip
      sudo mkdir themes
      cd themes
      sudo wget –no-check-certificate http://bitbucket.org/phpliteadmin/public/downloads/phpliteadmin_themes_2016-02-29.zip
      sudo unzip phpliteadmin_themes_2016-02-29.zip
      sudo rm *.zip

      then configure it, manually going to search for these 3 parameters, or via sed and following lines (IMPORTANT in the 2nd one, change $adminpass to your wanted pass)
      sudo sed -i -e ‘s#\$directory = \x27.\x27;#\$directory = \x27/home/pi/dbs/\x27;#g’ /var/www/html/phpliteadmin/phpliteadmin.config.php
      sudo sed -i -e “s#\$password = \x27admin\x27;#\$password = \x27$adminpass\x27;#g” /var/www/html/phpliteadmin/phpliteadmin.config.php
      sudo sed -i -e “s#\$subdirectories = false;#\$subdirectories = true;#g” /var/www/html/phpliteadmin/phpliteadmin.config.php

      1. Thanks Mr.Shark, ran everything and nothing new was installed, the last 3 items produced this error:

        sudo sed -i -e ‘s#\$directory = \x27.\x27;#\$directory = \x27/home/pi/dbs/\x27;#g’ /var/www/html/phpliteadmin/phpliteadmin.config.php
        sed: -e expression #1, char 1: unknown command: `�’

        I made the changes with nano, but when I run the web application, the old password gets me in and not the new one and it still doesn’t let me create any databases.

        “There was a problem setting up your database, /var/lib/phpliteadmin/. An attempt will be made to find out what’s going on so you can fix the problem more easily.

        Checking supported SQLite PHP extensions…

        PDO: installed
        PDO SQLite Driver: installed
        SQLite3: installed
        SQLiteDatabase: not installed”

        Has to be something simple, where is it starting from perhaps?

        1. yes, you can use an external config file, or put all in main file as i do, it’s the same… don’t copy paste those lines because blog formatted them wrong, just look at them in the sed part to see what to search and how to replace that, are just 3 lines…

  8. I unloaded everything to do with phplliteadmin and sqlite, removed everything I did with Mr.Shark’s instructions above. Then I redid the instructions above and I have blastoff!
    Now I just need to add the fields to the database.

    Thanks for the assistance.

Leave a Reply

Your email address will not be published. Required fields are marked *

Leave the field below empty!

This site uses Akismet to reduce spam. Learn how your comment data is processed.